# Recurrent Neural Networks and ARIMA Models for Euro/Dollar Exchange Rate Forecasting
## 2 Materials and Methods  
### Overview of Regression Techniques  
The classical statistical methods were historically used to analyze the behavior of time series. ARIMA models are well-known parametric techniques trained through linear regressions [6; 25]. The ARIMA algorithm in Figure 1a uses graphs, basic statistics, the Autocorrelation Function (ACF), the Partial Autocorrelation Function (PACF), and transformations to identify patterns and model components. This model provides estimates through least squares and maximum likelihood methods and uses the graphs, ACF, and PACF of residuals to verify the validity of the model--i.e., if the model is valid, then use that model; otherwise, go back to the first step. Lastly, the algorithm forecasts and tracks the model's performance using confidence intervals and simple statistics [26]. For a non-parametric technique such as ANN, the model is trained through non-linear algorithms. These self-adapting models do not require a priori assumptions of the series due to their flexibility in building model topologies and ability to easily identify and predict behavior patterns in the series [16; 25]. In this case, the training is conducted point-to-point; if the amount of atypical data is minimal, then the correct data fix the error generated by the atypical data, thereby converging to the exact model [27]. The Elman Neural Network (ENN) is a subclass of RNN. The ENN algorithm (Figure 2) starts with an input layer followed by a hidden layer and a context layer (delay layer) with the same number of neurons. The feedback gives temporality to the network, providing the system with short-term memory. This memory process occurs through delay units that are fed by the neurons of the hidden layer. The weights of the connections between the hidden layer and delay units are fixed and equal to 1, allowing one to obtain a copy of the output values of the hidden layer from the previous step [28]. In Figure 2, the input layer variables are represented by \(x_{t}\), and the hidden layer variables are represented by \(y_{t}\). The hidden layer vector is represented by the expression \(h_{t}=\sigma_{h}(W_{h}x_{t}+U_{h}h_{t-1}+b_{h})\), and the output weight matrix is represented by \(y_{t}=\sigma_{y}\big{(}W_{y}h_{t}+b_{y}\big{)}\), where \(W=\big{[}W_{x}\,W_{y}\big{]}^{T}\) and \(U_{h}\) are weight matrices, and \(b=\big{[}b_{h}\,b_{y}\big{]}^{T}\) is the bias. Here, \(\sigma_{h}\) and \(\sigma_{y}\) are the activation functions for the hidden and output layers, respectively [28]. Similar to ENN, the LSTM network can remember a relevant sequence of data and preserve it for several time instances. In this way, an LSTM network can achieve short-term memory similar to that of basic recurrent networks, as well as long-term memory. As shown in Figure 3a, each block of the LSTM network can contain several cells in a similar manner to an Elman network, only replacing the neurons and hidden units with a memory block (LSTM cell; Figure 3b). In Figure 3, the input and output are represented by \(x_{t}\) and \(y_{t}\), respectively, while the vector \(h_{t}\) represents short-term memory, and \(c_{t}\) represents long-term memory. For time series predictions of \(x_{t}\), the LSTM system updates the memory cell \(c_{t}\) and outputs a hidden state \(h_{t}\) for each step \(t\). Equation (1) represents the mechanism of LSTM [14; 29]:  
\[ \begin{split} i_{t}&=\sigma\big{(}W_{xi}^{T}\cdot x _{t}+W_{hi}^{T}\cdot h_{t-1}+b_{i}\big{)}\\ f_{t}&=\sigma\big{(}W_{xf}^{T}\cdot x_{t}+W_{hf}^{T} \cdot h_{t-1}+b_{f}\big{)}\\ o_{t}&=\sigma\big{(}W_{xo}^{T}\cdot x_{t}+W_{ho}^{T} \cdot h_{t-1}+b_{o}\big{)}\\ g_{t}&=tanh\big{(}W_{xg}^{T}\cdot x_{t}+W_{hg}^{T} \cdot h_{t-1}+b_{g}\big{)}\\ c_{t}&=f_{t}\otimes c_{t-1}+i_{t}\otimes gt\\ y_{t}&=h_{t}=o_{t}\otimes tanh(c_{t})\end{split}