## 3 Popular ANN Architectures Employed for Time Series Forecasting Purposes
### Recurrent Neural Networks  
The main characteristic of a recurrent neural network (RNN) is that it has at least one _feedback connection_, where the output of the previous step is fed as input to the current step. This recurrent connection system makes RNNs ideal for sequential or time series data, \"remembering\" past information. Another distinctive feature of an RNN is that each layer shares the same weight parameter. RNNs are not easy to train, but very accurate forecasts for time series can be obtained when trained correctly. There are several RNN architectures; however, they all have the following characteristics in common:  
1. RNNs contain a subsystem similar to a static FFNN;RNNs can take advantage of the nonlinear mapping abilities of an FFNN, with an added memory capacity for past information. RNN's learning can be performed by using the gradient descent method, similar to how it is used in the BP algorithm. Specifically, RNNs can be trained by using an algorithm called _backpropagation through time_ (BPTT). BPTT trains the network by computing errors from the output layer to the input layer, but unlike BP, it adds errors at each time step because it shares parameters at each layer. A basic RNN architecture, called the Elman network [50], has the inputs of the next time step together with its hidden unit activations that feed back on the network. Figure 3a shows the Elman network architecture. It is observed that it is necessary to discretize the time and update the activations step by step. In real neurons, this could correspond to the time scale on which they operate, and for artificial neurons, it could be any time step size related to the prediction to be made. In particular, for time series modeling applications, it seems like a natural choice to make the time-step size in an RNN equal to the time separation between any two consecutive time series values. A delay unit is introduced, which simply delays the signal/activation until the next time step. This delay unit can be regarded as a short-term memory unit. Suppose the vectors \(\vec{x}(t)\) and \(\vec{y}(t)\) are the inputs and outputs, \(\vec{W}_{IH}\), \(\vec{W}_{HH}\), and \(\vec{W}_{HO}\) are the three connection weight matrices, and \(f\) and \(g\) are the output and hidden unit activation functions of an Elman network; then, the operation of the said RNN can be described as a dynamic system characterized by the pair of nonlinear matrix equations:  
\[ \begin{split}\vec{h}(t)&=f\Big{(}\vec{W}_{IH}\vec{x} (t)+\vec{W}_{HH}\vec{h}(t-1)\Big{)}\hskip 42.679134 pt\text{State transition}\\ \vec{y}(t)&=g\Big{(}\vec{W}_{HO}\vec{h}(t)\Big{)} \hskip 42.679134 pt\text{Output equation}\end{split}  \tag{7}\]  
In a dynamical system, its state can be represented as a set of values that recapitulates all the information from the past about the system. The hidden unit activations \(\vec{h}(t)\) define the state of the dynamical system. Elman networks are useful in modeling chaotic time series, which are more closely related to chaos theory and dynamical systems. For further information on chaotic time series, see [PERSON] [51]. Some recent applications of Elman networks in time series forecasting can be found in [PERSON] and [PERSON] [52], [PERSON] and [PERSON] [53], and [PERSON], [PERSON], [PERSON], and [PERSON] [54]. Figure 3: Two simple types of recurrent neural networks. Each rectangle contains input units, artificial neurons or delay/memory units; their outputs being indicated by vector quantities \(\vec{x}(t)\), \(\vec{y}(t)\), \(\vec{h}(t)\), etc. A solid arrow connecting two rectangles represents the full set of connection weights among all involved units, which are encoded as matrices \(\vec{W}_{IH}\) and \(\vec{W}_{HO}\). Dashed arrows represent one-to-one connections between involved units; this means \(\vec{I}\) (identity matrix) and \(\vec{W}_{HH}\) are diagonal matrices. Another simple recurrent neural network architecture, similar to an Elman network, is the Jordan network [55].