## 3 Popular ANN Architectures Employed for Time Series Forecasting Purposes
### BP Problems in the Context of Time Series Modeling
#### 3.6.2 Alternatives to the BP Problems  
Attempts have been made to overcome these issues in the context of time series forecasting. See, for example, [PERSON], [PERSON], [PERSON], and [PERSON] [71] and [PERSON] [72]. Below is a brief outline of the main alternatives. **Batch normalization. ** [PERSON] and [PERSON] [73] described an _internal covariate shift_ as the effect of inputs with a corresponding distribution in each layer of a neural network, which is caused by the randomness that exists in the initialization of parameters and in the input data during the training process. They proposed to address the problem by normalizing the layer inputs, recentering and rescaling them, and applying the normalization to each training mini-batch. Batch normalization relaxes the care of parameter initialization, allows the application of much higher learning rates, and, in certain cases, eliminates the need for dropout to mitigate overfitting. Although batch normalization has been proposed to handle gradient explosion or vanishing problems, recently, [PERSON], [PERSON], [PERSON], [PERSON], and [PERSON] [74] showed that, at the initialization time, a deep batch norm network suffers from gradient explosion. **Gradient clipping. ** In 2013, [PERSON], [PERSON], and [PERSON] [70] assumed that a cliff-like structure appears on the error surface when gradients explode, and as a solution, they proposed clipping the norm of the exploded gradients. Furthermore, to solve the vanishing gradient problem, they use a regularization term to force the Jacobian matrices to preserve the norm only in relevant directions, keeping the error signal alive while it travels backwards in time. **Backpropagation through time (BPTT). ** This famous technique was proposed by [PERSON] [75] in 1990. If the computational graph of an RNN is expanded (unrolled RNN), it is basically an FFNN with the innovative characteristic that, throughout the unrolled RNN, the same parameters are repeated and these appear in each period. Then, the chain rule can be applied to propagate the gradients backward through the unrolled RNN, as would be performed in any FFNN. It should be considered that, in this unrolled RNN, for each parameter, the gradient with respect to itself must be added at all places where the parameter occurs. In summary, BPTT can be explained as using BP to RNN on sequential data, e.g., a time series [75]. The BPTT algorithm can be described as follows:  
* Introduce a time-step sequence of input and output pairs to the network. * Unroll the network. * For each time step, calculate and accumulate errors. * Roll-up the network. * Update weights. * Repeat. **Long short-term memory (LSTM). ** In 1997, [PERSON] and [PERSON] [76] introduced an RNN along with an appropriate gradient-based learning algorithm. Its goal is to introduce a short-term memory for RNN that can last for thousands of time steps, that is, a \"long short-term memory\". The main feature of LSTM is its _memory cell_ made up of three \"gates\": the input gate, output gate, and forget gate [77]. The flow of information is regulated by gates inside and outside the cell. First, the forget gate assigns a previous state a value between 0 and 1, compared to a current input. Then, it chooses what information to keep from a previous state. A value of 0 means deleting the information, and a value of 1 means keeping it. By applying the same system as the forget gate, the gateway determines the new information that will be stored in the current state. Finally, the output gate controls which pieces of information in the current state are generated by considering the previous and current states and assigning a value from 0 to 1 to the information. The LSTM network maintains useful long-term dependencies, generating relevant information about the current state. The goal of LSTM is to create an additional module in an ANN that can learn when to forget irrelevant information and when to remember relevant information [77].