## 3 Popular ANN Architectures Employed for Time Series Forecasting Purposes
### BP Problems in the Context of Time Series Modeling
#### 3.6.1 Vanishing and Exploding Gradient Problems  
One of the main problems encountered when training recurrent neural networks or deep neural networks (FFNNs with many hidden layers) with BP is the vanishing gradient problem. When the gradients become very small relative to the parameters, it can cause the weights in the initial layers not to change noticeably; this is known as the vanishing gradient problem [69]. This problem is commonly attributed to the architecture of the neural network, certain activation functions (sigmoid or hyperbolic tangent), and small initial values of the weights. The exploding gradient problem appears when the weights are greater than 1 and the gradient continues to increase, causing the gradient descent to diverge. Unlike the vanishing gradient problem, the exploding gradient problem is directly related to the weights in the neural network [69].  
For instance, a generic recurrent network has hidden states \(h_{1},h_{2},\dots\), inputs \(u_{1},u_{2},\dots\), and outputs \(x_{1},x_{2},\dots\). Let it be parametrized by \(\theta\), so that the system evolves as  
\[(h_{t},x_{t})=F(h_{t-1},u_{t},\theta) \tag{9}\]  
Often, the output \(x_{t}\) is a function of \(h_{t}\), as some \(x_{t}=G(h_{t})\). The vanishing gradient problem already presents itself clearly when \(x_{t}=h_{t}\), so we simplify our notation to the special case with  
\[x_{t}=F(x_{t-1},u_{t},\theta) \tag{10}\]  
Now, take its differential:  
\[dx_{t} =\
abla_{\theta}F(x_{t-1},u_{t},\theta)d\theta+\
abla_{x}F(x_{t-1 },u_{t},\theta)dx_{t-1} \tag{11}\] \[dx_{t-k} =\
abla_{\theta}F(x_{t-k-1},u_{t-k},\theta)d\theta+\
abla_{x}F(x _{t-k-1},u_{t-k},\theta)dx_{t-k-1}\] \[dx_{t} =(\
abla_{\theta}F(x_{t-1},u_{t},\theta)+\
abla_{x}F(x_{t-1},u_{ t},\theta)\
abla_{\theta}F(x_{t-2},u_{t-1},\theta)+\cdots)d\theta\]  
Training the network requires us to define a loss function to be minimized. Let it be \(L(x_{T},u_{1},...,u_{T})\), then minimizing it by gradient descent gives  
\[dL=\
abla_{x}L(x_{T},u_{1},...,u_{T})(\
abla_{\theta}F(x_{t-1},u_{t},\theta)+ \
abla_{x}F(x_{t-1},u_{t},\theta)\
abla_{\theta}F(x_{t-2},u_{t-1},\theta)+ \cdots)d\theta \tag{12}\]  
\[\Delta\theta=-\eta\cdot[\
abla_{x}L(x_{T})(\
abla_{\theta}F(x_{t-1},u_{t}, \theta)+\
abla_{x}F(x_{t-1},u_{t},\theta)\
abla_{\theta}F(x_{t-2},u_{t-1}, \theta)+\cdots)]^{T} \tag{13}\]  
where \(\eta\) is the learning rate. The vanishing/exploding gradient problem appears because there are repeated multiplications of the form  
\[\
abla_{x}F(x_{t-1},u_{t},\theta)\
abla_{x}F(x_{t-2},u_{t-1},\theta)\
abla_{ x}F(x_{t-3},u_{t-2},\theta)\cdots \tag{14}\]  
Specifically, the vanishing gradient problem arises when the neural network adds multiple layers with activation functions whose gradients approach zero. Since each layer contributes to the product of the activation functions and the layer weights, if the number of layers increases, the product quickly turns small [70].  
The explosive gradient problem arises when the network weights are multiplied by the activation functions, and as a result, we have a product with values greater than one, causing the values of the gradients to be large [70].