## 4 RNN Structures Used for Traffic Prediction
### LSTMs  
Theoretically, RNNs are simple and powerful models, effectively training them poses many challenges in practical applications. One major issue is the problem of vanishing and exploding gradients. Gradient explosion occurs during training when the norm of gradients sharply increases due to long-term dependencies, growing exponentially. Conversely, the vanishing gradient problem describes the opposite phenomenon, where gradients for long-term dependencies rapidly diminish to near-zero levels, making it difficult for the model to learn long-range dependencies. To overcome these issues, the introduction of LSTM addresses the problem by employing gate mechanisms to maintain long-term dependencies while mitigating gradient problems. These gate control systems include input gates, forget gates, and output gates, which work together to regulate information flow, retention, and output precisely.  
The basic formulas for LSTM are as follows:  
\[\begin{array}{l}\mathbf{f_{t}}=\sigma\Big{(}W_{f}[\mathbf{h_{t-1}},\mathbf{x_{t}}]+\mathbf{b_{f} }\Big{)}\\ \mathbf{i_{t}}=\sigma(W_{i}[\mathbf{h_{t-1}},\mathbf{x_{t}}]+\mathbf{b_{i}})\\ \mathbf{o_{t}}=\sigma(W_{o}[\mathbf{h_{t-1}},\mathbf{x_{t}}]+\mathbf{b_{o}})\\ \widetilde{\mathbf{C}_{t}}=\tanh(W_{\text{C}}[\mathbf{h_{t-1}},\mathbf{x_{t}}]+\mathbf{b_{c}}) \\ \mathbf{C_{t}}=\mathbf{f_{t}}\mathbf{C_{t-1}}+\mathbf{i_{t}}\widetilde{\mathbf{C_{t}}}\\ \mathbf{h_{t}}=\mathbf{o_{t}}\text{tanh}(\mathbf{C_{t}})\end{array} \tag{3}\]  
where \(\mathbf{x_{t}}\) represents the input vector at time step \(t\). \(\mathbf{h_{t-1}}\) represents the hidden state vector at time step \(t-1\) and will serve as part of the input for the next time step. \(\mathbf{f_{t}}\), \(\mathbf{i_{t}}\), and \(\mathbf{o_{t}}\) are the outputs of the forget gate, input gate, and output gate. \(\widetilde{\mathbf{C_{t}}}\) is the candidate layer at time step \(t\), representing potential new information that might be added to the current cell state. \(\mathbf{C_{t}}\) is the LSTM's internal state, containing the network's long-term memory. \(W_{f}\), \(W_{i}\), \(W_{o}\), and \(W_{\text{C}}\) are the weight matrices for the forget gate, input gate, output gate, and candidate layer. \(\mathbf{b_{f}}\), \(\mathbf{b_{i}}\), \(\mathbf{b_{o}}\), and \(\mathbf{b_{c}}\) are the bias vectors for the forget gate, input gate, output gate, and candidate layer, respectively. \(\sigma(\cdot)\) is the sigmoid activation function used for gating mechanisms and \(\tanh(\cdot)\) is the hyperbolic tangent activation function used for non-linear transformations.  
The forget gate \(\mathbf{f_{t}}\) decides which information to discard by using a sigmoid function that outputs values between \(\mathbf{0}\) and \(\mathbf{1}\) for each component in the cell state, where \(\mathbf{1}\) means \"retain this completely\" and \(\mathbf{0}\) means \"discard completely\". Simultaneously, the input gate \(\mathbf{i_{t}}\) and a candidate layer \(\widetilde{\mathbf{C_{t}}}\) decide which new information is stored by creating a vector of new candidate values. The cell state is then updated by combining the old state, multiplied by the output of the forget gate, with the new candidate values scaled by the output of the input gate. Lastly, the output gate \(\mathbf{f_{t}}\) together with the cell state passed through a \(\tanh\) function determines the next hidden state \(\mathbf{h_{t}}\), filtering the information to the output. This mechanism enables LSTMs to handle vanishing gradients and learn over extended sequences effectively.