# Traffic Noise Prediction Applying Multivariate Bi-Directional Recurrent Neural Network
## 2 Materials and Methods
### General Background of Recurrent Neural Network
The weights of the network at every iteration are calculated with the equation below:  
\[\theta_{j}:=\theta_{j}-\alpha\frac{\partial J(\theta)}{\partial\theta_{j}}, \tag{2}\]  
where \(\alpha\) is the learning rate, which is the step size for updating the weights at each iteration, multiplied by the gradients. \(\theta\) is the weight matrix (or parameters) to learn. \(J(\theta)\) is the loss function. A loss function measures how good the model predicts the target variable. In a regression task, the loss function is usually represented by mean squared error of the estimation against ground truth. The weights from early time steps are calculated sequentially according to the chain rule during backpropagation through time. When the gradients become smaller and smaller, the network weights will become almost no update anymore after each iteration. Therefore, the learning process cannot be further performed. This is the problem of vanishing gradient. To solve this problem, [PERSON] and [PERSON] [36] developed the LSTM network. The LSTM unit has not only the hidden state, but also an additional memory cell state that maintains selected information over long time. Keeping or dropping the maintained information is regulated by \"gates\", which allow a better control over the gradient flow in the long dependency. As illustrated in Figure 3, LSTM uses three gates, input gate, forget gate and output gate, to control when the information enters the memory cell, when to be dropped out and when to be sent to output. The \"gates\" are expressed by sigmoid functions; the three gates, memory cell state and hidden state can be explained as the equations below:  
Forget gate:  
\[f^{<t>}=\sigma(W_{f}[h^{<t-1>},x^{<t>}]+b_{f}), \tag{3}\]  
Input gate:  
\[i^{<t>}=\sigma(W_{i}[h^{<t-1>},x^{<t>}]+b_{i}), \tag{4}\]  
Output gate:  
\[o^{<t>}=\sigma(W_{o}[h^{<t-1>},x^{<t>}]+b_{o}), \tag{5}\]  
Hidden state:  
\[h^{<t>}=o^{<t>}*\tanh(c^{<t>}), \tag{6}\]  
Candidate memory cell state:  
Figure 1:0  
Figure 1:1 Illustration of long short-term memory (LSTM) unit, having three gates and a memory cell state. Memory cell state:  
Figure 1:2  
where the candidate memory cell state stores intermediate candidate values for the memory cell state, \(\sigma\) represents the sigmoid function, corresponding to the indication of'sigm' in Figure 3; and tanh indicates a hyperbolic tangent activation function. \(W_{f},W_{i},W_{o},b_{f},b_{i},b_{o}\) are the weights (weight matrices and bias vectors) of the network to train during the backpropagation with gradient descent, and \(*\) denotes element-wise multiplication. LSTM has a very high control ability over the gradient flow, which also brings complexity and additional operating costs. GRU is a simplified architecture of LSTM [37], as sketched in Figure 1:3  
GRU has only two \"gates\", called update gate and reset gate. The update gate decides how much of the past information needs to be further passed and which new information to add. The reset gate decides how much of the past information to forget. GRU does not have an additional memory cell and directly uses hidden state to transfer information. The mathematical expressions can be found in the equations below:  
Update gate:  
Figure 1:4  
Reset gate:  
Figure 1:5  
Candidate hidden state:  
Figure 1:6  
Hidden state:  
Figure 1:7  
where \(W_{z},W_{r},W_{z},b_{r},b\) are the parameters (weight matrices and bias vectors) to train, \(\sigma\) represents the sigmoid function, corresponding to the indication of'sigm' in Figure 4; and tanh indicates a hyperbolic tangent activation function. The formulas of the reset gate and update gate look quite similar. The differences lie in the weights and the usage of the gates. Because of the relative simplicity in the architecture, GRU is faster to train than LSTM, even without the cost of performance, depending on the concrete application scenario. The architectures of simple RNN, LSTM and GRU units show how the time signal in the recurrent neural network works with the adjacent time signals internally.