# Traffic Noise Prediction Applying Multivariate Bi-Directional Recurrent Neural Network
## 2 Materials and Methods  
### General Background of Recurrent Neural Network  
Time series data refer to a data sequence in the time order, which may contain specific properties such as a trend. The behavior and interactions of the variables over time are especially important for time series analysis [30]. Recurrent neural network has been a popular algorithm for modelling time series data, due to the feedback loop in the recurrent layer, as shown in Figure 1. The feedback loop enables a connection in time domain between nodes of the temporal sequence, as illustrated in Figure 1 after unfolding the feedback loop. In fully connected RNN, each node has input from all nodes. In partially connected RNN, only some nodes are in a feedforward loop [31]. The feedback loop maintains  
Figure 1: Recurrent neural network (RNN) architecture: recurrent layer with a feedback loop, indicating the additional time dimension after unfolding it. \"memory\" over time. The \"memory\" contains the temporal dependencies of the samples. The neurons in the recurrent layer are recurrent units. Three common recurrent units are namely simple RNN, LSTM and GRU units. Simple RNN is only capable of handling the short-term dependencies. LSTM and GRU can be also applied to long-term dependency problems, such as speech recognition or text generation, thanks to their complex structures [32]. They have additional \"gates\" to regulate the flow of the information. These gates are represented by sigmoid functions. The sigmoid function applied to \"gates\" is the logistic sigmoid function, having an \"S\"-shaped curve in the range of [0, 1], so that it can act like a gate, updating or forgetting information by squishing all coming values between 0 and 1. They learn which information is to keep or to forget and pass important information along the chain of the sequence to make predictions. The schemes of these three units are illustrated below. As illustrated in Figure 2, the simple RNN unit has only a hidden state that serves as the memory of RNNs [33]. There is simple operation of input and previous output, passing through the tanh (hyperbolic tangent) activation function. The activation function is a mathematical transformation from the input to output attached to each neuron in the neural networks. Non-linear activation functions make the network capable of learning very complex data and eventually achieving accurate predictions. The tanh activation squishes the values to the range of [\(-1,1\)], which regulates the information flowing through the network. No gate is embedded in simple RNN. The hidden state is the output of the unit cell, which can be expressed by the equation below:  
\[h^{<t>}=\tanh(W[h^{<t-1>},x^{<t>}]+b), \tag{1}\]  
where \(x^{<t>}\) is the input vector, \(h^{<t>}\) is the hidden state of current output, \(h^{<t-1>}\) is the hidden state of previous output. \(W\) is weight matrix, and \(b\) is bias vector, both of which are the parameters to train by backpropagation through time. Backpropagation is a widely used algorithm in feedforward neural networks for computing the gradients iteratively, backward from the last layer to previous layers [34]. In recurrent neural networks, it is called backpropagation through time (BPTT), as RNN is not a feedforward neural network due to its feedback loop, as illustrated in Figure 1. BPTT begins with unrolling all timesteps of the RNN and then computes the gradients like traditional backpropagation [35]. tanh indicates a hyperbolic tangent activation function. Figure 2: Illustration of simple RNN unit, having a simple structure without gate. A simple RNN unit cannot deal well with long-term dependencies, because of the vanishing gradient. The gradients carry information used for updating the network weights.