# TBRNet: Two-Stream BiLSTM Residual Network for Video Action Recognition
## 3 Proposed Approach
### BiLSTM Network  
Owing to the recurrent connections of each unit, RNNs yield good performances in modeling hidden sequential patterns of data. However, the \"internal memory\" property and the \"vanishing gradient\" problem of the RNN recurrent structure make it difficult to update the network parameters during backpropagation; thus, RNNs are deficient in interpreting early information and modeling the long-range temporal contexts of feature vector sequences. Fortunately, LSTM conquers this fundamental weakness through its unique cell structure, which contains forget, input, and output gates controlled by sigmoid units to decide what information to update and store in memory cells. Linear connections across these LSTM units help to transmit the previous information to the present time step. The structure of an LSTM unit is illustrated in Figure 7.  
Given the current input vector \(x_{t}\), the last hidden state \(h_{t-1}\), and the last memory cell state \(c_{t-1}\), the operations inside the LSTM cell can be formulated as  
\[i_{t}=s(U_{xt}x_{t}+W_{hi}h_{t-1}+b_{i}) \tag{8}\]  
\[f_{t}=s(U_{xt}x_{t}+W_{hf}h_{t-1}+b_{f}), \tag{9}\]  
\[o_{t}=s(U_{x0}x_{t}+W_{ho}h_{t-1}+b_{o}), \tag{10}\]  
\[g_{t}=y(U_{xt}x_{t}+W_{hc}h_{t-1}+b_{c}), \tag{11}\]  
Figure 6: The structure of the encoder. Each stream is constructed on the proposed Res-C3D, as depicted within the area of yellow and blue rounded rectangles. The orange arrows show the residual connections from the motion stream to the appearance stream, and \(\odot\) indicates the element-wise multiplicative operation. The final feature vector is the combination of outputs of fc6 in the appearance stream and fc7 in the motion stream. Figure 7: The structure of an LSTM unit. \[c_{t}=f_{t}\cdot c_{t-1}+g_{t}\cdot i_{t}, \tag{12}\]  
\[h_{t}=o_{t}\cdot y(c_{t}), \tag{13}\]  
where \(i_{t}\), \(f_{t}\), \(o_{t}\), and \(g_{t}\) denote the input gate, forget gate, output gate, and input modulation gate at time \(t\), respectively. \(U\), \(W\), and \(b\) denote the input weights, recursive weights, and bias vector. \(s(x)=(1+e^{-x})^{-1}\) denotes the sigmoid activation function, and \(y(x)=(e^{x}-e^{-x})/(e^{x}+e^{-x})\) denotes the hyperbolic tangent function. \(g_{t}\) is a vector that offers candidate values to update the memory cell, which is calculated from the present input and previous state by the tanh activation function. The input gate \(i_{t}\) and the input modulation gate \(g_{t}\) control what to write to the memory cell, while the forget gate \(f_{t}\) controls what previous information transmitted from the past to discard. The output gate \(o_{t}\) keeps the information for forthcoming operations that control the output of the cell at time \(t\). After updating the memory cell as \(c_{t}\), the hidden state \(h_{t}\) at time \(t\) can be calculated by element-wise multiplication of the output gate vector \(o_{t}\) and current memory cell state \(o_{t}\) after projection by the tanh function. Stacking two LSTM layers of opposite directions together in our network serves as the point of departure for the use of the bidirectional temporal structures of feature vector sequences. In contrast to the outputs of standard single-layer LSTM at time \(t\), the combined outputs of BiLSTM layers are decided by not only the cues from previous vectors but also cues from upcoming vectors. Therefore, by integrating the extra information from future data, BiLSTM networks are capable of generating higher-level global context relationships of sequential data from videos. The CNN feature vectors from the encoder serve as the visual representations of each video clip, which are ordered as temporal sequences. [PERSON] et al. (2017) employed the LSTM network to model the temporal dependencies of the features, which can be expressed as:  
\[v_{t}=g(f_{t}|v_{t-1}), \tag{14}\]  
where \(f_{t}\) indicates the feature vectors modeled at time \(t\), \(v_{t}\) denotes the temporal information, and \(g\) denotes the modeling function for temporal relationships.