# TBRNet: Two-Stream BiLSTM Residual Network for Video Action Recognition
## 3 Proposed Approach
### Temporal Attention Mechanism  
When we are looking at a dog, we usually focus on the body of the dog and try to observe it clearly while roughly glancing at other regions, such as the background. The attention mechanism is first implemented as a spatial version for image perception, which directs the visual attention to the salient part of images and pays less attention to other useless areas. Similarly, the contribution provided by every single input element from sequential data to abstract semantic content is not equal. For example, the start and the end segments of videos usually carry less vital information compared with some key segments located in the middle of videos. The principle of the temporal attention mechanism is to decide \"when\" to look by automatically directing high levels of focus to those vectors that contain the most valuable information while directing relatively low-level focus to those containing less information according to the importance and relevance of the tasks. In our work, we employed the temporal attention mechanism in the BiLSTM network to simulate the process of focusing attention among all segments of videos, where the gradients of the loss function of BiLSTM can be backpropagated through both the BiLSTM network and the attention network.  
The BiLSTM with the temporal attention mechanism is depicted in Figure 8. Specifically, given the input vector \(x_{it}\), where \(t\in[1,T]\), the hidden state of each time step encoded by BiLSTM can be formulated as  
\[\vec{h_{it}}=L\vec{STM}(x_{it}), \tag{16}\]  
\[\vec{h_{it}}=L\vec{STM}(x_{it}), \tag{17}\]  
where the forward and backward directions of the input sequences are denoted as \(\leftarrow\) and \(\rightarrow\), respectively. The final hidden state of BiLSTM can be expressed as:  
\[h_{it}=[\vec{h_{it}},\ \ \vec{h_{it}}]. \tag{18}\]  
Then, the attention-based temporal attribute vector \(o_{t}\) is calculated by the weighted average:  
\[o_{i}=\sum_{t=1}^{T}\beta_{it}\cdot h_{it} \tag{19}\]  
Here, \(\beta_{it}\) denotes the attention weights of the \(i\)-th BiLSTM output vector at time \(t\), which is computed by the following equations:  
\[\gamma_{it}=\tanh(W_{a}h_{it}+b_{a}), \tag{20}\]  
\[\beta_{it}=\frac{exp(\gamma_{it})}{\sum_{j=1}^{T}exp(\gamma_{it})}, \tag{21}\]  
where \(W_{a}\) is the weight matrix used to project the current input \(h_{it}\) of BiLSTM into another space, and \(b_{a}\) is a bias vector. We utilize the nonlinear _tanh_ activation function for the frame selection gate as the relevance vector \(\beta_{it}\) and normalize it through a softmax function. In other words, the temporal attention mechanism can be learned as a special representation of the time steps of interest to restrain the redundancy and noises.  
Then, we employed another LSTM for the final representations of videos by taking the integration of the CNN visual features extracted by the encoder and the merged outputs of BiLSTM together as inputs. As illustrated in Figure 9, there is an identical mapping residual connection from the output layer of the encoder to the bidirectional representations, which constructs a highway between layers and is thereby more conducive to gradient propagation through the network and the abstraction of global spatiotemporal representations.  
Figure 8: Connections depicting the structure of the BiLSTM model with the attention mechanism at a time step.