# TBRNet: Two-Stream BiLSTM Residual Network for Video Action Recognition
## 4 Experiments
### Comparison with State-of-the-Art Models  
In this section, we compare our proposed model with existing state-of-the-art video action recognition approaches on both UCF101 and HMDB51 benchmark datasets. We categorize these approaches into three types according to the types of extracted features, as reported in Table 5.  
Compared with the approaches that use hand-crafted features [17; 44], our proposed TBRNet yielded significant margins as high as 9.5% and 15.6% on UCF101 and HMDB51. Our encoder of TBRNet was based on a two-stream network [2] and C3D [4] which outperformed these two base models by 4.0% and 6.8% on UCF101. We note that some two-stream-based approaches, such as Two-stream fusion [3] and Spatiotemporal ResNets [38], lead to higher accuracies compared with our proposed two-stream encoder, and this can be explained by the fact that these two approaches with similar fusion strategies among streams use deeper CNNs as base models which need to learn more network parameters. However, with the help of the temporal attention-based BiLSTM decoder, our entire TBRNet achieved improvements of 2.9% and 2.0% on UCF101 and 7.4% and 6.4% on HMDB51, which verifies the benefit of our decoder in capturing long-term dependencies for temporal sequences. Compared with other 3D CNN methods, such as 3D Convolution [12], Res3D [26], and other LSTM-based methods, such as LSTM [5], LRCN [6], Two-stream + LSTM [28], and Multi-LSTM [45], our proposed TBRNet still performed better. From the performances of approaches that use hybrid features, such as TDD [44], C3D, and 3D Convolution, higher accuracies were obtained by taking iDT into consideration. Dense trajectories, calculated by the iDT algorithm, were stronger representations that contained more spatiotemporal features. However, they require higher computation resources compared with the usage of only optical flow. Note that without the BiLSTM decoder to model the long-range temporal dependencies, our two-stream CNN encoder still outperformed C3D and TDD with iDT versions by 1.6% and 0.5% \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Approach** & **UCF101** & **HMDB51** \\ \hline \multirow{4}{*}{Hand-crafted} & HOG [17; 44] & 72.4 & 40.2 \\  & HOF [17; 44] & 76.0 & 48.9 \\  & HOF+MBH [17; 44] & 82.2 & 54.7 \\  & iDT+ Fisher vector [17] & 85.9 & 57.2 \\ \hline \multirow{4}{*}{Deep-learned} & Two-stream [2] & 88.0 & 59.4 \\  & Two-stream fusion [3] & 92.5 & 65.4 \\  & Spatiotemporal resnets [38] & 93.4 & 66.4 \\  & TDD [44] & 90.0 & 63.2 \\  & C3D [4] & 85.2 & - \\  & 3D Convolution [12] & 91.8 & 64.6 \\  & Res3D [26] & 85.8 & 54.9 \\  & LSTM [5] & 88.6 & - \\  & LRCN [6] & 82.9 & - \\  & Two-stream + LSTM [28] & 88.6 & - \\  & Multi-LSTM [45] & 90.8 & - \\ \hline \multirow{4}{*}{Hybrid} & TDD + iDT [44] & 91.5 & 65.9 \\  & C3D + iDT [4] & 90.4 & - \\ \cline{1-1}  & 3D Convolution + iDT [12] & 93.5 & 69.2 \\ \cline{1-1}  & TSN [19] & 94.2 & 69.4 \\ \hline \multirow{2}{*}{Ours} & Encoder of TBRNet & 92.0 & 65.2 \\ \cline{1-1}  & TBRNet & 95.4 & 72.8 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Compared performances of our proposed TBRNet with state-of-the-art approaches on UCF101 and HMDB51. on UCF101, which shows the strong ability of our proposed encoder to capture spatiotemporal features from RGB frames and optical flow frames in a fixed-length temporal window. In addition, TSN [19] with iDT is pre-trained with the help of a large-scale ImageNet dataset which provides high diversity for training. Despite that, our proposed TBRNet also showed superior performance. Finally, we demonstrate that our TBRNet with an encoder-decoder framework provided global spatiotemporal information complementarily and achieved classification accuracies of 95.4% and 72.8% on UCF101 and HMDB51. The performances for all action classes in UCF101 and HMDB51 are given in Figure 10, and these confusion matrixes are well diagonalized. Some categories are easy to recognize, such as \"pull-up\" and \"ride bike\" in HMDB51, \"BenchPress\" and \"BandMarching\" in UCF101.