# TBRNet: Two-Stream BiLSTM Residual Network for Video Action Recognition
## 4 Experiments
### Datasets and Implement Details  
Created by Florida University, the UCF101 dataset is one of the most challenging datasets for video action recognition in realistic scenes, where the videos are captured with a large diversity of poses, viewpoints, illumination conditions, and cluttered background. It contains 13,320 videos with a spatial resolution of \(320\times 240\) that can be categorized into 101 classes. Each class includes 100-200 videos, and these classes of action are divided into five types: (1) human-object interaction, (2) body motion only, (3) human-human interaction, (4) the playing of musical instruments, and (5) sports.  
The HMDB51 dataset, a more challenging dataset published by American Brown University in 2011, contains more than 6000 video clips in total. The videos are organized as 51 distinct classes of action, and each class has more than 100 clips. The dataset covers highly diverse actions, such as object manipulations, facial actions, body movements, and human interactions. Problems such as poor video quality, a small number of training videos, significant camera motion, and a complex dynamic environment lead to higher intra-class variation, which makes it more challenging to reduce errors during training and testing periods. For both datasets, we adopted the original training/test splits and followed the standard evaluation protocols provided by the authors of these two datasets by averaging the accuracies of three splits as the final result.  
For the two-stream CNN encoder, we adopted the pre-computed RGB and optical flow frames of videos from Reference [3] as the inputs of our model. We divided each video into \(N=12\) segments, which also relates to the time steps of the BiLSTM decoder sequences. We sampled \(t=8\) frames of RGB frames and the corresponding \(t\) optical flow frames for each segment and used them as the input modalities. The settings of \(N\) and \(t\) are based on our experiments, which show optimal results in complexity and accuracy. Since the number of videos provided by datasets is limited, we adopted the same data augmentation strategy as [4] to mitigate the effect of overfitting. We resized the frame to \(128\times 171\) and randomly cropped them into \(112\times 112\). Thus, one input segment has a size of \(8\times 112\times 112\). Additionally, we horizontally flipped them with a 55% probability to further augment the training samples.  
As UCF101 is a larger dataset than HMDB51, we first trained our two-stream encoder on UCF101 to extract visual CNN features and then transferred it for HMDB51. As depicted in Figure 6, the merged output of fc6 from the appearance stream and fc7 from the motion stream were used to represent features of a video segment, which is a vector of 8192 dimensions. For each video, 12 feature vectors were fed into BiLSTM with 8192-dimensional hidden states. The outputs of the BiLSTM network with a weighted sum attention pooling were fed into another LSTM, together with the CNN visual features, to obtain the final global spatiotemporal representations. In the end, those spatiotemporal representations were optimized by the softmax cross-entropy loss to predict the actions.  
Figure 9: The approach to using the residual connection to combine the CNN features and the BiLSTM representations. \(\oplus\) denotes the attention mechanism effects on the BiLSTM model.  
We trained the model with the stochastic gradient descent (SGD) algorithm and set the batch size to 32 to perform the batch normalizations because of the memory constraints. The learning rate was set to 0.01 in the first 2000 iterations and then lowered at a fixed ratio every 100 iterations. For better training and validating, we also applied a momentum of 0.9 and an early-stopping strategy during the process of training. In addition, we applied a dropout probability of 0.8 to the fully connected layers in the encoder and also applied the decoder with a dropout of 0.5 to bidirectional connections.