# TBRNet: Two-Stream BiLSTM Residual Network for Video Action Recognition
## 2 Related Works  
### Video Action Recognition  
Before the dramatic success achieved by CNNs, approaches based on hand-crafted features have gained initial progress, nevertheless most of them only suitable for non-realistic actions performed in simple backgrounds. Those approaches utilize the feature detectors such as 3D-SIFT [13] to represent videos in the form of Histogram of Oriented Gradient (HOG) [14], Histogram of Optical Flow (HOF) [16], Motion Boundary Histogram (MBH) [15] or dense trajectories [24], and the local representations are taken into consideration using a spatio-temporal pyramid [25] with Bag-of-Words (BoW) [16] or Fisher  
Figure 2: The change in frames in a “basketball dunk” over time. The yellow circles mark the changes in action, and several discriminative frames are outlined in red.  
vector-based encoding [17] to feed into classifiers. Improved dense trajectories (iDT) [17], which uses Fisher vector as an encoder and SVM as a classifier, becomes one of the most significant approaches based on them. [PERSON] et al. [18] aggregated the temporal coherent descriptors, such as HOG, HOF, MBH and fisher vectors, into a multiple kernel learning algorithm which computes the optimal kernel and parameters from a large set of kernels to reduce the bias.  
Over several recent years, as CNNs have thrived, researchers tended to introduce them into video action recognition [2; 11; 12]. [PERSON] et al. [11] proposed a method to extract the spatial feature of each frame by using a pre-trained 2D CNN, and then fuse the spatial features of continuous frames in the last stage to obtain the classification result. However, the results turned out to be less satisfying, since the model was unable to effectively extract temporal features. Inspired by the two-stream networks proposed by [PERSON] et al. [2], many two-stream-based approaches, such as seen in Reference [19], were introduced which capture appearance features in spatial stream using RGB frames and motion features in temporal stream using optical flow frames, where each stream processes individually. With a simple but effective extension from 2D convolution to 3D convolution [12], 3D CNNs can encode the spatial and temporal information simultaneously and show promising results. However, the main deficiency of those 3D CNN methods [4; 12; 21; 26] is the lack of the ability for modeling long-term temporal dependencies, since they only capture motion features from fixed-number frames. [PERSON] et al. [20] point out that the long-term temporal patterns are important for action recognition. They encode the video using dynamic images which are designed to summarize a series of frames on the video level.  
Recurrent Neural Network (RNN), especially LSTM, attract much concern, since they are effective for handling temporal sequence encoding tasks. LSTMs show improvements when cooperating with features extracted by CNNs [1; 5; 6; 27; 28]. Long-term recurrent convolutional network (LRCN) [6] is an end-to-end framework that classifies the action in video sequences using LSTM with features abstracted by CNN. [PERSON] et al. [28] utilize a longer temporal pooling window with 3D convolution kernels. [PERSON] et al. [1] use bidirectional deep LSTM instead of single-layer LSTM to analyze a certain time interval feature sequence.