# Semi-CNN Architecture for Effective Spatio-Temporal Learning in Action Recognition
## 2 Related Work  
A two-stream architecture [3; 4; 5] consists of two separate 2D CNNs to train a classifier for each of the spatial and temporal features. The prediction scores from both streams will then be fused to form the final prediction. On the other hand, a 3D CNN [6; 7; 8] can directly learn spatio-temporal information when applied on consecutive frames, without explicitly computing the motion features. However, the number of trainable parameters increases substantially in deep models, leading to overfitting on smaller-scale datasets [9]. Different fusion methods have been investigated, such as fusing two-stream CNN and 3D CNN [10] or mixing 2D and 3D convolutions in an architecture [11], and splitting 3D convolution to 2D spatial and 1D temporal convolutions [12]. These methods have demonstrated improved performances over individual architectures that utilize full 2D or 3D convolutions.  
**3D CNN.** In order to capture both spatial and temporal features across video frames, [PERSON] et al. [6] proposed a 3D CNN model to perform 3D convolution on multiple frames and achieved better performance in action recognition as compared to 2D CNN. [PERSON] et al. [13] improved the model by capturing multiple timescales at different layers of the convolutional network; while [PERSON] et al. [7] experimented with different network architectures and kernel sizes for 3D convolutions. [PERSON] et al. [8] built a 3D ResNets architecture, by exploiting the effectiveness of residual learning in deep 2D ResNets [14]. [PERSON] et al. [15] trained 3D CNNs on both image frames and motion features using multiple spatio-temporal scales, to obtain combined results.  
**Two-stream CNN.** [PERSON] and [PERSON] [3] designed a two-stream CNN architecture, which captures spatial and temporal features in separate CNNs before combining them to train a classifier. [PERSON] and [PERSON] [5] trained separate CNNs for video frames and flow signals in region proposals for action prediction in individual frames. Instead of using the full image as input, their system search for region proposals and train the classifier for action prediction in individual frames. The identified action regions will be linked with consecutive frames to form connected motions. Similarly, [PERSON] et al. [16] also employed a human-based region proposals to train a multi-stream CNN that consists of appearance, motion and region features as input stream. [PERSON] et al. [4] achieved real-time action recognition by learning motion vector from a pre-trained optical flow network. [PERSON] et al. [17] encoded features from each CNN stream using temporal linear encoding and concatenate them to form a descriptor. Another work by [PERSON] et al. [18] learned aggregation of features for both spatial and temporal streams, to form a new encoded representation in action classification.  
**Spatio-temporal fusion.** [PERSON] et al. [19] introduced the factorizing of 3D spatio-temporal convolution kernels to 2D and 1D kernels, where their architecture starts from 2D convolution layers and splits into two streams for spatial and temporal encoding. [PERSON] and [PERSON] [10] introduced a fusion of two-stream CNN and 3D CNN, where fusion at different layers was investigated. [PERSON] et al. [11] introduced two variants of spatio-temporal learning. The first one contains both 2D and 3D convolutions in a ResNet model, while the second variant decomposes a 3D convolution into a 2D spatial convolution and a 1D temporal convolution.  
The variant of the architecture that is the most related to our proposed framework is the mixed convolutions network by [PERSON] et al. [11]. The difference is that our architecture performs spatial convolutions at the lower layers, followed by temporal convolutions and spatio-temporal convolutions at the higher layers. In addition, our model outperforms 3D ResNet by up to 18% in the top-1 accuracy for an input of 16 video clips when evaluated on the UCF-101 dataset, and it reduces overfitting issues as faced in 3D ResNet models [9].