# Semi-CNN Architecture for Effective Spatio-Temporal Learning in Action Recognition
## 4 Experiments
### Evaluation on Validation Dataset  
This section presented the evaluation results for our architecture and the comparison of its performance with the corresponding 3D-CNN model. Since a previous study [7] has demonstrated that 3D-CNN outperforms 2D-CNN in action recognition tasks, we did not explicitly train 2D-CNN models for our comparison here.  
The validation results on the UCF-101 Split-1 dataset are presented in Table 3. These results were obtained after training for 50 epochs, which might not have fully converged, but were presented here for comparison and reference. These results show a significant improvement on the prediction accuracy for our semi-CNN when applied on all the different models--VGG, ResNet and DenseNet. For the VGG-16 model, our architecture outperforms C3D [7] by 18% in accuracy. For ResNet models, an average of 16% improvement was achieved, while for the Densenet-121 model, the boost was almost 30%. Deeper networks of ResNet and DenseNet (such as ResNet-101 and DenseNet-169) were not presented here as the validation results deteriorate for both 3D- and semi-CNN architectures, demonstrating that the UCF-101 dataset was too small to be trained for a deep 3D network, as stated in [9; 22]. Figure 2 illustrates the comparison of the training performance for the models listed in Table 3.  
\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**Model**} & **2D-CNN** & **3D-CNN** & \multicolumn{2}{c}{**Semi-CNN**} \\ \cline{2-5}  & **Params** & **Params** & **Pre-Trained Params** & **Total Params** \\ \hline VGG-16 & 134.7 M & 179.1 M & 5.3 M & 82.2 M \\ ResNet-18 & 11.4 M & 33.3 M & 0.4 M & 31.7 M \\ ResNet-34 & 21.5 M & 63.6 M & 0.8 M & 60.5 M \\ ResNet-50 & 23.9 M & 46.4 M & 0.9 M & 45.8 M \\ ResNet-101 & 42.8 M & 85.5 M & 0.9 M & 84.8 M \\ ResNet-152 & 58.5 M & 117.6 M & 1.4 M & 115.6 M \\ DenseNet-121 & 7.2 M & 11.4 M & 0.8 M & 10.4 M \\ DenseNet-169 & 12.8 M & 18.8 M & 0.8 M & 17.9 M \\ \hline \hline \end{tabular}
\end{table}
Table 2: Number of training parameters in millions(M) for VGG, ResNet and DenseNet models.  
In Figure 2, the gap between the training and the validation plots (blue and orange lines) for 3D-CNN models was much bigger as compared to the plots in semi-CNN. This indicates that overfitting occurs when we train a more complex model (or a deeper network) on a relatively small dataset. Our architecture learned faster as could be seen in the steeper slope, which is a result of transfer learning, which initializes the parameters in our spatial convolution layers. For deeper networks such as ResNet-50 and DenseNet-121, huge fluctuations could be seen in the validation loss of the 3D architecture, which could mean that the network is overfitting as it could not generalize its training parameters on the validation set. Although fluctuations also occur in our semi-CNN architecture, it is more stable and the validation loss is progressively reduced when trained for more epochs.  
This empirical study provides strong support for our semi-CNN architecture in that a fusion of convolution layers outperforms a full 3D convolution network, with additional advantages on: (1) lower number of training parameters, (2) transfer learning from pre-trained models is feasible and (3) more stable training and faster convergence. Due to our limited computational resources, we only presented our validation results on shallower networks on UCF-101 dataset, and we expected that the performance could be generalized to deeper networks when applied on larger datasets.