# On the Use of Deep Learning for Video Classification
## 3 Video Classification
### Developments in Video Classification over Time
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Categories** & **Working Principle** & **References** \\ \hline \multirow{8}{*}{2D- CNNs} & These are image based models where & \\  & frame level feature extraction is & \\  & performed using CNN architecture and & \\  & classification is performed using & \\  & state-of-art classification models, for & \\  & example SVM. & \\ \hline \multirow{3}{*}{3D-CNNs} & 2D image classification extension to 3D & \\  & for video (For example the Inception 3D & [56] \\  & (I3D) architecture). & \\ \hline \multirow{3}{*}{Spatiotemporal Convolutional Networks} & To aggregate the temporal and the spatial & \\  & information, these methods primarily & [54; 57; 58] \\  & depend on convolution and pooling. & \\ \hline \multirow{3}{*}{Recurrent Spatial Networks} & To represent temporal information in & \\  & videos, recurrent neural networks such as & [47; 53; 59; 60] \\  & LSTM or GRU are used. & \\ \hline \multirow{3}{*}{Two/multi Stream Networks} & In addition to the context frame visuals, & \\  & these methods use layered optical flow to & [50; 61; 62; 63] \\  & identify movements. & \\ \hline \multirow{8}{*}{Mixed convolutional models} & Models built with the ResNet architecture & \\  & in mind. They are particularly interested & \\  & in models that utilize 3D convolution in & \\  & the bottom or top layers but 2D in the & \\  & remainder; these are referred to as & [64; 65] \\  & “mixed convolutional” models. Or the & \\  & methods based on mixed temporal & \\  & convolution with different kernel sizes. & \\ \hline \multirow{2}{*}{Hybrid Approaches} & These are models based on integration of & [66; 67; 68] \\  & CNN and RNN architectures. & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: _Cont._spatial networks use recurrent neural networks (RNN) to model temporal information in videos, such as LSTM or GRU. The ResNet architecture is used to build mixed convolutional models. They are particularly interested in models that utilize 3D convolution in the bottom or top layers but 2D in the remainder; these are referred to as \"mixed convolutional\" models. These also include methods based on mixed temporal convolution with different kernel sizes. Advanced architectures based on DenseNet have also shown promising results for the video classification task. Some of these notable architectures based on DenseNet include region-based CNN (R-CNN) [70; 71], faster R-CNN [72; 73], and YOLO [74]. Besides these architectures, there are also hybrid approaches based on the integration of CNN and RNN architectures. A summary of these architectures is provided in Figure 6.  
The different deep learning architectures described above employ different fusion strategies. These fusion strategies are either for the fusion of different features extracted from the video or for the fusion of different models used in the architecture. The fusion strategies mainly used for the extracted features are (i) concatenation, (ii) product, (iii) summation, (iv) maximum, and (v) weighted, where the concatenation approach simply combines all the features together, and all the features are used for classification. The product/summation approach performs the product/summation between the features extracted using different strategies and uses the result of product/summation to perform classification. The maximum approach takes the maximum value of the features extracted using different strategies and uses that for classification. The weighted approach gives different weights to different features and performs the classification using the weighted features. Different fusion methods are summarized in Figure 7.  
Figure 6: Summary of video classification approaches. Figure 7: Different Fusion Types.