# Temporal Modeling on Multi-Temporal-Scale Spatiotemporal Atoms for Action Recognition
## 4 Evaluation
### Model Visualization  
In CNN-based action recognition, it is difficult to explain the meaning of the extracted deep information using systemic and complete theory. However, visualization is an effective technique for gaining further insight into the trained CNN model and the extracted features. In addition to the visualization of the spatiotemporal atoms in Section 3.3, in this section, we visualize the convolution kernels and feature maps of Res3D and present the embedding visualization of the atoms. Figure 10 visualizes the extracted Conv1 feature maps of 16 consecutive frames (4, 2 and 1 clips for 4f, 8f and 16f Res3D, respectively). The figure shows that Res3D mainly captures the apparent (spatial) information from the first frame of the clip and captures the motion (temporal) information from the remaining 3, 7 and 15 frames of the clip. This observation confirms that different temporal scale information is extracted by the Res3D variants. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Methods** & **UCF101** & **HMDB51** & **CNN Architecture** & **Input of CNN** \\ \hline \multicolumn{5}{c}{**Handcrafted Representation Methods**} \\ \hline iDT [47] & 85.9\% & 57.2\% & – & – \\ iDT+HSV [48] & 87.9\% & 61.1\% & – & – \\ \hline \multicolumn{5}{c}{**Deep Learning Methods (2D CNN)**} \\ \hline SlowFusion [6] & 65.4\% & – & AlexNet & RGB \\ TDD [49] & 90.3\% & 63.2\% & ZFNet & RGB + OF \\ Two-stream & 86.9\%/ & 58.0\%/ & \\ (Averaging/SVM) [7] & 88.0\% & 59.4\% & CNN-M & RGB + OF \\ Improved Two-stream [14] & 92.5\% & 65.4\% & VGG-16 & RGB + OF \\ RankPooling [50] & 91.9\% & 65.4\% & VGG-16 & RGB + OF \\ AdaScan [51] & 89.4\% & 54.9\% & VGG-16 & RGB + OF \\ Transformation [52] & 92.4\% & 62.0\% & VGG-16 & RGB + OF \\ Trajectory Pooling [53] & 92.1\% & 65.6\% & VGG-16,CNN-M & RGB + OF \\ Multi-Source [54] & 89.1\% & 54.9\% & VGG-19 & RGB + OF \\ ConvPooling [23] & 88.2\% & – & GoogLeNet & RGB + OF \\ SlowFusion [6] & 65.4\% & – & AlexNet & RGB \\ \hline \multicolumn{5}{c}{**Deep Learning Methods (3D CNN)**} \\ \hline C3D [12] & 85.2\% & – & C3D & RGB \\ VLAD3 [55] & 90.5\% & – & C3D & RGB \\ LTC [20] & 91.7\% & 64.8\% & LTC & RGB + OF \\ Res3D [13] & 85.8\% & 54.9\% & Res3D & RGB \\ P3D-ResNet [56] & 88.6\% & – & P3D-ResNet & RGB \\ C3D [12] & 85.2\% & – & C3D & RGB \\ \hline \multicolumn{5}{c}{**Deep Learning Methods (2D CNN-RNN)**} \\ \hline LRCN [22] & 82.3\% & – & ZFNet & RGB + OF \\ LSTM [23] & 88.6\% & – & GoogLeNet & RGB + OF \\ Hybrid Deep [57] & 91.3\% & – & VGG-19,CNN-M & RGB + OF \\ Unsupervised [24] & 84.3\% & – & VGG-16 & RGB + OF \\ Multi-stream [58] & 92.6\% & – & VGG-19,CNN-M & RGB + OF \\ LRCN [22] & 82.3\% & – & ZFNet & RGB + OF \\ \hline \multicolumn{5}{c}{**Deep Learning Methods (3D CNN-RNN)**} \\ \hline Our method & 92.9\% & 66.7\% & Res3D & RGB + OF \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Comparison of our method with state-of-the-art methods. Figure 11 visualizes the learned conv1 kernel of the pre-trained Res3D, RGB 8f Res3D and OF 8f Res3D. Compared with RGB Res3D, OF Res3D is more different from the pre-trained Res3D. This result confirms that our settings speed up the process of kernel updating when the learned knowledge is transferred to a new modality. The embedding visualizations of the mined atoms in RGB stream of the 4f, 8f and 16f temporal scales are illustrated in Figure 12. We average the atoms for each action and project them to 2D space using t-SNE [59]. The figure shows that the mined atoms all are semantically separable and can be used to represent video effectively. Figure 11: Visualization of the kernels. The shape of the 64 Conv1 kernels is \(3\times 3\times 7\times 7\) (C*T*H*W). The visualization shows the learned Conv1 kernels with T = 2, which are arranged in \(8\times 8\) with a \(4\times\) upscale. Figure 12: Atoms embedding visualizations of the 4f, 8f and 16f temporal scales on UCF101 training split-1. The averaged atom of each action is visualized as a point and the averaged atoms that belong to the same action class have the same color.