# Modelling a Spatial-Motion Deep Learning Framework to Classify Dynamic Patterns of Videos
## 5 Experimental Evaluation
### Evaluation of HMDB51 Dataset  
The fireworks and hand gestures datasets are our own datasets, while HMDB51 is an accessible benchmark dataset. Therefore, it is necessary to compare it with recent similar DL-based works in order to illustrate the robustness of our models. Since HMDB51 is a popular benchmark, this dataset provides three training-testing splits with 70% per category for training and 30% per category for testing. Hence, we also used the same ratio for all the experiments. Comparisons of the average accuracies of our experimental models and the state-of-the-art methods are given in Table 2. Both the Dual-3 DCNNLSTM and Dual-CNNSELSTM architectures' layer-wise weights and neurons are given in Appendix A. In addition, all of the compared architectures in Table 2 are summarized in Figure A5. Compared to the original two-stream architecture [6], our proposed Dual-CNNSELSTM model and the baseline model improves by 10.46% and 8.44% with fusion by the SVM method respectively. Even though this two-stream network is able to do multi-task learning and SVM fusion, the novelty of our models is higher. Another interesting comparison is with Two-stream 3 DCNN [48], which exploits both RGB and optical flow independently and then concatenates them with an early fusion method. This work achieved an average accuracy of 62.54%, while our baseline model, which also used 3 DCNN on top of the LSTM layer, achieved 67.84%. Moreover, compared with the Two-stream Fusion ConvNet architecture [51], which used VGG16 for both spatial and temporal streams, our model achieved an accuracy increase of 4.46%--from 65.4% to 69.86%. Furthermore, the baseline model achieved a testing accuracy increase of 2.44%, from 65.45% to 67.84%.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Method** & **Accuracy** \\ \hline Two-stream model (fusion by averaging) [6] & 58.0\% \\ Two-stream SVM [6] & 59.4\% \\ Two-stream 3 DCNN [48] & 62.54\% \\ Two-stream Fusion [51] & 65.4\% \\ Two-Stream Fusion + IDT [51] & 69.2\% \\ ST-ResNet [52] & 66.41\% \\ ST-ResNet + IDT [52] & 70.03\% \\ Dual-3 DCNNLSTM (ours) & 67.84\% \\ Dual-CNNSELSTM (ours) & 69.86\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results comparison of HMDB51 dataset with our models and the state-of-the-art methods.  
Figure 7: Training accuracies of the Dual-CNNSELSTM model with the hand gestures, fireworks, and HMDB51 datasets.  
Moreover, though ST-ResNet [52], a novel implementation that generalizes ResNets for the spatiotemporal domain by introducing residual connections, achieved slightly higher accuracy than Two-stream Fusion [51], it could not outperform our baseline and proposed model. However, both Two-stream Fusion [51] and ST-ResNet [52], when further tested to see what benefit results from a late fusion of hand-crafted IDT features, achieved higher accuracies of 69.62% and 70.03%, respectively. Overall, among all of the state-of-the-art models compared, except for the Two-Stream Fusion + IDT [51] model's slightly increasing gap of 0.17%, our proposed Dual-CNNSELSTM model clearly shows a significant accuracy of 69.86% with the HMDB51 dataset.  
Since the fireworks and hand gestures datasets are balanced datasets, all the classes consist of an equal number of videos. However, the HMDB51 dataset is imbalanced in that its classes include a different number of sample videos. Therefore, for further clarification, we computed the F1-score of the HMDB51 dataset using class probability scores in the softmax layer of the networks for each class. We used this measurement to find the most beneficial classes. Figure 8 shows the best 10 classes' F1-score of the proposed Dual-CNNSELSTM model with identical class scores of the Dual-3 DCNNLSTM model. When we compared the top 10 beneficial classes of both models, we found that only the _eat_, _laugh_ and _stand_ classes had slightly higher scores with the Dual-3 DCNNLSTM model.