# Modelling a Spatial-Motion Deep Learning Framework to Classify Dynamic Patterns of Videos
## 7 Conclusions  
In this work, we propose a two-stream neural network concept, named SE-LSTM, considering the advantages of utilizing both spatial and motion information to classify dynamic patterns of videos. The SE-LSTM extends the general purpose of LSTM by exchanging information with previous cell states of both appearance and motion stream. The modified objective function of the LSTM unit is a new contribution of this work, presented in Section 3.1.2. Combining this SE-LSTM concept with CNNs, we investigated the advantages of communicating spatial and motion features to classify dynamic patterns by implementing the Dual-CNNSELSTM two-stream neural net architecture. This proposed architecture can model static visual features, short-term motion features, and long-term temporal features. In this architecture, first, we extracted spatial and motion features training over two CNNs on static RGB frames and stacked optical flows. Then, for long-term temporal modeling, the two extracted types of features were used separately as inputs of two LSTM networks. During the training, the time step cell states of both streams communicate with each other to share the spatiotemporal features.  
To validate this proposed Dual-CNNSELSTM model, we used fireworks, hand gestures, and HMDB51 datasets during this study. As baseline models of this study, we implemented two single streams and two two-stream learning models. When we were comparing the results of the baseline and the proposed Dual-CNNSELSTM model, the Dual-CNNSELSTM achieved significant accuracy with all experimental datasets. Furthermore, the datasets with both the baseline model and the proposed model significantly outperformed similar state-of-the-art methods with the HMDB51. Based on the conditions in which we created our fireworks and hand gesture datasets as well as the experimental results, we can see some limitations of our model. The videos should not have any other moving objects except the target objects and should have static backgrounds, to attain high classification accuracy. In future work, we plan to remove all the misclassified data from our experimental datasets to reduce the misclassifications. Another promising future direction is to classify the different kinds of dynamic patterns in the video data to enhance the robustness of our proposed model.  
Conceptualization, S.P.K.A.; Data curation, S.P.K.A. and N.L.H.; Funding acquisition, T.K.S.; Methodology, S.P.K.A.; Resources, N.L.H.; Supervision, T.K.S.; Validation, S.P.K.A. and N.L.H.; Writing--original draft, S.P.K.A.; Writing--review & editing, S.P.K.A. and T.K.S. All authors have read and agreed to the published version of the manuscript. This research is funded by the Ministry of Science and Technology (MOST) under the grant number MOST 109-2634-F-008-008. We acknowledge the support of the Ministry of Science and Technology, Taiwan. The authors declare no conflict of interest.  
Figure A2: Visualizing the Dual-3 DCNNLSTM Architecture.  
Figure A3: Dual-CNNSELSTM Output shape and number of weights in each layer.  
Figure A4: Dual-3 DCNNLSTM Output shape and number of weights in each layer.  
Figure 5: Comparison of the Different Architectures reported in Table 2.