# Modelling a Spatial-Motion Deep Learning Framework to Classify Dynamic Patterns of Videos
## 6 Discussion  
Our work has presented a novel two-stream architecture called State-exchanging LSTM (SE-LSTM), which is an extension of general LSTM to classify the dynamic patterns in videos more efficiently. To the best of our knowledge, no previous studies could distinguish the spatial and motion feature exchanging within long-term temporal phase while training. For this work, we selected videos under three conditions to maintain the fairness of our fireworks and hand gesture dataset and control the scope of our study. Each video should have one fireworks and gesture type, videos should not contain any other moving objects except the target objects, and the background should be static with invisible objects. Therefore, our model is not suitable for classifying multiple targeting objects--for instance, more than one type of firework or hand gesture within one video. The proposed model is the best fit with the video data, which have static backgrounds. Sample fireworks video types, which are not included in our dataset, are shown in Figure 9.  
Figure 8: Class-wise accuracy among the best 10 classes of HMDB51 dataset.  
Although previous works based on video classifications, discussed in Section 2, presented several good models, these prior studies did not address well the spatiotemporal relationship between consecutive time steps. However, our proposed model integrates short-term motion, spatial, and long-term temporal to classify the dynamic patterns. Moreover, the proposed Dual-CNNSELSTM architecture interact with the spatial and motion streams. The way that the cell states, which store the learned history of both streams' previous time steps, were extracted and averaged to be an input to the next time steps as additional information is a novel contribution of this work. Consequently, based on experimental results with different benchmarks, it is demonstrated that interaction with the spatial and motion streams within the training phase could more significantly improve performance than standalone two-stream networks.  
However, one crucial factor in this kind of classification problem is the mutual exclusiveness of the classes. For better classification results, it is necessary to have independent features among the classes but similar features inside a class. As we know, humans are error-prone. Thus, even though we categorize carefully while creating our datasets to maintain the uniqueness of each class, we had some mislabeled classification results, as shown in Figure 10, because of the relatively similar types of classes.  
In the fireworks dataset, some samples of the _Crossette_ class were wrongly classified as _WaterFlower_ class and some videos of _WaterFlower_ class were misclassified as _Palm_ class. Moreover, some samples were recognized as _Dasi_ class even though they were originally in the _Drop_ class. Apart from that, in the hand gesture dataset, there are also some mislabeled data. Some videos of the _up_ and _down_ and  
Figure 10: Instances of misclassified samples from (**a**) the hand gestures dataset; (**b**) the fireworks dataset; and (**c**) the HMDB51 dataset.  
Figure 9: Sample fireworks videos that did not satisfy our data selection conditions. (**a**) Videos containing multiple types of fireworks; (**b**) moving objects like ships and airplanes; (**c**) visible and static objects in the backgrounds.  
_zoomIn_ and _zoomOut_ classes are wrongly classified since these videos have late starting positions and early ending positions in the gesture performance. Not only the fireworks and hand gesture datasets but also the HMDB51 dataset had some mislabeled classes. We think this is because some classes consist of videos with a similar context or person present as in Figure 9c. For instance, some videos of _sword_exercise_ and _drow_sword_ classes are acted by the same person and the _swing_baseball_ and _throw_ classes generally contain video of the same baseball tournament. Moreover, _flic_flac_ and _cartwheel_ are two performances of the same gymnastics activity.