# Modelling a Spatial-Motion Deep Learning Framework to Classify Dynamic Patterns of Videos
## 1 Introduction
Therefore, to overcome the drawbacks of traditional deep learning methods, first, it is necessary to select a suitable neural network method to design an accurate two-stream architecture, which can utilize the spatial and motion information. Here, such a design should consider the spatiotemporal relationship to reach a higher performance than that of the existing two-stream models [17; 18]. In this study, as our first contribution, we introduce a novel concept state-exchanging long short-term memory (SE-LSTM). The proposed SE-LSTM concept considers the advantages of maintaining the spatial-motion relationship and Section 3.1 presents several functional parts of this concept in detail. It thus modifies the general internal structure of the LSTM unit. Explicitly, the cell state contains information learned from the previous time steps of a particular stream. However, the proposed SE-LSTM is designed by adding some additional information on cell states previously learned from both streams to the consecutive time steps of each stream. During this process, we encode previously learned information in memory cells, which are regulated with nonlinear gates to identify temporal dependencies. Hereinafter, this spatiotemporal information-sharing process is known as state-exchanging. Utilizing this SE-LSTM, as a second contribution, our study proposes a promising design: a deep neural network based on a two-stream architecture, named Dual-CNNSELSTM. The proposed Dual-CNNSELSTM architecture applies the SE-LSTM to overcome the limitations of CNN and RNN and integrates short-term motion, spatial, and long-term temporal to classify the dynamic patterns of videos. Two CNN streams were trained to model the spatial information of RGB videos and short-term motion features of stacked optical flows, and the dense optical flow method was used to create optical flow videos for the motion stream. The two LSTM networks followed by CNNs are employed to model the long-term temporal information, over the frame-level spatial and temporal information extracted by the top CNN phases. Afterwards, the proposed SE-LSTM concept interacts with this extracted information handling the cell states, which keeps the learned history of both streams'previous time step. Then we averaged the extracted cell information to be an input to recent time steps as additional information. Finally, we concatenate both spatial and motion features to evaluate the class probabilities to classify the dynamic patterns of input data. Since our study will classify the dynamic patterns of videos, this study considers fireworks [19] hand gestures [16], and human action videos from the HMDB51 dataset [20] to verify our proposed model. Furthermore, to demonstrate the significance of our proposed architecture, we conduct the performance examination under three datasets and standard HMDB51 benchmarking, finding that the performance boosts dramatically with the use of several state-of-the-art methods. Based on experimental results with different benchmarks, it is demonstrated that the interaction of the spatial and motion streams within the training phase could achieve a better performance than that of the previously discussed standalone two-stream networks. The rest of the paper is organized as follows. Sections 2 and 3 discuss the related work and methods. Next, datasets and experimental evaluations are discussed in Sections 4 and 5, respectively. Finally, Section 6 presents the conclusion and future work.