# Modelling a Spatial-Motion Deep Learning Framework to Classify Dynamic Patterns of Videos
## 3 Methods
### Dual-3 DCNNLSTM Model  
For video representation, it is easy to extend 2D CNNs by using them to extract features from each frame, then pooling their predictions through the whole video. This direct application is convenient but ignores the temporal connections among frames. To overcome this issue, some researchers add a recurrent layer such as LSTM to encode the state and capture the temporal ordering of frames [32].  
Figure 2: Process of state-exchanging package \((P_{t})\) working with the sequence of LSTMs. First, get both previous cell state values \(C_{t-1}\) and \(C^{\prime}_{t-1}\) of RGB and optical flow stream, average the value as in Equation (8) and calculate the value of the state-exchanging \(P_{t}\). Then, share this newly learned information between both streams’ next time step. Finally, update the next time step’s previous cell state via \([C_{t-1}+P_{t}]\). Likewise, update and reset previous cell states until the last time step.  
However, since 2 DCNN ignores the temporal information, it seems it is not a natural representation. Because of this drawback, the 3 DCNN concept, which can capture both spatial and temporal features using 2D spatial dimensions and one temporal dimension, is proposed [27]. Even though 2D CNN, together with LSTM, can extract high variation level features, it is unable to extract fine low-level features. Furthermore, 3 DCNN can extract fine motion features, but it does not support large variations since it depends on the filter [47]. Therefore, together with LSTM, a 3 DCNN model can give better classification results for the dynamic patterns of the videos. Moreover, the practical way to model short temporal video data is by fusion of the appearance features of a single frame and the stack of optical flow after passing through the two streams.  
Considering the advantages of combining CNN and LSTM networks, our baseline two-stream architecture Dual-3 DCNNLSTM, which was also used to classify hand gestures in our IC4 You project, consists of a 3 DCNN network followed by a stack LSTM layer [16]. The filter size of each Conv3D layer is \(3\times 3\times 3\), and the stride and padding are \(1\times 1\times 1\). The feature maps consist of four filter sizes doubled within each layer (32, 64, 64, and 128), and each number increases the depth. We applied batch normalization, which allows much higher learning rates and is less reliant on initialization to accelerate the model training. Each 3 DCNN layer is followed by a batch normalization layer, a ReLU layer, and a 3D max pooling layer with a pooling size of \(3\times 3\times 3\). Features are extracted from the 3 DCNN architecture and then fed into the one stack of LSTM layer with 256 hidden states. Also, we have added several dropout layers in every section with a value of 0.5 and then computed the output probability result, using the softmax layer in the end. Using both optical flow and RGB as the input data might produce a better result than using only one stream input. Thus, the two-stream model of the 3 DCNN + LSTM is considered as a baseline.