# 3D-CNN-Based Fused Feature Maps with LSTM Applied to Action Recognition
## 3 The Proposed Approach
### Extraction of Spatio-Temporal Fused Features  
#### 3.1.1 Generation of Motion Map  
A motion map is a powerful and compact representation of a video which can be useful in computer vision tasks. The motion map can visualize motion information in good manner, and can remove a large amount of information redundancy of the video, thereby revealing discriminative information. The calculation of the motion map is fast, and takes up fewer memory resources. Hence, using a map to represent the video has realistic requirements. Our propose model is very simple to implement and can be trained by increasing the training video length iteratively. Mainly, it is very helpful to solve the problem of videos of various lengths to get the same effect of the map representation, and also to integrate the temporal information into a map without losing the discriminative information of the video. Another advantage of this method is that we can extract a constant number of video frames per second, which improves the generalization performance of the network. We can utilize a 3D-convolutional neural network for the extraction of the motion map. 3D convolution and 3D pooling operations are adopted in 3D ConvNets. Three-dimensional convolution is the extension of 2D convolution. The output of the 2D convolution are two-dimensional feature maps, while the output volume of 3D convolution can have multiple-dimensions. Each feature map of the convolutional layer is connected with some successive adjacent frames in upper layer. As a result, the temporal information is not lost and the motion of the human body can be efficiently captured. Hence, multiple 3D convolutional layers can be used to handle the spatial and temporal information of the inputs in a hierarchal way.  
For a video \(V\) with \(N\) frames, we define the video frames as \(\textit{fi}\), \(i\) {\(1\dots\)\(\dots\)\(N\)}. \(F_{i}\) denotes the motion map from \(\textit{f}_{1}\) to \(\textit{f}_{i}\). In order to retain appearance and action information, we introduce an iterative method to generate a new motion map \(F_{i+1}\) using Equation (1) by combining the current motion map \(F_{i}\) with the future video frame \(\textit{f}_{i+1}\) by using MMN. Symbol \(\oplus\) is the pixel-wise addition between motion map and video frame. The process of generating our first and final motion map is shown in Figure 1a,b respective  
\[F_{i+1}=F_{i}\ \oplus f_{i+1} \tag{1}\]In the last iteration of our MMN network, we obtain the final motion map \(F_{N}\) of video \(V\), in which discriminative information is embodied and can be applied for action recognition tasks. Some motion maps generated by our C3D network are listed in Figure 2. Each map highlights the static object with its main features, and the superposed silhouette incarnates the different locations and postures of the actor and objects. For example, the action category playing violin shows that the actor, arms and violin are the main features, while the rest of the image is diluted. This shows the relationship between the arm movements and playing the violin. It reflects the motion relationship between the actor and the object, and proves that the dynamic information which is available in different sequences of the video can be retained and embodied in the motion map.