# 3D-CNN-Based Fused Feature Maps with LSTM Applied to Action Recognition
## 2 Related Work  
Over the last decade, researchers have presented many hand-crafted and deep-net-based methods for action recognition. Earlier works were based on hand-crafted features for non-realistic actions videos. Since the proposed method is based on deep neural network (DNN), in this section, we will only review related works based on DNN. In recent years, different variants of deep learning models have been proposed for human activity recognition in videos, and have achieved great performance for computer vision tasks. [PERSON] et al. [14]applied 3D convolutional kernels on video frames in a time axis to capture both spatial and temporal information. [PERSON] et al. (2015) directly applied CNNs to multiple frames in each sequence and obtained the temporal relations by pooling, using single, late, early and slow fusion; however, the results of this scheme were just marginally better than those of a single frame baseline. [PERSON] and [PERSON] (2016) used a two-stream CNN framework to incorporate both feature types, with one stream taking RGB image frames as the input and the other taking pre-computed stacked optical flows. Since optical flow contains only short-term motion information, adding it does not enable CNNs to learn long-term motion transitions. The additional stream significantly improved action recognition accuracy, indicating the importance of motion features. [PERSON] et al. (2013) avoided the need for pre-computing optical flow features through their 3D convolution (C3D) framework, which allows deep networks to learn temporal features in an end-to-end manner. However, C3D only covers a short range of the sequence. [PERSON] et al. (2017) introduced a temporal segment network (TSN) architecture, where a sparse temporal sampling strategy is adopted to model long-term temporal structures. In [PERSON] et al. (2018), [PERSON] et al. study a number of ways of fusing CNN towers in order to take advantage of this spatial-temporal information from the appearance and optical flow networks. However, the CNN-based method only extracts visual appearance features, and lacks the long-range temporal modeling capabilities. Moreover, the CNN-based method ignores the intrinsic difference between spatial and temporal domains. Some researchers have also presented methods by uniting the benefits of both hand-crafted and deep learned features, such as [PERSON] et al. (2018); [PERSON] et al. (2019), and obtained good results. They integrate the key factors from two successful video representations, namely improved trajectories [PERSON] et al. (2018) and two-stream ConvNets [PERSON] et al. (2018). How to combine the benefits of these two kinds of features to design good descriptors has been an active research area. Some research efforts have been carried out using depth imagery such as dynamic images and depth maps. [PERSON] et al. (2018) introduced the dynamic image network to generate dynamic images for action videos. The order of video frames is used as the supervisory information; however, this method loses some discrimination information. [PERSON] et al. (2019) represented a model in the form of depth maps in the context of action recognition. These contributions showed good action recognition results but were insufficient to capture dense and discriminative information in terms of shape, appearance and motion. [PERSON] et al. (2018) used a convolutional gated restricted Boltzmann machine to generate a flow field of the adjacent two frames in the video for action recognition, but this model could not generate a single map to represent a video. Rank pooling ([PERSON] et al., 2019) and Fisher Vector (1992) made an attempt to generate the desire length motion map. However, these methods are unable to model temporal dynamics among video frames. In order to model the temporal dynamics among video frames, RNNs have been considered for video-based HAR. RNN networks provide strength to find and process hidden patterns in time-space data.