## 3 Learning-Based Action Representation Approach
### Deep Learning-Based Approaches
#### 3.2.2 Discriminative/Supervised Models  
According to the literature survey of human action recognition, the most frequently used model under the supervised category is Convolutional Neural Networks (CNN or Convnet). The CNN [143] is a type of deep learning model which has shown excellent preformation at tasks such as pattern recognition, hand-written digit classification, image classification and human action recognition [142; 144]. This is a hierarchical learning model with multiple hidden layers to transform the input volume into output volume. Its architecture consists of three main types of layers: convolutional layer Convolution and Rectifier Linear Unit (CONV + ReLU), pooling layer, and fully-connected layer as shown in Figure 8. Understanding the operation of different layers of CNN require mapping back these activities into pixel space, this is done with the help of Deconvolutional Networks (Deconvnets) [145]. The Deconvnets use the same process as CNN but in reverse order for mapping from feature space to pixel space. Initially, the deep CNN [143] was used for representation and recognition of objects from still images [142]. This was extended to action recognition from videos in [147] using stacked video frames as input to the network but the results were worse than even the handcrafted shallow representations [48; 72]. This issue was investigated in [148] and they came up with the idea of two-stream (spatial and temporal) CNN for action recognition. An example of two-stream convolutional neural network is shown in Figure 9. These both streams were implemented as Convnet, the spatial stream recognizes the action from still video frames and the temporal stream performs action recognition from the motion in the form of dense optical flow. Afterwards, these two streams were combined using late fusion for action recognition. This method achieved superior results to one of the best shallow handcrafted-based representation methods [48]. However, the two-stream architecture may not be suitable for real-time applications due to its computational complexity. Figure 8: Convolutional Neural Networks layers (source [146]). Most of the deep CNN models for action recognition are limited to handling inputs in 2D form. However, some applications do have data in 3D form that requires a 3D CNN model. This problem was addressed in [149] by introducing the 3D convolutional neural networks model for the airport surveillance. This model uses features from both spatial and temporal dimensions by performing 3D convolutions at the convolutional layer. This method achieved state-of-the-art results in airport video surveillance datasets. The supervised learning CNN model 2D or 3D can also be accompanied by some unsupervised endeavours. One of the unsupervised endeavours is slow feature analysis (SFA) [150], which extracts slowly varying features from the input signal in an unsupervised manner. Beside other recognition problems, it has proved to be effective for human action recognition as well [151]. In [152], two-layered SFA learning was combined with 3D CNN for automated action representation and recognition. This method achieved state-of-the-art results on three public datasets including KTH, UCF sports, and Hollywood2. Other types of supervised models include Recurrent Neural Networks (RNNs). A method using RNN was proposed in [153] for skeleton-based action recognition. The human skeleton was divided into five parts and then separately fed into five subnets. The results of these subnets were fused into the higher layers and final representation was fed into the single layer. For further detail regrading this model reader may refer to [153]. The deep learning-based models for human action recognition require huge amount of video data for training. However, collecting and annotating huge amount of video data is immensely laborious and requires huge computational resources.