# Deep Learning-Based Human Action Recognition with Key-Frames Sampling Using Ranking Methods
## 3 Proposed Methodology
### Architecture of the Proposed Method
#### 3.2.2 Deep Leaning for Human Action Classification  
Over the past years, we have observed significant improvement in the field of machine learning, especially deep learning [49, 50, 51]. Nowadays, deep learning is being widely applied in every field, including detection, recognition, classification, and segmentation. Owing to the challenges of hand-crafted features in human action datasets, deep learning has also become very successful in human action recognition. However, there are very few methods that directly deal with raw human action sequences to recognize action classes. Even though some methods have been directly applied to raw sequences using RNN and LSTM, the performance was comparably lower. Most of the methods have encoded the  
Figure 4: The procedure of balancing a sequence into 16 frames from 13 frames.  
action sequences into spatial formats and then integrated a machine learning or deep learning model to discriminate the actions.  
Instead of representing a whole sequence in a single image for action, we consider sampling \(k\)-ranked frames as the input to the 3D-CNN model. The 3D-CNN can directly extract both spatial and temporal features from raw input frames without encoding them into another domain. We investigate the recognition performance using different deep learning models such as residual network (ResNet) [12], convolution 3D (C3D) [12], I3D [52], R2P1D [53], X3D [53], and 3D-FCNN [46]. Figure 5 depicts the average CCM and SSIM results for DHA, MSR-Action3D, and UTD-MHAD datasets. We experimentally find that a 3D ResNet with 101 layers called ResNet101 works well for the proposed method. As a result, we use ResNet101 for the backbone deep learning model for the whole experiment. A large number of convolutional layers in ResNet101 can learn high-level features and useful functions to obtain the hierarchical representation of action information. The average pooling layer helps extract discriminative features containing spatial and temporal directions.  
The ResNet101 is one of the most effective 3D-CNN models in which residual information of the previous layer is again connected to the current layer. It extracts deeper features as we consider 101 layers. A residual block in ResNet101 consists of convolution, batch normalization, rectified linear unit, convolution, and batch normalization operations, as shown in Figure 6. The input, \(I\), is passed through a residual function, \(F(I)\), and generates features, \(I^{\prime}\), by combining the output from \(F(I)\) with the input.  
Figure 5: Comparison of recognition results for different 3D-CNN models on the UTD-MHAD data using SSIM metric having \(k=24\).  
Figure 6: A residual block in ResNet.  
As we have trained the ResNet101 model for the multi-class classification, we have used the cross-entropy loss function \(Loss_{CE}\), defined as follows:  
\[Loss_{CE}=\sum_{i=1}^{C}y_{i}\log(p_{i}) \tag{3}\]  
where \(C\) is the number of classes and \(y_{i}\) and \(p_{i}\) indicate original and predicted labels, respectively.