# Deep Learning-Based Human Action Recognition with Key-Frames Sampling Using Ranking Methods
## 2 Related Works  
The drawbacks of the inertial, skeleton, and RGB sequences lead to the adoption of depth datasets to effectively recognize human actions. Depth sequences are introduced in the form of one-channel data, known as grayscale. Many researchers have converted the depth sequences into several other formats such as skeleton joints [24; 25; 26; 27], point clouds [28; 29; 30], and spatial-temporal images [31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44] and then extracted discriminative features for classification. Figure 1 shows three examples of depth sequence to the skeleton, point cloud, and dynamic images representation, respectively. Very few methods have conducted depth sequences directly to discriminate human activities [45; 46; 47; 48]. The skeleton data represent the 3D coordinate values of joints in the human body. Very few methods have been published in which skeleton joint information is extracted from depth videos for action classification. [PERSON] et al. [24] extracted joint information from depth sequences and computed the Eigen-joints feature, which combined static posture, motion, and offset of action. They conducted a naive Bayes nearest neighbor (NB-NN) classifier for recognizing actions. [PERSON] et al. [25] also estimated the joint information from depth maps to discriminate human action. They determined the histogram features from skeleton joints and used linear discriminant analysis for re-projection to perform the classification. [PERSON] et al. [26] partitioned the body parts embedded in skeleton information, called motion body partitioned. Then, they extracted local features for each part and aggregated them for classification. [PERSON] et al. [27] also conducted a similar approach in which they extracted the skeleton joint information depth sequences. They applied a 1D temporal CNN model along with a 2D spatial CNN for depth motion image (DMI) and a 3D volumetric CNN for raw depth sequences to accomplish action recognition. [PERSON] et al. [28] used an action graph to model the dynamics of actions and collected a bag of 3D points that corresponded to the nodes of the action graph for action discrimination. [PERSON] et al. [29] proposed a point cloud-based human action recognition system in which they introduced a descriptor and a key point detection algorithm. The descriptor was calculated based on a histogram of oriented principal components. In [30], [PERSON] et al. extracted human behavioral features called historical point cloud track features, including depth and skeleton information for action recognition. [PERSON] et al. [31] calculated motion information from depth difference and average depth images by dividing the silhouette bounding box hierarchically and encoded them into motion history images (MHIs). They used translation, scale, and orientation invariant Hu moments to encode the MHIs. [PERSON] et al. [32] applied a filtering method for determining the spatio-temporal interest points from the depth (DSTIP) sequence and used the depth cuboid similarity feature (DCSF) to describe the local depth cuboid around DSTIP. [PERSON] et al. [33] designed a feature map called depth histogram of oriented gradient (MHI  
Figure 1: Different techniques to encode the depth sequence into a spatio-temporal image; **(a)** skeleton, **(b)** point cloud, and **(c)** dynamic image from depth sequence. HOG) and modeled actions through k-means clustering. They evaluated the proposed method using HMM model. [PERSON] et al. [34] described a new feature descriptor based on depth motion map (DMM), contourlet transform (CT), and HOG. They applied the CT on DMM and computed HOG features for classification. [PERSON] et al. [35] extracted salient depth map (SDM) and binary shape map (BSM) features from depth sequences and formed a bag of map words for action classification. [PERSON] et al. [36] encoded the depth sequences into three DMMs along the front, side, and top projections. From the three DMMs, they computed contourlet-based HOG (CL-HOG) and local binary pattern (LBP) features.