# Deep Learning-Based Human Action Recognition with Key-Frames Sampling Using Ranking Methods
## 3 Proposed Methodology
### Architecture of the Proposed Method
#### 3.2.1 Key-Frames Sampling Using Ranking Methods  
Frame sampling is a procedure in which a fixed number of frames are chosen from a sequence of frames based on a criterion. In this research, we define the frame sampling criterion as the rank metric. The reason behind the fixed number of frames sampling is that there are several redundant frames in an action sequence. The redundant frames in most cases, the adjacent frames contain similar types of information in terms of spatial and temporal changes, as shown in Figure 3. Thus, it is very important to discard the unnecessary frames which have the same motion information. Sometimes, it is crucial to determine the best set of frames having better motion information to effectively recognize actions. We use the rank metric to reorganize the frames based on the rank values and then sample \(k\)-ranked frames as the significant set (\(k\) = 16, 20, 24). The rank metric can be different forms based on the intended works. As the frame sampling in an action recognition task, we choose SSIM and CCM metrics.  
The SSIM metric considers the structural properties between two adjacent frames to perform the comparisons. Let \(I_{t}\) and \(I_{t+1}\) be the two adjacent frames at times \(t\) and \(t+1\) with height and width H and W, respectively, in an action sequence. Then, the SSIM values (\(\psi\)) between \(I_{t}\) and \(I_{t+1}\) can be defined as follows:  
\[\psi(I_{t},\ I_{t+1})=\frac{\big{(}2\times\overline{I_{t}}\times\overline{I_{t +1}}+C_{1}\big{)}\big{(}2\times\mathit{var}\big{(}I_{t},\ I_{t+1}\big{)}+C_{2} \big{)}}{\Big{(}2\times\overline{I_{t}^{2}}+\overline{I_{t+1}^{2}}+C_{1} \big{)}\Big{(}\mathit{var}\big{(}I_{t}\big{)}^{2}+\mathit{var}\big{(}I_{t+1} \big{)}^{2}+C_{2}\Big{)}} \tag{1}\]  
Figure 3: 16-ranked key-frames selection using SSIM ranking method.  
Figure 2: \(k\)-ranked key-frames sampling using ranking methods (the background of each depth frames was removed).  
where \(var(I_{t},\ I_{t+1})\) indicates the inter-frame variance; \(var(I_{t})\) and \(var(I_{t+1})\) indicate the intra-frame variances. \(c_{1}\) and \(c_{2}\) are constants.  
Figure 3 illustrates how the 16-ranked frames sampling works. We have a total of 23 frames in the '_Bend_' action of the DHA dataset. We compute the SSIM value between the current frame and its immediate next frame. As given in Figure 3, we can say that the neighboring frames contain almost the same information. Since the larger SSIM values indicate more similarity, we sort the SSIM values in ascending order and sample the lowest 16-frames for the experiments. From Figure 3, we can assume that the sampled 16-frames also bear the same spatial and temporal information as the original sequences, with a little distortion. Similarly, we also perform the experiments and analysis with \(k=20\) and 24 for showing the generality of the proposed ranking method.  
To show the robustness of the proposed system, we also conduct the CCM metric to sample \(k\)-ranked frames. The CCM values (\(\rho\)) between \(I_{t}\) and \(I_{t+1}\) can be calculated by using Equation (2) as follows:  
\[\rho(I_{t},\ I_{t+1})=\frac{\sum_{i=1}^{H}\sum_{j=1}^{W}(I_{t}(i,j)-\overline{ I_{t}})(I_{t+1}(i,j)-\overline{I_{t+1}})}{\sqrt{\left(I_{t}(i,j)-\overline{I_{t}} \right)^{2}\left(I_{t+1}(i,j)-\overline{I_{t+1}}\right)^{2}}} \tag{2}\]  
where \(\overline{I}\) indicates mean value of \(I\). The overall procedure is the same as the sampling of frames based on SSIM values.  
For sequences with the number of frames less than \(k\), we duplicate every nth frame for making them the same length as \(k\). The overall scenario for balancing the samples in the short sequence is depicted in Figure 4. We use the '_kick_' action from the DHA dataset having 13 frames for the illustration of the balancing procedure. Since we need \(k\)-frames (\(k=16\)) for training the deep learning, we duplicate the 4 th, 8 th, and 12 th frames for balancing and insert them next to the neighboring frames.