# Best Frame Selection to Enhance Training Step Efficiency in Video-Based Human Action Recognition
## 3 System Overview
### Video Shortening  
According to previous research in this field, various methods have been used to find keyframes and to select a suitable sequence according to the application. The proposed method does not necessarily need to find the best sequence, but only the frames that convey the relevant information about human actions. The computational speed of the proposed method is an essential feature because many HAR systems work online. Based on these specifications, the proposed method uses a gradient operator to extract the images edges, i.e., the relevant places of each image frame, and then the difference between the edges of consecutive frames, i.e., the movement that is modeled by the difference between the edges of two consecutive frames, is taken into account in order to calculate an approximate estimate of the action information. The gradient difference of the frames can indicate the amount of movement in main locations. Due to the low calculations of the gradient operator relative to the usual motion detection operators, such as optical flow, this approach shows a lower runtime. The pseudo-code of the proposed method is given by Algorithm 1.  
Figure 1: Flowchart of the proposed method.  
Firstly, the input video is separated into its frames (\(F_{i}\)). In the next step, the gradient operator extracts the edges of all of the frames \(G_{F^{V}}\). The gradient operator can use different masks. In the Sobel and Prewitt masks, the gradient of each pixel is a weighted sum of a 3-by-3 neighborhood; the Roberts mask uses a 2-by-2 neighborhood; and the \"central difference\" and \"intermediate difference\" masks are column vectors. Figure 2 shows the different gradient masks in the vertical direction; in the horizontal direction, the masks are transposed.  
The results of different gradient masks are presented in the \"Result and Discussion\" section. The absolute value of the gradient difference of the consecutive frames is used as a fast, relatively low and simple operator in order to calculate the motion information (\(\Delta G_{F^{V}}\)).  
Figure 3 shows nine different frames of an input video, their gradient and the absolute value of the difference between consecutive frame gradients after normalization. The normalization is carried out based on the maximum value of all differences; in addition, if the gradient difference in a pixel is less than the specified threshold, it is discarded.  
Selecting a proper value for \(P_{\textit{thresh}}\) is important. If the gradient of the two consecutive frames difference in a pixel after normalization is less than \(P_{\textit{thresh}}\), it is assumed that no motion has occurred at that point between the two frames. The effect of \(P_{\textit{thresh}}\) on the accuracy of the HAR system that used the proposed pre-processing method in the input is given in the \"Result and Discussion\" section. The total energy of each difference is calculated in line 5 of Algorithm 1, and frames with energy lower than \(E_{M}\) are discarded. In the case of Figure 1, the proposed algorithm eliminated frames 2, 5 and 6. In step 6 (the last one) of Algorithm 1, the energy of the remaining frames is added together in a sliding window with length \(N_{frames}\). The window where the summation is maximum is the sequence with maximum motion information. This is because the zero-energy frames have been removed, and the selected window will contain more edge motion information than any other parts of the video. This edge motion, with an acceptable approximation, can include action information. Based on the simulation results, almost all of the videos that were separated from the original human action video in this way had relatively complete information about the involved action. After this step, the frames belonging to the selected window are submitted to the second part of the algorithm.  
Figure 2: Different Gradient masks: (**a**) Sobel, (**b**) Prewitt, (**c**) Roberts, (**d**) Central difference, and (**e**) Intermediate difference.