## 4 Methods and Experiment
### Keyframe Extraction  
Analyzing videos often involves processing a large number of frames, many of which may contain redundant information. This leads to a time-intensive undertaking, resulting in low accuracy. Nevertheless, using keyframes [30; 31] can substantially enhance the processing speed of video analysis. We adopt the SAD [32] in the LUV color space [33], focusing on the L channel and the K-mean algorithm (SADLK) for the frame extraction pre-processing and select a keyframe before transferring it to the multiple pretraining model for feature extraction. The keyframe selection is performed according to the Algorithm 1 given below [34]. Figure 3: Process pipeline of the proposed model. ``` Input: Video frames (RGB) Keyframe count (K) Output: Selected keyframes Procedure:
1. Step 1: Convert RGB to L1 IV in the L channel
2. For each framein range(\(1,\mathrm{N}+1\)):
3. // Given \(\alpha=0.2126729,\ \beta=0.7151522,\ \gamma=0.072175\)
4. \(\mathrm{Y_{i}=\alpha\cdot R_{i}+\beta\cdot G_{i}+\gamma\cdot B_{i}}\)
5. \(\mathrm{L_{i}=116\cdot f(Y_{i}/Y_{n})-16}\)
6. // Store \(\mathrm{L_{i}^{*}}\) values in \(\mathrm{LUV_{L}}\) array
7. End for
8. Step 2: SAD Calculation
9. For each framein range(\(1,\mathrm{N}+1\)):
10. \(\mathrm{SAD_{i}=\sum_{(x,y)}\ |\ \mathrm{LUV_{L}[i](x,y)-LUV_{L}[i+1](x,y)}\ |}\)
11. End for
12. Step 3: Keyframe Selection using K-means
13. // Initialize cluster centroids \(C\) with K random SAD values
14. While centroids are not converged
15. For each \(\mathrm{SAD_{i}}\) in \(\mathrm{SAD_{L}}\)
16. // Assign \(\mathrm{SAD_{i}}\) to the closest cluster \(\mathrm{C_{i}}\)
17. \(\mathrm{C_{i}=argmin}\big{|}\mathrm{SAD_{i}}-\mathrm{C_{i}}\big{|}\big{|}\)
18. End for
19. End while
20. Step 4:
21. Sort selected keyframes by frame sequence number (ID value)
22. // Output the selected keyframes
23. Return Selected keyframes in chronological order End Procedure

Figure 4: The overall framework for student behavior classification that used a multi-feature fusion network with keyframe selection to classify a behavior type. A hybrid approach network based on deep sequence deep neuron network focusing on the BiLSTM-AT model has been used for the classification. ### The Pretrained Models

The application of multiple features is beneficial for recognizing correct behaviors. In addition, color information is important, and is necessary to provide consistent scene luminosity, regardless of the condition of the captured image. Features of the optical flow also provide critical information regarding moving trajectories to distinguish behaviors. For each video, we extracted ten original RGB keyframes. We applied one technique of rotation, flipping, and shifting to each keyframe for geometric transformation and a color transformation for data augmentation purposes. Furthermore, we included an optical flow image for each keyframe. Thus, we obtained 40 keyframes per video clip in total. To obtain the necessary features for the classification of student behavior in the classroom, we use four pretrained models: Inception-V3, ResNet50-V, VGG-16, and EfficientNet-B7 for features including the RGB color, augmented data, luminosity in the L*a*b* color space, and optical flow, respectively. These multiple features are concatenated and fed into the classification network. Visualizations for RGB, DA, L*a*b*, and the optical flow keyframe are shown in Figure 5.

* **Inception-V3:** The Inception-V3 is a highly regarded CNN architecture developed by Google AI that has made significant strides in the field of computer vision. It is known for its ability to extract high-level features from images, which has consistently led to superior performance compared to other CNN architectures. The establishment of Inception-V3 was a turning point in network scaling strategies. According to [PERSON] et al. [35], the architects behind Inception-V3 aimed to leverage additional computational resources efficiently, resulting in innovative techniques such as factorized convolutions and advanced regularization.