## 4 Methods and Experiment
### Keyframe Extraction
To

Figure 4:1 Pretrained architecture for flow color visualization feature extraction. Forward LSTM equations:

Figure 4:2

Figure 4:3

At time step \(t\), \(h_{t}^{f}\) is the forward hidden state generated by \(LSTM_{f}\), by processing \(X_{t}\), an input feature vector typically from a video keyframe. \(h_{t-1}^{f}\) and \(c_{t-1}^{f}\) represent the prior forward hidden and cell states at \(t\)\(-\)1, capturing essential temporal information. \(c_{t}^{f}\), the forward cell state at time step \(t\), is the output of cell\({}_{\text{f}}\) processing \(X_{t}\), influenced by \(h_{t-1}^{f}\) and \(c_{t-1}^{f}\), collectively preserving critical information in the video sequence. Backward LSTM equations:

Figure 4:4

Figure 4:5

In time step \(t\), \(h_{t}^{b}\) represents the backward hidden state, which \(LSTM_{b}\) generates by processing \(X_{t}\) in a reverse sequence, often starting from the end of the input sequence \(h_{t+1}^{b}\) and \(c_{t+1}^{b}\) denote the subsequent backward hidden and cell states at \(t+1\), capturing information concerning the sequences' upcoming context. \(c_{t}^{b}\), the backward cell state at time step t, results from \(cell_{b}\) processing \(X_{t}\) in reverse order, under the influence of

Figure 4:6 BiLSTM-AT network architecture: the output from the BiLSTM module at each time step t is concatenated with the forward \(h_{t}^{f}\) and backward \(h_{t}^{b}\) hidden layers. This concatenated vector is then used as the input to the attention module. The attention module infers a variable length alignment weight vector \(a_{t}\) for each time step \(t\). This vector is used to compute a weighted sum \(C_{t}\) of the hidden states \(h_{t}\) of the BiLSTM module at each time step \(t\). This weighted sum is then used as the output \(Y_{t}\) of the attention module. and \(c_{t+1}^{b}\). This arrangement collectively preserves vital information about the sequence's future context. Concatenation of Hidden States: For each time step \(t\), we combine the forward and backward hidden states to create the final representation, \(h_{t}\):

Figure 4:7

This concatenation ensures that we capture the context in both the forward \(h_{t}^{f}\) and backward \(h_{t}^{b}\) directions, enabling a more comprehensive understanding of the keyframe sequence. Attention Mechanism: In the optional inclusion of an attention mechanism, we compute attention scores \(A_{t}\) for each keyframe \(X_{t}\) based on a query (\(Q\)), key (\(K\)), and value (\(V\)) vectors. This mechanism enhances the model's ability to focus on essential frames within the sequence:

Figure 4:8

where \(A_{t}\) is the attention score for the keyframe \(X_{t}\), reflecting the model's emphasis on the keyframe at time step \(t\). _Softmax_ is the mathematical function that converts the dot product of the query \(Q_{t}\) and transposed key \(K_{t}\) vectors into a probability distribution for easier interpretation. Context Vector Calculation: The context vector \(C\) is generated as a weighted sum of the value vectors \(V_{t}\) using the attention scores \(A_{t}\):

Figure 4:9

where \(C\) summarizes information from the selected keyframes by assigning more importance to those frames identified as crucial by the attention mechanism. As part of the attention mechanism, each keyframe feature vector \(X_{t}\) is element-wise multiplied by the context vector \(C\) to produce an attended representation \(Y_{t}\). This process allows the model to prioritize and enhance the significance of features within each keyframe, thus improving the accuracy of student behavior classification. ### Experiments

Our methodology employs multiple stages: First, pre-processing by frame extraction and selection are performed using the SADLK algorithm. Second, features are extracted from various pretrained models to obtain feature fusion. Lastly, behavior classification based on a hybrid model using a deep sequence neural network is utilized to classify student behavior based on the extracted features.