# Research on Students' Action Behavior Recognition Method Based on Classroom Time-Series Images
## 3 Proposed Method
### Detection of Students' Action Behavior
#### 3.1.4 MsTA Module  
In the field of video action recognition, existing research uses fixed convolution kernels or operations to learn the temporal features of videos. The AIA network used in this paper also uses fixed \(3\times 1\times 1\) convolution to learn the temporal features of students' action behaviors in the classroom scene. However, different action behaviors last for different durations; so, it is crucial to capture the multi-scale temporal features. This paper proposes a Multi-scale Temporal Convolution Unit (MsTCU) with different temporal convolution kernels, which utilizes different temporal sliding windows to extract the multi-scale temporal information. The Double Attention (DA) block of A\({}^{2}\)-Net [34] is applied to the fast pathway of the backbone network of the action detection model in this paper and cascaded with the abovementioned MsTCU to form an MsTA module, which extracts the local temporal features and global temporal features to model the long-distance interdependencies. The MsTCU and the DA block are described below. **MsTCU. ** As shown in Figure 4, the feature map \(X\in R^{C\times T\times H\times W}\) first passes through the three branches of MsTCU, which are equipped with different temporal convolution kernels, respectively, and perform temporal convolution processing on the feature map \(X\) to obtain multi-scale temporal features as follows:  
Figure 3: Interaction block and IA structure: (**a**) O-Block; (**b**) Dense Serial IA structure. (1) \(X\in R^{\mathbb{C}\times T\times H\times W}\) passes through three branches with temporal convolution kernel sizes of 3, 5, and 7, respectively, and the dimensionality of the output feature maps is constant in size, denoted as \(\left[X_{0}^{\prime},X_{1}^{\prime},X_{2}^{\prime}\right]\) Afterward, the feature maps of the three branches are summed up element by element:  
\[F_{1}=X_{0}^{\prime}+X_{1}^{\prime}+X_{2}^{\prime} \tag{2}\]  
(2) To maintain spatiotemporal consistency, \(F_{1}\) extracts the spatial features of the video through three dilated convolutional branches with convolutional kernel size 3 and dilation rates 1, 2, and 3, respectively. The output feature maps are represented using \(\left[X_{0}^{\prime\prime},X_{1}^{\prime\prime},X_{2}^{\prime\prime}\right]\). Afterward, the feature maps of the three branches are concatenated together and the channel is reduced in dimension by \(1\times 1\times 1\) convolution:  
\[F_{2}=f^{1\times 1\times 1}\big{(}Cat\big{(}\big{[}X_{0}^{\prime\prime},X_{1}^{ \prime\prime},X_{2}^{\prime\prime}\big{]}\big{)}\big{)} \tag{3}\]  
where \(f^{1\times 1\times 1}\) denotes the \(1\times 1\times 1\) convolution, \(F_{2}\in R^{\mathbb{C}\times T\times H\times W}\). **DA block. ** As shown in Figure 5, let the input be \(X\in R^{\mathbb{C}\times T\times H\times W}\) and the local feature of each spatiotemporal location \(i=1,\cdots,THW\) be \(v_{i}\). Then, define Equation (4):  
\[z_{i}=F_{distr}\Big{(}G_{gather}(X),v_{i}\Big{)} \tag{4}\]  
There are two operations in the equation. First, the \(G_{gather}\) operation adaptively aggregates the features of the global context; then, the \(F_{distr}\) operation assigns the aggregated global context features back to each location \(i\) based on the local features \(v_{i}\) and outputs \(z_{i}\). \(z_{i}\) can be obtained by steps (1) and (2):  
Figure 4: Multi-scale Temporal Convolution Unit. Figure 5: DA block. (1) Feature Gathering. The DA block uses bilinear pooling to capture the second-order statistics of features to generate a global representation. Bilinear pooling is the summation of the outer product of all feature vectors \((a_{i},b_{i})\) in the two input feature maps \(A\) and \(B\):  
\[G_{bilinear}(A,B)=AB^{T}=\sum_{\forall i}{a_{i}{b_{i}}^{T}} \tag{5}\]  
where \(A=[a_{1},\cdots,a_{thw}]\in R^{m\times thw}\) and \(B=[b_{1},\cdots,b_{thw}]\in R^{n\times thw}\).