# Multi-Scale Audio Spectrogram Transformer for Classroom Teaching Interaction Recognition
## 1 Introduction
In terms of performance, CRNN methods lack global modeling of audio and are limited in terms of recognition accuracy; the Transformer-based audio classification method is computationally intensive and has poor applicability in limited device scenarios. In application, human emotions play an important role in classroom learning. Based on AffectNet [26], the largest database of facial expressions, values and aproalsals in the wild, excellent visual analysis methods for facial emotions have emerged. Emotion-GCN [27] uses graph convolutional networks (GCNs) to recognize facial expressions in the wild. EmoAffectNet [28] consists of a robust pre-trained backbone model and a temporal subsystem to many video frames with good generalization by modeling the temporal dependence of the frames. These methods have been transferred and used to identify student behavior and academic emotions in the classroom [29; 30; 31; 32]. Among them, Multi-task EfficientNet-B2 [33] identifies emotions and engagement in online learning and is more innovative. Existing approaches to intelligent classroom analysis are often based on a single expression or behavioral feature, but visual features are just as important as audio features within a real classroom environment. Students' behavior and emotions provide an initial indication of their learning status, and students' verbal interactions with the teacher reflect a progressive experience within the classroom. The development of multimodal machine learning informs this, with multimodal recognition enhancing the interactive use of different information. In [34], multimodal fusion networks performed modal complementation and weighting, with more robust emotion recognition results. Therefore, the introduction of audio modality for classroom audio interaction recognition is necessary to provide a more integrated analysis of classroom assessment based on classroom behavior and academic emotion recognition. In summary, we propose an audio classification method based on the Transformer [35] architecture: Multi-scale Audio Spectrogram Transformer (MAST), which further improves the recognition performance through a multiscale self-attention mechanism and a pre-trained model of ImageNet [36]. Unlike the method of AST, which requires long training time and consumes a large amount of GPU resources, the multi-scale approach reduces the number of parameters and arithmetic power requirements of the model. At the same time, in order to obtain rich feature representation, multi-channel spectrogram features are constructed to enhance the robustness of the model and realize the migration from the visual model to the audio model. Finally, based on global modeling, the time and frequency axes are separately reinforced with local enrichment to extract the time and frequency band features corresponding to each token output from the backbone network. This method is applied to the smart classroom to identify classroom interactions by automated means. The main contributions and innovations of this paper are as follows:  
* The multi-channel features enhance the representation of the original signal by the audio spectrogram features while solving the problem of inconsistent input data dimensions during the transition from the visual model to the audio model. * We propose the MAST model to achieve long-range global modeling of the audio spectrogram. The multi-scale self-attentive mechanism reduces the complexity of the model, as well as focuses on the vital information of the audio spectrogram and eliminates the interference of redundant information caused by multi-channel features. * We constructed a classroom teacher-student interaction recognition dataset based on audio features and a real classroom audio database, providing basic data for the training of classroom interaction recognition models and classroom interactive evaluation. We also pioneered the application of audio classification models to smart education, which provides a certain reference for subsequent smart classroom research.