# Multi-Scale Audio Spectrogram Transformer for Classroom Teaching Interaction Recognition
## 3 Methods
### Multi-Scale Audio Spectrogram Transformer  
#### 3.3.1 Pooling Attention  
Similar to traditional convolutional networks, the network construction process continuously reduces the time-frequency resolution and increases the channel dimensions, in  
Figure 3: Construction of Log-Mels feature maps.  
Figure 2: Extraction process of multi-channel spectrogram features.  
which Multi-Head Pooled Self-Attention (MHPSA) is the basis for flexible scaling, and each Transformer block contains Multi-Head Pooled Self-Attention (MHPSA). The left part of Figure 4 is a single Multi-scale Audio Spectrogram Transformer block, where Multi-Head Pooled Self-Attention (MHPSA) is shown in the right half of Figure 4. Unlike AST [25], in which the channel dimension and the time-frequency resolution are kept constant, MHPSA pools the feature tensor to reduce the sequence length of the data. For a specific MHPSA, the input tensor \(\mathrm{X}\in\mathbb{R}^{L\times D}\) goes through a linear layer mapping tensor of \(\hat{\mathrm{Q}},\hat{\mathrm{K}},\hat{\mathrm{V}}\), the weight dimension of the linear layer is \(\mathrm{D}\times\mathrm{D}\) and the output mapping tensor is still \(\mathbb{R}^{L\times D}\).  
\[\hat{\mathrm{Q}}\ =\mathrm{XW_{Q}}\ \ \ \hat{\mathrm{K}}\ =\mathrm{XW_{K}}\ \ \ \hat{\mathrm{V}}=\mathrm{XW_{V}} \tag{3}\]  
The pooling convolution is next computed in each dimension using \(\mathcal{P}(\cdot,\Theta)\) for \(\hat{\mathrm{Q}},\hat{\mathrm{K}},\hat{\mathrm{V}}\). \(\Theta\) denotes the use of a pooling kernel with parameter (k,s,p), k denotes the size of the pooling kernel, s is the pooling kernel step size, p is the pooling padding scale and the pooled sequence length \(\tilde{\mathrm{L}}\) is as in Formula (4). For the individual \(\hat{\mathrm{Q}},\hat{\mathrm{K}},\hat{\mathrm{V}}\) intermediate tensor, the pooling operation produces the pooled scaled \(\mathrm{Q}=\mathcal{P}(\hat{\mathrm{Q}},\Theta_{\mathrm{Q}})\), \(\mathrm{K}=\mathcal{P}(\hat{\mathrm{K}},\Theta_{\mathrm{K}})\) and \(\mathcal{P}\mathrm{V}=(\hat{\mathrm{V}},\Theta_{\mathrm{V}})\). The pool attention is calculated as in Formula (5), \(\mathrm{E}^{(\mathrm{red})}\) is the relative position encoding and \(\sqrt{\hat{\mathrm{d}}}\) is used for row-by-row normalization of the inner product matrix.  
\[\tilde{\mathrm{L}}=\left\lfloor\frac{\mathrm{L}+2\mathrm{p}-\mathrm{k}}{ \mathrm{s}}\right\rfloor \tag{4}\]  
\[\mathrm{Attention(Q,K,V)}=\mathrm{Softmax}(\mathcal{P}(\hat{\mathrm{Q}}, \Theta_{\mathrm{Q}})\mathcal{P}(\hat{\mathrm{K}}\,,\Theta_{\mathrm{K}})^{ \mathrm{T}}+\mathrm{E}^{(\mathrm{red})}/\sqrt{\hat{\mathrm{d}}})\mathcal{P}( \hat{\mathrm{V}}\,,\Theta_{\mathrm{V}}) \tag{5}\]  
Different from the traditional single invariant scale attention calculation (NA) in AST, in the pooling attention calculation (PA) the time-frequency resolution is continuously reduced in the Transformer block. Thus, the corresponding reduction in computational complexity and \(\mathrm{s_{Q},s_{K},s_{V}}\) are the scaling factors for the time-frequency resolution. The  
Figure 4: Multi-scale self-attention block.  
computational complexity of the two mechanisms for an audio patch tagging of f \(\times\) t with data dimension D is:  
\[\text{NA}: \mathcal{O}(\text{ftD}^{2}+(\text{ft})^{2}\text{D}) \tag{6}\] \[\text{PA}: \mathcal{O}(\text{ftD}^{2}+\frac{(\text{ft})^{2}}{\text{sqSk}}\text{D}) \tag{7}\]  
where pooling attention reduces the second complexity term by a factor of \(\text{sqSk}_{\text{S}}\). As the network deepens, the pooling attention module extracts key audio patch tokens by filtering, keeping the relationships calculated in consecutive frequency bands and time frames across the global range.