## 4 Proposed Methodology
### Expression and Speech Feature Fusion Model with Attention Mechanism  
The expression and speech feature fusion model based on the attention mechanism is shown in Figure 9. This study selects CvT, the recognition model with the best performance in both facial expression and speech modalities, as the feature extraction network through comparison experiments. The expression and speech data are transmitted to the attention feature fusion module following their respective feature extraction networks to form a novel fusion sequence. Finally, the corresponding emotion categories are output through the fully connected layer. Figure 8: Bi-LSTM model structure. Figure 9: Multimodal feature fusion model structure. The asterisks (*) in the image represent the number of convolutional projection blocks in each stage: *1, *2, and *10 indicate there are 1, 2, and 10 convolutional projection blocks in the first, second, and third stages, respectively. The attention feature fusion network is the core part of the model which uses the Multiscale Channel Attention Module (MS-CAM) to compute the weights of the two modalities and to generate the foci of the facial and speech features. The high-dimensional features of facial expressions and the Mel spectrogram acquired in the previous stage are utilized as input to the network, i.e., \(X_{face}\) and \(X_{speech}\in R^{\mathbb{C}\times H\times W}\), respectively. The module employs two branches to derive the channel attention weights. One branch uses global average pooling to extract the attention of the global features and the other branch directly employs pointwise convolution to extract the attention of the local features, thus acquiring the weight values \(W_{face}\) and \(W_{speech}\), respectively. The MS-CAM computes attention weights through the following two parallel branches:  
1. Global Attention Branch: \[F_{global}=\sigma(W_{2}\delta(W_{1}\mathrm{GAP}(X)))\] (7) where \(GAP\) denotes Global Average Pooling, \(W_{1}\in\mathbb{R}^{\mathbb{C}/r\times\mathbb{C}}\) and \(W_{2}\in\mathbb{R}^{\mathbb{C}\times\mathbb{C}/r}\) are the weights of two convolution layers, \(\delta\) represents the ReLU activation function, and \(\sigma\) is the sigmoid function. 2. Local Attention Branch: \[F_{local}=\sigma(W_{4}\delta(W_{3}X))\] (8) where \(W_{3}\in\mathbb{R}^{\mathbb{C}/r\times\mathbb{C}}\) and \(W_{4}\in\mathbb{R}^{\mathbb{C}\times\mathbb{C}/r}\) are the weights of the convolution layers. 3. The final attention weights are computed as a weighted sum of these two branches as follows: \[M(X_{face}\oplus X_{speech})=\alpha F_{global}+(1-\alpha)F_{local}\] (9) where \(\alpha\) is a learnable parameter, and \(\oplus\) denotes the concatenation operation. Finally the feature vector \(Z_{fusion}\) is computed as follows: \[Z_{fusion}=M(X_{face}\oplus X_{speech})\otimes X_{face}+(1-M(X_{face} \oplus X_{speech}))\otimes X_{speech}.\] (10)  
The specific processing and parameter variations of the attention feature fusion network are shown in Table 5. The input image, with size (3, 224, 224), is transformed into feature vectors \(X_{face}\) and \(X_{speech}\) of size (384, 14, 14) using the CvT feature extraction network. After the initial feature fusion, it is fed into the two branches of local attention and global attention, respectively. The extracted local feature shapes are (384, 14, 14) while the global feature shapes are (384, 1, 1). The outputs from the two attention branches are then linearly summed and passed through the Sigmoid function to calculate the weight values \(W_{face}\) and \(W_{speech}\), which are subsequently multiplied by \(X_{face}\) and \(X_{speech}\), respectively, resulting in the fusion features with the shape of (384, 14, 14). Finally, through the fully connected layer, the four classification results are output to predict the emotion of a child with autism. In the feature fusion model, we use ReLU activation functions in the local attention branch to introduce non-linearity while maintaining computational efficiency.