## 4 Proposed Methodology  
### Model Architecture Selection Rationale  
When designing emotion recognition systems for children with autism, the selection of appropriate model architectures is crucial due to the unique characteristics of emotional expressions in this population. After carefully evaluating various deep learning architectures, we selected specific models for each modality based on their theoretical advantages and empirical performance. For facial expression recognition, we compared pure transformer architectures (ViT) with hybrid convolutional-transformer models (CvT). While we considered other advanced architectures such as the Swin Transformer [31], with its hierarchical structure and shifted window approach, and ConvNeXt [32], which modernizes traditional CNN architectures, we selected ViT and CvT for our comparative analysis for several reasons. First, they represent two fundamental paradigms in vision processing (pure self-attention vs. convolution-enhanced attention). Second, their architectural simplicity facilitates the direct comparison of how different feature extraction mechanisms perform on the uniquely challenging facial expressions of children with autism, who often display reduced dynamic complexity and increased facial asymmetry [9]. For speech emotion recognition, we evaluated the following two complementary approaches: spectral representation (Mel spectrograms) processed by CvT and sequential feature representation (MFCCs) processed by Bi-LSTM. Although alternatives such as wavelet transforms, Linear Predictive Coding (LPC), and raw audio models like WaveNet ([PERSON] et al., 2016 [33]) were considered, we selected Mel spectrograms and MFCCs, as they remain the gold standard in speech emotion recognition and effectively approximate human auditory perception. The Bi-LSTM architecture was specifically chosen for MFCCs due to its proven capability to model sequential dependencies in temporal data, while CvT was applied to Mel spectrograms to leverage its ability to capture both local spectral patterns and global frequency relationships. Our multimodal fusion approach employs a feature-level fusion strategy with an attention mechanism, which allows the model to dynamically weight the contributions of each modality. This design choice was motivated by the observation that children with autism often express emotions asynchronously across different channels ([PERSON] et al., \begin{table}
\begin{tabular}{c c c c c} \hline
**Emotion Type** & **Training Set** & **Validation Set** & **Test Set** & **Total** \\ \hline Calm & 11,328 & 2834 & 1574 & 15,736 \\ Happy & 7002 & 1752 & 974 & 9728 \\ Sad & 9106 & 2278 & 1266 & 12,650 \\ Angry & 8258 & 2066 & 1148 & 11,472 \\ Total & 35,694 & 8930 & 4962 & 49,586 \\ \hline \end{tabular}
\end{table} 
Table 2: Numbers of Facial expression image and speech Mel spectrogram datasets. \begin{table}
\begin{tabular}{c c c c c} \hline
**Emotion Type** & **Training Set** & **Validation Set** & **Test Set** & **Total** \\ \hline Calm & 160,474 & 96,672 & 39,444 & 296,590 \\ Happy & 138,168 & 29,982 & 19,494 & 187,644 \\ Sad & 203,224 & 22,192 & 14,592 & 240,008 \\ Angry & 176,662 & 20,482 & 20,748 & 217,892 \\ Total & 678,528 & 169,328 & 94,278 & 942,134 \\ \hline \end{tabular}
\end{table} 
Table 3: Number of MFCC datasets. 2013 [8]), necessitating a flexible fusion approach rather than the simple concatenation or averaging of features. We propose an emotion recognition architecture for children with autism based on the fusion of expression and speech features as shown in Figure 5, which consists of the following three stages: comparison, fusion, and prediction. The comparison stage uses single-modal data to conduct model experiments. (1) Emotion recognition based on facial expression involves comparing the performance of two models with different structures, i.e., ViT and CvT, and selecting the best one as the facial expression feature extraction network in the following stage.