## 4 Experimental Setup and Analysis of Results
### Ablation Experiment
The adaptive fusion model (blue triangular dotted line) outperforms the others in handling synchronization issues, exhibiting a low WER not only at a frame shift of 0 but also across a wider range of frame shifts. This indicates strong resilience to time synchronization errors. However, the sparse adaptive fusion model (green square dotted line) achieves the lowest WER at a frame shift of 0, slightly outperforming the adaptive fusion model. This suggests that the inclusion of the sparse attention mechanism offers advantages when precise time synchronization is critical. In summary, the adaptive attention fusion mechanism excels at addressing time synchronization errors in AVSR tasks, significantly outperforming the sparse normal fusion model without the adaptive mechanism. Additionally, the sparse adaptive model  
Figure 8: Comparison of the word error rates (WERs) of the models under different signal-to-noise ratios; avg denotes the average signal-to-noise ratio. demonstrates strong robustness within a specific frame shift range, further confirming the effectiveness of the adaptive mechanism in managing audio-video desynchronization. For training, the model was pretrained before the formal training phase to allow it to learn basic features and accelerate convergence. A total of 11 iterations were performed, with each iteration enabling the model to learn an increasing number of words. The WER after each iteration is shown in Figure 10.  
Initially, the increase in the number of words was gradual, allowing the model to adapt progressively. The increment was later accelerated, doubling every few steps, to identify the inflection point in the model's performance. Overall, all three models exhibited relatively high WERs with a small number of words, likely due to the insu  
Figure 10: Comparison of the word error rates (WERs) for different numbers of pretrained words in the AV modality, with one pretrained word for the 1 st iteration, two pretrained words for the 2 nd iteration, and 37 pretrained words for the 11 th iteration. Figure 9: Comparison of the word error rates (WERs) of the model in the AV modality with different frame shifts, where a positive frame shift indicates that the video frame is shifted backward (left), and a negative frame shift indicates that the video frame is shifted forward (right). contextual information. The best performance was achieved when the number of words ranged from 9 to 21, allowing the models to effectively leverage contextual cues for more efficient and accurate recognition. Among the models, sparse attention and adaptive attention each have their respective strengths and weaknesses. Sparse attention performs well with a small to medium number of words, effectively focusing on important information and reducing computational complexity. In contrast, adaptive attention excels with a medium volume of data, dynamically adjusting attention weights to improve recognition accuracy. With larger datasets, sparse attention maintains stability, whereas adaptive attention shows a slightly higher WER, yet still retains its adaptability. Combining both approaches capitalizes on their individual strengths, thereby enhancing the overall performance and stability of the model across different data volumes.