## 2 Related Work
### Sentiment Analysis Based on Unaligned Data Sequence Scenarios  
The following is an introduction to sentiment analysis based on word alignment and unaligned data sequences.  
1. Sentiment analysis based on word alignment: In the scenario of unaligned data sequences, previous studies usually used P2 FA [32] to perform the forced word alignment. Previous studies on the fusion of unaligned data sequences have mined the correlation between data of different modalities on the basis of effectively representing unimodal information [33]. Nevertheless, since the performance of sentiment analysis using shallow learning architecture is far from satisfactory, more complex models have been proposed successively [4; 5; 6; 7; 8; 9; 10; 11; 12; 13], such as RMFN, MFM, HFFN, RAVEN, etc.  
We note that word alignment requires not only detailed information about the domain but also meta-information about the exact time range of words in the datasets, which leads to sentiment analysis not always being feasible in practical applications.
2. Sentiment analysis based on unaligned words: This means that sentiment analysis is performed directly without explicitly aligning sequence data. In 2019, [PERSON] et al. [14] first extended the NMT Transformer [34] to multimodal sentiment analysis and proposed a multimodal MulT that can learn the interaction information between different modalities and directly perform the sentiment analysis of text, video, and audio without explicitly aligning the sequence data. Although better results are obtained, the complexity of the model increases. Given the above problems, [PERSON] et al. [35] proposed a low-rank fusion-based Transformer (LMT-MULTI) to avoid any excessive parameterization of the model in 2020. Based on this, [PERSON] et al. [15] designed a new lightweight network in 2021, which adopts a Transformer with cross-modal blocks to realize the complementary learning of different modalities. Although the accuracy is not the best, better results are obtained with the least model parameters. However, since the above studies use the same annotation methods in different modalities, the performance in capturing modal differentiation information is poor, and the additional unimodal annotation requires a lot of manpower and time. Based on this, [PERSON] et al. [24] proposed a sentiment analysis method combining self-supervision and multi-task learning in 2021. The difference between different modalities is obtained through independent single-peak supervision, but using fused modal information for sentiment analysis may result in issues such as loss of unimodal information or inclusion of noise in the fused information. To address this problem, in 2021, [PERSON] et al. [17] proposed a single-peak enhanced Transformer method to carry out sentiment analysis for text, video, and audio. Notwithstanding, the method of feature enhancement based on unimodal is far from enough. In 2022, [PERSON] et al. [18] proposed the progressive modal reinforcement (PMR) method with a message center to enhance the accuracy of sentiment analysis of text, video, and audio through the complementary process of unimodal information and fusion information but ignored the impact of the interaction between different modalities on the final result of sentiment analysis.  
Currently, the sentiment analysis in unaligned data sequence scenarios is mostly based on text, video, and audio data, while the above studies (1) ignore the role of considering effective unimodal feature representation and (2) ignore that the correlation and interaction between different modalities in important time steps will lead to certain deviations in the accuracy of sentiment analysis results. Therefore, extracting the heterogeneous relationship between different modality features effectively is still a difficult problem in cross-modal fusion.