# Bimodal Fusion Network with Multi-Head Attention for Multimodal Sentiment Analysis
## 4 Experiments
### Multimodal Sentiment Analysis Results  
We conducted experiments on unaligned and aligned CMU-SOSEI and report the results as shown in Table 1. The proposed BMAN model presented overall advantages over the baselines.  
Obviously, BMAN outperformed unimodal models. Compared with Transformer-T, Transformer-A and Transformer-V, BMAN achieved average improvements of 4.32% (Acc-7), 9.15% (Acc-2) and 5.74% (F1), respectively. This is because these unimodal models focus on capturing intermodal dependency but lack support from other modal information. Moreover, we noticed that the unimodal model using audio only presented a great performance decline compared with the model using text only. This is consistent with our previous assumption that text plays the most important role in various modalities and implies the rationality of taking the text as the main information and audio as auxiliary information.  
Compared with RNN-based models, such as CTC+EF-LSTM and CTC+RAVEN, BMAN achieved average improvements of 4.52% (Acc-7), 8.6% (Acc-2) and 1.99% (F1) on the unaligned dataset. This is reasonable because CTC+EF-LSTM fuses multimodal information via direct concatenation, ignoring the importance among various modalities as well as the vector offset during the fusion process, thereby presenting disappointing performance. CTC+RAVEN improved the flexibility of multimodal information fusion by placing multimodal attention gating after unimodal LSTM encoders; however, thin attention cannot fit the complex cross-modal interaction relationship. In contrast, BMAN achieved full cross-modal interactions via deep multi-head attention and abstractly fused acoustic information into textual representation in the form of joint decoding, alleviating the vector offset caused by audio modality to text modality.  
In addition, we also observed that the proposed BMAN on the unaligned dataset has greater advantages over that on the aligned dataset. On unaligned CMU-MOSEI, BMAN outperformed CTC-LSTM with a considerable improvement of 6.42% (Acc-7), 13.99% (Acc-2) and 2.06% (F1), while this advantage appeared to be reduced on the aligned dataset. This is ascribed to our joint decoder being better at solving the alignment problem of multiple modalities. Its cross-modal attention assigns weights to all the audio frames according to the text samples. In this way, the text information of a certain sample will focus on the acoustic information of several frames related to it with high scores, and other irrelevant frame information will not be paid attention to, thereby achieving alignment between text samples and audio frames. Compared with models using CTC to obtain alignment sequences, our model internalized the data alignment into an intermediate step  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Model** & **Acc-7** & **Acc-2** & **F1-Score** & **MAE** & **Corr** & **Data Setting** \\ \hline Transformer-T & 46.5 & \(-/77.4\) & \(-/78.2\) & 0.653 & 0.631 & Unaligned \\ Transformer-A & 41.4 & \(-/65.6\) & \(-/68.8\) & 0.764 & 0.31 & Unaligned \\ Transformer-V & 43.5 & \(-/66.4\) & \(-/69.3\) & 0.759 & 0.343 & Unaligned \\ CTC+EF-LSTM & 41.7 & 65.3/\(-\) & 76.0/\(-\) & 0.799 & 0.265 & Unaligned \\ CTC+RAVEN & 45.5 & \(-/75.4\) & \(-/75.7\) & 0.664 & 0.599 & Unaligned \\ BMAN & **48.12** & **79.29/78.95** & **78.06/77.84** & **0.6471** & **0.640** & Unaligned \\ \hline Graph-MFN & 45.0 & 76.9/\(-\) & 77.0/\(-\) & 0.71 & 0.54 & Aligned \\ EF-LSTM & 46.7 & **79.1**/72.02 & **79.89**/61.89 & 0.674 & **0.704** & Aligned \\ BBFN(VA) & 41.1 & \(-/71.1\) & \(-/64.5\) & 0.816 & 0.261 & Aligned \\ BMAN & **46.84** & 75.55/**78.5** & 76.11/**78.32** & **0.656** & 0.624 & Aligned \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison results (%) of multimodal sentiment analysis models. Bold indicates the optimal value of the corresponding evaluation indicators.  
of representation learning to facilitate the learning of the overall model, thereby presenting strong competitiveness on unaligned datasets.