# Bimodal Fusion Network with Multi-Head Attention for Multimodal Sentiment Analysis
## 3 Proposed Approach
### Unimodal Encoders  
We utilize two independent transformer encoders to construct the text encoder and audio encoder, capturing text information and audio information, respectively. Given the language input \(l_{l}=\left[l_{1},l_{2},\cdots,l_{N}\right]\), where \(l_{n}=\left[i_{1},i_{2},\cdots,i_{T_{l}}\right]\), \(N\) and \(T_{l}\) are the total sample number and word number of each sample, respectively. We first use the GloVe embeddingtool to obtain the \(d_{l}\)-dimensional textual embeddings \(X_{n}=[x_{1},x_{2},\cdots,x_{T_{l}}]\in\mathbb{R}^{T_{l}\times d_{l}}\). Similarly, given the audio input \(I_{A}=[A_{1},A_{2},\cdots,A_{M}]\), where \(A_{m}=[j_{1},j_{2},\cdots,j_{T_{a}}]\), \(M\) and \(T_{a}\) are the sample number of audio and the frame number of each sample, respectively. Their initial embeddings are represented as \(X_{a}=[a_{1},a_{2},\cdots,a_{T_{a}}]\in\mathbb{R}^{T_{a}\times d_{a}}\), where \(T_{a}\) and \(d_{a}\) are the frame number and acoustic embedding dimension, respectively. Taking the text encoder as an example, it extracts the relations of word pairs in parallel with the aid of attention and outputs the textual representation after full inter-word interaction. It consists of \(N\) stacked encoders, the core function layers of which are a multi-head attention sublayer and a feedforward sublayer, as shown in the right part of Figure 2.  
The multi-head attention learns the token weights within the sequence in parallel, which indeed improves the computational efficiency but also means there is no order between words. However, natural language is a sequence of knowledge arranged in a certain order to express semantics, and this naturally determines the importance of order [30]. To make the model maintain efficient parallel learning while accounting for order information, we transform the absolute token position into position embedding (_PE_) following the previous work [31] and add it into textual embedding via a linear operation, called the position-aware feedforward network (_PFN_):  
\[X_{PFN}=W(x_{n}+PE(x_{n}))+b \tag{1}\]  
\(X_{l}\) then serves as the input of the multi-head attention mechanism. The position information is updated along with the network and assists attention in weight assignment:  
Figure 2: The overall architecture of BMAN. It includes two important components: unimodal encoders for capturing text and audio information and a bimodal decoder for jointly learning bimodal fused representation. The right part shows the detailed structure of the unimodal encoder, where \(Z\) can denote both textual and acoustic representations. \[X^{h}=Softmax\left(\frac{QW_{q}^{(h)}\times\left(KW_{k}^{(h)}\right)^{T}}{\sqrt{d_{ l}}}\right)V \tag{2}\]  
where \(Q\), \(K\) and \(V\) are equal to \(X_{PFN}\), and the \(h\) denotes the head number. The matrix multiplication of \(Q\) and \(K\) obtains the interword correlation coefficient, and \(X^{h}\) is the representation after weighting. The necessity of the multi-head is to set up multiple semantic spaces to learn the complex relationship between words. We apply a linear operation on the concatenation of these heads to obtain the new representation:  
\[X_{MHA}=concat\left(X^{1},X^{2},\cdots,X^{H}\right)\times W_{o} \tag{3}\]  
Since we set \(N\) encoders, there are \(N\) multi-head attentions in the text encoder, leading to a deep network. Excessive layers may cause vanishing or exploding gradients, leaving the whole network unable to update. The residual connection adds the previous layer output to the current layer, preventing the vanishing gradient in backpropagation. Layer normalization can eliminate gradient explosion by limiting the output of each layer in a small numerical range. These two operations ensure the normal update of the network, and we apply them in every sublayer.