# Bimodal Fusion Network with Multi-Head Attention for Multimodal Sentiment Analysis
## 3 Proposed Approach
### Bimodal Decoder  
As shown in Figure 3, the bimodal decoder takes the text encoder output \(X_{l}\) and audio encoder output \(X_{a}\) as input and conducts \(N\) joint decoding steps via \(N\) decoders in series to intelligently fuse the cross-modal information. Specifically, it matches the samples of each time step with that of another modality via multi-head attention. Although the alignment relationship between modalities is complicated, attention can adaptively allocate different weights to adapt to different alignment situations:  
\[attention(Q,K,V)=softmax\left(\frac{QW_{q}^{(h)}\times\left(KW_{k}^{(h)}\right)^ {T}}{\sqrt{d_{k}}}\right)V \tag{7}\]We set the textual information \(X_{l}\) as query \(Q\) and the acoustic information \(X_{a}\) as \(K\) and \(V\) to assign weights to \(V\) according to \(Q\). Equation (7) can be transformed as follows:  
\[X_{h}^{[i]}=softmax\left(\frac{X_{l}^{[i-1]}W_{l}^{(h)}\times\left(X_{a}W_{a}^{( h)}\right)^{T}}{\sqrt{d_{a}}}\right)X_{a} \tag{8}\]  
\[X_{a\to l}^{*[i]}=concat\left(X_{1}^{[i]},X_{2}^{[i]},\cdots,X_{H}^{[i]} \right)\times W_{o} \tag{9}\]  
where \(i\in[1,2,\cdots,N]\) denotes the \(i\)-th decoder and \(X_{l}^{[i-1]}\) is the output of \(i\)-1-th encoder. \(W_{l}^{(h)}\) and \(W_{a}^{(h)}\) denote the trainable weight matrices of the \(h\)-th attention head, and \(W_{o}\) is used to fuse the outputs of the \(H\) heads. Equation (8) alleviates the alignment problem and vector offset problem simultaneously. The \(i\)-th encoder obtains two vector sequences, one is the text information representation of each sample \(X_{l}^{[i]}(l\in[1,N])\), and the other is the acoustic information representation of each frame \(X_{a}^{[i]}(a\in[1,M])\). Attention will focus on the weighted fusion of these two sequences. Specifically, take the text sequence as the key, query the acoustic sequence, find the \(N\times M\) weight matrix and multiply it by the text vector sequence to find the integrated representation. In this process, the text information of each sample will focus on the acoustic information of several frames related to it with high scores, and other irrelevant frame information will not be paid attention to. This enables acoustic-text alignment. In addition, attention can intelligently learn the correlation between acoustic information and text information. It realizes acoustic-to-text integration according to weights, which is more learnable than the direct concatenation of text sequence and acoustic sequence, thereby alleviating the uncontrolled vector offset.  
Figure 3: Overall architecture of the bimodal joint decoder (**left**) and detailed structure of the cross-modal attention mechanism (**right**).  
We take textual information as the main representation by adding the weighted result with the input textual vectors instead of acoustic vectors:  
\[X_{l}^{[i]}=LayerNorm\Big{(}X_{l}^{[i-1]}+X_{a-l}^{*[i]}\Big{)} \tag{10}\]  
After decoding \(N\) times, we finally obtain the intermodal textual representation \(X_{l}^{[N]}\), which serves as the input of the prediction layer to calculate the probabilities of sentiment categories:  
\[\overline{y}=softmax\Big{(}W_{c}X_{l}^{[N]}+b_{c}\Big{)} \tag{11}\]  
where \(W_{c}\) and \(b_{c}\) are trainable parameters for the prediction layer.