## 3 Methods
### Multi-Task Multimodal Fusion Network with Conformer  
Inspired by [PERSON] et al. [73], we propose the Conformer-based fusion method. The Conformer [73] (see Figure 4) is a convolution-augmented Transformer for speech recognition. It can combine convolution neural networks and Transformers to model both local and global dependencies of audio sequences in a parameter-efficient way. The model exhibits better accuracy with fewer parameters than previous work on the LibriSpeech dataset. The Conformer block (see Figure 4a) is composed of four modules stacked together, i.e., a feed-forward module, a self-attention module, a convolution module, and a second feed-forward module at the end. In the multi-head self-attention module (see Figure 4d), we employ pre-norm residual units with dropout, which helps in training and regularizing deeper models. The convolution module (see Figure 4c) is stacked in the order of layernorm layer, convolution layer, gated linear unit (GLU), and so on. Batchnorm is deployed just after the convolution for training deep models. As for the feed-forward module (see Figure 4b), it starts with a layernorm layer, which is followed by two interleaved linear layers and dropout layers. We also apply Swish activation to regularize the network. Especially, we first load the unimodal models saved above and set these models as untrainable to obtain unimodal representations from their hidden layers. We then concatenate these representations to form the multimodal representation, which can be formulated as  
\[X_{M}=[H_{V}:H_{A}:H_{T}], \tag{15}\]  
We use a Conformer-based method to fuse the multimodal representation and predict the emotional annotation of video. Especially, we send \(X_{M}\) into the first feed-forward module and define the result as \(R_{FFM_{1}}\), which is fed into the multi-head self-attention module. The operation of the multi-head self-attention module in the Conformer can be formulated as follows:  
\[R_{LN}^{M}=LN\Big{(}R_{f/m_{1}}\Big{)}, \tag{16}\] \[Que_{M}=W_{Que}^{M}.R_{LN}^{M},\] \[Key_{M}=W_{Key}^{M}.R_{LN}^{M},\] \[score_{M}=softmax\big{(}W_{attn}^{M}.tanh(Que_{M}.Key_{M})\big{)},\] \[R_{attn}^{M}=score_{M}.\big{(}W_{Val}^{M}.Val_{M}\big{)},\] \[R_{mha}^{M}=W_{MHA}^{M}.Concat\Big{(}R_{attn}^{M},\cdot\cdot\cdot,R_{attn_{h}}^{M}\Big{)},\] \[R_{DP}^{M}=DP\big{(}R_{mha}^{M}\big{)},\]  
where \(LN(\cdot)\) is the layer normalization operation, and \(Que_{M}\), \(Key_{M}\), and \(Val_{M}\) stand for query, key, and value, respectively. \(Val_{M}\) is equal to \(Key_{M}\), and \(score_{M}\) represents the weight on the value. \(Softmax\) is the softmax function and \(tanh(\cdot)\) is the hyperbolic tangent function. \(R_{attn}^{M}\) is the output of the attention module of one head and \(R_{mha}^{M}\) is the output of the multi-head attention module. \(W_{Que}^{M}\), \(W_{Key}^{M}\), \(W_{attn}^{M}\), and \(W_{Val}^{M}\) represent the weight matrices. \(Concat(\cdot)\) represents the concatenate operation and \(DP(\cdot)\) represents the dropout operation. Figure 4: Conformer architecture. (**a**) is Conformer architecture, (**b**) is feed-forward module, (**c**) is convolution module, and (**d**) is multi-head self-attention module. \(\oplus\): operation of addition. \(R_{DP}^{M}\) is then fed into the convolution module, and the operations in the convolution module can be formulated as  
\[ \begin{array}{c}R_{cm_{1}}^{M}=W_{cm_{1}}^{M}\_LN(R_{DP}^{M})+b_{cm_{1}}^{M},\\ R_{cm_{2}}^{M}=W_{cm_{2}}^{M}\_BN(GLU(R_{cm_{1}}^{M}))+b_{cm_{2}}^{M},\\ R_{CON}=DP\big{(}R_{cm_{2}}^{M})+H_{DP}^{M},\end{array}  \tag{17}\]  
where \(LN(\cdot)\) is the layer normalization operation, \(GLU(\cdot)\) represents the gated linear unit operation, \(BN(\cdot)\) is the batch normalization operation, and \(DP(\cdot)\) represents the dropout operation. \(R_{cm_{1}}^{M}\) and \(R_{cm_{2}}^{M}\) are convolution operation results. \(W_{cm_{1}}^{M}\) and \(W_{cm_{2}}^{M}\) represent the weight matrices, and \(b_{cm_{1}}^{M}\) and \(b_{cm_{2}}^{M}\) represent the bias vectors.