## 3 Methods
### Unimodal Neural Networks
#### 3.1.2 Audio  
Wav2 Vec 2.0 is a self-supervised framework for speech representation learning. It can capture the information about the speaker and performs well in speech recognition tasks, especially in ultra-low-resource cases [46]. Therefore, we use it as a feature extractor for acoustic data. Inspired by [PERSON] et al. [41], we propose a network that consists of a CNN, followed by a self-attention mechanism and a Bi-LSTM.  
Firstly, we employ Wav2 Vec 2.0 to extract acoustic features from the waveform signal and define the acoustic feature as \(X_{A}\). We then feed \(X_{A}\) into a CNN layer:  
\[R_{cnn}^{A}=W_{cnn}^{A}\cdot X_{A}+b_{cnn}^{A}, \tag{8}\]  
where \(R_{cnn}^{A}\) represents the CNN output, \(W_{cnn}^{A}\) represents the weight matrix, and \(b_{cnn}^{A}\) represents the bias vector.  
We use the \"Scaled Dot-Product Attention\" from [56] as the attention mechanism. The matrix of outputs is computed as  
\[\begin{array}{c}Que_{A}=W_{Que}^{A}\cdot R_{cnn}^{A},\\ Key_{A}=W_{Kev}^{A}\cdot R_{cnn}^{A},\\ score_{A}=softmax(W_{attn}^{A}\cdot tanh(Que_{A}\cdot Key_{A})),\\ R_{attn}^{A}=score_{A}\cdot(W_{Val}^{V}\cdot Val_{A}),\end{array} \tag{9}\]  
where \(Que_{A}\), \(Key_{A}\), and \(Val_{A}\) stand for query, key, and value, respectively. \(Val_{A}\) is equal to \(Key_{A}\), and \(score_{A}\) represents the weight on the value. \(Softmax\) is the softmax function, and \(tanh(\cdot)\) is the hyperbolic tangent function. \(R_{attn}^{A}\) is the output of the attention module and can be defined as \(R_{attn}^{A}=\big{[}r_{0}^{A},r_{1}^{A},\cdots,r_{1}^{A}\big{]}\), where \(t\) is the timestep. \(R_{attn}^{A}\) is fed into BiLSTM to extract deep representations. \(W_{Que}^{A}\), \(W_{Kev}^{A}\), \(W_{Att}^{A}\), and \(W_{Val}^{V}\) represent the weight matrices. The outputs of BiLSTM can be computed as follows:  
\[\begin{array}{c}i_{t}^{A}=\sigma\big{(}W_{i}^{A}\big{[}h_{t-1}^{A}:r_{t}^{A }\big{]}+b_{i}^{A}\big{)},\\ f_{t}^{A}=\sigma\big{(}W_{a}^{A}\big{[}h_{t-1}^{A}:r_{t}^{A}\big{]}+b_{t}^{A} \big{)},\\ o_{t}^{A}=\sigma\big{(}W_{o}^{A}\big{[}h_{t-1}^{A}:r_{t}^{A}\big{]}+b_{o}^{A} \big{)},\\ C_{t}^{A}=f_{t}^{A}\cdot C_{t-1}^{A}+i_{t}^{A}\cdot tanh\big{(}W_{c}^{A} \big{[}h_{t-1}^{A}:r_{t}^{A}\big{]}+b_{c}^{A}\big{)},\\ H_{t}^{A}=o_{t}^{A}\cdot tanh\big{(}C_{t}^{A}\big{)},\end{array} \tag{10}\]  
where \(i_{t}^{A}\), \(f_{t}^{A}\), and \(o_{t}^{A}\) represent outputs of the input gate, forget gate, and output gate, respectively, in LSTM; \(C_{t}^{A}\) represents the current moment cell state; and \(H_{t}^{A}\) represents the hidden output of LSTM and can be defined as \(H_{t}^{A}=\big{[}h_{1}^{A},\cdots,h_{t}^{A}\big{]}\), where \(t\) is the timestep. \(\sigma(\cdot)\) denotes the sigmoid activation function and \(tanh(\cdot)\) is the hyperbolic tangent function. \(W_{i}^{A}\), \(W_{f}^{A}\), \(W_{o}^{A}\), and \(W_{c}^{A}\) represent the weight matrices and \(b_{i}^{A}\), \(b_{f}^{A}\), \(b_{o}^{A}\), and \(b_{c}^{A}\) represent the bias vectors.  
Finally, we feed the output of BiLSTM into a fully connected layer for acoustic emotional classification. The operation process of the acoustic emotional classification can be formulated as  
\[H_{A}=\left[\begin{array}{c}\overset{\rightarrow}{H_{t}^{A}}:\overset{\leftarrow }{H_{t}^{A}}\end{array}\right]\!Y_{A}=W_{A}\!\cdot\!H_{A}+b_{A}, \tag{11}\]  
\(H_{A}\) and \(Y_{A}\) represent hidden representation and annotation of the acoustic modality, respectively. \(W_{A}\) represents the weight matrix, and \(b_{A}\) represents the bias vector.