## 3 Proposed Model  
Relying on the advantages that MTL can learn the shared features between tasks, an explainable multitask-based shared feature learning model using language recognition as an auxiliary task is proposed to establish the relationship between languages and emotions. The MSFL incorporates a task-sharing module for feature representation and a task-specific module for multi-task classification. Figure 2 illustrates our proposed model. Previous studies show that utterance-level features are usually transformed into deeper representations by applying the DNN, while LSTM generally handles frame-level features, and its time step is typically related to sentence length. Different from previous research, 88-dimensional utterance-level features are extracted in this study, which are also called High-level Statistics Functions (HSFs), and they are input as time steps into LSTM to obtain the dependencies of features. However, not all features contribute equally to the emotional representation in an utterance, so the attention mechanism layer is stacked on  
Figure 1: The framework of multi-task SER based on SIMO and MIMO. LSTM to make the model better understand the important contribution of each feature and focus on the important features by assigning different weights. To further process the features selected by the attention mechanism and transform them into deeper feature representation, we utilized the non-linear transformation operations of DNN stacking two fully connected layers with a ReLU activation function. Finally, to transform the feature vectors into a posterior probability distribution for classification, the more emotionally discriminative MSFs are input into the task-specific module, which consists of two softmax layers concerning specific tasks of emotion and language recognition. Specifically, three labeled corpora are used in this study, where \(\mathbf{X}_{e}\) represents the EMO-DB corpus, \(\mathbf{X}_{c}\) represents the CASIA corpus, and \(\mathbf{X}_{s}\) represents the SAVEE corpus. Let \(\mathbf{X}=[\mathbf{X}_{e},\mathbf{X}_{c},\mathbf{X}_{s}]\in\mathbf{R}^{n\times d}\), where \(n=n_{e}+n_{c}+n_{s}\), \(d\) is the dimension of features. \(\mathbf{Y}_{e}=[y_{e,1},y_{e,2},\dots,y_{e,n}]\in\mathbf{R}^{n\times c_{e}}\) represents the label of emotion recognition task, \(\mathbf{Y}_{l}=[y_{l,1},y_{l,2},\dots,y_{l,n}]\in\mathbf{R}^{n\times c_{l}}\) represents the label of language recognition task, where \(c_{e}\) and \(c_{l}\) represent the number of emotion categories and language categories, respectively. Each row in matrix \(\mathbf{X}\) represents a feature vector \(x=\{x_{1},x_{2},\cdots,x_{i},\cdots,x_{d}\}\), \(x_{i}\) represents an acoustic feature. The 88-dimensional acoustic features are extracted, so the value of \(d\) is 88. For our proposed model, feature vector \(x\) with 88 time-steps with one input dimension is input into an LSTM network containing 32 nodes to obtain the output \(\mathbf{B}(\mathbf{B}=[b_{1},b_{2},\cdots,b_{i},\cdots,b_{d}])\), where \(b_{i}\) is a 32-dimensional vector. Then, to correspond the 88-dimensional features to the nodes of the fully connected layer containing 88 nodes in the attention mechanism, the matrix \(\mathbf{B}\) is transposed and input to the attention mechanism layer, which is calculated as follows. \[f(b_{i})=\mathbf{W}^{\text{T}}b_{i} \tag{1}\]  
\[\alpha_{i}=\frac{\exp(f(b_{i}))}{\sum_{t}\exp(f(b_{n}))} \tag{2}\]  
\[\mathbf{Z}=\sum\alpha_{i}b_{i} \tag{3}\]  
Here, \(\mathbf{W}^{\text{T}}\) is the trainable parameter, \(\alpha_{i}\) is the normalized attention weight of each feature calculated by the softmax function operating on the attention scores, and \(\mathbf{Z}\) is the output weighted sum of the LSTM.