## 3 Proposed Method
### Extended Multimodal Bottleneck Transformer
\tag{2}\]  
Each sequence \(X_{m}\) is considered a two-dimensional tensor with a size of \((M+N_{m})\times D\), where \((M+N_{m})\) denotes the extended sequence length, and \(D\) represents the common latent dimension. For each modality \(m^{\text{th}}\), the corresponding matrix \(X_{m}\) is input into a dedicated transformer encoder layer. \[\begin{array}{l}X^{l+1}_{m}=\text{TransformerLayer}_{m}^{l}(X^{l}_{m})=\\ \qquad\qquad\qquad\qquad\qquad  
between this number and the ground truth is set as the core loss function because it is a standard loss function for regression tasks [11]. For classification tasks, \(C\) is assigned to the number of emotional annotations. The mean average of the binary cross-entropy loss over all emotional annotations is set as the core objective function, formulated as follows:  
\[\mathcal{L}_{bce}(z,y)=\frac{1}{c}\sum_{c=1}^{C}(y(c)\log\sigma(z(c))+(1-y(c)) \log(1-\sigma(z(c)))) \tag{7}\]  
\[\mathcal{L}_{ae}(z,y)=|z-y| \tag{8}\]  
where \(z\) and \(y\) denote the input and target tensors, respectively, and \(\sigma(\cdot)\) denotes the sigmoid function. Based on these core loss functions, \((M+1)\) loss function elements \(\mathcal{L}_{1},\mathcal{L}_{2},\ldots,\mathcal{L}_{M}\) and \(\mathcal{L}_{M+1}\) take \(z_{1},z_{2},\ldots,z_{M}\) and \(z_{\text{fuse}}\) as input, respectively, formulated as follows:  
\[\mathcal{L}_{m}= \begin{cases}\mathcal{L}_{bce}(z_{m},y),&\text{(classification)} \\ \mathcal{L}_{ae}(z_{m},y),&\text{(regression)}\end{cases} ,\quad m\in\{1,\ldots,M\} \tag{9}\]  
\[\mathcal{L}_{M+1}=\mathcal{L}_{fuse}= \begin{cases}\mathcal{L}_{bce}(z_{fuse}, y),&\text{(classification)}\\ \mathcal{L}_{ae}(z_{fuse},y),&\text{(regression)}\end{cases}  \tag{10}\]  
Regarding the final loss function, [PERSON] et al. [20] applied adaptive loss for multimodal learning by summing the loss components from all modalities with dedicated  
Figure 3: Illustration of proposed XMBT. trainable coefficients \(\alpha_{m}\). [PERSON] et al. [19] first introduced this strategy as uncertainty weighting (UW) for multitask learning. When a task has higher uncertainty and loss than others, this strategy can prevent the model from making a significant decision lacking clarity or insight. The formulation for adaptive loss is  
\[\mathcal{L}_{adapt}=\sum_{m=1}^{M+1}\frac{1}{\alpha_{m}^{2}}\mathcal{L}_{m}+ \sum_{m=1}^{M+1}\log\alpha_{m} \tag{11}\]  
The parameter \(\alpha_{m}\) is interpreted as learning the relative weight of the loss \(\mathcal{L}_{m}\), where \(m\) ranges from \(1\) to \((M+1)\). The multimodal adaptive loss employed by [PERSON] et al. [20] shares the same problems as the UW loss employed by [PERSON] et al. [19]. First, the regularization terms \(\log\alpha_{m}\) can be negative and lead the loss to a negative value. Second, all trained parameters \(\alpha_{m}\) could decrease and lead to a decrease in the learning rate of the training process. When all coefficients of loss functions decrease, the overall loss decreases; therefore, the parameter updating process slows. This work resolves these disadvantages by adopting the approach by [PERSON] et al. [25]. The first problem can be addressed by replacing \(\log\alpha_{m}\) with \(\log(1+\alpha_{m}^{2})\). The new value \((1+\alpha_{m}^{2})\) guarantees that the logarithm term is always positive. The second problem is solved using a positive value \(\varphi\) to constrain the weights in a controlled manner. This work enforces the summation of the absolute value of all \(\alpha_{m}\), which equals approximately \(\varphi\), preventing them from simultaneous decrement. Therefore, the restrained revised adaptive (RRA) loss for multimodal learning is developed and formulated as follows:  
\[\mathcal{L}_{\tau a}=\sum_{m=1}^{M+1}\frac{1}{\alpha_{m}^{2}}\mathcal{L}_{m}+ \sum_{m=1}^{M+1}\log(1+\alpha_{m}^{2})+|\varphi-\sum_{m=1}^{M+1}|\alpha_{m}|| \tag{12}\]  
Moreover, this work proposes DRA loss, a combination of RRA and dynamic weighted averaging (DWA) strategies from multitask learning disciplines for multimodal learning.