Venue: NAACL 2025

# Mixture of Multimodal Adapters for Sentiment Analysis

 Kezhou Chen1, Huixia Ben2*, Shuo Wang1, Shengeng Tang3, Yanbin Hao3

###### Abstract

Pre-trained language model (PLM) have achieved great success in text sentiment analysis. However, in practical applications, sentiment is not only conveyed through language but also hidden in other modalities. Therefore, multimodal sentiment analysis (MSA) has attracted increasing research interest. Compared to text sentiment analysis, MSA is challenging since (1) emotions hidden in body movements or vocal timbres eclipse traditional analytical methods, and (2) transferring PLM to MSA task requires huge training parameters. Thus, to solve these issues, we introduce the **Mixture** of **Mu**ltimodal **A**dapters (**MMA**) into the PLM. Specifically, we first design a mixture-of-multimodal-experts module to capture and fuse emotional movements from different data. Meanwhile, we use a compression parameter for each expert to reduce the training burden. We apply our method to two benchmark datasets and achieve state-of-the-art performance with a tiny trainable parameter count. For example, compared to the current state-of-the-art method, AcFormer, we only need 1/22 of its training parameters amount (\(130{\rm M}\to 6{\rm M}\)) to achieve better results. The code is available at [https://github.com/MMA4MSa/MMA](https://github.com/MMA4MSa/MMA).

1University of Science and Technology of China, 2Anhui University of Science and Technology,

3Hefei University of Technology

chenkezhou@mail.ustc.edu.cn, shuowang.edu@gmail.com, huixiaben@gmail.com, tangsghfut.edu.cn, haoyanbinhehotmail.com

Footnote 2: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 1822–1833

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Sentiment analysis aims to identify emotional expressions from human conversations. In this task, recent pre-trained language models (PLMs) and their related structures, _e.g._ transformer, RNNs, have shown a powerful ability for textual analysis Devlin et al. (2018); Raffel et al. (2020); Wang et al. (2018); Guo et al. (2019); Peng et al. (2023); Ben et al. (2024); Wang et al. (2020, 2022); Lu et al. (2023). Thus, they incidentally improve the recognition performance of emotions from the textual information. However, only processing textual information cannot accurately express people's emotions. This is because emotional expression not only contains textual descriptions but also includes body language and voice, where these contents in different modalities can more accurately reflect human emotion. Therefore, multimodal sentiment analysis (MSA) has been proposed to fuse these contents to analyze sentiment.

Unlike traditional multimodal analysis or fusion tasks Zhu et al. (2024); Wang et al. (2024); Zhu et al. (2024); b; Wang et al. (2024), video and audio in MSA are used to express emotions rather than to identify objects, which weakens the ability of traditional recognition modules. Thus, the semantic gap between these and textual data is a challenge in the MSA tasks Hazarika et al. (2020); Chen et al. (2024). To solve this issue, ConFEDE Yang et al. (2023) designs a unified learning framework for contrastive representation learning and contrastive feature decomposition to robust multimodal representation. AcFormer Zong et al. (2023) designed a compact transformer using contrastive learning and pivot fusion strategies. However, (1) the fusion strategies of these recent methods are similar to those of traditional multimodal recognition methods which focus on achieving a more comprehensive fusion of multimodal features. While the MSA task needs to analyze the emotion hidden in vision and audio, where both visual and audio modalities may include noise and task-irrelevant features, such as background movements in videos and ambient noise in audio recordings Zong et al. (2023). Consequently, analyzing irrelevant or misleading features can impede the accurate assessment of emotional states Hazarika et al. (2022). And (2) these methods and their modules are fully fine-tuned to cater to multimodal representations. Although this strategy has brought great performance advantages over training from scratch, it has become a resource-consuming and parameterized process with increasing models and data.

In the sparsely activated mixture of experts

(MOE) module, different parts of the model, namely experts, are used to handle various tasks or aspects of the data. In this way, experts not proficient in handling a particular task do not participate in the forward process, resulting in more precise task-related representations [14]. Inspired by the MOE system, we extend it to MSA task, enabling selective multimodal feature fusion, thereby obtaining more task-relevant multimodal representation. Specifically, we design a mixture of multimodal experts, consisting of the multimodal experts and the multimodal attention router, where each expert is used to deal with a specific modality, and the multimodal attention router is responsible for determining which expert will be activated at each time step of the text sequence. Compared to traditional MSA methods, our approach enables selective fusion, only the features from the activated multimodal experts will be fused with the textual hidden states, thereby minimizing the negative impact of irrelevant noise in multimodal data while retaining task-related features in audio and video.

To reduce the training burden, a popular strategy in natural language processing is parameter-efficient fine-tuning (PEFT) [23, 50, 79]. It enables the efficient adaptation of large pre-trained models to various downstream applications by only training a small number of (extra) model parameters instead of all the model's parameters [68], which significantly decreases the computational and storage costs. Inspired by the PEFT strategy, we designed the aforementioned MOE structure as a parameter-efficient method. Specifically, the adapter is designed to serve as the expert to capture and fuse the different data (video, audio, and text) in each transformer block, where the textual information is modeled by a PLM, and the video and audio are extracted to assist in the analysis of textual content. Compared to the full training method [79, 100, 101], we only need to train the parameters in the adapter, which greatly reduces the required computing resources. Meanwhile, our adapter is a plug-and-play strategy. In other words, we can flexibly apply it to various language models and directly transform these models into a multimodal model that can handle MSA tasks.

In summary, we call our mixture-of-multimodal-experts with adapter strategy as Mixture of **M**ultimodal **A**dapters (MMA). The contributions are summarized as follows:

* We extend the MOE structure into the mixture of multimodal experts to selectively integrate useful multimodal features into the PLM for emotional recognition.
* We design a plug-and-play adapter to help the PLM to analyze the multimodal sentiment in a lightweight calculation.
* Experiments on multiple benchmarks validate the superiority of our method. Moreover, we apply MMA to large language model (LLM) and validate the versatility of our method.

## 2 Related Works

### Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) aims to perceive human emotions in videos. The previous MSA methods can be divided into two categories: methods with sophisticated fusion mechanisms and multimodal representation learning.

Fusion-based methods focus on implementing the interaction of multimodal information through intricate fusion mechanisms. For example, Zadeh et al. design the Tensor Fusion Network to model intra-modality and inter-modality dynamic [101, 102]. Then, to improve the efficiency of TFN, Liu et al. propose low-rank Fusion using low-rank tensors [103]. Tsai et al. first introduce cross-modal attention into the MSA task. Sun et al. propose CubeMLP to reduce memory consumption by mixing features on sequence, modality, and channel three levels [104]. Zong et al. design pivot attention fusion to reduce the computational complexity [101].

Different from fusion-based methods, multimodal representation learning-based approaches emphasize the significance of multimodal representation before fusion. MISA focuses on learning the modality-invariant and -specific subspaces before fusion [79]. Yu et al. propose a method in which modality-specific representations are learned through joint training in unimodal subtasks and a multimodal main task [68]. Some MSA works explore the use of contrastive learning to improve multimodal representations. For example, Yang et al. introduce contrastive learning into MSA to learn similarity and dissimilarity features for each modality [101]. Zong et al. employ contrastive learning to improve 

the unimodal representation and facilitate cross-modal alignment (Zong et al., 2023).

While the aforementioned methods can effectively fuse multimodal features for sentiment analysis, they ignore the noise features in video and audio data and the training burden brought by fully fine-tuning pre-trained models. Our MMA method strives to selectively fuse multimodal features for better integration of video and audio data and can be incorporated into a frozen PLM without introducing excessive trainable parameters.

### Parameter-Efficient Fine-Tuning

Many natural language processing tasks have benefited from transformer-based PLMs and LLMs (Devlin et al., 2018; Liu et al., 2019; Brown et al., 2020). However, the large scale of LLMs leads to significant computational costs (Xu et al., 2023). To this end, parameter-efficient fine-tuning (PEFT) methods are proposed to efficiently adapt the LLMs over downstream tasks by training a small part of additional parameters, such as Adapter (Houlsby et al., 2019), Prefix Tuning Li and Liang (2021), and LoRA Hu et al. (2021). Then, He et al. present a unified view of the above methods and propose the MAM adapter by combining their advantages (He et al., 2021). Soon, PEFT methods have also been emerged in the multimodal field, such as methods for vision-language foundation models (Wang et al., 2024a) and methods for multimodal large language models (Liang et al., 2024). In our work, we propose a PEFT method to convert a PLM into a multimodal model that can handle the MSA task.

### Mixture of Experts

The Mixture of Experts (MOE) architecture distributes each example to a subset of the parameters instead of reusing all parameters for each input (Fedus et al., 2022a). Eigen et al. first treat a component of a neural network as an expert (Eigen et al., 2013). Then, Shazeer et al. try MOE on a larger scale with LSTM layers by learning a routing mechanism (Shazeer et al., 2016). After the transformer achieves great success (Vaswani et al., 2017b), some works explore the possibility of applying MOE to the transformer by replacing the fully-connected layer in transformer block with MOE structure, such as GShaard (Lepikhin et al., 2020) and Switch Transformer (Fedus et al., 2022b). Since MOE can benefit from specialized knowledge while keeping a low computational demand, it achieves great success in large language models (Jiang et al., 2024; Wei et al., 2024). Then, MOE is applied in various domains, including computer vision (Riquelme et al., 2021), parameter-efficient fine-tuning (Dou et al., 2023), and multi-modal learning (Li et al., 2024; Xie et al., 2024). In this paper, we aim to utilize the MOE structure to achieve selective fusion for MSA.

## 3 Approach

In this section, we first briefly revisit the preliminaries of the MSA tasks and give an overview of our framework. Then, we illustrate our model in detail. Finally, we describe the training and inference procedures of our method.

### Preliminaries

The data corpus of Multimodal Sentiment Analysis (MSA) task can be categorically segmented into three distinct subsets: the **textual set**\({\cal D}_{t}\), the **visual set**\({\cal D}_{v}\), and the **audio set**\({\cal D}_{a}\). Each of these datasets serves as a conduit for conveying sentiment through vocal intonations, facial expressions, or subtleties of body language. The primary objective of MSA is to synthesize these disparate data streams to forge a more nuanced and comprehensive understanding of sentiment. It leverages the richness of multimodal inputs to uncover the obscured emotional nuance.

An overview of our method is shown in Figure 1. First, we employ an \(L\)-layer transformer-based pre-trained language model (PLM) to encode the given textual content \({\cal D}_{t}\). Then, we use a prediction head to predict the output of the transformer into sentiment. To integrate information from different modalities \({\cal D}_{v}\) and \({\cal D}_{a}\), we design a plug-and-play module within each transformer block, called **Mixture** of **Multimodal Adapters** (MMA), and help the PLM capture emotional content effectively.

### Mixture of Multimodal Adapters (Mma)

**Blocks Designing.** Our method is integrated into the pre-trained transformer block (Vaswani et al., 2017a). Thus, we first introduce our transformer block and describe the different modules in detail. Specifically, given a multimodal set \({\cal D}_{t}\), \({\cal D}_{v}\), and \({\cal D}_{a}\), we first embed it into features by different pretrained feature extraction toolkits:

\[\mathbf{T}=\Phi_{t}({\cal D}_{t}),~{}\mathbf{V}=\Phi_{v}({\cal D}_{v}),~{} \mathbf{A}=\Phi_{a}({\cal D}_{a}), \tag{1}\]

where \(\{\mathbf{T},\mathbf{V},\mathbf{A}\}\in\mathbb{R}^{M_{\{t,v,a\}}\times d_{\{ t,v,a\}}}\), \(M_{\{t,v,a\}}\) is the sequence length of {text, audio}. \(d_{\{t,v,a\}}\) 

is the feature dimension of {text, video, audio}. For text modality, \(\Phi_{t}\) is the embedding layer of a specific PLM, while \(\Phi_{v}\) and \(\Phi_{a}\) are commonly used feature extraction toolkits for video and audio in previous works Yu et al. (2021); Yang et al. (2023). Then we use \(L\) layers PLM to analyze the sentiment. For convenience, we use \(\mathbf{B}^{l}_{\mathrm{in}}\) and \(\mathbf{B}^{l}_{\mathrm{out}}\) to represent the input and output of the \(l_{\mathrm{th}}\) transformer block, respectively. Thus, the inputs of the first and other layers are represented as:

\[\mathbf{B}^{l}_{\mathrm{in}}=\left\{\begin{array}{ll}\mathbf{T},&l=1,\\ \mathbf{B}^{l-1}_{\mathrm{out}},&l>1.\end{array}\right. \tag{2}\]

As depicted in Figure 1(a), in each transformer block, we first use the pre-trained Self-Attention Module with a few training parameters (LoRA) to refine the input \(\mathbf{B}^{l}_{\mathrm{in}}\) as:

\[\mathbf{X}^{l}_{t}=\operatorname*{Self-Attention}(\mathbf{B}^{l}_{\mathrm{in }}). \tag{3}\]

Then we pass visual feature \(X_{v}\) and acoustic feature \(X_{a}\) through the 1D convolutional layers:

\[\hat{\mathbf{V}}^{l}=\operatorname*{Conv1D}(\mathbf{V}),\hat{\mathbf{A}}^{l} =\operatorname*{Conv1D}(\mathbf{A}), \tag{4}\]

where \(\hat{\mathbf{V}}^{l}\in\mathbb{R}^{M_{v}\times d_{t}}\) and \(\hat{\mathbf{A}}^{l}\in\mathbb{R}^{M_{a}\times d_{t}}\). The 1D temporal convolutional layers can not only help multimodal sequences contain the local structure but also project the features of video and audio to the same dimension \(d_{t}\). Then we fuse text features with video and audio features using a cross-attention strategy. To reduce the training cost, there is no parameter in our cross-attention module:

\[\mathbf{X}^{l}_{v} =\operatorname*{softmax}(\frac{\mathbf{T}^{l}\hat{\mathbf{V}}^{ l^{\top}}}{\sqrt{d_{t}}})\hat{\mathbf{V}}^{l}+\mathbf{T}^{l}, \tag{5}\] \[\mathbf{X}^{l}_{a} =\operatorname*{softmax}(\frac{\mathbf{T}^{l}\hat{\mathbf{A}}^{ l^{\top}}}{\sqrt{d_{t}}})\hat{\mathbf{A}}^{l}+\mathbf{T}^{l},\]

where \(\mathbf{X}^{l}_{v}\in\mathbb{R}^{M_{t}\times d_{t}}\) and \(\mathbf{X}^{l}_{a}\in\mathbb{R}^{M_{t}\times d_{t}}\) are cross token fusion features. Note that after the cross-attention module, \(\mathbf{X}^{l}_{v}\) and \(\mathbf{X}^{l}_{a}\) have the same sequence length with \(\mathbf{X}^{l}_{t}\), thus we design a token-level selective fusion to select appropriate multimodal features for the PLM, named Mixture of Multimodal Experts (MOME), and the output of \(l_{\mathrm{th}}\) transformer block \(\mathbf{B}^{l}_{\mathrm{out}}\) can be calculated as the sum of the Feed-Forward Network (FFN) Vaswani et al. (2017) and the MOME:

\[\mathbf{B}^{l}_{\mathrm{out}}=\operatorname*{FFN}(\mathbf{X}^{l}_{t})+ \operatorname*{MOME}(\mathbf{X}^{l}_{t},\mathbf{X}^{l}_{v},\mathbf{X}^{l}_{a}). \tag{6}\]

Note that in this block, only LoRA, Conv1D, and MOME modules have a few training parameters.

**Mixture of Multimodal Experts.** Through conv1D and cross attention, the PLM has gained the capability to handle non-linguistic sequences at every transformer block. However, in MSA task, non-linguistic features may not be beneficial, and at different time steps, the noisy modalities and beneficial modalities may vary. Considering these complexities, we have introduced the mixture of multimodal experts (MOME) module consisting of **multimodal experts** and **multimodal attention router**, as shown in Figure 1(b). For **multimodal experts**, to save as many parameters as possible, we let the adapters serve as experts. For

Figure 1: The overall architecture of our method, where (a) is the blocks designing of MMA, and (b) is the mixture of multimodal experts in MMA.

 

the \(i_{\rm th}\) time step of the text and fused sequences \(({\bf X}^{l}_{v,i},{\bf X}^{l}_{t,i},{\bf X}^{l}_{a,i})\), we use three types of adapters (vision, text, and audio) to capture the features:

\[{\bf H}^{I,n}_{m,i}=\delta({\bf D}^{l,n}_{m}{\bf X}^{l}_{m,i}+b^{l,n}_{\rm m, down}), \tag{7}\]

where \({\bf H}^{I,n}_{m,i}\in\mathbb{R}^{r}\), \(r\) is the intrinsic dimension (rank) of the adapter, \(\delta(x)\) is a non-linear activation function, \(n\in\{1,2,\cdots,N\}\) is the index of experts of modality \(m\), \(N\) is the number of experts per modality, \({\bf D}^{l,n}_{m}\) is the down-project matrix and \(b^{l,n}_{m,down}\) is the bias vector, \(m\in\{t,a,v\}\). \(r\ll d_{t}\), makes sure that the number of parameters in experts is far less than that in frozen pre-trained weights. Then we use the up-project matrix \({\bf U}^{l,n}_{m}\) to reconstruct the \({\bf H}^{I,n}_{m,i}\) into \(d_{t}\) dimension:

\[{\bf E}^{I,n}_{m,i}=s^{l,n}_{m}({\bf U}^{l,m}_{m}{\bf H}^{I,n}_{m,i}+b^{l,n}_{ \rm m,up}), \tag{8}\]

where \({\bf E}^{l,n}_{m,i}\in\mathbb{R}^{d_{t}}\) is the output of the \(n_{\rm th}\) experts of modality \(m\) based on the \(i_{\rm th}\) token, \({\bf U}^{l,n}_{m}\in\mathbb{R}^{d_{t}\times r}\) is the up-projection matrix and \(b^{l,n}_{m,up}\) is the bias vector. \(s^{l,n}_{m}\) is a learnable scalar initialized as one to control the impact of this adapter. Thus, this module can process features of various modalities while maintaining a tiny parameter count.

Different from unimodal MOE, our multimodal MOE needs to consider features from various modalities for routing. Therefore, we propose the **multimodal attention router** in which we utilize the multimodal attention gate to compute the gating vector for multimodal experts. First, we stack the three modality features at time step \(i\) into a matrix \({\bf M}^{l}_{i}=[{\bf X}^{l}_{v,i},{\bf X}^{l}_{t,i},{\bf X}^{l}_{a,i}]\in \mathbb{R}^{3\times d_{t}}\). Then we pass it into a parameter-free Self-Attention module:

\[{\bf G}^{l}_{i}={\rm softmax}(\frac{{\bf M}^{l}_{i}{\bf M}^{l^{\top}}_{i}}{ \sqrt{d_{t}}}){\bf M}^{l}_{i}. \tag{9}\]

Then a linear layer is used to calculate the gating vector of multimodal experts:

\[{\bf g}^{l}_{i}={\bf W}^{l}_{g}{\bf G}^{l}_{i}+b^{l}_{g}, \tag{10}\]

where \({\bf g}^{l}_{i}\in\mathbb{R}^{3\times N}\) is the gating vector of the \(i_{\rm th}\) token. \({\bf W}^{l}_{g}\) is the projection matrix and \(b^{l}_{g}\) is the bias. Then, we merge multimodal experts according to the gating vector \({\bf g}^{l}_{i}\) and calculate the adapter change of \(i_{\rm th}\) token \({\bf C}^{l}_{i}\). For the convenience of subsequent descriptions, we denote \(\{{\bf F}^{l,1}_{v,i},\cdots,{\bf E}^{l,N}_{v,i},{\bf E}^{l,1}_{t,i},\cdots,{ \bf E}^{l,N}_{t,i},{\bf E}^{l,1}_{a,i},\cdots,{\bf E}^{l,N}_{a,i}\}\) as \(\{\overline{{\bf E}}^{l}_{i,1},\cdots,\overline{{\bf E}}^{l}_{i,3\times N}\}\), and utilize discrete routing to dispatch tokens into appropriate experts.

\[\hat{{\bf g}}^{l}_{i},\mathbb{I}^{l}_{i}={\rm Top\mbox{-}}~{}K({\bf g}^{l}_{ i}), \tag{11}\]

where \(\hat{{\bf g}}^{l}_{i}\in\mathbb{R}^{K}\) is the Top-\(K\) values of \({\bf g}^{l}\). \(\mathbb{I}^{l}_{i}\) is the indices of Top-\(K\) values in \({\bf g}^{l}_{i}\). Suppose \(\{\hat{{\bf E}}^{l}_{i,1},\cdots,\hat{{\bf E}}^{l}_{i,K}\}=\{\overline{{\bf E}} ^{l}_{i,j}|j\in\mathbb{I}^{l}_{i}\}\) are selected experts, we pass the gating values through softmax operation and calculate the weighted sum between gating values and selected experts output:

\[{\bf C}^{l}_{i}=\frac{\alpha}{r}\sum_{j=1}^{K}{\rm softmax}(\hat{{\bf g}}^{l}_{ i})_{j}\hat{{\bf E}}^{l}_{i,j}, \tag{12}\]

where \(\alpha\) is a hyper-parameter controlling the impact of adapter change \({\bf C}^{l}_{i}\). \({\bf C}^{l}_{i}\) is the \(i_{th}\) time step of \({\bf C}^{l}\), which is the output of the \({\rm MODE}({\bf X}^{l}_{t},{\bf X}^{l}_{v},{\bf X}^{l}_{a})\) used in equation (6). Under such a routing strategy, the total number of activated experts is fixed, but the number of activated experts per modality may vary. We hope that the attention router can select the most helpful features for sentiment analysis and avoid useless features or noise in video and audio data.

**Intra-Modality Load Balancing.** In our MOME, experts of different modalities are differentiated through different inputs. To ensure that experts of the same modality can capture diverse features and achieve a balanced load, we apply the load balancing loss in switch transformer Fedus et al. (2022b) for each modality. Given a batch \(\mathcal{B}\) with \(T\) tokens, for modality \(m\in\{t,v,a\}\), we first calculate the fraction of tokens dispatched to expert \(n\) in each transformer layer, denoted as \(f^{l,n}_{m}\):

\[f^{l,n}_{m}=\frac{1}{T}\sum_{i\in\mathcal{B}}I(n\in{\rm Top\mbox{-}}~{}K({\bf g }^{l}_{i})), \tag{13}\]

where \(I(\cdot)\) is the indicator function. When the load of experts for modality \(m\) is completely balanced, \(f^{l,n}_{m}\) is equal for different \(n\). Then, we calculate \(P^{l,n}_{m}\), the fraction of the router probability allocated for expert \(n\) of modality \(m\):

\[P^{l,n}_{m}=\frac{1}{T}\sum_{i\in\mathcal{B}}{\rm softmax}({\bf g}^{l}_{i})^{ n}. \tag{14}\]

Similarly, when the routing is uniform, \(P^{l,n}_{m}\) is equal for different \(n\). Thus, the load-balancing loss in the \(l_{\rm th}\) layer for modality \(m\) is the product between \(f^{l,n}_{m}\) and \(P^{l,n}_{m}\):

\[\mathcal{L}^{l}_{lb,m}=N\sum_{n\in\mathbb{I}_{m}}f^{l,n}_{m}P^{l,n}_{m}, \tag{15}\]

where \(\mathbb{I}_{m}\) is the index of experts for modality \(m\). When experts for modality \(m\) are under uniform routing, the auxiliary loading balancing loss 

achieves its minimum. The overall load balancing loss is the average of the load balancing losses from all layers for all three modalities:

\[{\mathcal{L}_{lb}=\frac{1}{L}\sum_{l=1}^{L}({\mathcal{L}_{lb,t}^{l}+ \mathcal{L}_{lb,v}^{l}+\mathcal{L}_{lb,a}^{l})}.} \tag{16}\]

### Training and Inference

After combining MMA with the PLM, the output of \(L_{\rm th}\) transformer block can be considered as a multimodal representation. To predict the sentiment strength, we feed the multimodal representation into a prediction head, consisting of two linear transforms and a tanh in between:

\[{\mathbf{p}=\mathbf{W}_{2}\tanh(\mathbf{W}_{1}\mathbf{B}_{\text{out},[\text{cls}] }^{L}+b_{1})+b_{2}}, \tag{17}\]

where \(\mathbf{B}_{\text{out},[\text{cls}]}^{L}\) is the [cls] token of the output of \(L_{\rm th}\) transformer block. Given the sentiment label \({\mathbf{y}}\), we adopt the mean absolute error (MAE) between the prediction \({\mathbf{p}}\) and the label \({\mathbf{y}}\) as the task loss, and the main loss function is the weighted sum of task loss and load balancing loss:

\[{\mathcal{L}=\mathrm{MAE}({\mathbf{y}},{\mathbf{p}})+\lambda{\mathcal{ L}_{lb}}}, \tag{18}\]

where \(\lambda\) is the hyper-parameter controlling the impact of load balancing loss. During the training process, only the parameters of the adapters and prediction head are updated, while the pre-trained weights remain fixed. During the inference process, only the activated experts participate in the forward process to achieve selective multimodal fusion.

## 4 Experiments

In this section, we conduct experiments to evaluate the performance of our method. First, we introduce the detailed setting about our experiments. Then, we compare our method with the previous methods. Finally, we conduct experiments to validate the effectiveness of different components in MMA.

### Experimental Settings

Datasets.We conducted experiments on two widely used datasets: MOSI Zadeh et al. (2016) and MOSEI Zadeh et al. (2018).

The MOSI is a dataset comprising 93 videos sourced from YouTube, each ranging from 2 to 5 minutes in length. These videos are from 89 different speakers and segmented into 2199 short video clips, each annotated by five different annotators. The annotations in MOSI are real numbers within the range of [-3, +3], where the sign of the value represents the polarity of the sentiment, and the magnitude signifies the intensity of the sentiment.

MOSEI comprises 3228 videos across 250 different topics sourced from YouTube, featuring 1000 speakers, which ensures a variety of samples. These videos are divided into 23453 short video segments. The annotation format in MOSFET dataset is consistent with MOSI, using real-valued annotations within the range of [-3, +3].

Evaluation Metrics.To evaluate our model, we adopt metrics commonly used in previous works Zong et al. (2023); Hazarika et al. (2020), including mean absolute error (MAE), pearson correlation coefficient (Corr), seven-class accuracy (ACC-7), binary classification accuracy (ACC-2), and F1 score. Except for MAE, higher values of these metrics indicate stronger performance of the model. Additionally, we considered the number of trainable parameters (denoted as TP) to estimate the resource consumption during model training.

Implementation Details.For experiments on MOSI and MOSEI, the general hyper-parameters are as follows: number of experts per modality is 2, \(\alpha\) is 32, and \(\lambda\) is 0.01. The rank of LoRA is 32. The feature extraction toolkits for audio and video modality are COVAREP Degottex et al. (2014) and FACET Baltrusaitis et al. (2016). A more detailed explanation can be found in Appendix B.1.

### Comparisons with Other Methods

Table 1 is the performance of our MMA on the MOSI and MOSEI datasets, and the compared methods include TFN Zadeh et al. (2017), LMF Liu et al. (2018), MulT Tsai et al. (2019), MISA Hazarika et al. (2020), MAG, Rahman et al. (2020), Self-MM Yu et al. (2021), CubeMLP Sun et al. (2022), ConFEDE Yang et al. (2023), and Acformer Zong et al. (2023). The detailed description of these methods is shown in Appendix A. (B) and (L) indicates that the pre-trained language models are BERT-base-uncased Devlin et al. (2018) and LLAMA2-7B Touvron et al. (2023), respectively. We bold the best results in each metric. When the language model is BERT, MMA yields better or comparable results to many baseline methods. Specifically, MMA significantly outperforms SOTA in all metrics on MOSI and in ACC-7 on MOSEI with much fewer trainable parameters. For other metrics on the MOSEI dataset, MMA achieves a very close performance to SOTA. These results verify the effectiveness of our method



To test MMA's capabilities further, We employ MMA on LLAMA2, which is the most commonly used open-source LLM. Following [143], we use the last token \({\bf B}_{\rm out,M_{t}}^{L}\) as the representation of the output sequence. We apply two strong baselines, namely ConFEDE, and AcFormer, to LLAMA2. For a fair comparison, we utilize LoRA to transfer LLAMA to MSA task and keep the number of trainable parameters comparable when training ConFEDE and AcFormer. We observe that MMA surpasses ConFEDE and AcFormer in all metrics on both datasets.

### Ablation Study

In ablation study, we use the MOSI dataset to evaluate the effectiveness of multimodal experts, multimodal attention router, and load balancing loss.

**Multimodal Experts.** First, we investigate the impact of multimodal experts on model performance by adding one kind of expert at a time, as shown in Table 2. We observe that employing a combination of three types of experts yields the best performance. Moreover, the absence of multimodal experts (first line) corresponds to the poorest outcome, underscoring the effective integration of multimodal information into the language model facilitated by these experts.

As illustrated in Figure 2(a), we investigate how the model performance varies with the rank of adapters \(r\) and the number of experts \(N\). These two hyper-parameters determine the number of trainable parameters in MMA. We conduct experiments with \(r=[2,4,8,16,32,64]\) and \(N=[1,2,3,4]\). The experimental results demonstrate a gradual improvement in model performance as intrinsic dimension \(r\) increases. This improvement is attributed to the increased parameter count of each adapter, thereby enhancing its capability. However, once intrinsic dimension \(r\) reaches \(32\), further increments do not yield improvements, suggesting that larger intrinsic dimensions of multimodal adapters may not be necessary for MSA. For the number of experts per modality \(N\), the performance keeps increasing as \(N\) increases to \(2\), but does not continue to improve as \(N\) further increases. Surprisingly, when \(N\) is \(4\) the model performs worse than when \(N\) is \(2\) or \(3\). We believe that one expert per modal

Figure 2: Ablation study on rank \(r\), number of experts \(N\), and number of activated experts \(K\).

 

ity is insufficient for capturing diverse patterns in multimodal data, whereas an excessive number of experts risks the overfitting of MMA, consequently diminishing model performance.

We also studied the impact of \(K\) (number of activated experts) on the performance when \(N\) is 2. The results are shown in Figure 2 (b). We can observe that in MOSI, as \(K\) increases, the model performance initially improves and then begins to deteriorate. The performance of MMA is best when \(K\) is \(3\). We believe that when the number of activated experts is too small, some effective multimodal information is excluded, and when the number of activated experts value is too large, too many irrelevant multimodal features may be introduced into the PLM. Therefore, the model performance is best when \(K\) is an intermediate value.

**Multimodal Attention Router.** Then we unveil the effect of our multimodal attention router by comparing it with the following routing strategies: (1) Without Router (w/o Router): randomly select the activated multimodal experts. (2) Linear Router (L.R.): After averaging features from different modalities, apply a linear layer to calculate the weights of all experts. The experimental results are shown in Table 3. It can be observed that our multimodal attention router exhibits superior performance. We speculate that this is because, without a router, the model cannot activate the appropriate experts, making the MOE structure less effective. And the commonly used linear router in MOE struggles to enable the interaction of different modalities at different time steps, making it difficult to select effective multimodal features for the PLM. Our method can better consider multimodal information when calculating the weights for different experts, thereby achieving better results.

At last, we investigate the loading situation of multimodal experts. The visualization results are shown in Figure 3. First, we can see that the load of MMA is generally balanced across different modalities. For experts across 12 layers of BERT, the percentage of tokens choosing vision, text, and audio experts is 35.6%, 33.4%, and 31.0%. Secondly, we can observe that the load situation in each layer varies, indicating its ability to select appropriate experts for different layers of the PLM.

**Loading Balancing Loss.** We investigate the influence of our intra-modality load balancing loss in Table 4. We compare the following three strategies: (1) without load balancing loss (w/o LBL): do not use load balancing auxiliary loss during training. (2) Unified load balancing loss (U.LBL): apply a unified load balancing loss across all experts, regardless of whether their inputs are the same. (3) Intra-modality load balancing loss (Ours): only apply load balancing loss among experts with the same modality (the same input). We can observe that our method performs better, and compared to the case without load balancing loss, the unified load balancing loss did not improve the model's performance. This is because the multimodal router achieves selective fusion by activating different multimodal experts to integrate useful multimodal features while discarding misleading or noisy ones. Forcing the load to be balanced among experts of different modalities would impair the router's ability to select effective multimodal features.

## 5 Conclusion

In this paper, we have proposed the Mixture of Multimodal Adapters to tackle the problem of multimodal sentiment analysis. Specifically, (1) multimodal experts are used to insert multimodal content into the pre-trained language model. (2) The

Figure 3: Percentage of activated multimodal adapters across different layers after training MMA on BERT.

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|} \hline Methods & MAE & Corr & ACC-7 & ACC-2 & F1 \\ \hline w/o Router & 0.739 & 0.791 & 45.0 & 84.9 & 85.0 \\ L.R. & 0.705 & 0.799 & **48.3** & 85.4 & 85.3 \\ \hline
**Ours** & **0.693** & **0.803** & 46.9 & **86.4** & **86.4** \\ \hline \end{tabular}
\end{table}
Table 3: Evaluation of the multimodal router on MOSI.

 

multimodal attention router is designed to dispatch features in textual sequence to appropriate experts. (3) Our method is a plug-and-play adapter that converts a transformer-based language model into an MSA model with few trainable parameters. Extensive experiments demonstrate the effectiveness of our method. Note that video and audio representations are limited by the feature extraction tools. In future work, we plan to build an end-to-end network with pre-trained video and audio models.

## Limitations

While our MMA method has shown promising results in multimodal sentiment analysis task, there are still some limitations. First, to ensure a fair comparison with the baselines, we used FACET Baltrusaitis et al. (2016) and COVAREP Degottex et al. (2014) to extract video and audio features. These extraction tools may have limited the quality of video and audio representations, thereby affecting the model's accuracy. A better approach would be to leverage pre-trained video and audio models, which we plan to explore in our future work. Secondly, our work is based on the scenario where all three modalities (_i.e._ , text, video, and audio) are present, without considering cases of missing modalities. As a result, the model's robustness to missing modalities may be insufficient.

## Acknowledgments

The work was supported by the National Natural Science Foundation of China (Grants No. 62202439), the Anhui Provincial Natural Science Foundation, China (Grant No. 2408085QF191), the Fundamental Research Funds for the Central Universities (Grant No. JZ2024HGTA0178). This work was also supported by the advanced computing resources provided by the Supercomputing Center of the USTC.

## References

* Balthwaitis et al. (2016) Tadas Balthwaitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In _2016 IEEE winter conference on applications of computer vision (WACV)_, pages 1-10. IEEE.
* Ben et al. (2024) Huixia Ben, Shuo Wang, Meng Wang, and Richang Hong. 2024. Pseudo content hallucination for unpaired image captioning. In _Proceedings of the 2024 International Conference on Multimedia Retrieval_, pages 320-329.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Cai et al. (2024) Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2024. A survey on mixture of experts. _arXiv preprint arXiv:2407.06204_.
* Chen et al. (2024) Kezhou Chen, Shuo Wang, and Yanbin Hao. 2024. Hierarchical supervised contrastive learning for multi-modal sentiment analysis. In _International Conference on Multimedia Modeling_, pages 56-69. Springer Nature Switzerland.
* Degottex et al. (2014) Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarep--a collaborative voice analysis repository for speech technologies. In _2014 ieee international conference on acoustics, speech and signal processing (icassp)_, pages 960-964. IEEE.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Dou et al. (2023) Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et al. 2023. Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment. _arXiv preprint arXiv:2312.09979_, 4(7).
* Eigen et al. (2013) David Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. 2013. Learning factored representations in a deep mixture of experts. _arXiv preprint arXiv:1312.4314_.
* Fedus et al. (2022a) William Fedus, Jeff Dean, and Barret Zoph. 2022a. A review of sparse expert models in deep learning. _arXiv preprint arXiv:2209.01667_.
* Fedus et al. (2022b) William Fedus, Barret Zoph, and Noam Shazeer. 2022b. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. _Journal of Machine Learning Research_, 23(120):1-39.
* Guo et al. (2019) Dan Guo, Shuo Wang, Qi Tian, and Meng Wang. 2019. Dense temporal convolution network for sign language translation. In _IJCAI_, pages 744-750.
* Hazarika et al. (2022) Devaramyu Hazarika, Yingting Li, Bo Cheng, Shuai Zhao, Roger Zimmermann, and Soujanya Poria. 2022. Analyzing modality robustness in multimodal sentiment analysis. _arXiv preprint arXiv:2205.15465_.
* Hazarika et al. (2020) Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020. Misa: Modality-invariant and specific representations for multimodal sentiment analysis. In _Proceedings of the 28th ACM international conference on multimedia_, pages 1122-1131.

 

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. In _International Conference on Learning Representations_.
* Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilho, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In _International conference on machine learning_, pages 2790-2799. PMLR.
* Hu et al. (2021) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In _International Conference on Learning Representations_.
* Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mikral of experts. _arXiv preprint arXiv:2401.04088_.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Lepikhin et al. (2020) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. In _International Conference on Learning Representations_.
* Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. _arXiv preprint arXiv:2101.00190_.
* Li et al. (2024) Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. 2024. Uni-moe: Scaling unified multimodal lms with mixture of experts. _arXiv preprint arXiv:2405.11273_.
* Liang et al. (2024) Tian Liang, Jing Huang, Ming Kong, Luyuan Chen, and Qiang Zhu. 2024. Querying as prompt: Parameter-efficient learning for multimodal language model. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 26855-26865.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Liu et al. (2018) Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018. Efficient low-rank multimodal fusion with modality-specific factors. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2247-2256.
* Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. _arXiv preprint arXiv:1711.05101_.
* Lu et al. (2023) Jinda Lu, Shuo Wang, Xinyu Zhang, Yanbin Hao, and Xiangnan He. 2023. Semantic-based selection, synthesis, and supervision for few-shot learning. In _Proceedings of the 31st ACM International Conference on Multimedia_, pages 3569-3578.
* Peng et al. (2023) Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. 2023. Rwkv: Reinventing rnns for the transformer era. _arXiv preprint arXiv:2305.13048_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of machine learning research_, 21(140):1-67.
* Rahman et al. (2020) Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque. 2020. Integrating multi-modal information in large pretrained transformers. In _Proceedings of the conference. Association for Computational Linguistics. Meeting_, volume 2020, page 2359. NIH Public Access.
* Riquelme et al. (2021) Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andres Susano Pinto, Daniel Keysers, and Neil Houlsby. 2021. Scaling vision with sparse mixture of experts. _Advances in Neural Information Processing Systems_, 34:8583-8595.
* Shazeer et al. (2016) Noam Shazeer, Azalia Mirhoseini, Krzyszlof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2016. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In _International Conference on Learning Representations_.
* Sun et al. (2022) Hao Sun, Hongyi Wang, Jiaqing Liu, Yen-Wei Chen, and Lanfen Lin. 2022. Cubemlp: An mlp-based model for multimodal sentiment analysis and depression estimation. In _Proceedings of the 30th ACM international conference on multimedia_, pages 3722-3729.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashkyok, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Tsai et al. (2019) Yao-Hung Hubert Tsai, Shaoijie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philipe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In _Proceedings of the conference. Association for computational linguistics. Meeting_, volume 2019, page 6558. NIH Public Access.

 

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017a. Attention is all you need. _Advances in neural information processing systems_, 30.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, AidannN. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017b. Attention is all you need. _Neural Information Processing Systems_.
* Wang et al. (2024) Haixin Wang, Xinlong Yang, Jianlong Chang, Dian Jin, Jinn Sun, Shikun Zhang, Xiao Luo, and Qi Tian. 2024a. Parameter-efficient tuning of large-scale multimodal foundation model. _Advances in Neural Information Processing Systems_, 36.
* Wang et al. (2019) Shuo Wang, Dan Guo, Xin Xu, Li Zhuo, and Meng Wang. 2019. Cross-modality retrieval by joint correlation learning. _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_, 15(2s):1-16.
* Wang et al. (2018) Shuo Wang, Dan Guo, Wen gang Zhou, Zheng jun Zha, and Meng Wang. 2018. Connectionist temporal fusion for sign language translation. In _Proceedings of the 26th ACM international conference on Multimedia_, pages 1483-1491.
* Wang et al. (2024b)Shouo Wang, Jinda Lu, Haiyang Xu, Yanbin Hao, and Xiangnan He. 2024b. Feature mixture on pre-trained model for few-shot learning. _IEEE Transactions on Image Processing_, 33:4104-4115.
* Wang et al. (2020)Shuo Wang, Jun Yue, Jianzhuang Liu, Qi Tian, and Meng Wang. 2020. Large-scale few-shot learning via multi-modal knowledge discovery. In _Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16_, pages 718-734. Springer International Publishing.
* Wang et al. (2022)Shao Wang, Xinyu Zhang, Yanbin Hao, Chengbing Wang, and Xiangnan He. 2022. Multi-directional knowledge transfer for few-shot learning. In _Proceedings of the 30th ACM International Conference on Multimedia_, pages 3993-4002.
* Wang et al. (2024c)Shoue Wang, Xinyu Zhang, Meng Wang, and Xiangnan He. 2024c. Symmetric hallucination with knowledge transfer for few-shot learning. _IEEE Transactions on Multimedia_.
* Wei et al. (2024)Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei Lu, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, et al. 2024. Skyworkmoe: A deep dive into training techniques for mixture-of-experts language models. _arXiv preprint arXiv:2406.06563_.
* Xie et al. (2024)Yifeng Xie, Zhihong Zhu, Xin Chen, Zhanpeng Chen, and Zhiqi Huang. 2024. Moba: Mixture of bi-directional adapter for multi-modal sarcasm detection. In _Proceedings of the 32nd ACM International Conference on Multimedia_, pages 4264-4272.
* Xu et al. (2023) Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2023. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. _arXiv preprint arXiv:2312.12148_.
* Yang et al. (2023)Jiuding Yang, Yakun Yu, Di Niu, Weidong Guo, and Yu Xu. 2023. Confede: Contrastive feature decomposition for multimodal sentiment analysis. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7617-7630.
* Yu et al. (2021)Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. 2021. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In _Proceedings of the AAAI conference on artificial intelligence_, volume 35, pages 10790-10797.
* Zadeh et al. (2017) Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1103-1114.
* Zadeh et al. (2016)Amir Zadeh, Rowan Zellers, Eli Pincus, and Louisiana-Philippe Morency. 2016. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. _IEEE Intelligent Systems_, 31(6):82-88.
* Zaghefzadeh et al. (2018) AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: Cmumosei dataset and interpretable dynamic fusion graph. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2236-2246.
* Zaken et al. (2022)Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 1-9.
* Zhu et al. (2024a) Xingyu Zhu, Shuo Wang, Jingda Lu, Yanbin Hao, Haifeng Liu, and Xiangnan He. 2024a. Boosting few-shot learning via attentive feature regularization. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 7793-7801.
* Zhu et al. (2024b)Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. 2024b. Enhancing zero-shot vision models by label-free prompt distribution learning and bias correcting. In _NeurIPS 2024_.
* Zhu et al. (2024c) Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. 2024c. Selective vision-language subspace projection for few-shot clip. In _Proceedings of the 32nd ACM International Conference on Multimedia_, pages 3848-3857.
* Zong et al. (2023)Daoming Zong, Chaoyue Ding, Baoxiang Li, Jiaksui Li, Ken Zheng, and Qunyan Zhou. 2023. Acformer: 

An aligned and compact transformer for multimodal sentiment analysis. In _Proceedings of the 31st ACM International Conference on Multimedia_, pages 833-842.

## Appendix A Baselines

**TFN**: Tensor Fusion Network Zadeh et al. (2017) models intra-modality and inter-modality dynamics by three-fold Cartesian product.

**LMF**: Low-rank Multimodal Fusion Liu et al. (2018) performs multimodal fusion using low-rank tensors to improve efficiency and avoid exponential increase in dimensions.

**MuIf**: Multimodal transformer Tsai et al. (2019) is a transformer-based model requiring no alignment assumption by learning a latent cross-modal adaptation through a pairwise cross-modal attention mechanism.

**MISA**: MISA Hazarika et al. (2020) learns modality invariant subspace and modality-specific subspace to learn effective modality representations which is conducive to the fusion process.

**MAG**: Multimodal Adaptation Gate Rahman et al. (2020) integrates multimodal information into pretrained Bert by changing the position of words in the semantic space.

**Self-MM**: Self-Supervised Multi-task Multimodal sentiment analysis network Yu et al. (2021) can auto-generate unimodal labels and learn modality-specific representation by joint training on unimodal subtasks and multimodal main task.

**CubeMLP**: CubeMLP Sun et al. (2022) is an MLP-based model that performs feature mixing on sequence, modality, and channel to reduce computational costs while maintaining high performance

**ConFEDE**: ConFEDE Yang et al. (2023) introduces contrastive learning into MSA to learn similarity and dissimilarity features for each modality and predicts sentiment depending on decomposed modality representations.

**AcFormer**: Aligned and Compact transformer Zong et al. (2023) is a model that aligns different modalities by contrastive learning and performs fusion using a pivot attention module.

## Appendix B Experiments

### Experimental Details

Here we introduce the detailed settings of our experiments. When MMA is based on BERT, the pre-trained model is bert-base-uncased1. We conduct experiments on a single NVIDIA RTX 3090. For experiments on MOSI and MOSFET, the batch size is 128, the learning rates are {1e-3, 2e-4}, the optimizers are {AdamW Loshchilov and Hutter (2017), Adam Kingma and Ba (2014)}. When MMA is based on LLAMA2, the pre-trained model is LLAMA2-7B-base 2. We conduct experiments on a single NVIDIA A100. For experiments on MOSI and MOSEI, the batch size is 16,4, the learning rates are {1e-4, 1e-5}, and the optimizer is AdamW Loshchilov and Hutter (2017). All models are trained for 25 epochs.

Footnote 1: [https://huggingface.co/meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf)

### Position of MMA

In our method, we integrate the output of self-attention with multimodal features and then add the fused result to the output of FFN. However, the position of MMA can be variable. To explore the best position of MMA in the transformer block, following He et al. (2021), we categorize the positions in the transformer block into four types based on the insertion form (sequential or parallel) and modified representation (attention or FFN), as shown in the Figure 4. The insertion form and modified representation of position {(a),(b),(c),(d)} are {(sequential, attention), (parallel, attention), (sequential, FFN), (parallel, FFN)}. The experimental results are shown in Table 5. Firstly, the best performance is achieved when MMA is in position **(d)**. In a closer look, we observed that the parallel insertion form outperforms the sequential insertion form, and modifying the representation of FFN is better than modifying the representation of attention, which is consistent with the experience from unimodal adapters He et al. (2021).

Figure 4: Four different positions of MMA.

\begin{table}
\begin{tabular}{|p{28.5pt}|p{28.5pt}|p{28.5pt}|p{28.5pt}|p{28.5pt}|} \hline Types & MAE & Corr & ACC-7 & ACC-2 & F1 \\ \hline (a) & 0.717 & **0.806** & 46.4 & 84.0 & 84.0 \\ (b) & 0.697 & 0.797 & **48.0** & 86.0 & 85.9 \\ (c) & 0.707 & 0.796 & 47.5 & 84.9 & 84.9 \\ (d) & **0.693** & 0.803 & 46.9 & **86.4** & **86.4** \\ \hline  & \multicolumn{1}{p{28.5pt}}{} & \multicolumn{1}{p{28.5pt}}{} & \multicolumn{1}{p{28.5pt}}{} & \multicolumn{1}{p{28.5pt}}{} \\ \end{tabular}
\end{table}
Table 5: Evaluation of the position of MMA on MOSI.