DaNet: Dual-Aware Enhanced Alignment Network for Multimodal Aspect-Based Sentiment Analysis
Authors: Aoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang, Yiming Tang, Ning An
Accepted at: acl25

Proxy-Driven Robust Multimodal Sentiment Analysis with Incomplete Data

Aoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang, Yiming Tang, Ning An
School of Computer Science and Information Engineering, Hefei University of Technology
zhuaqiang@mail.hfut.edu.cn, {jsjxhumin, xh_wang, jiaoyun, ymtang}@hfut.edu.cn, ning.g.an@acm.org

Abstract
Multimodal Sentiment Analysis (MSA) with incomplete data has gained significant attention recently. Existing studies focus on optimizing model structures to handle modality missingness, but models still face challenges in robustness when dealing with uncertain missingness. To this end, we propose a data-centric robust multimodal sentiment analysis method, Proxy-Driven Robust Multimodal Fusion (P-RMF). First, we map unimodal data to the latent space of Gaussian distributions to capture core features and structure, thereby learn stable modality representation. Then, we combine the quantified modality intrinsic uncertainty to learn stable multimodal joint representation (i.e., proxy modality), which is further enhanced through multi-layer dynamic cross-modal injection to increase its diversity. Extensive experimental results show that P-RMF outperforms existing models in noise resistance and achieves state-of-the-art performance on multiple benchmark datasets. Code will be available at https://github.com/aoqzhu/P-RMF.

1 Introduction
Multimodal Sentiment Analysis (MSA) integrates information from multiple modalities to understand and recognize human emotions (Singh et al., 2024). By fusing complementary information, multimodal learning generates richer joint representation (Xu et al., 2023). However, in real-world scenarios, modality data is often missing due to factors such as background noise, sensor limitations, and privacy concerns. Data incompleteness significantly reduces the effectiveness of models trained on complete data (Wang et al., 2023b).

In recent years, many studies (Wang et al., 2023c; Zhang et al., 2024; Sun et al., 2023b; Li et al., 2024b) have proposed influential solutions to the missing data problem in MSA. These methods can be broadly categorized into two main types: reconstruction-based methods (Lian et al., 2023; Sun et al., 2023b; Zhao et al., 2021) and joint representation-based methods (Li et al., 2024a,b; Kim and Kim, 2024). Although these methods have made significant progress in addressing data incompleteness, models focusing on model structure optimization still face challenges in performance and robustness when handling data with uncertain missingness. These challenges mainly stem from distributional discrepancies in incomplete data and the epistemic uncertainty of deep learning models (Kendall and Gal, 2017). Moreover, existing models for incomplete data often fail to balance the effective processing of complete data.

The randomly missing incomplete data further increases the arbitrary uncertainty of the data and the epistemic uncertainty of the model (Kendall and Gal, 2017). Therefore, we explore whether a data-centric method, focusing on the core features and structures of the data, can enhance model robustness to input variations and noise. Inspired by probability distributions (Ho et al., 2020; Chen et al., 2022; Yang et al., 2024), we argue that semantically related data, even across different modalities, should exhibit similar distributions in the latent Gaussian space despite varying noise levels. This latent space constructs regularized, compact, and semantically consistent representations, abstracting

Step1: Generate Robust Proxy Modality
\( \mathcal{N}(\mu^m, \sigma^m) \)

Step2: Proxy-Driven Dynamic Injection

![The core steps of P-RMF.](page_820_629_563_312.png)

Figure 1: The core steps of P-RMF.
high-level features while weakening task-irrelevant details, thereby enhancing robustness to input variations and noise. Given the stable and semantically consistent unimodal representation, should we disregard the intrinsic uncertainty of the data? Prior studies (Sun et al., 2023a; Yu et al., 2020) highlight the unique characteristics of unimodal data and the varying contributions of different modalities to sentiment recognition. We argue that the uncertainty arising from missing noise in heterogeneous data reflects inter-modal information discrepancies, embodying the distinct properties of each modality. Thus, we quantify this uncertainty by measuring feature space distribution discrepancies and integrate it into multimodal fusion to learn dynamic and robust representation.

Based on the above analysis, we propose Proxy-Driven Robust Multimodal Fusion (P-RMF) for robust sentiment analysis under uncertain missing data. As shown in Fig. 1, P-RMF consists of two core steps: generate robust proxy modality and proxy-driven dynamic injection. Specifically, we employ variational inference and design an independent Variational Autoencoder (VAE) for each modality. The encoder maps data into a Gaussian distributions latent space, learning its mean and variance, while the reparameterization trick enables sampling of latent variables as compact representation of missing data. The decoder then reconstructs the complete input, learning to recover missing information. The variance captures modality uncertainty, whereas the mean provides a stable representation. By integrating both, We learn a consistent and stable joint modality representation (i.e., proxy modality). Finally, with proxy modality as the dominant, combined with quantized uncertainty, we enhance its diversity by multi-layer dynamic cross-modal injection. The main contributions of this paper are as follows:

• We propose a data-centric robust multimodal sentiment analysis method to address the robustness problem under uncertain missing data. This method learns stable unimodal representations from missing data and incorporates quantization uncertainty to guide the learning of dynamic and robust multimodal representation.

• We propose a robust proxy modality generation module that learns stable data representations in a latent space based on Gaussian distributions and generates a robust joint proxy modality representation by incorporating quantized uncertainty.

• We propose a proxy modality-driven dynamic injection module that incorporates quantized modality uncertainty, iteratively injecting modality-specific semantics with varying weights into the proxy modality to enhance the diversity of the joint representation.

2 Related Work

Multimodal Sentiment Analysis (MSA) can be categorized into context-based MSA and noise-aware robust MSA based on modeling methods. Early context-based MSA studies assumed that all data are complete and available during both training and inference phases (Han et al., 2021; Qian et al., 2023; Yu et al., 2023; Li et al., 2023b; Zhu et al., 2024; Zhang et al., 2023). These studies primarily focused on learning unified multimodal representations by analyzing intra- and inter-modal contextual relationships. Despite the progress made by these methods, incomplete data can significantly reduce the effectiveness of models trained on complete data.

In recent years, many studies (Li et al., 2023a, 2024a; Zeng et al., 2022; Zhang et al., 2024; Sun et al., 2023b; Li et al., 2024b) have attempted to solve the missing data problem in MSA by methods such as data reconstruction networks and joint representation learning. For example, EMT-DLFR (Sun et al., 2023b) encouraged the model to learn semantic information from missing data by performing low-level feature reconstruction. These reconstruction-based methods focus on complementing missing modalities using existing modalities, but the quality of the reconstructed data is often difficult to guarantee and is less interpretable. Therefore, UMDF (Li et al., 2024a) and CorrKD (Li et al., 2024b) introduced a knowledge distillation mechanism to learn joint multimodal representations by optimizing the fusion strategy and model architecture. Further, LNLN (Zhang et al., 2024) shows that stabilizing the text-dominant modality under varying levels of missing noise improves model robustness, but ensuring the integrity and quality of the text modality is challenging in multimodal data with arbitrary missing uncertainties.

Inspired by LNLN (Zhang et al., 2024), we argue that the stable dominant modality need not be confined to the original modality (e.g., text or
visual), but can instead be represented by the semantic consistency of different modalities in the high-level feature space. Therefore, we propose a Proxy-Driven Robust Multimodal Fusion (P-RMF).

3 Methodology

3.1 Overall Framework

Fig. 2 shows the key components and workflow of the proposed Proxy-Driven Robust Multimodal Fusion (P-RMF) method. P-RMF consists of five main components: Multimodal Encoder, Proxy Modality Generation (PMG), Proxy-Driven Dynamic Injection (PDDI), Data Reconstruction (DR), and Sentiment Prediction. During training, P-RMF jointly inputs complete data and randomly missing incomplete data. With the joint constraint of KL divergence loss and variational sample reconstruction loss, PMG generates a consistent and robust proxy modality representation by combining the quantized intrinsic modality uncertainty. The PDDI incorporates quantized uncertainty to enhance the diversity of proxy modality representations through dynamic cross-modal injection iterations in the multi-layer. In addition, to complement the missing fine-grained sentiment semantics, P-RMF designs a reconstructor for each modality to reconstruct the missing data. During testing, the trained P-RMF is applied to scenarios involving random intra-modal and inter-modal missing data to evaluate its robustness and effectiveness.

![The framework of the P-RMF.](page_68_120_1347_579.png)

Figure 2: The framework of the P-RMF.

3.2 Input Construction and Multimodal Encoder

We conduct experimental analysis on the MOSI (Zadeh et al., 2016), MOSEI (Bagher Zadeh et al., 2018) and SIMS (Yu et al., 2020) datasets. To simulate random data missing scenarios, we follow the settings of previous work (Zhang et al., 2024), randomly erasing 0% to 100% of information in each modality for every sample. Specifically, 0 padding is applied to the erased portions of the visual and audio modalities, while the text modality is filled with the unknown word token [UNK].

For feature extraction, this study follows prior work (Zhang et al., 2024): text modality is encoded using BERT (Devlin et al., 2019), visual features are extracted via OpenFace (Baltrusaitis et al., 2018), and audio features are obtained using Librosa (Brian McFee et al., 2015). For a given multimodal input, \( X^m \in \mathbb{R}^{L_m \times d_m} \) represents the complete multimodal sequence, while \( H^m \in \mathbb{R}^{L_m \times d_m} \) denotes the incomplete sequence with random missing data. Here, \( m \in \{t, a, v\} \) represents the modality type (i.e., text, audio, and visual), \( L_m \) denotes the sequence length, and \( d_m \) denotes the feature vector dimension.

3.3 Proxy Modality Generation

Previous studies have shown that maintaining the integrity of the dominant Modality significantly enhances the robustness of the model (Zhang et al., 2024). Inspired by probability distributions (Ho et al., 2020; Chen et al., 2022; Yang et al., 2024),
Figure 3: The overall architecture of the Variational Autoencoder.

we learn stable joint multimodal representation from Gaussian distributions of the data as the dominant Modality.

The key to Gaussian distribution analysis lies in how to map the data to the parameterized form of the mean and variance according to the task requirements. To achieve this, we have designed a Variational Autoencoder (VAE) for each modality, as shown in Fig. 3. In the VAE framework, the encoder learns the parameterized mean \( u(h) \) and log-variance \( \sigma^2(h) \) of the input data in the Gaussian-distributed latent space, and \( \mathcal{N}(u(h), \sigma^2(h)I) \) represents the latent distribution of the data. The decoder then reconstructs the input data based on the generated latent variables \( z \) output from the encoder, recovering the core features.

Formally, the variational posterior distribution of the unimodal feature \( h \) can be expressed as
\[
q(z|h) = \mathcal{N}\left[ z|u(h), \sigma^2(h)I \right].
\]
Thus, for the multimodal representation \( H_n^m \) of the \( n \)-th sample, the variational posterior distributions for different modalities can be expressed as:
\[
q(z_n^{m}|h_n^m) \sim \mathcal{N}[z_n^{m}|u(h_n^m), \sigma^2(h_n^m)I] \tag{1}
\]
where \( z_n^m, m \in \{t, v, a\} \) represents the latent variable for modality \( m \) of sample \( n \), and \( I \) denotes the identity matrix. To make the learning of the latent space smoother and more stable, we introduce the Kullback-Leibler (KL) divergence as a regularization term to align the discrepancy between the posterior distribution \( q(z_n^m|h_n^m) \) and the standard prior distribution \( p(z_n^m) = \mathcal{N}(z_n^m|0, I) \). Additionally, we introduce cross-modal KL to align the distributional representations across modalities. The total KL constraint is computed as follows:
\[
\mathcal{L}_{KL} = \sum_{m \in M} D_{KL}(q(z_n^{m}|h_n^m)||q(z_n^{m^*}|h_n^{m^*})) \\
+ \sum_{m \in M} D_{KL}(q(z_n^{m}|h_n^m)||p(z_n^m)) \tag{2}
\]
where \( m \in M = \{t, v, a\} \) and \( m_1 \neq m_2 \). To effectively model the latent space, we employ the reparameterization trick, enabling sampling from the Gaussian distribution in the latent space. Specifically, the latent variable \( z \) is expressed as:
\[
z = u(h) + \varepsilon \cdot \sigma(h), \; \varepsilon \sim \mathcal{N}(0, I) \tag{3}
\]
where \( \varepsilon \) represents the noise sampled from the standard normal distribution. With this reparameterization method, we can convert the sampling process into a conductive operation, which allows the gradient to be efficiently transmitted via back-propagation to support model training. The latent variable \( z \), as an abstract representation of the input data, captures its essential features, reflecting the effectiveness and validity of the latent space parameterization. By evaluating the reconstruction error between the reconstructed samples \( \hat{h}_n^m = \text{Decoder}(z_n^m) \) and the complete data \( x \), we ensure that the samples generated from the latent space closely approximate the complete data.
\[
\mathcal{L}_{SR} = \frac{1}{N} \sum_{m \in M} \sum_{n=1}^N \| x_n^m - \hat{h}_n^m \|^2 \tag{4}
\]
where \( x_n^m \) represents the complete data, \( \hat{h}_n^m \) is the reconstruction of the incomplete input data, and \( N \) is the number of samples. Thus, the total loss for variational inference in multimodal data can be formulated as:
\[
\mathcal{L}_{VAE} = \mathcal{L}_{KL} + \mathcal{L}_{SR} \tag{5}
\]
The \( \mathcal{L}_{VAE} \) constraint ensures effective and consistent stable representations in the latent space. The mean \( u(h) \) of the Gaussian distribution represents stable features, while the variance \( \sigma^2(h) \) reflects distribution uncertainty. Given the stable unimodal representation \( u(h_n^m) \), we compute fusion weights \( \omega_n^m \) for cross-modal representations based on modality uncertainty \( \sigma(h_n^m) \), and apply them to the joint multimodal representation.
\[
H^p = \sum_{m \in M} \omega_n^m \cdot u(h_n^m), \\
\omega_n^m = \frac{\exp(1/\sigma(h_n^m))}{\sum_{m \in M} \exp(1/\sigma(h_n^m))} \tag{6}
\]
where \( H^p \) represents the stable joint multimodal representation, termed the proxy modality.

3.4 Proxy-Driven Dynamic Injection
The proxy modality \( H^p \) maintains the stability and semantic consistency of multimodal data by preserving the core features and structure of the data. However, the specific features of unimodal data reflect the diversity of multimodal representations. Therefore, we propose the PDDI. As shown in Fig. 2, the PDDI incorporates quantized uncertainty weights \( \omega^m_n \) to enhance the diversity of proxy modality representation through dynamic cross-modal injection iterations in the multi-layer.

As in Fig. 4, we keep the multimodal features \( H^m, m \in \{t, a, v\} \) unchanged and use Cross-Modal Attention (CMA) to inject multimodal information. Given the previous layer’s output \( H^p_{l-1} \) and \( H^m \), CMA is defined as:

\[
U^{pm}_l = \text{softmax}\left(\frac{Q^p \cdot K^m}{\sqrt{d_k}}\right) \cdot V^m
\]

where \( Q^p = LN(H^p_{l-1}) \cdot W^p_Q, K^m = LN(H^m) \cdot W^K_V, V^a = LN(H^m) \cdot W^a_V, LN \) is layer normalization. Then, \( U^{pm}_l, m \in \{t, a, v\} \) is weighted by weights \( \omega^m \) of each modality and combined with the previous layer’s output \( H^p_{l-1} \) to obtain \( U_l \).

\[
U_l = LN(H^p_{l-1}) + \sum_{m \in M} \omega^m \cdot U^{pm}_l
\]

Finally, \( U_l \) undergoes layer normalization, a feedforward neural network (FFNN), and a residual to obtain the output \( H^p_l \) of the current layer.

\[
H^p_l = \text{FFNN}(LN(U_l)) + U_l
\]

(9)

3.5 Data Reconstruction
To maximize the potential of available data and reduce information loss due to missing data, we design a data reconstructor \( R^m \) consisting of two Transformer layers for each modality to rebuild the missing information. For incomplete data \( H^m \), we combine the complete data \( X^m \) with a mean square error loss function to optimize the performance of the reconstructor.

\[
\mathcal{L}_{DR} = \frac{1}{N} \sum_{m \in M} \sum_{n=1}^N ||x^m_n - R^m(h^m_n)||^2
\]

(10)

The data reconstruction loss \( \mathcal{L}_{DR} \) enhances the model’s ability to capture sentiment information in the data by minimizing the discrepancies between the original and reconstructed features, thereby enhancing robustness.

3.6 Overall Learning Objectives
We feed the learned robust multimodal joint representation \( H^p_L \) into a Multi-Layer Perceptron (MLP) consisting of two fully connected layers and a ReLU activation function to output the sentiment prediction. The model is optimized using the L1 loss to minimize sentiment prediction error.

\[
\mathcal{L}_{SP} = \frac{1}{N} \sum_{n=1}^N |\hat{y}_n - y_n|
\]

(11)

where \( \mathcal{L}_{SP} \) represents the sentiment prediction loss, \( \hat{y}_n = \text{MLP}(H^p_L) \) denotes the predicted value for sample \( n \), and \( y_n \) represents the label for the sample.

In summary, our method P-RMF is designed with three learning objectives: \( \mathcal{L}_{VAE}, \mathcal{L}_{DR} \), and \( \mathcal{L}_{SP} \). Therefore, the total loss can be expressed as:

\[
\mathcal{L} = \lambda_1 \mathcal{L}_{SP} + \lambda_2 \mathcal{L}_{VAE} + \lambda_3 \mathcal{L}_{DR}
\]

(12)

where \( \lambda_1, \lambda_2 \) and \( \lambda_3 \) are weighted hyperparameters.

4 Experiments
4.1 Datasets
We performed a comprehensive experimental analysis on three widely used public datasets MOSI (Zadeh et al., 2016), MOSEI (Bagher Zadeh et al., 2018) and SIMS (Yu et al., 2020). The details of the datasets are shown in Appendix A.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="5">MOSI</th>
    <th colspan="5">MOSEI</th>
  </tr>
  <tr>
    <th>Acc-2</th>
    <th>F1</th>
    <th>Acc-5</th>
    <th>Acc-7</th>
    <th>MAE</th>
    <th>Corr</th>
    <th>Acc-2</th>
    <th>F1</th>
    <th>Acc-5</th>
    <th>Acc-7</th>
    <th>MAE</th>
    <th>Corr</th>
  </tr>
  <tr>
    <td>MISA</td>
    <td>70.33/71.49</td>
    <td>70.00/71.28</td>
    <td>33.08</td>
    <td>29.85</td>
    <td>1.085</td>
    <td>0.524</td>
    <td>75.82/71.27</td>
    <td>68.73/63.85</td>
    <td>39.39</td>
    <td>40.84</td>
    <td>0.780</td>
    <td>0.503</td>
  </tr>
  <tr>
    <td>Self-MM</td>
    <td>69.26/70.51</td>
    <td>67.54/66.60</td>
    <td>34.67</td>
    <td>29.55</td>
    <td>1.070</td>
    <td>0.512</td>
    <td>77.42/73.89</td>
    <td>72.31/68.92</td>
    <td>45.38</td>
    <td>44.70</td>
    <td>0.695</td>
    <td>0.498</td>
  </tr>
  <tr>
    <td>MMIM</td>
    <td>67.06/69.14</td>
    <td>64.04/66.65</td>
    <td>33.77</td>
    <td>31.30</td>
    <td>1.077</td>
    <td>0.507</td>
    <td>75.89/73.32</td>
    <td>70.32/68.72</td>
    <td>41.74</td>
    <td>40.75</td>
    <td>0.739</td>
    <td>0.489</td>
  </tr>
  <tr>
    <td>CENET</td>
    <td>67.73/71.46</td>
    <td>64.85/68.41</td>
    <td>37.25</td>
    <td>30.38</td>
    <td>1.080</td>
    <td>0.504</td>
    <td>77.34/74.67</td>
    <td>74.08/70.68</td>
    <td><b>47.83</b></td>
    <td><b>47.18</b></td>
    <td>0.685</td>
    <td>0.535</td>
  </tr>
  <tr>
    <td>TETFN</td>
    <td>67.68/69.76</td>
    <td>63.29/65.69</td>
    <td>34.34</td>
    <td>30.30</td>
    <td>1.087</td>
    <td>0.507</td>
    <td>67.68/69.76</td>
    <td>63.29/65.69</td>
    <td>47.70</td>
    <td>30.30</td>
    <td>1.087</td>
    <td>0.508</td>
  </tr>
  <tr>
    <td>TFR-Net</td>
    <td>66.35/68.15</td>
    <td>60.06/61.73</td>
    <td>34.67</td>
    <td>29.54</td>
    <td>1.200</td>
    <td>0.459</td>
    <td>77.23/73.62</td>
    <td>71.99/68.80</td>
    <td>34.67</td>
    <td>46.83</td>
    <td>0.697</td>
    <td>0.489</td>
  </tr>
  <tr>
    <td>ALMT</td>
    <td>68.39/70.40</td>
    <td><b>71.80/72.57</b></td>
    <td>33.42</td>
    <td>30.30</td>
    <td>1.083</td>
    <td>0.498</td>
    <td>77.54/76.64</td>
    <td>78.03/77.14</td>
    <td>41.64</td>
    <td>40.92</td>
    <td>0.674</td>
    <td>0.481</td>
  </tr>
  <tr>
    <td>LNLN</td>
    <td>70.94/72.55</td>
    <td>71.25/72.73</td>
    <td>38.27</td>
    <td><b>34.26</b></td>
    <td>1.046</td>
    <td><b>0.527</b></td>
    <td>78.19/76.30</td>
    <td>79.95/77.77</td>
    <td>46.17</td>
    <td>45.42</td>
    <td>0.692</td>
    <td>0.530</td>
  </tr>
  <tr>
    <td><b>P-RMF</b></td>
    <td><b>71.53/72.81</b></td>
    <td>71.69/72.93</td>
    <td><b>38.50</b></td>
    <td>34.19</td>
    <td><b>1.038</b></td>
    <td>0.525</td>
    <td><b>78.83/78.14</b></td>
    <td><b>80.39/79.33</b></td>
    <td>45.87</td>
    <td>44.63</td>
    <td><b>0.658</b></td>
    <td><b>0.589</b></td>
  </tr>
</table>

Table 1: Robustness comparison of overall performance on MOSI and MOSEI under intra-modal missingness.

<table>
  <tr>
    <th>Model</th>
    <th>Acc-2</th>
    <th>F1</th>
    <th>Acc-3</th>
    <th>Acc-5</th>
    <th>MAE</th>
    <th>Corr</th>
  </tr>
  <tr>
    <td>MISA</td>
    <td>72.71</td>
    <td>66.30</td>
    <td>56.87</td>
    <td>31.53</td>
    <td>0.539</td>
    <td>0.348</td>
  </tr>
  <tr>
    <td>Self-MM</td>
    <td>72.81</td>
    <td>68.43</td>
    <td>56.75</td>
    <td>32.28</td>
    <td>0.508</td>
    <td>0.376</td>
  </tr>
  <tr>
    <td>MMIM</td>
    <td>69.86</td>
    <td>66.21</td>
    <td>52.76</td>
    <td>31.81</td>
    <td>0.544</td>
    <td>0.339</td>
  </tr>
  <tr>
    <td>CENET</td>
    <td>68.13</td>
    <td>57.90</td>
    <td>53.17</td>
    <td>22.29</td>
    <td>0.589</td>
    <td>0.107</td>
  </tr>
  <tr>
    <td>TETFN</td>
    <td>73.58</td>
    <td>68.67</td>
    <td>59.61</td>
    <td>33.42</td>
    <td>0.505</td>
    <td>0.387</td>
  </tr>
  <tr>
    <td>TFR-Net</td>
    <td>68.13</td>
    <td>58.70</td>
    <td>52.89</td>
    <td>26.52</td>
    <td>0.661</td>
    <td>0.169</td>
  </tr>
  <tr>
    <td>ALMT</td>
    <td>71.85</td>
    <td>76.21</td>
    <td>56.47</td>
    <td>34.16</td>
    <td>0.509</td>
    <td>0.372</td>
  </tr>
  <tr>
    <td>LNLN</td>
    <td>72.73</td>
    <td><b>79.43</b></td>
    <td><b>57.14</b></td>
    <td>34.64</td>
    <td>0.514</td>
    <td>0.397</td>
  </tr>
  <tr>
    <td><b>P-RMF</b></td>
    <td><b>73.64</b></td>
    <td>74.65</td>
    <td>54.75</td>
    <td><b>34.83</b></td>
    <td><b>0.500</b></td>
    <td><b>0.414</b></td>
  </tr>
</table>

Table 2: Robustness comparison of overall performance on SIMS under intra-modal missingness.

4.2 Implementation Details

Evaluation Metrics. MOSI and MOSEI: Acc-2, Acc-5, Acc-7, F1 scores, MAE, and Corr. SIMS: Acc-2, Acc-3, Acc-5, F1 scores, MAE, and Corr. Appendix B provides detailed of the metrics.

Experimental Setup: Detailed Experimental Setup and parameters are provided in Appendix B.

4.3 Baselines

We conduct a fair comparison with several advanced and state-of-the-art methods, including complete-modality methods: MISA (Hazarika et al., 2020), Self-MM (Yu et al., 2021), MMIM (Han et al., 2021), CENET (Wang et al., 2022), TETFN (Wang et al., 2023a), ALMT (Zhang et al., 2023), CubeMLP (Sun et al., 2022), and DMD (Li et al., 2023b); as well as missing-modality methods: TFR-Net (Yuan et al., 2021), LNLN (Zhang et al., 2024), MCTN (Pham et al., 2019), TransM (Wang et al., 2020), SMIL (Ma et al., 2021), GCNet (Lian et al., 2023), and CorrKD (Li et al., 2024b).

4.4 Robustness Comparison

We evaluate the robustness and effectiveness of P-RMF under two scenarios: Intra-modal missingness and Inter-modal missingness.

Robustness Comparison for Intra-modal Missingness. Following previous work (Zhang et al., 2024), we set the missing rate to predefined values ranging from 0 to 0.9, with an increment of 0.1, to simulate the test conditions of intra-modal random missingness. For each method, we calculate the average results at different missing rates, reflecting the model’s overall performance under varying noise levels. More detailed test results are provided in Appendix D.

Tables 1 and 2 show the average robustness evaluation results of different models on the MOSI, MOSEI, and SIMS datasets under varying intra-modal missingness. As shown in these tables, P-RMF significantly outperforms existing models in noise resistance and achieves SOTA or competitive results across multiple evaluation metrics.

As shown in Table 1, compared to the SOTA missing-modality method LNLN, P-RMF improves all metrics on the MOSI and MOSEI datasets by an average of 0.36% and 2.46%, respectively. This indicates that P-RMF, with its proxy-dominated joint multimodal representation, outperforms the LNLN text-dominated approach in noise resistance. However, P-RMF achieves only suboptimal performance on metrics such as Acc-7, possibly due to an imbalance in data distribution. For example, in the MOSEI dataset, 67.85% of sentiment values fall within the range of -1 to 1, with imbalances further exacerbated in noisy scenarios. Since P-RMF focuses more on learning consistent features, the imbalanced distribution increases its difficulty in learning fine-grained tasks when data is missing.

As shown in Table 2, P-RMF achieves significant improvements in several metrics on the SIMS dataset. Compared with the suboptimal results of LNLN, P-RMF improves 1.25% and 4.28% on Acc-2 and Corr, respectively, which verifies its robustness under different noise scenarios. However, the
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Model</th>
    <th colspan="6">Testing Condition</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>{t}</th>
    <th>{a}</th>
    <th>{v}</th>
    <th>{t, a}</th>
    <th>{t, v}</th>
    <th>{a, v}</th>
    <th>{t, a, v}</th>
  </tr>
  <tr>
    <td rowspan="7">MOSI</td>
    <td>Self-MM</td>
    <td>67.80</td>
    <td>40.95</td>
    <td>38.52</td>
    <td>69.81</td>
    <td>74.97</td>
    <td>47.12</td>
    <td><b>84.64</b></td>
    <td>60.54</td>
  </tr>
  <tr>
    <td>CubeMLP</td>
    <td>64.15</td>
    <td>38.91</td>
    <td>43.24</td>
    <td>63.76</td>
    <td>65.12</td>
    <td>47.92</td>
    <td>84.57</td>
    <td>58.24</td>
  </tr>
  <tr>
    <td>DMD</td>
    <td>68.97</td>
    <td>43.33</td>
    <td>42.26</td>
    <td>70.51</td>
    <td>68.45</td>
    <td>50.47</td>
    <td>84.50</td>
    <td>61.21</td>
  </tr>
  <tr>
    <td>MCTN</td>
    <td>75.21</td>
    <td>59.25</td>
    <td>58.57</td>
    <td>77.81</td>
    <td>74.82</td>
    <td>64.21</td>
    <td>80.12</td>
    <td>70.00</td>
  </tr>
  <tr>
    <td>TransM</td>
    <td>77.64</td>
    <td>63.57</td>
    <td>56.48</td>
    <td>82.07</td>
    <td>80.90</td>
    <td>67.24</td>
    <td>82.57</td>
    <td>72.92</td>
  </tr>
  <tr>
    <td>SMIL</td>
    <td>78.26</td>
    <td>67.69</td>
    <td>59.67</td>
    <td>79.82</td>
    <td>79.15</td>
    <td>71.24</td>
    <td>82.85</td>
    <td>74.10</td>
  </tr>
  <tr>
    <td>GCNet</td>
    <td>80.91</td>
    <td>65.07</td>
    <td>59.67</td>
    <td><b>84.73</b></td>
    <td><b>83.58</b></td>
    <td>70.02</td>
    <td>83.20</td>
    <td>75.31</td>
  </tr>
  <tr>
    <td>CorrKD</td>
    <td>81.20</td>
    <td>66.52</td>
    <td>60.72</td>
    <td>83.56</td>
    <td>82.41</td>
    <td><b>73.74</b></td>
    <td>83.94</td>
    <td>76.01</td>
  </tr>
  <tr>
    <td colspan="2"><b>P-RMF</b></td>
    <td><b>81.36</b></td>
    <td><b>71.44</b></td>
    <td><b>70.32</b></td>
    <td>82.10</td>
    <td>81.94</td>
    <td>73.11</td>
    <td>84.37</td>
    <td><b>77.81</b></td>
  </tr>
  <tr>
    <td rowspan="7">MOSEI</td>
    <td>Self-MM</td>
    <td>71.53</td>
    <td>43.57</td>
    <td>37.61</td>
    <td>75.91</td>
    <td>74.62</td>
    <td>49.52</td>
    <td>83.69</td>
    <td>62.35</td>
  </tr>
  <tr>
    <td>CubeMLP</td>
    <td>67.52</td>
    <td>39.54</td>
    <td>32.58</td>
    <td>71.69</td>
    <td>70.06</td>
    <td>48.54</td>
    <td>83.17</td>
    <td>59.01</td>
  </tr>
  <tr>
    <td>DMD</td>
    <td>70.26</td>
    <td>46.18</td>
    <td>39.84</td>
    <td>74.78</td>
    <td>72.45</td>
    <td>52.70</td>
    <td>84.78</td>
    <td>63.00</td>
  </tr>
  <tr>
    <td>MCTN</td>
    <td>75.50</td>
    <td>62.72</td>
    <td>59.46</td>
    <td>76.64</td>
    <td>77.13</td>
    <td>64.84</td>
    <td>81.75</td>
    <td>71.15</td>
  </tr>
  <tr>
    <td>TransM</td>
    <td>77.98</td>
    <td>63.68</td>
    <td>58.67</td>
    <td>80.46</td>
    <td>78.61</td>
    <td>62.24</td>
    <td>81.48</td>
    <td>71.87</td>
  </tr>
  <tr>
    <td>SMIL</td>
    <td>76.57</td>
    <td>65.96</td>
    <td>60.57</td>
    <td>77.68</td>
    <td>76.24</td>
    <td>66.87</td>
    <td>80.74</td>
    <td>72.09</td>
  </tr>
  <tr>
    <td>GCNet</td>
    <td>80.52</td>
    <td>66.54</td>
    <td>61.83</td>
    <td>81.96</td>
    <td>81.15</td>
    <td>69.21</td>
    <td>82.35</td>
    <td>74.79</td>
  </tr>
  <tr>
    <td>CorrKD</td>
    <td>80.76</td>
    <td>66.09</td>
    <td>62.30</td>
    <td>81.74</td>
    <td>81.28</td>
    <td>71.92</td>
    <td>82.16</td>
    <td>75.18</td>
  </tr>
  <tr>
    <td colspan="2"><b>P-RMF</b></td>
    <td><b>81.91</b></td>
    <td><b>75.91</b></td>
    <td><b>73.19</b></td>
    <td><b>84.61</b></td>
    <td><b>85.17</b></td>
    <td><b>76.88</b></td>
    <td><b>85.48</b></td>
    <td><b>80.45</b></td>
  </tr>
</table>

Table 3: Performance comparison under varying inter-modal missingness conditions on MOSI and MOSEI. For example, the symbol "{t}" indicates that only the text modality is available.

F1 performance of P-RMF is inferior to that of LNLN. As can be seen from Fig. 5(c) and (f), the F1 performance of LNLN improves as the missing rate increases while the MAE performance continues to decrease. This suggests that the LNLN is biased in the face of missing data and the unbalanced data makes the model tend to perform lazy behavior in high-noise scenarios, i.e., predicting a higher proportion of categories in the training set (Zhang et al., 2024). This shows that the model struggles to learn effective predictive knowledge and instead relies on lazy to maintain accuracy.

Fig. 5 shows the performance curves of several advanced methods under different missing rates, providing an intuitive reflection of model robustness. It can be observed that as the missing rate increases, the performance of all models declines. Model structures optimized for modal missingness (e.g., TFR-Net) show a significant drop in robustness under varying levels of random data missingness. However, we proposed P-RMF consistently outperforms other models in most cases, demonstrating exceptional robustness.

Robustness Comparison for Inter-modal Missingness. Following previous work (Li et al., 2024b), we removed entire modalities from samples in the MOSI and MOSEI datasets to simulate inter-modal missingness test conditions. We evaluate model performance in different inter-modal missingness scenarios using the F1 score. Results for other metrics can be found in Appendix E.

The following key conclusions can be drawn from Table 3: (i) Inter-modal missingness leads to performance degradation across all models, indicating that complementary information from hetero-

![Performance curves of various missing rates. (a), (b) and (c) is the F1 curves on MOSI, MOSEI and SIMS. (d), (e) and (f) is the MAE curves on MOSI, MOSEI and SIMS.](page_1012_1342_410_246.png)

Figure 5: Performance curves of various missing rates. (a), (b) and (c) is the F1 curves on MOSI, MOSEI and SIMS. (d), (e) and (f) is the MAE curves on MOSI, MOSEI and SIMS.
<table>
  <tr>
    <th>Method</th>
    <th>Acc-2</th>
    <th>F1</th>
    <th>Acc-5</th>
    <th>Acc-7</th>
    <th>MAE</th>
    <th>Corr</th>
  </tr>
  <tr>
    <td>P-RMF</td>
    <td><b>71.53 / 72.81</b></td>
    <td>71.69 / 72.93</td>
    <td><b>38.50</b></td>
    <td><b>34.19</b></td>
    <td><b>1.038</b></td>
    <td>0.525</td>
  </tr>
  <tr>
    <td>w/o PMG</td>
    <td>70.38 / 71.79</td>
    <td>70.54 / 71.82</td>
    <td>37.35</td>
    <td>33.53</td>
    <td>1.055</td>
    <td>0.517</td>
  </tr>
  <tr>
    <td>w/o PDDI</td>
    <td>71.24 / 72.15</td>
    <td>71.10 / 72.34</td>
    <td>34.49</td>
    <td>30.95</td>
    <td>1.107</td>
    <td>0.476</td>
  </tr>
  <tr>
    <td>w/o DR</td>
    <td>71.51 / 72.07</td>
    <td><b>71.76 / 72.14</b></td>
    <td>35.85</td>
    <td>32.10</td>
    <td>1.066</td>
    <td><b>0.526</b></td>
  </tr>
</table>

Table 4: Ablation experiments on MOSI.

geneous modalities enhances sentiment semantics in joint representation; (ii) The performance of the text modality outperforms other modalities, suggesting that the text modality contains more rich knowledge information; (iii) Our proposed P-RMF exhibits significantly better noise resistance under different inter-modal missingness conditions compared to other models. Compared with the suboptimal model CorrKD, P-RMF improves the average performance on the MOSI and MOSEI datasets by 2.37% and 7.01%, respectively; (iv) Existing training models for missing data perform well in robustness tests for missing modalities, while complete modality training models perform better in complete modality tests. Our method, P-RMF, demonstrates optimal robustness in modality missing scenarios and high performance in complete modality scenarios. For example, in the complete modal test on MOSEI, P-RMF achieves an F1 score of 85.48, improving by 0.83% over the second-best method, DMD (a complete modality method).

4.5 Ablation Study

To investigate the contribution of the core modules of P-RMF, we conducted ablation experiments on MOSI, calculating the average test results across different missing rates.

As shown in Table 4, removing different modules of P-RMF leads to a decrease in overall performance. Specifically, when the PMG module is removed, binary classification tasks (such as Acc-2 and F1) show a significant decline, while multi-class tasks (such as Acc-5 and Acc-7) remain relatively stable. In contrast, removing the PDDI module has the opposite effect compared to removing PMG. This indicates that the PMG module helps to learn stable and consistent multimodal representation, while the introduction of PDDI enhances the diversity of the joint representation by injecting unimodal feature information into the proxy joint representation generated by PMG. Thus, PDDI and PMG complement each other to guarantee stability across different tasks. In addition, Fig. 6 shows the experimental results of P-RMF ablation on the MOSI dataset with different missing rates.

As observed in Fig. 6, even at lower missing rates, removing the PMG module leads to a significant drop in F1 performance, indicating that the PMG module contributes to the model’s stability. When the PDDI module is removed, F1 performance remains relatively stable, but the MAE value is higher, suggesting that the absence of PDDI increases prediction errors.

![Ablation results of different various missing ratios on MOSI.](page_1012_1012_388_246.png)

Figure 6: Ablation results of different various missing ratios on MOSI.

5 Conclusion

This paper proposes a data-centric robust multimodal sentiment analysis method, Proxy-Driven Robust Multimodal Fusion (P-RMF). P-RMF shows outstanding robustness in both uncertain missing and complete multimodal data scenarios. Specifically, P-RMF learns stable modality representation from the latent space of the data’s Gaussian distribution and quantifies intrinsic modality uncertainty to obtain stable and robust multimodal joint representation (i.e., proxy modality). In the proxy-driven cross-modal injection framework, modality-specific features with varying weights is iteratively injected into the proxy modality, enhancing its diversity representation. Comprehensive experiments demonstrate that P-RMF outperforms existing models in noise resistance and robustness. P-RMF considers both the consistency of data distribution and intrinsic uncertainty, offering a novel method for robust MSA with incomplete data.
Limitations

Although P-RMF achieves performance improvements in exploring robust multimodal representation of arbitrarily uncertain missing data, there are still some limitations that need to be addressed or clarified in future work. Firstly, due to limitations in resources and experimental conditions, we did not replicate all baseline methods but instead referred to the detailed test results from (Zhang et al., 2024) and (Li et al., 2024b) in various missing data scenarios. To ensure fairness in the experiments, we used the same datasets and experimental design as in (Zhang et al., 2024) and (Li et al., 2024b), with the complete experimental setup and detailed results provided in Appendices B, D, and E. Secondly, while this paper simulates both inter-modal and intra-modal missing scenarios, real-world environments may involve more complex joint missingness, which imposes higher demands on model performance and offers opportunities for future optimization. Finally, although P-RMF achieves significant improvement in overall robustness, P-RMF does not always outperform other methods in some metrics due to the stochastic nature of the missing data noise. It is also worth investigating how to balance the performance of the model under different noise levels in the future.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 62176084, and Grant 62176083, and in part by the National Key Research and Development Program of China under Grant 2023YFC3604704.

References

AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236–2246, Melbourne, Australia. Association for Computational Linguistics.

Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. Openface 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 59–66.

Brian McFee, Colin Raffel, Dawen Liang, Daniel P.W. Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. 2015. librosa: Audio and Music Signal Analysis in Python. In Proceedings of the 14th Python in Science Conference, pages 18 – 24.

Yixuan Chen, Dongsheng Li, Peng Zhang, Jie Sui, Qin Lv, Lu Tun, and Li Shang. 2022. Cross-modal ambiguity learning for multimodal fake news detection. In Proceedings of the ACM web conference 2022, pages 2897–2905.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Wei Han, Hui Chen, and Soujanya Poria. 2021. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9180–9192, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM international conference on multimedia, pages 1122–1131.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc.

Alex Kendall and Yarin Gal. 2017. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30.

Donggeun Kim and Taesup Kim. 2024. Missing modality prediction for unpaired multimodal learning via joint embedding of unimodal models. In European Conference on Computer Vision, pages 171–187. Springer.

Mingcheng Li, Dingkang Yang, Yuxuan Lei, Shunli Wang, Shuaibing Wang, Liuzhen Su, Kun Yang, Yuzheng Wang, Mingyang Sun, and Lihua Zhang. 2024a. A unified self-distillation framework for multimodal sentiment analysis with uncertain missing modalities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10074–10082.

Mingcheng Li, Dingkang Yang, and Lihua Zhang. 2023a. Towards robust multimodal sentiment analysis under uncertain signal missing. IEEE Signal Processing Letters.
Mingcheng Li, Dingkang Yang, Xiao Zhao, Shuaibing Wang, Yan Wang, Kun Yang, Mingyang Sun, Dongliang Kou, Ziyun Qian, and Lihua Zhang. 2024b. Correlation-decoupled knowledge distillation for multimodal sentiment analysis with incomplete modalities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12458–12468.

Yong Li, Yuanzhi Wang, and Zhen Cui. 2023b. Decoupled multimodal distilling for emotion recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6631–6640.

Zheng Lian, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. 2023. Gcnet: Graph completion network for incomplete multimodal learning in conversation. IEEE Transactions on pattern analysis and machine intelligence, 45:8419–8432.

Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. 2021. Smil: Multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2302–2310.

Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póczos. 2019. Found in translation: Learning robust joint representations by cyclic translations between modalities. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 6892–6899.

Fan Qian, Jiqing Han, Yongjun He, Tieran Zheng, and Guibin Zheng. 2023. Sentiment knowledge enhanced self-supervised learning for multimodal sentiment analysis. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12966–12978, Toronto, Canada. Association for Computational Linguistics.

Upendra Singh, Kumar Abhishek, and Hiteshwar Kumar Azad. 2024. A survey of cutting-edge multimodal sentiment analysis. ACM Computing Surveys, 56:1–38.

Hao Sun, Hongyi Wang, Jiaqing Liu, Yen-Wei Chen, and Lanfen Lin. 2022. Cubemlp: An mlp-based model for multimodal sentiment analysis and depression estimation. In Proceedings of the 30th ACM international conference on multimedia, pages 3722–3729.

Jun Sun, Shoukang Han, Yu-Ping Ruan, Xiaoning Zhang, Shu-Kai Zheng, Yulong Liu, Yuxin Huang, and Taihao Li. 2023a. Layer-wise fusion with modality independence modeling for multi-modal emotion recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 658–670, Toronto, Canada. Association for Computational Linguistics.

Licai Sun, Zheng Lian, Bin Liu, and Jianhua Tao. 2023b. Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis. IEEE Transactions on Affective Computing, 15:309–325.

Di Wang, Xutong Guo, Yumin Tian, Jinhui Liu, LiHuo He, and Xuemei Luo. 2023a. Tetfn: A text enhanced transformer fusion network for multimodal sentiment analysis. Pattern Recognition, 136:109259.

Di Wang, Shuai Liu, Quan Wang, Yumin Tian, Lihuo He, and Xinbo Gao. 2022. Cross-modal enhancement network for multimodal sentiment analysis. IEEE Transactions on Multimedia, 25:4909–4921.

Yuanzhi Wang, Zhen Cui, and Yong Li. 2023b. Distribution-consistent modal recovering for incomplete multimodal learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22025–22034.

Yuanzhi Wang, Zhen Cui, and Yong Li. 2023c. Distribution-consistent modal recovering for incomplete multimodal learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22025–22034.

Zilong Wang, Zhaohong Wan, and Xiaojun Wan. 2020. Transmodality: An end2end fusion method with transformer for multimodal sentiment analysis. In Proceedings of the web conference 2020, pages 2514–2520.

Peng Xu, Xiatian Zhu, and David A Clifton. 2023. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45:12113–12132.

Chuanpeng Yang, Fuqing Zhu, Yaxin Liu, Jizhong Han, and Songlin Hu. 2024. Uncertainty-aware cross-modal alignment for hate speech detection. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16973–16983, Torino, Italia. ELRA and ICCL.

Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. 2020. CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3718–3727, Online. Association for Computational Linguistics.

Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. 2021. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 10790–10797.

Yakun Yu, Mingjun Zhao, Shi-ang Qi, Feiran Sun, Baoxun Wang, Weidong Guo, Xiaoli Wang, Lei Yang, and Di Niu. 2023. ConKI: Contrastive knowledge injection for multimodal sentiment analysis. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13610–13624, Toronto, Canada. Association for Computational Linguistics.
Ziqi Yuan, Wei Li, Hua Xu, and Wenmeng Yu. 2021. Transformer-based feature reconstruction network for robust multimodal sentiment analysis. In Proceedings of the 29th ACM International Conference on Multimedia, pages 4400–4407.

Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259.

Jiandan Zeng, Tianyi Liu, and Jiantao Zhou. 2022. Tag-assisted multimodal sentiment analysis under uncertain missing modalities. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1545–1554.

Haoyu Zhang, Wenbin Wang, and Tianshu Yu. 2024. Towards robust multimodal sentiment analysis with incomplete data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Haoyu Zhang, Yu Wang, Guanghao Yin, Kejun Liu, Yuanyuan Liu, and Tianshu Yu. 2023. Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 756–767.

Jinming Zhao, Ruichen Li, and Qin Jin. 2021. Missing modality imagination network for emotion recognition with uncertain missing modalities. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2608–2618, Online. Association for Computational Linguistics.

Aoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang, Yiming Tang, and Fuji Ren. 2024. Kehr: Knowledge enhanced self-supervised balanced representation for multimodal sentiment analysis. In Proceedings of the 32nd ACM International Conference on Multimedia, MM ’24, page 5732–5741.
A Datasets

MOSI. The MOSI dataset consists of speakers expressing their opinions on topics such as movies across 93 YouTube videos, containing 2,199 video clips with sentiment annotations. The sentiment intensity of each clip is labeled on a scale from -3 (strongly negative) to +3 (strongly positive).

MOSEI. The MOSEI dataset is an extended version of MOSI, covering 250 different topics and containing 22,856 annotated video clips. The sentiment annotation method is the same as that of MOSI.

SIMS. The SIMS dataset consists of 2,281 video clips. Each sample has one multimodal sentiment label and three unimodal sentiment labels, with sentiment scores ranging from -1 (negative) to +1 (positive).

The statistics of these datasets are summarized in Table 5.

B Implementation Details

Evaluation Metrics: For the MOSI and MOSEI datasets, we report binary classification accuracy (Acc-2), five-class classification accuracy (Acc-5), seven-class classification accuracy (Acc-7), as well as F1 scores and mean absolute error (MAE) associated with Acc-2. The Acc-2 and F1 scores are reported in two forms, using the split marker "-/-": the first score represents negative/non-negative (including 0), and the second score represents negative/positive. For the SIMS dataset, we report Acc-2, Acc-3, Acc-5, F1 scores, MAE, and correlation (Corr). Acc-2, Acc-3, Acc-5, and Acc-7 represent the percentage of correct predictions within their respective sentiment intervals. For example, Acc-7 indicates the accuracy across seven sentiment intervals from -3 to +3. Except for MAE, higher values for all metrics indicate better model performance.

Experimental Setup: All models are trained using the PyTorch framework on an NVIDIA RTX A40 with 60GB of memory. For each dataset, training is performed with different random seeds (1111, 1112, and 1113), and results are averaged. The Adam optimizer is used, with BERT as the backbone. The learning rate is set to 1e-4, the batch size to 32, and training lasts for 100 epochs. During each epoch, 50% of the training samples are randomly selected, and for each modality, a random percentage of information (from 0% to 100%) is erased to simulate data missingness. Detailed parameters are provided in Table 6.

C Efficiency Analysis

Table 7 shows the computational overhead of the proposed P-RMF model on the MOSI dataset (Zadeh et al., 2016), and compares it with the state-of-the-art model LNLN (Zhang et al., 2024) for incomplete modality under the same experimental setup. Compared to LNLN, P-RMF achieves a shorter runtime per epoch while maintaining a similar number of parameters, indicating that its model structure is more optimized and computationally efficient.

D Robustness Evaluation of P-RMF under Intra-modal Missingness

Following previous work (Zhang et al., 2024), we set the missing rate to predefined values ranging from 0 to 0.9, with an increment of 0.1, to simulate the test conditions of intra-modal random missingness. Like previous work(Zhang et al., 2024), we did not evaluate at \( r = 1.0 \), as this would imply complete data erasure from each modality, rendering the experiment non-informative.

Tables 8, 9, and 10 present the detailed results of the robustness tests for our model, P-RMF, on the MOSI, MOSEI, and SIMS datasets under various data missing rates. It can be observed that the model performance decreases continuously with the increase of missing rate. This is primarily due to the increase in missing modality rates, which not only reduces the available information for the model but also introduces indirect issues such as distribution shifts, weakened feature correlations, and amplified noise. The combined impact of these factors leads to a continuous decline in model performance, making it challenging for the model to maintain stable, state-of-the-art results.

E Robustness Evaluation of P-RMF under Inter-modal Missingness

Following previous work (Li et al., 2024b), we removed entire modalities from samples in the MOSI and MOSEI datasets to simulate inter-modal missingness test conditions.

Tables 11, 12, and 13 present the robustness evaluation results of our model, P-RMF, on the MOSI, MOSEI, and SIMS datasets under intermodal missing conditions. It can be observed that model performance generally improves with the increase in modality information. The performance of the text modality typically surpasses that of other unimodal, indicating that the text modality contains
<table>
  <tr>
    <th>Dataset</th>
    <th>Speaker</th>
    <th>Video clip</th>
    <th>Train</th>
    <th>Valid</th>
    <th>Test</th>
    <th>Language</th>
  </tr>
  <tr>
    <td>MOSI</td>
    <td>93</td>
    <td>2199</td>
    <td>1284</td>
    <td>229</td>
    <td>686</td>
    <td>English</td>
  </tr>
  <tr>
    <td>MOSEI</td>
    <td>1000</td>
    <td>22856</td>
    <td>16326</td>
    <td>1871</td>
    <td>4659</td>
    <td>English</td>
  </tr>
  <tr>
    <td>SIMS</td>
    <td>474</td>
    <td>2281</td>
    <td>1368</td>
    <td>456</td>
    <td>457</td>
    <td>Chinese</td>
  </tr>
</table>

Table 5: Dataset statistics.

<table>
  <tr>
    <th>Hyper-parameter</th>
    <th>MOSI</th>
    <th>MOSEI</th>
    <th>SIMS</th>
  </tr>
  <tr>
    <td>\( d_t \)</td>
    <td>768</td>
    <td>768</td>
    <td>768</td>
  </tr>
  <tr>
    <td>\( d_a \)</td>
    <td>5</td>
    <td>74</td>
    <td>33</td>
  </tr>
  <tr>
    <td>\( d_v \)</td>
    <td>20</td>
    <td>35</td>
    <td>709</td>
  </tr>
  <tr>
    <td>\( L \)</td>
    <td>4</td>
    <td>4</td>
    <td>4</td>
  </tr>
  <tr>
    <td>\( \lambda_1, \lambda_2, \lambda_3 \)</td>
    <td>1,0,5,0,1</td>
    <td>1,0,5,0,1</td>
    <td>1,0,5,0,1</td>
  </tr>
  <tr>
    <td>Batch size</td>
    <td>32</td>
    <td>32</td>
    <td>32</td>
  </tr>
  <tr>
    <td>Epoch</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>Adam</td>
    <td>Adam</td>
    <td>Adam</td>
  </tr>
  <tr>
    <td>Vector Length T</td>
    <td>8</td>
    <td>8</td>
    <td>39</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>1e-4</td>
    <td>1e-4</td>
    <td>1e-4</td>
  </tr>
  <tr>
    <td>Fully connected layer</td>
    <td>128</td>
    <td>128</td>
    <td>128</td>
  </tr>
</table>

Table 6: Hyper-parameters setting.

<table>
  <tr>
    <th>Model</th>
    <th>Parameters</th>
    <th>Time / Epoch</th>
  </tr>
  <tr>
    <td>P-RMF</td>
    <td>117 M</td>
    <td>18 s</td>
  </tr>
  <tr>
    <td>LNLN</td>
    <td>116 M</td>
    <td>24 s</td>
  </tr>
</table>

Table 7: Computational overhead.

richer knowledge information. It is worth noting that in all test scenarios across the datasets, even though the Acc-2 or F1 scores improve, the model’s other performance metrics significantly drop whenever the text modality is missing. For example, in Table 13, under testing conditions such as {a}, {v}, or {a,v}, although the F1 score is higher, metrics like Acc-5, Acc-7, MAE, and Corr see a notable decline. This indicates that missing the text modality causes the model to lose crucial semantic information, while the sentiment cues provided by the audio and visual modalities are relatively weak and cannot compensate for the loss of the text modality. Therefore, even with a higher F1 score in binary classification tasks, due to the loss of key features, the model’s performance in more complex multi-classification and regression tasks still significantly declines.

F Independent ablation on MOSI.

Independent ablation of the three loss functions in the Proxy Modality Generation (PMG) module, As shown in Table 14. SA, CMA, and SR represent Standardized Alignment, Cross-Modal Alignment, and Sample Reconstruction within the PMG module.

G Hyperparameter sensitivity analysis.

The model has three \( \lambda_1, \lambda_2, \lambda_3 \) weight hyperparameters. Preliminary tests showed that it was difficult to find a set of hyperparameters that performed optimally across all datasets. Based on prior experience [1], we set the main task weight \( \lambda_1 \) to 1 and the reconstruction loss weight \( \lambda_3 \) to 0.1. We mainly analyzed the impact of different weights \( \lambda_2 \) for constraint losses in Proxy Modality Generation. The detailed analysis is shown in Table 15.
<table>
  <tr>
    <th rowspan="2">Missing Rate \( r \)</th>
    <th colspan="6">MOSI</th>
  </tr>
  <tr>
    <th>Acc-2</th>
    <th>F1</th>
    <th>Acc-5</th>
    <th>Acc-7</th>
    <th>MAE</th>
    <th>Corr</th>
  </tr>
  <tr><td>0.0</td><td>82.65 / 84.15</td><td>82.69 / 84.37</td><td>48.83</td><td>44.31</td><td>0.726</td><td>0.782</td></tr>
  <tr><td>0.1</td><td>81.34 / 82.62</td><td>81.35 / 82.89</td><td>47.52</td><td>42.13</td><td>0.800</td><td>0.730</td></tr>
  <tr><td>0.2</td><td>78.13 / 79.57</td><td>78.11 / 80.97</td><td>44.75</td><td>40.38</td><td>0.853</td><td>0.668</td></tr>
  <tr><td>0.3</td><td>75.80 / 76.83</td><td>75.82 / 79.27</td><td>42.71</td><td>39.21</td><td>0.922</td><td>0.621</td></tr>
  <tr><td>0.4</td><td>73.76 / 75.46</td><td>74.09 / 77.71</td><td>40.67</td><td>35.59</td><td>1.001</td><td>0.584</td></tr>
  <tr><td>0.5</td><td>71.28 / 73.02</td><td>71.66 / 73.33</td><td>37.90</td><td>33.67</td><td>1.077</td><td>0.523</td></tr>
  <tr><td>0.6</td><td>67.35 / 68.75</td><td>67.64 / 68.64</td><td>33.24</td><td>29.30</td><td>1.147</td><td>0.432</td></tr>
  <tr><td>0.7</td><td>65.16 / 66.16</td><td>65.33 / 64.69</td><td>32.94</td><td>27.84</td><td>1.229</td><td>0.383</td></tr>
  <tr><td>0.8</td><td>61.08 / 62.04</td><td>61.22 / 60.76</td><td>29.74</td><td>25.97</td><td>1.275</td><td>0.316</td></tr>
  <tr><td>0.9</td><td>58.75 / 59.45</td><td>59.01 / 56.66</td><td>26.68</td><td>23.49</td><td>1.346</td><td>0.212</td></tr>
  <tr><th>Avg.</th><th>71.53 / 72.81</th><th>71.69 / 72.93</th><th>38.50</th><th>34.19</th><th>1.038</th><th>0.525</th></tr>
</table>

Table 8: Robustness evaluation results of P-RMF on MOSI under various rates of intra-modal missing data.

<table>
  <tr>
    <th rowspan="2">Missing Rate \( r \)</th>
    <th colspan="6">MOSEI</th>
  </tr>
  <tr>
    <th>Acc-2</th>
    <th>F1</th>
    <th>Acc-5</th>
    <th>Acc-7</th>
    <th>MAE</th>
    <th>Corr</th>
  </tr>
  <tr><td>0.0</td><td>83.62 / 85.20</td><td>83.68 / 85.48</td><td>52.09</td><td>49.77</td><td>0.539</td><td>0.767</td></tr>
  <tr><td>0.1</td><td>82.79 / 83.98</td><td>82.94 / 84.37</td><td>51.45</td><td>49.04</td><td>0.556</td><td>0.748</td></tr>
  <tr><td>0.2</td><td>82.25 / 82.97</td><td>82.58 / 83.37</td><td>49.35</td><td>47.91</td><td>0.576</td><td>0.722</td></tr>
  <tr><td>0.3</td><td>80.88 / 81.26</td><td>81.40 / 81.86</td><td>47.78</td><td>45.95</td><td>0.611</td><td>0.683</td></tr>
  <tr><td>0.4</td><td>79.76 / 79.97</td><td>80.58 / 80.74</td><td>46.73</td><td>45.59</td><td>0.631</td><td>0.653</td></tr>
  <tr><td>0.5</td><td>78.64 / 78.62</td><td>79.74 / 79.49</td><td>44.90</td><td>43.94</td><td>0.666</td><td>0.601</td></tr>
  <tr><td>0.6</td><td>77.44 / 76.11</td><td>78.97 / 77.58</td><td>43.12</td><td>42.26</td><td>0.703</td><td>0.545</td></tr>
  <tr><td>0.7</td><td>75.87 / 74.64</td><td>78.12 / 75.88</td><td>42.41</td><td>41.73</td><td>0.733</td><td>0.481</td></tr>
  <tr><td>0.8</td><td>74.46 / 70.75</td><td>77.91 / 73.03</td><td>41.00</td><td>40.46</td><td>0.764</td><td>0.401</td></tr>
  <tr><td>0.9</td><td>72.59 / 67.86</td><td>77.95 / 71.51</td><td>39.90</td><td>39.62</td><td>0.805</td><td>0.289</td></tr>
  <tr><th>Avg.</th><th>78.83 / 78.14</th><th>80.39 / 79.33</th><th>45.87</th><th>44.63</th><th>0.658</th><th>0.589</th></tr>
</table>

Table 9: Robustness evaluation results of P-RMF on MOSEI under various rates of intra-modal missing data.

<table>
  <tr>
    <th rowspan="2">Missing Rate \( r \)</th>
    <th colspan="6">SIMS</th>
  </tr>
  <tr>
    <th>Acc-2</th>
    <th>F1</th>
    <th>Acc-3</th>
    <th>Acc-5</th>
    <th>MAE</th>
    <th>Corr</th>
  </tr>
  <tr><td>0.0</td><td>78.34</td><td>79.69</td><td>60.61</td><td>38.95</td><td>0.441</td><td>0.550</td></tr>
  <tr><td>0.1</td><td>77.24</td><td>78.88</td><td>59.52</td><td>37.72</td><td>0.454</td><td>0.530</td></tr>
  <tr><td>0.2</td><td>76.23</td><td>77.46</td><td>59.30</td><td>37.05</td><td>0.460</td><td>0.512</td></tr>
  <tr><td>0.3</td><td>75.66</td><td>75.89</td><td>56.89</td><td>36.89</td><td>0.475</td><td>0.483</td></tr>
  <tr><td>0.4</td><td>75.32</td><td>76.30</td><td>55.8</td><td>36.54</td><td>0.482</td><td>0.472</td></tr>
  <tr><td>0.5</td><td>73.61</td><td>74.83</td><td>54.83</td><td>34.79</td><td>0.495</td><td>0.404</td></tr>
  <tr><td>0.6</td><td>72.12</td><td>73.32</td><td>53.05</td><td>33.92</td><td>0.515</td><td>0.383</td></tr>
  <tr><td>0.7</td><td>71.58</td><td>72.22</td><td>52.52</td><td>32.23</td><td>0.531</td><td>0.369</td></tr>
  <tr><td>0.8</td><td>69.51</td><td>70.35</td><td>49.89</td><td>31.07</td><td>0.568</td><td>0.269</td></tr>
  <tr><td>0.9</td><td>66.77</td><td>67.54</td><td>45.08</td><td>29.10</td><td>0.581</td><td>0.168</td></tr>
  <tr><th>Avg.</th><th>73.64</th><th>74.65</th><th>54.75</th><th>34.83</th><th>0.500</th><th>0.414</th></tr>
</table>

Table 10: Robustness evaluation results of P-RMF on SIMS under various rates of intra-modal missing data.
<table>
  <tr>
    <th rowspan="2">Testing Condition</th>
    <th colspan="7">MOSI</th>
  </tr>
  <tr>
    <th>Acc-2</th>
    <th>F1</th>
    <th>Acc-5</th>
    <th>Acc-7</th>
    <th>MAE</th>
    <th>Corr</th>
  </tr>
  <tr>
    <td>{l}</td>
    <td>80.90 / 81.01</td>
    <td>80.89 / 81.36</td>
    <td>47.46</td>
    <td>42.57</td>
    <td>0.777</td>
    <td>0.759</td>
  </tr>
  <tr>
    <td>{a}</td>
    <td>55.03 / 56.85</td>
    <td>71.72 / 71.44</td>
    <td>20.99</td>
    <td>20.55</td>
    <td>1.367</td>
    <td>0.107</td>
  </tr>
  <tr>
    <td>{v}</td>
    <td>54.96 / 56.01</td>
    <td>70.72 / 70.32</td>
    <td>20.26</td>
    <td>19.83</td>
    <td>1.368</td>
    <td>0.099</td>
  </tr>
  <tr>
    <td>{l, a}</td>
    <td>81.05 / 82.16</td>
    <td>81.03 / 82.10</td>
    <td>48.69</td>
    <td>43.29</td>
    <td>0.776</td>
    <td>0.760</td>
  </tr>
  <tr>
    <td>{l, v}</td>
    <td>80.90 / 82.01</td>
    <td>80.89 / 81.94</td>
    <td>48.40</td>
    <td>43.15</td>
    <td>0.777</td>
    <td>0.759</td>
  </tr>
  <tr>
    <td>{a, v}</td>
    <td>55.95 / 58.42</td>
    <td>71.98 / 73.11</td>
    <td>20.85</td>
    <td>20.41</td>
    <td>1.366</td>
    <td>0.109</td>
  </tr>
  <tr>
    <td>{l, a, v}</td>
    <td>82.65 / 84.15</td>
    <td>82.69 / 84.37</td>
    <td>48.83</td>
    <td>44.31</td>
    <td>0.726</td>
    <td>0.782</td>
  </tr>
  <tr>
    <td>Avg.</td>
    <td>70.21 / 71.52</td>
    <td>77.13 / 77.81</td>
    <td>36.50</td>
    <td>33.44</td>
    <td>1.022</td>
    <td>0.482</td>
  </tr>
</table>

Table 11: Robustness evaluation results of P-RMF under inte-modality missingness on the MOSI dataset.

<table>
  <tr>
    <th rowspan="2">Testing Condition</th>
    <th colspan="7">MOSEI</th>
  </tr>
  <tr>
    <th>Acc-2</th>
    <th>F1</th>
    <th>Acc-5</th>
    <th>Acc-7</th>
    <th>MAE</th>
    <th>Corr</th>
  </tr>
  <tr>
    <td>{l}</td>
    <td>81.52 / 82.26</td>
    <td>82.11 / 81.91</td>
    <td>50.46</td>
    <td>49.09</td>
    <td>0.562</td>
    <td>0.757</td>
  </tr>
  <tr>
    <td>{a}</td>
    <td>71.02 / 62.85</td>
    <td>83.06 / 75.91</td>
    <td>41.74</td>
    <td>41.25</td>
    <td>0.838</td>
    <td>0.115</td>
  </tr>
  <tr>
    <td>{v}</td>
    <td>70.52 / 61.77</td>
    <td>82.14 / 73.19</td>
    <td>33.68</td>
    <td>33.59</td>
    <td>0.828</td>
    <td>0.209</td>
  </tr>
  <tr>
    <td>{l, a}</td>
    <td>83.58 / 84.62</td>
    <td>83.60 / 84.61</td>
    <td>50.42</td>
    <td>49.04</td>
    <td>0.565</td>
    <td>0.757</td>
  </tr>
  <tr>
    <td>{l, v}</td>
    <td>83.52 / 85.20</td>
    <td>83.49 / 85.17</td>
    <td>51.87</td>
    <td>49.41</td>
    <td>0.559</td>
    <td>0.765</td>
  </tr>
  <tr>
    <td>{a, v}</td>
    <td>70.68 / 63.34</td>
    <td>82.16 / 76.88</td>
    <td>37.63</td>
    <td>33.59</td>
    <td>0.822</td>
    <td>0.211</td>
  </tr>
  <tr>
    <td>{l, a, v}</td>
    <td>83.62 / 85.20</td>
    <td>83.68 / 85.48</td>
    <td>52.09</td>
    <td>49.77</td>
    <td>0.539</td>
    <td>0.767</td>
  </tr>
  <tr>
    <td>Avg.</td>
    <td>77.78 / 75.03</td>
    <td>82.89 / 80.45</td>
    <td>45.41</td>
    <td>43.68</td>
    <td>0.673</td>
    <td>0.512</td>
  </tr>
</table>

Table 12: Robustness evaluation results of P-RMF under inte-modal missingness on the MOSEI dataset.

<table>
  <tr>
    <th rowspan="2">Testing Condition</th>
    <th colspan="7">SIMS</th>
  </tr>
  <tr>
    <th>Acc-2</th>
    <th>F1</th>
    <th>Acc-3</th>
    <th>Acc-5</th>
    <th>MAE</th>
    <th>Corr</th>
  </tr>
  <tr>
    <td>{l}</td>
    <td>77.90</td>
    <td>79.37</td>
    <td>60.34</td>
    <td>38.29</td>
    <td>0.441</td>
    <td>0.550</td>
  </tr>
  <tr>
    <td>{a}</td>
    <td>69.37</td>
    <td>81.91</td>
    <td>54.27</td>
    <td>21.66</td>
    <td>0.640</td>
    <td>0.052</td>
  </tr>
  <tr>
    <td>{v}</td>
    <td>69.37</td>
    <td>81.91</td>
    <td>54.27</td>
    <td>21.44</td>
    <td>0.639</td>
    <td>-0.013</td>
  </tr>
  <tr>
    <td>{l, a}</td>
    <td>77.02</td>
    <td>78.22</td>
    <td>60.61</td>
    <td>38.29</td>
    <td>0.440</td>
    <td>0.537</td>
  </tr>
  <tr>
    <td>{l, v}</td>
    <td>78.12</td>
    <td>79.87</td>
    <td>60.39</td>
    <td>38.29</td>
    <td>0.442</td>
    <td>0.537</td>
  </tr>
  <tr>
    <td>{a, v}</td>
    <td>69.37</td>
    <td>81.91</td>
    <td>54.27</td>
    <td>21.44</td>
    <td>0.636</td>
    <td>0.021</td>
  </tr>
  <tr>
    <td>{l, a, v}</td>
    <td>78.34</td>
    <td>79.69</td>
    <td>60.61</td>
    <td>38.95</td>
    <td>0.441</td>
    <td>0.550</td>
  </tr>
  <tr>
    <td>Avg.</td>
    <td>74.21</td>
    <td>80.41</td>
    <td>57.82</td>
    <td>31.19</td>
    <td>0.526</td>
    <td>0.319</td>
  </tr>
</table>

Table 13: Robustness evaluation results of P-RMF under inte-modal missingness on the SIMS dataset.

<table>
  <tr>
    <th>Method</th>
    <th>Acc-2</th>
    <th>F1</th>
    <th>Acc-5</th>
    <th>Acc-7</th>
    <th>MAE</th>
    <th>Corr</th>
  </tr>
  <tr>
    <td><b>P-RMF</b></td>
    <td><b>71.53 / 72.81</b></td>
    <td><b>71.69 / 72.93</b></td>
    <td><b>38.50</b></td>
    <td><b>34.19</b></td>
    <td>1.038</td>
    <td><b>0.525</b></td>
  </tr>
  <tr>
    <td>w/o PMG</td>
    <td>70.38 / 71.79</td>
    <td>70.54 / 71.82</td>
    <td>37.35</td>
    <td>33.53</td>
    <td>1.055</td>
    <td>0.517</td>
  </tr>
  <tr>
    <td>w/o SA</td>
    <td>70.84 / 72.11</td>
    <td>70.98 / 72.16</td>
    <td>37.88</td>
    <td>33.87</td>
    <td>1.134</td>
    <td>0.516</td>
  </tr>
  <tr>
    <td>w/o CMA</td>
    <td>70.77 / 71.79</td>
    <td>70.54 / 71.82</td>
    <td>37.76</td>
    <td>33.53</td>
    <td>1.120</td>
    <td>0.508</td>
  </tr>
  <tr>
    <td>w/o SR</td>
    <td>71.35 / 71.55</td>
    <td>71.14 / 71.46</td>
    <td>38.17</td>
    <td>33.80</td>
    <td><b>1.033</b></td>
    <td>0.520</td>
  </tr>
</table>

Table 14: Independent ablation on MOSI.
<table>
  <tr>
    <th>\( \lambda_2 \)</th>
    <th>Acc-2</th>
    <th>F1</th>
    <th>Acc-5</th>
    <th>Acc-7</th>
    <th>MAE</th>
    <th>Corr</th>
  </tr>
  <tr>
    <td>0.1</td>
    <td>71.20 / 72.44</td>
    <td>71.31 / 72.65</td>
    <td>37.97</td>
    <td>34.15</td>
    <td>1.055</td>
    <td>0.500</td>
  </tr>
  <tr>
    <td>0.2</td>
    <td>71.43 / 72.65</td>
    <td>71.50 / 72.77</td>
    <td>38.31</td>
    <td>33.88</td>
    <td>1.053</td>
    <td>0.507</td>
  </tr>
  <tr>
    <td>0.3</td>
    <td>71.50 / 72.76</td>
    <td>71.63 / 72.78</td>
    <td>38.33</td>
    <td>34.11</td>
    <td>1.047</td>
    <td>0.515</td>
  </tr>
  <tr>
    <td>0.4</td>
    <td>71.42 / 72.73</td>
    <td>71.34 / 72.51</td>
    <td><b>38.62</b></td>
    <td>34.02</td>
    <td>1.039</td>
    <td>0.522</td>
  </tr>
  <tr>
    <td>0.5</td>
    <td>71.53 / <b>72.81</b></td>
    <td><b>71.69</b> / <b>72.93</b></td>
    <td>38.50</td>
    <td><b>34.19</b></td>
    <td>1.038</td>
    <td>0.525</td>
  </tr>
  <tr>
    <td>0.6</td>
    <td>71.51 / 72.80</td>
    <td>71.57 / 72.88</td>
    <td>38.55</td>
    <td>34.11</td>
    <td><b>1.033</b></td>
    <td><b>0.526</b></td>
  </tr>
  <tr>
    <td>0.7</td>
    <td><b>71.55</b> / 72.77</td>
    <td>71.66 / 72.63</td>
    <td>37.99</td>
    <td>33.75</td>
    <td>1.040</td>
    <td>0.516</td>
  </tr>
  <tr>
    <td>0.8</td>
    <td>71.44 / 72.61</td>
    <td>71.22 / 72.56</td>
    <td>38.33</td>
    <td>34.01</td>
    <td>1.051</td>
    <td>0.514</td>
  </tr>
  <tr>
    <td>0.9</td>
    <td>71.32 / 72.53</td>
    <td>71.27 / 72.44</td>
    <td>37.66</td>
    <td>33.55</td>
    <td>1.063</td>
    <td>0.505</td>
  </tr>
  <tr>
    <td>1.0</td>
    <td>71.43 / 74.56</td>
    <td>71.33 / 72.62</td>
    <td>37.87</td>
    <td>33.76</td>
    <td>1.056</td>
    <td>0.508</td>
  </tr>
</table>

Table 15: Hyperparameter sensitivity analysis.

Reviewer paper summaries: This paper proposes DaNet, a dual-aware enhanced alignment network for Multimodal Aspect-Based Sentiment Analysis (MABSA), aiming to address challenges in fine-grained cross-modal alignment and noise reduction. The core contributions include a Multimodal Denoising Encoder for filtering irrelevant visual regions, dual aspect-aware and sentiment-aware alignment modules for cross-modal interaction, and two pre-training tasks to enhance implicit aspect-opinion alignment. Experiments across three MABSA subtasks show performance improvements over baselines like VLP-MABSA and AoM.

This paper proposes DaNet, a Dual-Aware Enhanced Alignment Network for Multimodal Aspect-Based Sentiment Analysis (MABSA), which aims to jointly extract aspect-sentiment pairs from text and image data. Existing approaches often assume explicit aspect terms and align the entire image with text, introducing noise and failing to capture fine-grained or implicit sentiment-aspect relationships. DaNet addresses these limitations through a Multimodal Denoising Encoder (MDE) that filters irrelevant visual regions, and a dual-aware alignment module comprising an Aspect-Aware Alignment Module (AAAM) and a Sentiment-Aware Alignment Module (SAAM) that works together to enhance the alignment of text and visual cues while reducing inter-aspect interference. To further support implicit aspect detection, DaNet incorporates two pretraining tasks—Implicit Aspect-Opinion Generation (IAOG) using ChatGPT and Visual Aspect-Opinion Generation (VAOG) using DeepSentiBank—which improve its understanding of sentiment patterns across modalities. Experimental results on Twitter-2015 and Twitter-2017 show that DaNet outperforms state-of-the-art baselines in MABSA, MATE, and MASC tasks, including against large language models, and demonstrates strong robustness in implicit aspect scenarios.

This paper proposes two novel modules and two specific pre-training tasks to facilitate MABSA. The Dual-Aware Enhanced Alignment Network (DaNet) is designed to achieve fine-grained alignment between image and text modalities, while the Multimodal Denoising Encoder (MDE) aims to reduce noise caused by irrelevant information. To further enhance the model’s ability to capture common sentiment patterns and aspect-sentiment semantics within text-image data, the authors introduce two pre-training tasks: Implicit Aspect-Opinion Generation (IAOG) and Visual Aspect-Opinion Generation (VAOG).
Summary of strengths: - The dual-aware alignment framework provides a structured approach to decouple aspect-sentiment interactions, which is intuitive and reasonable. 
- The model achieves competitive performance on standard benchmarks (Twitter-2015/2017), particularly in MASC.

- Dual-Aware Enhanced Alignment: The proposed DaNet introduces a unique dual-aware mechanism combining Aspect-Aware Alignment Module (AAAM) and Sentiment-Aware Alignment Module (SAAM). This dual-view approach captures fine-grained interactions and addresses mutual interference among aspects and sentiments.

- Multimodal Denoising Encoder (MDE): A key innovation is the joint text-image guided denoising encoder, which filters irrelevant visual regions based on attention scores, thus improving alignment precision and reducing multimodal noise.

- Pretraining with LLM Guidance: The design of Implicit Aspect-Opinion Generation (IAOG) using ChatGPT and Visual Aspect-Opinion Generation (VAOG) using DeepSentiBank introduces practical and scalable methods for supervising implicit sentiment reasoning.

- Across three MABSA subtasks (MABSA, MATE, MASC), DaNet achieves superior or competitive results on Twitter-2015 and Twitter-2017 datasets, outperforming recent baselines including AESAL, AoM, and VLP-MABSA.

1. The authors present a comprehensive comprehension of MABSA, acknowledge the challenges inherent in MABSA task and propose a tailored solution. Ablation studies validate the effectiveness of proposed method.
2. The authors provide detailed experimental settings, which facilitates the reproducibility of this work.
Summary of weaknesses: - The proposed MDE and alignment modules closely resemble prior work (e.g., TMFN [Wang et al., 2024a], AoM [Zhou et al., 2023]). For instance, TMFN already employs multi-granular fusion to reduce cross-modal noise, while AoM uses token-region alignment—a core idea behind AAAM. The incremental tweaks (e.g., "weakly correlated region injection") lack sufficient theoretical or architectural innovation.
- The claim that "global alignment introduces noise" (§1) overlooks recent work like DPFN [Wang et al., 2024b], which explicitly models inter-aspect interference. The distinction between existing methods needs further clarification.
- The IAOG task relies on ChatGPT-3.5 to generate synthetic aspect-opinion pairs, introducing potential bias (e.g., ChatGPT’s tendency to hallucinate). While the authors acknowledge this limitation (§5.1), they provide no quantitative analysis of label noise impact. Similarly, VAOG depends on DeepSentiBank—an outdated sentiment detector (2014)—which likely mislabels modern social media imagery. These choices undermine the reliability of pretraining, especially compared to self-supervised methods like CLIP.
- The implicit aspect experiments (§4.5) simulate implicit scenarios by masking explicit aspect terms (e.g., replacing "hamburger" with "<mask>"). However, this approach is artificial—implicit aspects (e.g., "delicious" implying "food") require contextual inference, not token deletion.
- While DaNet outperforms VLP-MABSA and AoM, its gains are modest (e.g., +0.5% F1 on Twitter-2017 MASC vs. AESAL). More critically, it underperforms DQPSA in single-aspect scenarios (Table 4), raising concerns about its robustness. The ablation study (Table 5) shows minimal impact from IAOG/VAOG (≤1% F1 drop), questioning their necessity.
- How does "nearest-neighbor region merging" (Eq. 5–8) prevent over-smoothing of visual features?
- The literature review is insufficient, as several relevant works on the same task are missing (e.g., [a, b])

[a] Vanessa: Visual connotation and aesthetic attributes understanding network for multimodal aspect-based sentiment analysis. EMNLP 2024

[b] Atlantis: Aesthetic-oriented multiple granularities fusion network for joint multimodal aspect-based sentiment analysis. Information Fusion 2024

1. The baselines for comparison are not up-to-date, mostly from before 2023.

2. The authors design two specific pre-training tasks on MVSA-Multi to further enhance DaNet’s ability to learn joint multimodal representation and the relationships between aspect-sentiment. However, they do not test how the two tasks may affect the performance of previous methods, e.g., AoM, VLP-MABSA, and AESAL, leaving concerns about generalizability.

3. What are the design ideas behind using the tanh and sigmoid activation in Formula (10) and Formula (13) respectively? The paper lacks the corresponding explanations.

4. The improvements of DaNet are somewhat marginal compared with the strong baseline AESAL.

1. This work compares an insufficient number of models and fails to incorporate some of the latest high-performing SOTA models. 
2. The proposed modules in this paper demonstrate limited novelty. Notably, the AAAM module exhibits similarities to the A3M module from AoM: Detecting Detecting aspect-oriented information for multimodal aspect-based sentiment analysis. The authors should explicitly delineate the distinctions between AAAM and A3M and formally acknowledge the foundational contributions of prior work.
3. In Section 3.4, the authors claim ChatGPT is used to generate the matched sentiment-aspect pairs. The impact of these sentiment-aspect pair quality on model performance requires discussion in this study.
Comments, suggestions and typos: There is a missing space after the closing parenthesis in lines 81 and 84.

1. Lack of comparison with some strong baselines, e.g., ADAR [1], CORSA [2], and RNG [3].

2. Line 217-218: The citation (Liu et al.) doesn't direct to the reference.

3. The authors are advised to consider a more appropriate symbol for the projected image features $V$, which is easily confused with the value matrix in the attention mechanism.

4. The authors are advised to write in the simple present tense in the academic paper.


References:

[1] Chen Z, Zhu Z, Xu W, et al. Aspects are Anchors: Towards Multimodal Aspect-based Sentiment Analysis via Aspect-driven Alignment and Refinement[C]//Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 2292-2300.

[2] Liu X, Li R, Ye S, et al. Multimodal Aspect-Based Sentiment Analysis under Conditional Relation[C]//Proceedings of the 31st International Conference on Computational Linguistics. 2025: 313-323.

[3] Liu Y, Zhou Y, Li Z, et al. RNG: Reducing Multi-level Noise and Multi-grained Semantic Gap for Joint Multimodal Aspect-Sentiment Analysis[C]//2024 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2024: 1-6.

See Weaknesses section. 

In addition,  Fig. 4 is cited before Fig.3. I suggest that the figure numbering be re-arranged accordingly.
Overall assessment scores: 2

3

2.5
Metareview score: 3
Metareview: This paper introduces DaNet, a Dual-Aware Enhanced Alignment Network for Multimodal Aspect-Based Sentiment Analysis (MABSA). Unlike prior methods that align entire images with text—often adding noise—DaNet uses a Multimodal Denoising Encoder (MDE) to filter irrelevant visual information. It features two key modules: an Aspect-Aware Alignment Module (AAAM) and a Sentiment-Aware Alignment Module (SAAM), which jointly enhance fine-grained alignment and reduce cross-aspect interference. DaNet also leverages two pretraining tasks—Implicit Aspect-Opinion Generation (IAOG) via ChatGPT and Visual Aspect-Opinion Generation (VAOG) via DeepSentiBank—to better detect implicit aspects. Experiments on Twitter-2015/2017 datasets show DaNet outperforms state-of-the-art models, including large language models, especially in implicit aspect scenarios.
Summary of reasons to publish: - DaNet introduces a novel dual-aware mechanism that combines the Aspect-Aware Alignment Module (AAAM) and the Sentiment-Aware Alignment Module (SAAM). This dual-view strategy effectively captures fine-grained aspect-sentiment interactions while mitigating mutual interference between them.

- DaNet achieves state-of-the-art or competitive results across three key MABSA subtasks—MABSA, MATE, and MASC—on the Twitter-2015 and Twitter-2017 datasets, outperforming strong baselines such as AESAL, AoM, and VLP-MABSA.