Wasserstein Modality Alignment Makes Your Multimodal Transformer More Robust
Authors: zhuo zhi
Yuxuan Sun
Qiangqiang Wu
Ziquan Liu
Miguel R. D. Rodrigues
Venue: Accepted by TMLR ()

Abstract: Multimodal fusion with a multimodal transformer is an effective method for both early and late fusion paradigms. However, in a multimodal transformer, the modality fusion is performed solely through the self-attention mechanism, which is originally designed for unimodal token sequences. To improve the self-attention mechanism for handling multimodal input, a parametric adapter model, like the Q-former in BLIP-2, is often used to align tokens from different modalities. Our empirical study unveils that only using the self-attention layer to perform the modality fusion makes the model less robust to missing modalities and input noise, as the model will overly rely on one certain modality. To improve the robustness of the transformer, our paper proposes an implicit approach based on Wasserstein distance that aligns tokens from different modalities without using any additional trainable parameters. Our empirical study shows that the implicit modality alignment improves the effectiveness of the multimodal Transformer in discriminative tasks, as well as its robustness to input noise and missing modalities. We conduct experiments on four downstream task datasets, including 2-modalities and 3-modalities tasks. We also consider different fusion paradigms, i.e., early and late fusion. The experimental results show that our proposed method has a significant improvement in both performance and robustness over all baselines across all datasets and fusion paradigms.

Published in Transactions on Machine Learning Research (01/2025)
Wasserstein Modality Alignment Makes Your Multimodal Transformer More Robust
Zhuo Zhi
ahuo. @uul. uk
Deportment of Electronic and Electrionl Engineering University College London
Yuxuan Sun
ywm.an.suu. 22w.cl2..222.cccooomom
Department Electronic and Electricol Engineering University College London
Qiangqiang Wu
giangyuuu2@ciipu.cee edu.
Department of Computer Science City University of Hong Kong
Ziquan Liu
ziquan. www.liiillll....mmmmm.o uk
School of Electronic Engineering and Computer Science Queen Mary Universily of London"
Miguel Rodrigues
m.roriigues &ucl.. acc.cll.ccccccoco uk
Department of Electronic and Electrical Engineering University College London
Reviewed OpenReview: /ttps://openreviiiee net/ formf id= 2IKuU2BB62
Abstract
Multimodal fusion with a multimodal transformer is an effective method for both early and late fusion paradigms. However, in a multimodal transformer, the modality fusion is performed solely through the self-attention mechanism, which is originally designed for uni- modal token sequences. To improve the self-attention mechanism for handling multimodal input, parametric adapter model, like the Q-former in BLIP-2, is often used to align tokens from different modalities. Our empirical study unveils that only using the self-attention layer to perform the modality fusion makes the model less robust to missing modalities and input noise, as the model will overly rely on one certain modality. To improve the robustness of the transformer, our paper proposes an implicit approach based on Wasserstein distance that aligns tokens from different. modalities without using any additional trainable parameters. Our empirical study shows that the implicit modality alignment improves the effectiveness of the multimodal Transformer in discriminative tasks, as well as its robustness to input noise and missing modalities. We conduct experiments four downstream task datasets, including 2-modalities and 3-modalities tasks. We also consider different fusion paradigms, i.e, early and late fusion. The experimental results show that our proposed method has a significant improvement in both performance and robustness over all baselines across all datasets and fusion paradigms.
1 Introduction
Multimodal machine learning (MML) mimics human perception by integrating multiple modalities such as text, audio, images, video, and sensor data to form a comprehensive understanding of the world. Many
"Corresponding author

---

Published in Transactions on Machine Learning Research (01/2025)
multimodal models have been applied to various tasks like multimodal medical diagnostics Hayat et al. (022a)), sentiment analysis Zadeh et al. (2018) and malicious speech detection Kiela et al. (2020).
Aligning heterogeneous data in multimodal learning is crucial since such data often exhibit distinct dis- tributions, representations, and noise levels. Proper alignment enhances the uniform representation of these diverse data types, leading to improved performance and robustness multimodal tasks Ghahre- mani Boozandani & Wachinger (2024); Liang et al. (2024); Kim al. (2020). To achieve better modality alignment, various strategies are applied in large-scale multimodal models, such the Q-Former in BLIP-2
Li et al. 2023 contrastive learning in CLIP Radford et al. [2021) and Imagebind Girdhar et al. 2023
Multimodal fusion based on one-tower transformer, named as multimodal transformer (MT), by its flex- ibility and simplicity, are widely used for a variely of multimodal learning tasks Lee et. al. [0023 Nagrani et al. 2021); Zhi et al. 2024]; Ma et al. (2021). Although the multimodal transformer can handle multi- modal tokens as the input due to the flexibility of self-attention layers, lacks mechanism for modality alignment during the line-tuning process. In other words, it not optimal to rely on pre-trained multi-
modal transformer to achieve the modal alignment Kim et al. (2021); Wang et al. (2021) Taking ViLT Kim et al. 2021 as example, tokens from different modalities the multimodal transformer. Several learning tasks, such as image text matching and word patch alignment are applied during the pre-training phase to ensure the modality alignment. However, the aligmment process is not enforced in the fine-tuning stage if the input tokens are from multimodal sources, which leads to deteriorated performance in a downstream task.
concatenated together and processed by
(a) Fine-tuning process of multimodal transformer
(b) Fine-tuning process of multimodal transformer with WMA regularization
II,
WMA regularization
H
cb
H.
H
y
fg
Multimodal Transformer fo,
Multimodal Transformer fo.
Averago of COA
Averago ofl tost cauok
0,675
0.706
+2.0%
0.644
8.481
0.428
alme
AUROX
Raterid Mexses
MMf-MDB
UB- UR-FUNNY
MOSBI
Ratefol Momea
MM MMM-1000
UR-PUNNY
MOSEI
Figure 1: The overview of the proposed method (using early fusion structures as example). (a) Fine- tuning the multimodal transformer on the downstream task. The image data x;"" and text data 2: are firstly processed by the embedding operation ea.,, and ea... (in late fusion structure, ebm,, and es... refer to the feature extraction by the pretrained encoders, i.e., Bert for text and ViT for image. ) to get the token sequence xith and -,m which then are concatenated and processed by the multimodal transformer fox. H, in the multimodal transformer is input into the classifier fe_ for predicting the label й. Note that there is specific module to adjust modality alignment in (a). (b) We propose the WMA regularization method to search for the optimal modality alignment for the target task. WMA calculates the OT distance between t2 to represent the alignment degree of two modalitics. The target OT distance range is set to achieve the optimal alignment. Our proposed WMA method elfectively improves both the performance and robustness of the model, as shown in the lower left corner (1) and (2) ( 'Average of all test cases' refers the average result of normal test, test with noise and the test with missing modality. See more details Table. and Table. 2
". and H,
2

---

Published in Transactions on Machine Learning Research (01/2025)
To address this issue, propose Wasserstein Modality Alignment (WMA), an implicit regnlarization method to align the Wasserstein distance bet ween two modalities within a multimodal transformer, as shown in Fig. 1 To make the computation feasible, we use the popular optimal transport (OT) distance Peyré et al. 2019) as the instantiation of Wasscrstein distance. We regularize the degree of modality aligument by adjusting the OT distance between different modalities' feature distributions. Interestingly, our empirical study demonstrates that directly minimizing the OT distance between two modalities often leads to inferior performance, so our WMA aligns two modalities with task-dependent modality distance. In the practical sense, the proposed WMA is a plug-an wong-and-play method and does not introduce any additional trainable parameters. Our main contributions are three-fold:
We propose to perform the modality aligmment in the fine-tuning process of a pre-trained multimodal transformer without any additional adapter and design the Wasscrstein Modality Alignment based on the optimal transport distance to achieve light modality alignment in the feature tokens of the transformer.
Instead of minimizing the OT distance between any two modalities, our WMA is lask-dependent modality alignment method that can handle different requirements for the degree of modality align- ment.
We evaluate proposed WMA with three strong baselines four datasets, including 2- moddlities and 3-modalities tasks. We also consider different fusion paradigms, including both carly fusion and late fusion. Our experimental results demonstrate significant improvements in performance and especially robustness across all tasks and test cases. In early fusion paradigm, the averaged improvement of our method compared with the best result from all baselines over all datasets are 1.4%, 1.5%, 1.4%, 0.7% and 1.4%, respectively. For the late fusson paradigm, the improvement are 1.5%, 1.8%, 1.3%, 2.3% and 1.9%.
The paper is organized as follows. Sec. 2 gives an overview of related research. Sec. introduces the proposed method. Sec. 4 shows the experiment results and the analysis. Finally, Sec. 5 summarizes the paper, its limitations, and future work.
2 Related Work
Modality alignment in multimodal learning. Almost all large-scale multimodal models use specific strategies for modal alignment. Contrastive loss is popular approach that promotes related modality alignment by boosting the similarity of positive sample pairs Li et al. (2021); Radford et al. (2021): Girdhar
ct al. 2023). LLaVA trains an image-to-text adapter to align the two modalitios Liu et al. (2024). Flamingo achieves modality aligument. by combining a pretrained vision encoder and a language model through a series of gated cross-attention layers, allowing for effective interaction between visual and textual inputs Alayrac et al. (2022). In Li et al. (2023). : BLIP-2 employs a lightweight. Querying Transformer to connect frozen image encoders with large language models for modality alignment. In contrast, the multimodal (2021) and SimVTMWang et al. (2021) only employ the agent task such
transformer, i.e., ViLT Kim et al. as image text matching and word patch alignment for aligning modality during pre-training, lacking such approach at fine-tuning stage. To solve this problem, we propose an implicit regularization method to adjust the alignment of the multimodal transformer during the fine-tuning process.
Robustness of multimodal learning. Modality noise and absence are two challenges to the robustness of multimodal learning. In ML for healthcare, the patient may be missing the data such an X-ray due to economic/ tiiiingiinggg issues Zhi et al. (2024). In addition, some sensor data may be accompanied by a lot of noise duc to improper wear. A similar situation occurs in vision-language tasks. For example, some online recommender models are unable to receive images uploaded by users or receive blurry images with a lot noise due to network issues. Sijie et al. Mai et al. (2022) propose the multinodal information bottleneck to filter out noisy information in unimodal representations. Md et al. Islam & Iqbal (2022) apply cooperative multitask learning- guided multimodal fusion approach to get robust performance noisy and misaligned sensor data. For the missing modality problem, Ma et al. (2021) reconstructs the
3

---

Published in Transactions on Machine Learning Research (01/2025)
missing modalities using modality priors and Bayesian Meta-Learning during the inference phase. Lee et al. (2023) propose the missing-aware prompts to learn the patterns of complete and incomplete samples. In Zhi et al. (2024), an approach inspired by in-context learning is proposed to improve the data efficiency for multimodal learning under missing modality and data scarcity. However, these methods require additional parameters to enhance the incomplete samples. Differently, we employ the nonparametric regularization approach to obtain robust multimodal representation.
3 Proposed Method
We first describe the problem definition, our motivation and the proposed method is claborated on later.
3.1 Problem setting
We consider the multimodal transfer learning problem with downstream dataset containing multi- modal input samples. For notation simplicity, we assume there are two modalities in the dataset, i.c. D " ) where y is the label. Note that our framework can handle any number of modalities in principle and We will describe how to extend it to 3-modalities tasks later. When we employ pretrained multimodal transformer for solving the larget lask, some embedding operations feature extraction are performed firstly performed on the input data x," and x;
x c0m1
(x:²1t) [Mictsi M111 miL
(1)
x. p22 = c0my (I = [m2dds: Meni M2Lmg
(2)
where etm, and Cimy refer to the embedding operation for two modalities such as linear projection, position embedding and modality type embedding Kim al. (2021) in early fusion structure. In late fusion structure, com, and eomg, refer to the feature extraction by the pretrained encoders, i.c., Bert for text and ViT for image. miets and m2ets are the added classification head token and Lmy and Lmg are the number of embedded tokens tokens/feature. feature. [:] means the concatenate operation. Then, the multimodal transformer fax inference the output tokens H, by
(3) where H; E Iemm xd and H E IR Tmmg xd are the processed features for two modalities, d is the emn als E xi is the final classification head token which can be input into an added classifier fregressor fo, for predicting the label and minimizing the loss:
H1 = fax (1 Thi ; 11 ) = [H, cls, H, "1; H.m"...
bedding dimension. H,ds
J = fo. (H, clock = Lets (0t.
(4)
where lets is the task-dependent loss function such as cross-cntropy and (()) is the loss value for the ith sample.
The issue with the fine-tuning process described above that lacks the approach for aligning क; and x;" or their reprosentation in this multimodal transformer. We will introduce our proposed method for solving this problem later.
3.2
Motivation
Although the strength of multimodal learning unimodal learning has been proved both in theory Huang et al. (2021); Lu (2023) and in practice Radford et al. (2021), it is plagued by missing modalities Zhi et al. (2024): h Lee et al. (022)) and input noise Papandreou et al. (2009). One of the reasons for the brittlencss of multimodal learning is that it heavily relies on one certain modality to make the prediction as a result of the distributional gap between different modalities. In other words, some modalities are more easily by the training process bocause those modalities are more learnable than others. The phenomenon relevant shortcut learning Geirhos et al. (2020) where the model will use the shorteut features to complete the given learning task and fail to generalize to out-of-listribution scenarios. To illustrate the phenomenon
4

---

Published in Transactions on Machine Learning Research (01/2025)
(a) The histogram of integrated gradients value
MT-early
MT MT-early RegBN
MT-early-WMA
Density
10
10
1.0
0.5
0.0
0.5
1.0
Integrated gradients value of text tokens
MI-cadly
MT-eatlyRegBN
Density
10"
MT early WMA
10
-1.0
0.5
0.0
0.5
1.0
Integrated gradients value of image tokens
(b) The statistics of integrated gradients value
Text
mean
std
NLL
sum
MT-carly
1.59
43331.60
690.63
MTTcarly-RegBN
0.03
1.52
41435.76
602.70
MTTearly-WMA
0.04
1.75
45527.80
811.38
Image
std
NLL
MT-carly
0.00
2.56
433552,72
113.28
MTTeadyy-RgBBN
293459.95
143.68
M'Fearryy MA
0.04
0.45
107685.24
702.77
The Integrated gradients value of input token under Hateful Memes dataset and early fusion structure. We calculate the Integrated gradients value for every input token for all samples and put them together to obtain the distribution analysis. The bigger Integrated gradients value refers to the bigger contribution to the final classification. (a) The histogram of the Intograted gradicnts values. (b) The statistics of the distribution of the Integrated gradients values. MT-early, MT-eary-ReBBN and MT-early- WMA refer to the original MT, MT with RegBN regularization Ghahremani Boozandani Wachinger (2024) and MT with our proposed WMA regularization. The performance and robustness of three methods
Figure 2:
be find in Table I From the statistics, we find that MT-carly heavy relies text for decision due to the sum of the Integrated gradients values of the lext token 690.63 is much higher than that of the image token 113.28 The RegBN method alleviates the problem slightly with the numbers 602.70 and 143.68. Our proposed WMA method solves this problem the contribution of text token and image token is almost identical-811. and 702.77. In WMA, the image modality has more Gaussian-like altribution value distribution compared with others, meaning that the model exploits the weak modality to learn the task.

---

Published in Transactions on Machine Learning Research (01/2025)
in multimodal learning, analyze fine-tuned multimodal transformer fusion structure, as shown in Fig. 2 Integrated Gradients is an attribution method that computes the contribution of cach imput feature to the output of a neural network by integrating gradients along a path from a baseline input to the actual input. With this lool, we oblain the contribution of each input token the final classification.
the explanation tool Integrated Gradients Sundararajan et al. (2017 to
two-modality dataset. Hateful Memes under the early
It is observed from Fig. that standard fine-tuning mainly uses the strong modality (text) to make the prediction while the weak modality (image) is not sufficiently exploited. To address the observed shorteut learning issuc and make the model more robust to novel test environments, we propose to control the distributional gap between different modalities. As the distribution of easily learned modalities is regularized to be close to other modalities, the model is forced to avoid shortcut learning and use all modalities to learn the task.
0.750
0.725
normal test
image noise - text noise
0.700
image miss
text miss
0.675
AUROC
0.650
0.625
0.600
0.575
0.550
0.1
0.2
0.4
0.6
0.8
1.2
1.4
1.6
1.8
2
4
10
Value of
Figure 3: The performance of MT-Marll--l-aaee WMA Hateful Memes datasets in early fusion structure under with different value of B. When 3 = 0, it equals to minimize the OT distance. The best overall performance is gained at 3 = 1.8. The performance and robustness of the model are not exactly positively vositively/negativel)) /negatively correlated with the value of 3, indicating that the alignment of the model should be task-dependent, instead of minimizing the OT value. See more results in Table 5
3.3 Wasserstein Modality Alignment
We propose the Wasserstein Modality Alignment (WMA), an implicit regularization for adjusting the align- ment of different modalities in the multimodal transformer. For keeping computation efficient, we use the OT distance as the instantiation of Wasserstein distance and represent the alignment degree of two modalities by it. Uniquely, WMA searches the lask-dependent optimal alignment. through two hyperparameters rather than directly minimizing the OT distance.
For the feature H, and H, the optimal transport problem is defined as
W(H H min (C,T), Te
(5)
where C RJom Tong is a manually defined cost matrix, with each element Cpy representing the distance between the pth token of Hm and the ęth token of HMa. The optimal solution 7 is known the optimal

---

Published in Transactions on Machine Learning Research (01/2025)
transport plan. The set E(o, 8) is defined as:
E(,,6) {T E E x XLma [1img o, T bmn =ō].
(6)
where a and å are the normalized distributions for H and H respectively, which are given by:
1 Amm Liva
(7)
Lim
where Iim, and Lmg are vectors of ones with lengths Lm, and Lmn, respectively. To keep the computation efficient, apply the IPOT algorithm Xic et al. (2018) to solve this problem.
The optimal transport cost D Dimm2 is calculated as
D D (C,
We use Di Di ing as the reference for the aligmment degree and manually set the target value to regularize the model parameters during the fine-tuning phase by modifying the loss function
task Lato v:) a(D -D Dmmmma? )²
(9)
We the average D mn of the first batch model initialization (by applying the pretrained weight) as a basis and set the search range by combination of hyperparameters and B. By this strategy, we do not min- imize Dj mm instead, we search for optimal alignment for the different modalities in the fine-tuning phase. Fig. 3 shows the search results for the Hateful Memes dataset in early fusion structure under a = 1 with different values of B. The best overall performance is achieved at 8 = 1.8 which demonstrates the superiority of the proposed WMA over minimizing OT values. For 3-modalities tasks H₃ [H, cts H, mi, H, m²2 H, ma]
we can casily modify Eq. g
task Eclo v1) a((Di i
-3D m, mmz
+(Di Maims im
3D sng)
+ +(D mD 3 ma )
(10)
4 Experiment
We first introduce the experimental settings and then present the experimental results of our methods and baselines on five datasets and two fusion paradigms, demonstrating the effectiveness of our method.
4.1
Experimental Setting
Datasets. We select two 2-modalities datasets and two 3-modalities datasets across different downsstrem tasks to evaluate our proposed method.
Hateful Memes Kiela et. al. (2020). This a binary classification task with two modalities, image and text. The task detect the maliciousness of memes. The numbers of the samples in the sainiing/vvaltestinng dataset are 8500, 500 and 1000.
MM-IMDb Arevalo et al. (2017). This is multi-label (25 labels) classification task with two modalities, image and text. The task is to tag the film. The numbers of the samples in the saining/vaaltestinng datasct are 32278, 5411 and 16120.
HR-FFUNYYHHsan al. (2019). This is binary classification task with three modalitics, text, video and audio. The task is detect humor in talks. The number of samples in the training/ meaniingvvaatestinnn dataset. 8074, 1034, 1058.

---

Published in Transactions on Machine Learning Research (01/2025)
MOSEI Zadeh et al. (2018). This is a regression task with three modalities, text, video and audio.
The lask is to recognize the degree of sentiment. The number of samples in the training/ val/vesting dataset are 16265, 1869, 4643.
MedFuse-I Hayat et al. (022b). This a real-world dataset which contains EHR and X-ray data for each patient. This binary classification task aims to predict in-hospital mortality after the first 48 hours spent in the ICU. The numbers of the samples in the training/val/testti /val/testing dataset are 18845, 2138 and 5243.
Metrics. We set the metrics for each dataset according to the tasks. For Hateful Memes, UR-FUNNY and Medfuse-1, we use the AUROC as evaluation metrics. For MM-IMDb and MOSEI, we use F1 score and MAE, respectively.
Multimodal Transformer for different fusion paradigms. We consider two popular multimodal fusion paradigms by multimodal transformer: carly fusion and late fusion. For carly fusion, we use a 12-layer and 12-head transformer and initialize it with the pretrained weight ViLT-B Kim et al. (2021). J. For late fusion, we employ a 2-layer and 8-head transformer and randomly initialize it. All input tokens are added with modality embedding and a cls token by following Kim al. (202)).
Input data processing. In early fusion structure, for vision-language tasks Hateful Memes and MM-IMDb, we follow the operation in ViLT Kim et al. (2021): the text is embedded by Bert and the image is split into patches (same with ViT). For UR-FUNNY and MOSEI, we use MultiBench Liang et al. (2021) to get the embedded feature of three modalities. For Medfuse-I, we use a linear layer to embed the EHR data and the same way with ViLT to process the X-ray image. In late fusion structure, for vision-language tasks Hateful Memes and MM-IMDD, a ViT-base and a Bert-base are employed to extract the feature of two modalities, while for UR-FUNNY and MOSEI, three GRU models are involved for feature extraction by following Liang et al. (022I). For Medfuse-I, we follow the operation in Hayat et al. (2022b).
Baseline. We select the following strong baseline for comparison. These methods focus on efficiently adjusting modal fusion to improve performance in multimodal learning.
Standard transfer learning. We add a classifier/regresso for the target task and fine-tune all layers of the models, which is denoted as MTT-eally and MT-late in the latter part.
RegBN Ghahremani Boozandani & Wachinger 2024 Batch normalization of multimodal data with regularization (RegBN) uses the Frobenius norm as a regularizer term to address the side effects of confounders and underlying dependencics among different modality sources. RegBN provides a plug-and-play approach to early and late fusion.
GWMAC Gong et al. 2022). Gromov- Wasserstein multi-modal alignment and clustering (GWMAC) learns the Gromov Wasserstein barycenter of their kernel matrices. GW barycenter is associated with the GW distances between the different modalities to the clusters, and the optimal transport plans corresponding to the GW distances help to achieve the alignment and the clustering of the multi- modal data jointly. This method only provides a late fusion version, and we extend it to an early fusion framework.
MIB Mai et al. (2022). The multimodal information bottleneck (MIB) learns the minimal sufficient representation for given task by maximizing the mutual information between the represcntation and the larget and simultaneously constraining the mutual information between the representation and the input data. The method is also compatible with early fusion and late fusion frameworks.
Hyperparameter settings. We set the batch size for Hateful Memes, MM-IMDB, UR-FUNNY and MOSEI as 128, 64, 256, 256. The learning rate search range is [le-3, 5c-4, 1c-4, 5c-5, 1c-5). The learning rate strategy is linear decay with warm-up. The search range of a is set as [0.1, 0.2, 1.0, 5.0). The search range of B is [0.1, 0,2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0, 4.0, 10.0]. Early stopping with patience 5 is applied for selecting the weight. Experiments are running on Tesla V100 GPUs.
Robustness test setting. We test the robustness of the model in two ways: test with modality noise and test with missing modality. When simulating noise in a particular modality, we consider three kinds
8

---

Published in Transactions on Machine Learning Research (01/2025)
of noise for image data: Gaussian noise, salt and pepper noise and Poisson noise. For text data, use swap adjacent letters, randomly permute middle section, keyboard typos, sticky keys and omission to create the noisy input. For time series data (i.e., some features from MultiBench [iang et al. (2021) and EHR data), we add the Gaussian Noise, uniform noise and Poisson noise. When simulating a missing modality, we randomily remove a proportion of the samples for that modality by setting all pixels to for the image, replacing the text with an empty string and setting all values to for time series. For Gaussian noise, the noise level is set to the standard deviation of the noise. For the salt and pepper noise, the noise level refers to the proportion of total pixels perturbed. For the uniform noise, the noise level refers to the noise range. For the text data noise, the noise level is the probability of randomly applying noise to a word. We set the noise level to be 0.25, 0.5, 0.75 to the image data, text data and time series data. The final result is the average of the performance at different noise levels. For the missing modality Lest, We set 25%, 50% and 75% as the missing proportion and calculate the average performance. Note that the real-world dataset Medfuse-I lacks X-ray images for 74% of samples and we do not manually set additional missing cases to simulate rcal application scenarios.
Table 1: Results of our proposed method with the baseline on all datasets and test cases for early fusion. The bold numbers mean the best performance. The bigger AUROC and F1 and smaller MAF refer to better performance.
socmal test
test with coddality mobse
test with missing aociality
Metric
Wis,EERR
Imaage/ miserng
XIT-marly MTT-eely-GWMAAC
0.790
0.541
01298
0.626
0.653
0.641
0.752
0.561
0.680
0.626
0.009
0.654
Batefal Memor
AUROC1
MTT-muly-RagHH
0.736
1.546
0.680
0.620
0.650
0.546
AfT-muly-Mti8
0.743
11.530
0.661
0.619
0.648
0.546
0.748
0.383
0.682
0.645
0.629
0.663
XIT -ouul
0.051
1.251
0.402
0.364
0.312
0.128
0.003
1.354
0.470
0.472
01:42
0.454
MM-IMDS
F
MT onlly RegBNN
0,658
0.463
0.451
0.429
0,423
0.563
11.38
0.464
0.473
0.423
0,424
MIL-ouly- W3BA
0.563
0.356
0.471
0.477
0.437
0.461
MaT-matly
0,700
11,645
0.678
0.6455
0.612
0,670
1.500
0,062
0.705
0.078
0.966
AUROCT UR-FUNNY
MTT-oulyy Roggh
0.714
11614
0.865
0.615
0.018
0.645
16000
0,822
MI carly MID
0,711
1.641
0.682
0.63
0.006
0,670
0,000
01000
MT early WBTA
0,712
0.681
0.686
0.643
0.624
0.674
0.703
0.675
ArT-muuly
0.633
11,991
0.514
0.672
0.651
eatly GWMAC
0.624
0.679
0.827
0.657
0.817
0
MOSEI
MAET
MT early RogDN
0.617
0.714
0.877
0.665
0.622
0.848
0.620
0.704
MT-earl-XIBB
0.619
0.670
0.852
0.066
0.625
0.851
0.620
0.201
XTT-mulyyWMA
0.615
0.676
0.824
0.661
0.619
0.813
0.623
0.691
early
0,820
0,762
0.725
0.864
0.787
0.749
0.753
Medfise-1
AUROC1
ATT-nulyyReefNN
0.855
0.751
0.722
0.776
XTT-muly-3iIB
0,802
0.744
0.715
0.770
AFT-muly- Wouully--WA..
0.861
0.796
0.755
0.504
Table I and 2 present the quantitative results of our proposed method WMA with the basclincs across all the datasets and Lest cases under early and late fusion, respectively. From Table and we summarize the following observations:
Our proposed WMA is more performant than the MT in all datasets and test cases under both early and late fusion paradigms. Specifically, in the average performance of all test cases, the improvements are 3.0%, 7.7%, 2.0%, 2.1% 5.4% for five datasets in early fusion and 2.4%, 2.8%, 1.8%, 3.6%, 3.4% in late fusion. Note that WMA in early fusion benefits the model more than in late fusion, indicating that the distributional regularization is an essential component to make the self-allention--bssed modality fusion more robust.. Our proposed WMA method is also competitive compared to the best results from all baselines. To be specific, MT-early-Wii WMA surpasses the best result at 1.4%,1.5%0 1..%,11%%%% 0.7% and 1.4% in five datasets. For MT-late- WMA, the number comes to 1.5%, 188%, 1.3%, 2.3% and 1.9%.
GWMAC exhibits a similar trend as the proposed WMA method in most datasets, which shows improvement in both normal test and robustness compared with the MT, but not as much as WMA. What's worse, GWMAC introduces more parameters and computing burden. It's worth noting that MIB and RegBN show the opposite trend with GWMAC and WMA, i.., in most of the datasets, the performance of the normal tests is improved (some of them even achieve the highest score),
9

---

Published in Transactions on Machine Learning Research (01/2025)
Table 2: Results of our proposed method with the baseline on all datasets and test cases for late fusion. The bold numbers mcan the best performance. The bigger AUROC and F1 and smaller MAE refer to better performance.
sormal test
test with modaaliny mosse
test waih missing moccalit
Metric
audia imuge/ miselin
misning
missing
MrT-late
0.718
0.632
0.480
0.680
0.622
0.068
0.721
11.633
0.676
0.681
0.622
0.073
Halefal Menos
AURUCT
MTT-late-Ragh
0.791
11,64)
0.568
0.688
0,022
0.068
MT--late Mild
0.727
11,055
0.662
0.675
0.640
0.674
MT lote
0.728
0.671
0.507
0.692
0.684
MTT-late
0.612
11.631
0.488
0.553
0.434
0.505
MFT SVT-late-GWWal GWLCC
0,603
11474
0.405
0.559
0,419
0.520
MM IMDs
Pit
MT lote RegDNN
0.612
0.438
0.505
0.550
0.411
0.307
MT-late-MID
0.415
MTT-late-W WALA
0.610
0.459
0.510
0.559
0.426
0.510
XET late
0,7000
0.651
0.686
0,095
0.654
0.641
0.670
0.872
0,200
0.653
0.602
0.000
0.658
0.666
0.660
0.675
UR-PUNNY
AUROC1
late ReglNN
0.709
0.604
0.083
0.084
0,670
MT late MID
0.712
0.627
0.684
0.002
0.643
0.636
0.670
0.671
-MMA
0,710
0.680
0.698
0.702
0.060
0.672
0.683
0.684
MFT late
0,638
11,85
0,826
0,638
0,020
late GWMAC
0.627
0.811
0.812
MOSEI
MAET
MT-late-Roggh RegDN
0.610
0
0.820
0.627
0.620
0.829
0.637
0.882
MT--at--MRR
0.610
0.6811
0.844
0.619
0.624
0.831
0.620
0.650
-MMA
0.008
0.648
0.807
0.015
0.612
0
0.609
0.671
MT--arly
0.853
0.794
0.768
0.802
Molfice]1
AUROC1
AfT-nuly-RagHhh
0,859
0.785
0.745
0.796
AFT-muty- MiH
0,800
I..74
0.718
0.791
MT onlly W3EA
0.564
0.775
0.817
while the robustness is compromised. We attribute it to the fact that both MIB and RegBN aim to learn a multimodal represcntation without redundant information, making the model mainly learn from a certain modality. In our proposed WMA, the model adaptively achieves the proper modality alignment to balance multiple modalities to improve both performance and robustness.
In summary, our WMA method simultaneously improves the performance and robustness of the multimodal transformer, outperforming all baselines across all datasets and fusion structures. In addition, compared to most parametric methods, WMA docs not introduce any additional trainable parameters.
We also report all the searching results under various combinations of a (weight for the WMA loss) and B (weight for the target distance) in Table 5 for early fusion and Table 12 for late fusion. We have the following observations:
The modality alignment can be effectively adjusted by using our proposed WMA method. The obvious performance improvements can be achieved in almost half of the settings.
The performance and robustness of the model are not exactly positively/negativee /negatively correlated with the value of B. Diffcrent tasks and even different fusion structures require different degrecs of modality aligument. For example, 5 better performance and robust at bigger target OT values for the Hateful Memes dataset under early fusion and the opposite trend observed from Table for Hateful Memes datasets under late fusion. Nevertheless, our proposed method can achieve task-dependent optimal alignment.
4.2 Ablation study
Minimizing the OT distance. We compare minimizing the OT distance (used an agent task in many pre-train tasks Kim et al. (2021)) with our proposed WMA method. We simulate this strategy by setting to 1 and 33 to 0. The comparison of this method with our proposed method on the Hateful Memes dataset is shown in Fig. 4 All results are shown in Table 13 and Table 14 for early fusion and late fusion, respectively. Table 13 and Table [4 indicate that our proposed WMA outperforms this strategy in most of the test cases. The average value of all test cases WMA exceods that in Minimizing the OT at 2.0%, 5.3%, 1.2% and 2.3% for all datasets under carly fusion. For late fusion, the improvements are 1.2% 2.0%, 19.0% and 2.6%. We assume that different tasks require different levels of modality heterogeneity and alignment,
10

---

Published in Transactions on Machine Learning Research (01/2025)
and over-alignment could cause loss of modality heterogeneity which might be important to the model performance and robustness. Through only two hyperparameters, our proposed WMA method adaptively achieves optimal modal alignment, guaranteeing both performance and robustness.
normal test
normal test
Minumine OT WMA
Minimize OT
4.65
0.65
text
6
image noise
text missing
0.6
image
missing
noise
image missing
text noise
image missing
text noise
(a) early fusion
(b) late fusion
Figure 4: Results of our proposed method with minimizing OT distance on Hateful Memes datasets. See the result for all datasets in Table 13 and Table 14
normal test
normal test
MT. bite Iat_cs
MT.
MT_ CS
MT. Iaie
.66
MT. oall__5 38
MT. TV
MTL
MT. WMA
16
MiLoxly_WMe
text missing
image
text missing
image
noise
noise
image missing
text noise
image missing
text noise
(a) early fusion
(b) late fusion
Figure 5: Results of our proposed method with different distance metrics Hateful Memes datasets. See the result for all datasets in Table 15 and Table 16
Different distance metrics. We also explore different distance metrics in the proposed regularization method. Specifically, we select the cosine similarity (CS), Jensen Shannon (JS) divergence and total variation (TV) distance. The comparison of all methods on the Hateful Memes dataset shown Fig. All results under early fusion and late fusion are shown in Table 15 Table respectively. From Table 15 and Table 16 we find that while the WMA method performs best across almost all datasets and fusion paradigms, some methods come close in certain datasets, i.e., cosine similarity in Hateful Memes datasets. This proves the validity of our principle idea: obtaining task-dependent optimal fusion representations by adjusting the modal alignment in the multimodal transformer.
Computational efficiency. To provide more intuitive demonstration of the computational efficiency of our proposed method, we compare the training times of all methods on the UR-FUNNY and MOSEI datasets under early fusion and late fusion. Specifically, report the time required for training step, including both the forward and backpropagation processes for single batch of data. As shown in Table 4 our method surpasses the strong basclines GWMAC, RegBN, and MIB in terms of training speed. This improvement can be altributed to the fact. that GWMAC, RegBN, and MIB introduce additional parameters, while our WMA method operates without training any new parameters. Although the fast OT method Xie et al. (2018) used in the paper increases the computation time for distance calculations, the overall impact on runtime remains minimal. During inference, our method retains the same number of parameters and execution speed as the basic MT. Therefore, the proposed WMA method demonstrates excellent computational efficiency.
11

---

Published in Transactions on Machine Learning Research (01/2025)
Robustness to different random seeds. The WMA algorithm utilizes the average OT value of the first batch of samples as the search basis. To verify the robustness of the method under different random seeds, we conduct an ablation study using the UR-FUNNY dataset with early fusion. For each experiment, we use 10 different random seeds: [0, 10, 20, 30, 40, 50, 60, 70, 80, 90]. We report the average value along with the standard deviation across all baselines Table 3 From the table, we observe that the results of all met exhibit slight variations under different random seeds. However, the proposed WMA method consistently demonstrates clear advantage in terms of both performance and robustness. Additionally, we find that the optimal values for and & in the WMA method vary slightly depending the seed (e.g., [a=5, 3=1.8) for random seed 0, and [a=1, 3=2.2 for random seed = 20). This adaptability highlights the WMA method's strength in achieving optimal modality alignment under varying conditions.
Table 3: The performance of different methods on UR-FUNNY dataset under early fusion with different
random seeds.
normal Test
best
WILE modility BoDe
tes with missing modility
auddio
Emage
mitsing
atdiio
miesingg
M -endy
BJRIS 0.00
0.42 = 1.3844
0.627 = (11K3
0.634 2: 0.004
0.615 = 0.006
0.673 2: 0.000
0.631
0U000
MT-aaly-GWMAC
0.701 0.005
0.653
0.004
0.675 0.004
0.628 0.006
0.610 + 0.008
0.671 0.004
0.700
0.003
UR-PUNNY
MT-eudl- BegBN
MT AiT--oyy--a MIB
0,710 0.00
0.60
0,10S
0.082 0.006
0,632 0.005
0.611
= 0.003
0.095
0.001
0.692 0.003
MT eadl WMA
0.713 0.00
0.681
0.004
0.688
0.004
0.626 1 0.004
0.626 1 0.003
0.675
0.004
0,708
0.004
Table 4: Computational time a step for different methods on UR-FUNNY and MOSEI datasets under
early fusion and late fusion.
early fusion
late fusson
models/datasete
UR-FUNNY
MOSEI
UR-FUNNY
MOSEI
MT-late
2.364s
2,288s
1.735s
1.506s
MT-late-GWMAC
3.290s
2.801s
2.172s
2.054s
MT-late-RegBN
3.962s
3.287s
3.298s
2.990s
MT-late-MIB
3.476ss
2.947s
2.786s
2.409s
MT-late-WML
2.788s
2.570s
2.098s
1.920s
MT-early-CS
2.381s
2.312s
1.778s
1.554s
MT-carly-JS
2.398s
2.472s
1.816s
1.609s
MT-early-TV
2.409s
2.3358
1.983a
1.7618
5
Discussion
This paper primarily investigates the application of WMA to multimodal transformers in discriminative tasks. Additionally, we explore the potential applications of WMA in generative tasks. Through a compre- hensive literature survey, we observed that current generative multimodal large language models (MLLMs) continue to rely on the transformer architecture, supplemented by connectors that bridge different modality tokens for example, the MLP used in LLAVA Liu al. (2024) and the Q-Former used in BLIIP2 Li et al. (2023). We propose that our WMA method can be seamlessly integrated with these existing approaches for the following reasons:
Identical frameworks. Most generative MLLMs also use the transformer architecture, but involve a decoder on the output side. There is no architectural conflict for implementing the WMA in such MLLMs.
Non Marammrait approach. WMA is non-parametric method that does not require additional training, thereby avoiding any extra overhead in the training process of MLLMs.
Token-level implementation. WMA operates at the token level within the input to the transformer, ensuring compatibility with the connectors employed in MLLMs.
We believe that further regularization of MILMs using WMA will enhance their performance as well robustness, representing significant direction for future work.
12

---

Published in Transactions on Machine Learning Research (01/2025)
specific characteristic in hiring, credit scoring system Geirhos et al. (2020); ; Adewumi et al. (2024). Here We would like to discuss the impact of the proposed methodology on this issue. From Fig. 2 we observe that MT-early heavily relies on text for the decision due to the sum of the Integrated gradients values of the text token 690.63 is much higher than that of the image loken 113.28. The RegBN method alleviates the problem slightly with the numbers 602.70 and 143.68. Our proposed WMA method solves this problem the contribution of text token and image token is almost identical 811.38 and 702.77. In WMA, the image modality has more Gaussian-like attribution value distribution compared with others, meaning that the model exploits the weak modality to learn the task. Through this case study, we demonstrate that the proposed WMA approach enables the model to allocate more equitable attention to features across different modalities, rather than concentrating on data from a single modality. Our proposed method effectively mitigales bias arising from amplified data in multimodal systems.
Multimodal systems can amplify biases in the data, i... focusing
6 Conclusion
This paper addresses pivotal challenge multimodal transformers: the absence of a modality alignment approach during the fine-tuning phase. We introduce a Wasserstein distance-based regularization method to adjust the modality alignment degree. The proposed method does not require training more parameters and can be easily integrated into the multimodal transformer. The experimental results demonstrate significant improvements on performance and especially robustness on both 2-modalities and 3-modalities tasks, as well as for early and late fusion paradigms. In early fusion paradigm, the averaged improvement of our method compared with the best result from all baselines over all datasets are 1.4%, 1.5%, 1.4%, 0.7% and 1.4%, respectively. For the late fusion paradigm, the improvements are 1.5%, 1.8%. 1.3%, 2.3% and 1.9%. Meanwhile, our experimental results show that modality alignment needs to be task-dependent, rather than forced alignment, i.e., minimizing the OT distance between modalities, which provides valuable insights for related work. Our future work will focus more theoretical analyses of our proposed method.
References
Tosin Adewumi, Lama Alkhaled, Namrata Gurung, Goya van Boven, and Trene Pagliai. Fairness and bias in multimodal ai: A survey, XrXiv preprint arXiv:2406. 19097, 2024.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 11-37766-33736, 2022.
John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multimodal units for information fusion. arXiv preprinl Xrxi::1102 01992, 2017.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2611):665 673, 2020.
Morteza Ghahremani Boozandani and Christian Wachinger. Regbn: Batch normalization of multimodal data with regularization. Advances in Neural Information Processing Systems, 36, 2024.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
Fengjiao Gong, Yuzhou Nie, and Hongteng Xu. Gromov In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 603-613, 2022.
molti-modal alignment and clustering.
Md Kamrul Hasan, Wasifur Rahman, Amir Zadch, Jianyuan Zhong, Md Iftckhar Tanvoer, Louis- Louis-Philippe Morency, et al. Ur-funny: A multimodal language dataset for understanding humor. ar preprint 1904:0190.00611 2019.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Nasir Hayat, Krzysztof J Geras, and Farah E Shamout. Medfuse: Multi-modal fusion with clinical time-series data and chest x-ray images. In Machine Learning for Healthoore Conference, pp. PML.R, 2022a.
Nasir Hayat, Krzysatof J. Geras, and Farah E. Shamout. Medfuse: Multi-modal fusion with clinical time- series data and chest x-ray images, 2022b. URL
Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi- modal learning belter than single (provably). Advances in Neural Information Processing Systems, 34: 2021.
Md Mofijul Islam and Tariq Iqbal. Mumu: Cooperative multitask learning-based guided multimodal fusion. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 1043 1051, 2022.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, 33:2611 12221126244--1 2020.
Eun-Sol Kim, Woo Young Kang, Kyoung Woon On, Yu-Jung Heo, and Syoung-Tak Zhang, Hypergraph attention networks for multimodal learning. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognilion, pp. 14581 14581-14590, 2020.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Iision-and-languaee transformer without convolution or region supervision. In International Conference on Machine Learning, pp. 5583-5500 PMLR, 2021.
Yi-Lam Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen-Yu Lee. Multimodal prompting with missing modal- itics for visual recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognilion, pp. 14943 14952, 2023.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joly, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705, 2021.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19740-19742. PMLR, 2023.
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Yufan Chen, Peter Wu, Michelle Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. In Thirly-fifth Conference on Neural Information Processing Systems Dalnsels and Benchmarks Track (Round 1), 2021.
Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nic, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, et al. Quantifying & modeling multimodal interactions: An information decomposition framework. Advances in Neural Information Processing Systems, 36, 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 26296 26306, 2024.
Zhou Lu. A theory of multimodal learning. In Thirty-seventh Conference Neural Information Processing http://wwwrriii...oo.ooooooaaaaa
Systems, 2023. URL
Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learn- ing with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2302 2310, 2021.
Sijie Mai, Ying Zeng, and Haifeng Hu. Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations. IEF. Transactions Mullimedia, 2022.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Arsha Nagrani, Shan Yang, Amurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances in neural information processing systems, 34:14200- 142113 2021.
George Papandreou, Athanassios Katsamanis, Vassilis Pitsikalis, and Petros Maragos. Adaptive multimodal fusion by uncertainty compensation with application to audiovisual speech recognition. IEEE Transactions on Audio, Speech, und Language Processing, 17(3):423 435, 2009.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications data science. Foundalions and Trendst in Machine Learning, (1(56)):355 607, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp.
PMLR, 2017.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXic preprint arXiv:2108. 2021.
Yujia Xie, Xiangfeng Wang, Ruijia Wang, and Hongyuan Zha. A fast proximal point method for computing exact wasserstein distance. ar preprint Xix: 102..882.000 04307, 2018.
Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. Multi- attention recurrent network for human communication comprehension. Shcrtt-Second AAAI Conference on Artificial Intelligence, 2018.
Zhuo Zhi, Ziquan Liu, Moe Elbadawi, Adam Daneshmend, Mine Orlu, Abdul Basit, Andreas Demosthenous, and Miguel Rodrigues. Borrowing treasures from neighbors: In-context learning for multimodal learning with missing modalities and data scarcity. Xin preprint arXiv:2403.004 2024.
15

---

Published in Transactions on Machine Learning Research (01/2025)
A More experimental results
We report all the scarching results under various combination of a and 3 for all datascts in Table 8 for early fusion and g 12 for late fusion.
Table 5: Results of different a and B on Hateful Mernss dataset under early fusion. Bold indicates better or equal performance than MTrearly, and red font is the weight we select.
test
[ 0.1
0.2
P= 0.4
A
A=
3
A 1.6
1.8
0 2
10
normal
0.735
0.728
0.731
0.735
0.726
0.789
0.731
0.728
0.725
0.724
0.733
0.721
0.732
0.714
image polse
0.373
0.564
0.560
0.565
0.557
0.553
0.565
0.555
0.570
0.568
0.564
0.582
0.565
0.668
0.1
text noise
0.686
0.682
0.686
0.086
0.676
0.683
0.679
0.083
0.609
0.965
0.681
0.068
0.683
0.061
image missing
0.611
0.618
0.614
0.618
0.625
0.651
0.627
0.021
0.018
0.617
0.612
0.602
0.606
0.619
text missing
0.671
0.665
0.666
0.668
0.657
0.664
0.664
0.666
0.655
0.654
0.657
0.653
0.659
0.648
normal
0.726
0.731
0.732
0.731
0.738
0.735
0.736
0.732
0.736
0.740
0.735
0.726
0.717
0.721
image poice
0.571
0.572
0.581
0.554
0.553
0.563
0.584
0,571
0.532
0.564
0.556
0.548
0.2
text noise
0.678
0.685
0.682
0.686
0.692
0.684
0.689
0.676
0.682
0.6855
0.679
0.676
0.677
0.680
Image misding
0.600
0.006
0.610
0.617
0.615
0.635
0.635
0.643
0.635
0.642
0.647
0.615
0.006
0.634
text missing
0.659
0.004
0.062
0.066
0.069
0.669
0.667
0.668
0.667
0.062
0.658
0.662
0.648
0.654
normal
0.724
0.730
0.729
0.729
0.732
0.730
0.730
0.735
0.736
0,740
0.748
0.741
0.725
0.730
image coice
0.574
0.571
0.557
0.567
0.580
0.563
0.573
0.556
0.568
0.552
0.583
0.556
0.561
0.562
text noise
0.674
0.688
0.684
0.682
0.677
0,684
0.670
0.681
0,680
0.677
0.682
0.676
0.070
0.675
Image missing
0.621
0.608
0.610
0.613
0.613
0.618
0.620
0.624
0.626
0.625
0.645
0.620
0.007
0.616
text missing
0.658
0.609
0.066
0.670
0,606
0.663
0.662
0.662
0,660
0.656
0.659
0.665
0.648
0.647
normal
0.728
0.721
0.727
0.733
0.732
0.734
0.738
0.733
0.735
0.721
0.741
0.737
0,731
0.728
imago noier
0.577
0.578
0.575
0.561
0.571
0.574
0.569
0.546
0.564
0.562
0.575
0.559
0.577
0.546
text noise
0.670
0.684
0.063
0.680
0.686
0.689
0.676
0.681
0.684
0.675
0.689
0.673
0.665
0.675
image miesing
0.021
0.612
0.621
0.623
(1.03
0.821
0.632
0.817
0.621
0,803
0.624
0.026
0.646
0.621
lexl missing
0.658
0.652
0.660
0,861
0.662
0,668
0.660
0.657
0,866
0.652
0,663
0.652
0.655
0.667
nocmad 0.720
0.726
0.721
0.718
0.730
0.724
0.732
0.726
0.724
0.732
0.736
0.732
0.716
0.705
imago noise
0.576
0.563
0.580
0.540
0.157
0.555
0.565
0.171
0.555
0.570
0.676
0.5677
0.577
0.529
noise
0.672
0.607
0.068
0.663
0.676
0.674
0.681
0.677
0.671
0.682
0.669
0.673
0.669
0.655
image minsing
0.634
0.623
0.611
0.619
0.041
0.615
0.611
0.632
0.643
0.628
0.620
0.019
0.651
0.612
missing
0.661
0.660
0.651
0.647
0.662
0.658
0.687
0.055
0.660
0.652
0.659
0.658
0.636
Table 6: Results of different a and 8 on MM-IMDb dataset under early fusion. equal performance than MT-carly, and red font is the weight we select.
Bold indicates better or
Lest A 0
8= 0.1
==02
B
8 0.6
8 0.8
8
1
1.2
$
A 1.6
== 188
2
10
normal
0.552
0.549
0.548
0.551
0.552
0.551
0.550
0.550
0,550
0.553
0.548
0.551
0.546
0.551
image anise
0.289
0.317
0.316
0.304
0.319
0.298
0.252
0.284
0.314
0,273
0.263
0.227
0.1
lest noise
0.461
0.456
0.456
0.464
0.161
0:458
0.457
0.456
0.159
0.464
0.454
0.458
0.118
0.158
image missing,
0.464
0.462
01.466
0.456
0L470
0.463
0.460
0.472
0.454
0.477
0.470
0.476
0.451
texi missing
0.122
0.418
0.415
0.414
0.112
0.414
0.408
0.412
0.114
0.417
0.410
0.411
0.112
0-413
normal
0.549
0.552
0.552
0.350
0.550
0.551
0.350
0.551
0.548
0.351
0.548
0.350
0.550
0.548
Image walse
0.326
0.329
0.271
0,292
01321
0.226
0.275
0,281
0.326
0.288
0.294
0.323
0.176
0.225
0.2
text noise
0.461
0.465
0.462
0.461
0.461
0.461
0.458
0.459
0.456
0.459
0.455
0.450
0.452
0.462
Image missing
0.465
0.457
01.474
0.468
(L.666
0.466
0.461
0.480
0l.422
0.474
0.480
0.484
0.446
0.452
text missing
0.417
0.419
0.422
0.412
0.415
0.413
0.412
0.418
0.412
0.417
0.410
0.407
0.406
0.421
normal
0.551
0.553
0.552
0.548
0.552
0.549
0.550
0.549
0.550
0.546
0.545
0.544
0.546
0.545
image aoise
0.309
0,276
0.286
0.276
0,296
0.237
0.329
0.247
0,270
0,269
0.272
0.235
0.227
0.291
0
text noise
0.462
0.459
0.458
0.458
0.462
0.459
0.457
0.457
0.459
0.449
0.445
0.449
0.456
0.452
image missing
0.455
0.414
0.153
0.451
0.470
0.466
0.462
0.469
0,161
0.450
0.472
0.466
0,459
0.111
fexi missing
0.115
0.415
0.415
0.400
0.421
0.415
0.416
0.410
0-115
0.412
0.413
0.430
0.414
0.108
nornal
0.555
0.558
0.553
0.553
0.548
0.550
0.351
0.549
0.545
0.544
0.545
0.545
0.544
0.544
image auise
0.263
0.267
0.226
0.279
0.276
0.335
0.344
0.257
0.250
0.278
0,271
0.206
0.279
0,216
text noise
0.461
0.470
0. 0.160
0.4600
0.455
0:454
0.455
0.454
0,162
0.452
0.463
0.451
0.418
0.450
Image missing
0.470
0.459
0L.668
0.470
0.472
0.476
0.488
0.486
0..42
0.481
0.476
0.465
0.441
0.447
text missing
0.420
0.432
0.423
0.416
0.417
0.414
0.419
0.417
0.4133
0.411
0.416
0.419
0.400
0.406
normal
0.555
0.563
0.553
0.554
0.554
0.548
0.352
0.551
0.548
0.544
0.548
0.550
0.549
0.540
Image wolse
0.282
0.355
0.254
0.332
0,243
0:230
0.238
0.250
0.219
0.298
0.206
0.222
0.284
0.215
text noise
0.469
0.471
0.463
0.461
0.464
0.457
0.456
0.482
0.461
0.453
0.451
0.455
0.456
0.433
image missing
0.173
0.477
0.481
0.473
0.469
0.169
0.474
0.470
0.470
0.476
0.475
0.479
0.169
0.135
text missing
0.430
0.437
0.419
0.421
0.420
0.405
0.415
0.416
0.425
0.413
0.427
0.425
0.415
0.396
16

---

Published in Transactions on Machine Learning Research (01/2025)
Table 7: Results of different a and 8 UR-FUNNY dataset under early fusion. Bold indicates better
equal performance than MT-early, and red font is the weight select.
test case
8 0
0.1
8 2--00
0.2
0.4
3
0.6
8 -
0.8
88-
1.2
3
1.4
1.8
8 1.8
8
3
10
normal
0,702
0.701
0.701
0.700
0.608
0.700
0.702
0.704
0.704
0.659
0.639
0.699
0.608
0.037
hmagg noise
0.646
0.648
0.643
0.649
0.647
0.654
0.662
0.663
0.652
0.651
0.648
0.654
0.656
0.657
0.1
text
0,680
0.682
0.681
0.681
0,680
0.679
0.684
0.683
0.686
0.681
0,680
0.674
0.683
0.670
audio moise
0,630
0.633
0.630
0.635
0.630
0.633
0.626
0.631
0,633
0.629
0.627
0.635
0.643
0.631
image missing
0.608
0.608
0.609
0.606
0.6009
0,608
0.813
0.611
0.614
0.608
0.612
0.597
0.611
0.601
tex missing
0.673
0.682
0.684
0.689
0.674
0.068
0.882
0.678
0.680
0.670
0.682
0.647
0.066
0.088
audio missing
0.700
0.699
0.697
0.006
0.694
0.697
0.697
0.699
0.701
0.606
0.005
0.801
0.996
0.002
normal
0.609
0.701
0.600
0.701
0.608
0.699
0.704
0.704
0.702
0.702
0.009
0.702
0.705
0.700
hmage noise
0.644
0.648
0.653
0.653
0.658
0.648
0.645
0.650
0.651
0.655
0.658
0.643
0.646
0.662
text noise
0.680
0.682
0.678
0.683
0.683
0.684
0.681
0.683
0,680
0.678
0.682
0.681
0.672
0.681
audio noise
0.626
0.630
0.632
0.636
0.638
0.631
0.626
0.629
0.127
0.632
0.632
0.628
0.620
0.635
image missing
0.611
0.610
0.627
tex: missing
0.682
0.685
0.677
0.680
0.670
0.677
0.878
0.672
0.683
0.672
0.689
0.676
0.686
0.689
audio missing
0.695
0.698
0.695
0.697
0.695
0.695
0.699
0.699
0.697
0.639
0.096
0.698
0.700
0.006
normal
0.705
0.700
0.702
0.609
0.608
0.701
0.701
0.805
0.702
0.705
0.702
0.705
0.704
0.704
Image 0.661
0.665
0.653
0.648
0.656
0.638
0.660
0.649
0.657
0.666
0.671
text noise
0.679
0,683
0.679
0.681
0.679
0.683
0.679
0.876
0,680
0.667
0.686
0.688
0.667
0.686
andio noise
0.633
0.643
0.634
0.635
0.627
0.634
0.631
0.629
0.634
0.636
0.630
0.639
0.640
0.633
image missing
0.628
0.808
0..001
0.607
0.608
0.607
0.613
0.802
0.609
0.583
0.611
0.616
0.621
0.620
tex: uissing
0,638
0.664
0.675
0.667
0.681
0.682
0.653
0.676
0.675
0.582
0.671
0,680
0.631
0.647
andio missing
0.699
0.697
0.638
0.605
0.696
0.698
0.697
0.600
0.699
0.704
0.698
0.702
0.697
0.700
normal
0.704
0.685
0,600
0.007
0.637
0.706
0.702
0.705
0.701
0.702
0.712
0.609
0.704
0.617
image noise
0.664
0.671
0.607
0.062
0.605
0.655
0.652
0.657
0.681
0.671
0.671
0.614
text nolse
0.675
0.661
0.676
0.679
0.624
0.656
0.679
0.682
0.685
0.688
0.686
0.684
0.681
0.621
audio noise
0.631
0.644
0.637
0.633
0.632
0.643
0.637
0.634
0.598
image massing
0.601
0.606
0.611
0.608
0.555
0.619
0.601
0.609
0,020
0,623
0.624
0.605
0.620
0.549
texi missing
0:658
0.669
0.672
0.643
0.555
audio missing
0.703
0.693
0.606
0.625
0.618
0.702
0.697
0.702
0.698
0.698
0.703
0.696
0.700
0.506
normal
0.608
0.7014
0.609
0.701
0.700
0.703
0.609
0.701
0.704
0.696
0.608
0.709
image noise
0.665
0.675
0.667
0.653
0.669
0.659
0.642
0.062
0.6G8
0.671
0.066
0.671
0.668
0.063
text
0.676
0.681
0.685
0.696
0.675
0.679
0.680
0.683
0.650
0.682
0.674
0.064
andio noise
0.637
0.649
0.644
0.631
0.645
0.647
0.629
0.635
0.635
0.638
0.637
0.634
0.661
0.620
image missing
0.605
0.029
0.613
0.611
0.001
0,618
0.600
0.643
0,621
0.619
0.612
0.626
0.639
0.602
texi miseing
0,650
0.667
0.673
0.652
0.657
0.665
0.875
0.662
0.658
0.692
0.651
0.662
0.626
0.646
audio missing,
0,6855
0.699
0.606
0.605
0.699
0.697
0.683
0.699
0.696
0.606
0.700
0.682
0.687
0.706
17

---

Published in Transactions on Machine Learning Research (01/2025)
Table 8: Results of different and / on MOSEI dataset under early fusion. Bold indicates better or equal
performance than MTr-aarly, and red font is the weight we select.
test case
8 3-60 0
0.1
B 2--00
0.2
0.4
A -
0.6
0.8
3
1
1.2
1.4
1.6
8 1.8
2
4
B 10
normal
0.068
0.628
0.647
0.627
0.624
0.626
0.639
0.647
0.622
0.634
0.635
0.634
0.605
0.709
image moise
0.725
0.897
0.698
0.700
0.678
0.725
0.209
0.731
0.741
0.208
0.724
0.739
0.737
0.784
0.1 text noise
0.849
0.863
0.844
0.813
0.828
0.832
0.833
0.839
0.822
0,857
0.826
0,825
0.827
0.903
aadso nosse
0.691
0.678
0.674
0.685
0.653
0.689
0.096
0.693
0.678
0.685
0.679
0.092
1,890
0.709
image missing
0.678
0.635
0.653
0.612
0,629
0.635
0.648
0.652
0.632
0.655
0.046
0.616
0,044
0.771
text missing
0.829
0.835
0.528
0.813
0.818
0.825
0.821
0.825
0.819
0.838
0.821
0.818
0.927
0.872
audio miseing
0.669
0,630
0.651
0.628
0.628
0.625
0.610
0.619
0.624
0.653
0.056
0.633
0.050
0.792
normal
0.624
0.628
0.631
0.683
0.630
0.626
0.622
0.628
0.657
0.636
0.623
0.629
0.652
0.635
image moise
U.722
0.686
0.691
0.602
0.689
0.735
0.703
0.680
0.737
0.741
0.720
0. 0.708
0.721
0.745
0.2
text noise
0.831
0.831
0.830
0.852
0.824
0.823
0.817
0.850
0.912
0.830
0.816
0.811
0.847
0.830
wadio noise
0.686
0.654
0.681
0.674
0.679
0.687
0.601
0.695
0.731
0.602
0.657
0.668
0.687
0.095
Image missing
0.634
0.635
0.637
0.653
0,632
0.635
0.633
0.633
0.657
0.645
0.636
0.641
0.656
0.644
text missing
0.822
0.821
0.817
0.830
0.820
0.817
0.852
0.827
0.568
0.821
0.820
0.821
0.527
0.822
audio missing
0.628
0.640
0.640
0.650
0.629
0.627
0.628
0.636
0.656
0.641
0.634
0.638
0.668
0.652
normal
0.624
0.689
0.628
0.615
0.630
0.833
0.632
0.640
0.621
0.642
0.642
0.630
0.653
0.829
image anise
0.711
0.740
0.732
0.721
0.088
0.838
0.724
0.707
0.710
0.753
0.715
0.755
0,686
0.834
0. 1
test noise
0.855
0.833
0.841
0.827
0.845
0.830
0.827
0. 0.838
0.821
0.822
0.821
0,969
0.843
0.836
andio nosse
0.676
0.684
0.662
0.655
0.839
0.679
0.664
0.671
0.664
0.679
0.712
0.676
0.834
imnge missing
0.629
0.691
0.635
0.621
0.640
0.833
0.640
0.617
0.627
0.655
0.648
0.658
0.656
0.830
fext missing
0.831
0.832
0.817
0.826
0.835
0.817
0.824
0.820
0.819
0.816
0.876
0.835
audio missing
0.623
0.690
0.627
0.621
0.637
0.837
0.634
0.646
0.634
0.646
0.658
0.649
0.718
0.900
normal
0.632
0.658
0.615
0.631
0.627
0.624
0.613
0.624
0.627
0.638
0.610
0.612
0,633
0.832
image woise
0.743
0.710
0.676
0.710
0.691
0.687
0.697
0.714
0.707
0.716
0.743
0.736
0.686
0.834
text noise
0.818
0.865
0.824
0.835
0.849
0.841
0.839
0.837
0.852
0.890
0.871
0.855
0.823
0.834
wadso nosse
0.685
0.685
0.684
0.678
0.689
0.606
0.663
0.675
0.678
0.680
0.659
0.095
0.664
0.834
Image missing
0.643
0.680
0.619
0.636
0.635
0.635
0.653
0.631
0.637
0.687
0.652
0.632
0.643
0.833
text missing
0.812
0.842
0.813
0.821
0.529
0.824
0.826
0.824
0.530
0.836
0.944
0.871
0.910
0.834
audio miseing
0.630
0.651
0.623
0.639
0.641
0.629
0.616
0,633
0.628
0.658
0.611
0,613
0.897
0.913
normal
0.629
0.626
016655
0.619
0.627
0.627
0.631
0.631
0.622
0.630
0.032
0.621
0.648
0.832
image noise
0.207
0.717
0.718
0.728
0.740
0.735
0.700
0.720
0.698
0.715
0.717
0,703
0,705
0.833
a= = 10
text noise
0.831
0.815
0.860
0.825
0.816
0.824
0.810
0.820
0.529
0.911
0.531
0.831
0.823
0.833
watio noise
0.665
0.661
0.674
0.667
0.696
0.602
0:681
0.674
0.680
0.674
0.699
0.667
0.682
0.33
image missing
0.632
0.629
0.661
0.626
0.631
0.643
0.634
0.636
0.626
0.637
0.639
0.625
0.665
0.832
teel missing
0.822
0.809
0.838
0.814
0.812
0.817
0.828
0.816
0.820
0.800
0.829
0,838
0.904
0.832
audio missing
0.628
0.626
0.663
0.628
0.626
0.641
0.647
0.637
0.631
0.630
0.634
0.624
1.029
0.834
Table 9: Results of different a and ,3 on Hateful Memes dataset under late fusion. Bold indicates better or
equal performance than MT-larly, and red font is the weight we select.
test oise
A 0
á
0.1
/ 0.2
0.4
A =
0.6
0.8
3
1
J 3= 1.2
1.4
A
1.6
8 A= 1.8
8
2
8
4
P 10
normal
0.685
0.724
0.725
0.722
0.722
0.723
0.680
0.680
0.681
0.710
0.707
0.721
0.709
0.686
image aoise
0.661
0.666
0.666
0.663
0.662
0.662
0.658
0.658
0.659
0.661
0.659
0.672
0.654
0.685
0.1
text noise
B
0.670
0.672
0.670
0.671
0.671
0:633
0.632
0.633
0.634
0.663
0.668
0.652
0.635
image missing
0.674
0.690
0.682
0.680
0.674
0.687
0.671
0.672
0.672
0.687
0.685
0.692
0.678
0.673
text missing
0.583
0.636
0.636
0.633
0.632
0.610
0:581
0.579
0.580
0.004
0.504
0.614
0.606
0.576
normal
0.685
0.684
0.684
0.685
0.724
0.723
0.680
0.681
0,708
0.722
0.723
0.721
0.707
0.715
image anise
0.647
0.600
0.660
0.661
0.605
0.656
0.058
0.659
0.660
0.672
0.674
0.675
0.664
0.674
0.2
text noise
0.650
0.635
0.636
0.636
0.672
0.671
0.653
0.633
0.658
0.670
0.669
0.666
0.058
0.661
image missing
0.648
0.673
0.674
0.684
0.683
0.671
0.671
0.650
0.697
0.696
0.694
0.681
0.683
text missing
0.607
0.583
0.583
0.583
0.629
0.630
0.580
0.581
0.560
0.606
0.618
0,609
0.622
0.609
normal
0.719
0.712
0.717
0.722
0.661
0.690
0.728
0.723
0.716
0.718
0.058
0.722
0.589
0.591
imsge muise
0.664
0.652
0.657
0.665
0.661
0.659
0.671
0.672
0.669
0.663
0.664
0.675
0.569
0.671
text noise
0.668
0.607
0.673
0.635
0.633
0.675
0.670
0.662
0.670
0.637
0,661
0.507
0.574
imnge missing
0.678
0.675
0.673
0.680
0.674
0.672
0.697
0.694
0.693
0.684
0.668
0.684
0.577
0.677
text missing
0.632
0.609
0.625
0.623
0.582
0.580
0.613
0.623
0.604
0.616
0.584
0.624
0.562
0.567
normal
0.716
0.717
0.717
0.713
0.717
0.712
0.709
0.722
0.728
0.719
0.676
0.580
0.599
0.601
imsag
0.650
0.671
0.674
0.661
0.659
0.684
0.660
0.676
0.686
0.656
0.659
0.370
0.569
0.560
0
test noise
0.688
0.659
0.600
0.658
0.663
0.669
0.661
0.686
0.664
0.664
0.626
0.567
0.580
0.584
image missing
0.678
0.691
0.695
0.680
0.684
0.687
0.685
0.695
0.678
0.685
0.667
0.577
0.570
0.549
text miscing
0.626
0.626
0.631
0.614
0.617
0.599
0.603
0.615
0.597
0.625
0.583
0.562
0.575
0.581
normal
0.724
0.728
0.720
0.717
0.681
0.681
0.711
0.721
0.715
0.713
0.590
0.501
0.601
0.597
image moise
0.062
0.671
0.660
0.671
0.660
0.661
0.600
0.676
0.657
0.654
0.570
0.570
0.564
0.551
10
text noise
0.663
0.667
0.658
0.680
0.632
0.631
0.661
0.685
0.662
0.685
0.568
0.570
0.583
0.582
image miseing
0.708
0.692
0.682
0.694
0.674
0.671
0.680
0.692
0.680
0.675
0.577
0.578
0.556
0.537
text missing
0.646
0.663
0.632
0.626
0.584
0.583
0.606
0.620
0.626
0.625
0.561
0.365
0.580
0.578
18

---

Published in Transactions on Machine Learning Research (01/2025)
Table 10: Results of different a and 33 on MM-IMDb dataset under late fusion. Bold indicates better or
equal performance than MT-early, and red font is the weight we select.
leel case
3= 0
=
0.1
_ 0.2
/= 0.4
8 0.6
8 =
0.8
8=1
== 1.2
3= 1.4
== 1.6
== 1.8
==2
8 4
[ 10
noruaa
0.603
0.603
0.602
0.603
0.602
0.602
0.602
0.601
0.601
0.601
0.601
0.601
0.602
0,600
haage noise
0.470
0.470
0.470
0.471
0.473
0.474
0.474
0.473
0.471
0.460
0.468
0.468
0.450
0.427
text noise
0.498
0.496
0.495
0.496
0.495
0.495
0.494
0.494
0.494
0.495
0.495
0.495
0.497
0.493
image massing
0.357
0.557
0.537
0.557
0.558
0.539
0.537
0.554
0.354
0.554
0.354
0.354
0.556
0.555
Less wnisiing
0.415
0.416
0.417
0.416
0.418
0.419
0.419
0.417
0.417
0.417
0.117
0.117
0.417
0.111
noruaal
0.603
0.603
0.603
0.603
0.600
0.602
0.602
0.801
0.602
0.600
0.600
0.601
0.601
0.599
Image noise
0.472
0.470
0,472
0.471
0.471
0.474
0.475
0,472
0.009
0.467
0,464
0,462
0.438
0.428
0.2
lexi mike
0.498
0.497
0.498
0.495
0.492
0.191
0.491
0.495
0.495
0.496
0.196
0.196
0.196
0.186
image missing
0.555
0.555
0.556
0.557
0.555
0.558
0.558
0.554
0.554
0.553
0.653
0.560
0.656
text missing
0.416
0.413
0.416
0.417
0.416
0.419
0.419
0.417
0.418
0.416
0.417
0.417
0.417
0.420
normal
0.609
0.606
0.605
0.603
0.601
0.602
0.602
0.600
0.602
0.603
0.602
0.002
0.601
0.508
image noise
0.463
0.470
0.472
0.472
0.472
0.473
0.474
0.489
0.484
0.465
0.468
0.461
0.441
0.422
lexi moiee
0.506
0.505
0.502
0.499
0.194
0.491
0.496
0.495
0.497
0.498
0.197
0.196
0.186
0:477
image missing
0.554
0.554
0.554
0.556
0.554
0.557
0.554
0.552
0,562
0.554
0.554
0.553
0.549
0.531
tex, unieiing
0.415
0.417
0.418
0.417
0.416
0.418
0.417
0.417
0.418
0.419
0.118
0.416
0.423
0,423
nuruaal
0.802
0.610
0.609
0.605
0.600
0.601
0.601
0.509
0.600
0.589
0.309
0.595
0.582
0.591
Emagg noise
0.411
0..489
0.463
0.474
0.474
0.475
0.473
0.465
0.461
0.463
0.455
0.442
0.414
0.400)
text noise
0.482
0.510
0.509
0.502
0.496
0.493
0.495
0.496
0.497
0.496
0.494
0.493
0.475
0.464
image missing
0.555
0.559
0.536
0.553
0.552
0.555
0.554
0.551
0,550
0.551
0.550
0.542
0.515
0,507
texs missing
0.397
0.426
0.419
0.420
0.416
0.416
0.416
0.416
0.415
0.113
0:412
0.401
0.411
D.113
noruaal
0.600
0.609
0.612
0.604
0.599
0.600
0.600
0.589
0.598
0.596
0.594
0.591
0.592
0.589
image noise
0.449
0.432
0.439
0.474
0.473
0.475
0.471
0,664
0,061
0.458
0.446
0.406
0.405
= 10
lexi moiee
0:488
0.501
0.508
0.502
0.497
0.191
0.195
0.496
0.496
0.492
0.491
0.485
0.168
0.160
image missing
0:553
0.556
0.555
0.532
0.351
0.554
0.549
0.549
0.538
0.610
0.507
text missing
0.393
0.414
0.422
0.420
0.416
0.414
0.416
0.415
0.413
0.411
0.401
0.408
0.413
0.410
Table 11: Results of different a and | on UR-FUNNY datasot under late fusion. Bold indicates better or
equal performance than MT-early, and red font is the weight we select.
test case
2= 0
3= 0.1
3 0.2
8
0.4
8 3- 0.6
0.8
8
1
5
1.2
5
1.4
5
166
8 1.8
5 2
2
10
normal
0.823
0.590
0.701
0.701
0.701
0.701
0.702
0.702
0.700
0.699
0.702
0.006
image noise
0.506
0.506
0.197
0.671
0.663
0.655
0.652
0.611
0.632
0.019
0.611
0.607
0.578
0.578
tedt
0.626
0.588
0.695
0.693
0.692
0.691
0.689
0.686
0.890
0.686
0.682
audio noke
0.629
0.622
0.586
0.700
0.700
0.699
0.699
0.700
0.700
0.699
0.697
0.694
0.693
0.689
image missing
0.551
0.548
0.540
0.655
0.656
0.657
0.658
0.658
0.654
0.650
0.651
0.644
0.646
0.637
text missing
0.814
0.606
0.575
0.652
0.653
0.646
0.647
0.649
0.651
0.649
0.647
0.657
0.672
0.608
audio missing
0.601
0.585
0.570
0.681
0.683
0.680
0.680
0.660
0.678
0.076
0,677
0.674
0.659
0.654
sorual
0.502
0.618
0.634
0.700
0.710
0.701
0.701
0.701
0.702
0.700
0.701
0.702
0.686
0.630
Imange nolao
0.485
0.504
0.505
0.682
0.660
0.654
0.651
0.644
0.633
0.620
0.606
0.603
0.579
0.513
text noise
0.592
0.620
0.637
0.696
0.698
0.693
0.692
0.691
0.688
0.696
0.692
0.690
0.682
0.034
audio noise
0.585
0.614
0.630
0.701
0.702
0.699
0.700
0.699
0.699
0.698
0,896
0.694
0.689
0.631
image missing
0.551
0.551
0.660
0.657
0.657
0.659
0.656
0.611
0.611
0.638
0.319
text missing
0.580
0.612
0.625
0.658
0.672
0.647
0.648
0.649
0.653
0.647
0.661
0.660
0.667
0.624
andio miscing
0.586
0.607
0.611
0.684
0.683
0.680
0.680
0.679
0.677
0.676
0.675
0.673
0.652
0.006
normal
0.804
0.617
0.629
0.816
0.701
0.703
0.700
0.704
0.700
0.702
0.706
0.702
0.603
0.306
image noise
0.492
0.492
0.492
0.493
0.060
0.055
0.651
0.644
0.632
0.819
0.601
0.590
0.510
0.508
a
text noise
0.611
0.621
0.637
0.621
0.694
0.693
0.695
0.690
0.687
0.693
0.693
0.687
0.603
0.596
audio noise
0.586
0.607
0.615
0.607
0.701
0.700
0.700
0.701
0.697
0.698
0.700
0.696
0.600
0.300
image missing
0.542
0.346
0.540
0.547
0.655
0.655
0.656
0.655
0.643
0.641
0.640
0.643
0.544
0.542
text missing
0.588
0.597
0.606
0.604
0.648
0.648
0.649
0.654
0.651
0.661
0.066
0.671
0.597
0.588
andio missing
0.594
0.604
0.614
0.615
0.680
0.678
0.681
0.677
0.679
0.675
0.666
0.658
0.605
0.507
cormal
0.013
0.802
0.595
0.699
0.701
0.706
0.701
0.701
0.704
0.706
0.635
0.636
0.591
0.390
image noise
0-402
0.483
0.680
0.655
0.641
0.649
0.626
0.632
0.516
0.315
0.507
0.507
5
test noise
0.621
0.609
0.601
0.692
0.688
0.696
0.893
0.083
0.694
0.693
0.635
0.638
0.592
0.391
audio noler
0.605
0.504
0.587
0.699
0.700
0.703
0.701
0.697
0.700
0.700
0.629
0.631
0.584
0.382
image missing
0.554
0.350
0.545
0.637
0.652
0.658
0.654
0.651
0.651
0.637
0.546
0.547
0.540
0.339
text missing
0.606
0.596
0.587
0.630
0.631
0.050
0.645
0.638
0.667
0.065
0.614
0.620
0.583
0.583
audio missing
0.008
0.599
0.587
0.681
0.670
0.676
0.678
0.674
0.008
0.850
0.601
0.003
0.583
0.393
normal
0.586
0.591
0.610
0.595
0.702
0.702
0.702
0.704
0.704
0.638
0.631
0.597
0.591
0.590
iunge noise
0.187
0-488
0.490
0.496
0.641
0.625
0.647
0.621
0.621
0.502
0.515
0.508
0.506
0.306
10
trxt noise
0.592
0.509
0.624
0.601
0.686
0.685
0.694
0.683
0.692
0.544
0.634
0.297
0.501
0.301
audio noise
0.578
0.586
0.605
0.587
0.703
0.697
0.699
0.699
0.702
0.632
0.592
0.582
0.582
image miseing
0.533
0.5d1
0.517
0.541
0.651
0.656
0.664
0.619
0.617
0.553
0.545
0.542
0.539
0.539
missing
0.585
0.600
0.626
0.631
0.611
0.659
0.620
0.388
0.583
0.383
andio missingg
0.582
0.388
0.606
0.586
0.665
0.671
0.682
0.668
0.658
0.631
0.602
0.500
0.503
0.300
19

---

Published in Transactions on Machine Learning Research (01/2025)
Table 12: Results of different a and 3 MOSEI dataset under late fusion. Bold indicates better or equal
performance than MT-early, and red font is the weight we select.
test case
B- 0
A-00 0.1
/
0.2
$ ---0 0.4
0.6
0.9
/ 1
5
1.2
3
1.4
B -
1.6
B
1.8
2
8 --4
10
normal
0.748
0.624
0.652
0.750
0.702
0.617
0.628
0.777
0.724
0.634
0,621
0.774
0.740
0.620
image noise
0.775
0.642
0.673
0.874
0.718
0.619
0.663
0.878
0.764
0.655
0.863
0.807
0.819
0.715
0.1 cedt noise
0.811
0.823
0.814
0.815
0.819
0,840
0.810
0.849
0.820
0.531
0.819
0.812
1.007
0.809
audio noise
0.761
0.627
0.655
8.775
0.704
0.618
0.633
0.772
0.723
0.640
0.623
0.780
0.739
0.622
image missing
0.748
0.626
0.658
0.797
0.708
0.619
0.632
0.797
0.728
0.639
0.629
0.781
0.741
0.632
text missing
0,812
0.814
0.812
0.848
0.820
0.852
0.810
0.856
0,825
0,827
0.817
0.813
1,011
0.810
audio missing
0.753
0.625
016511
0,796
0.707
0.620
0.627
0.795
0.726
0.632
0.023
0.773
0.742
0.624
sornal
0.653
0.630
0.781
0.721
0.730
0.712
0.629
0,630
0.645
0,617
0.625
0.628
lunge noise
0.686
0.659
0.855
8.77
0.727
0.742
0.746
0.608
0.086
0.724
0.688
0.659
0.067
0.771
0.2
text noise
0.825
0.818
0.864
0.837
0.852
0.884
0.824
0.818
0.816
0.807
0.827
0.821
0.817
audio nosse
0.655
0.635
0.793
0.777
0.720
0.732
0.710
0.632
0.634
0.707
0.649
0.618
0.625
0.632
image missing
0.661
0.635
0,800
0.719
0.736
0.711
0.633
0.627
0.708
0.656
0,622
0.627
0.628
text missing
0.526
0.817
01.665
0.843
0.864
0.840
0.903
0.818
0.817
0.819
0.908
0.814
0.817
0.816
audio missing
0.656
0.629
0.508
0.799
0.727
0.729
0.718
0.637
0.632
0.711
0.613
0.622
0.628
0.628
normal
0.626
0.762
0.635
0.670
0.640
0.640
0.619
0.629
0.608
0.624
0.625
0.643
0.635
0.781
iunge noise
0.675
0.786
0.663
0.684
0.672
0.660
0.673
0.667
0.643
0.665
0,658
0.606
0.055
0.815
text nofar
0.816
0.823
0.807
0.814
0.819
0.835
0.815
0.815
0.807
0,818
0.822
0,829
0.820
0.808
audio nosse
0.629
0.764
0.637
0.674
0.641
0.644
0.622
0.631
0.615
0.623
0.630
0.647
0,640
0
image mireing
0.026
0.765
0.672
0.643
0.639
0.623
0.629
0.612
0.629
0.625
0.615
0.635
0.789
Lext missing
0.816
0.826
0.807
0.814
0.817
0.828
0.814
0.809
0.801
0.812
0.820
0.825
0.819
0.809
andio missing
0.633
0.739
0.642
0.623
0.638
0.609
0.627
0.628
0.642
0.631
0.790
normal
0.776
0.832
0.621
0.625
0.621
0.625
0.708
0.643
0.720
0L635
0.707
0.718
0.790
0.810
image nolac
0.779
0.852
0.670
0.662
0.677
0.648
0.732
0.678
0.816
0.886
0.721
0.754
0.842
0.832
a
text moise
0.549
0.533
0.810
0.821
0.813
0.814
0.812
0.852
1.008
0.815
0,819
0.806
0.809
0.811
audio noise
0.777
0.833
0.626
0.627
0.621
0.628
0.711
0.615
0.728
0.641
0.711
0.720
0.792
0.812
image missing,
0.774
0.835
0.627
0.635
0.634
0.631
0.718
0.646
0.721
0.642
0.708
0.718
0.708
0.820
text missing
0.851
01833
0.811
0.821
0.811
0.813
0.811
0.849
1.013
0.812
0.817
0.809
0.809
0,811
andoo missing
0.782
0.862
0.621
0.628
0.632
0.626
0.713
0.636
0.728
0.632
0.706
0.720
0.794
0.814
normal 0.5.0
0.841
0.628
0.731
0.645
0.742
0.628
0.663
0.629
0.6:0
0,616
0.712
0.736
0,817
image noise
0.840
0.749
0.670
0.759
0.847
0.608
0.075
0.071
0.742
0.831
a = 10
text boise
0.839
0.840
0.809
0.826
0.811
0.868
0.829
0.828
0.813
0.815
0.812
0.806
0.818
0.817
audloo noise
0.840
0.842
0.631
0.736
0.646
0.745
0.631
0.6688
0.635
0.647
0.618
0.713
0.744
0.819
image missing
0.855
0.556
0.632
0.738
0.647
0.742
0.626
0.642
0.631
0.637
0.621
0
0.760
0.825
text missing
0.889
0.640
0.811
0,838
0.814
0,850
0.820
0.824
0.812
0,811
0.811
0.509
0,818
0.817
audio missing 0.842
0.846
0.631
0.731
0.647
0,730
0.629
0.648
0.629
0.640
0.618
0.716
0.749
0.820
Table 13: Results of our proposed method with minimizing OT distance on all datasets under early fusion. The bold numbers mean the best performance. The bigger AUROC and Fl and smaller MAE refer to better performance.
noriall tos
test with mociality Bolse
sost with missing mociality
Datasets
Metrie
Methods
normal
/video
text
audio poise
Imange/ /sideo
missing
tex: maisting
andio miisiing
averaage
Hateful Menes
AUROC1
mamimize OT
0.72t
0.374
0.074
0.621
0.558
0.650
WMA
0.748
0.583
0.682
0.645
0.659
0.663
MDA-MDb
Fin
minimbre OT
0.351
0.300
0.462
0.453
0.45
0.438
WMA
0.563
0.355
0.471
0.477
0.437
0.161
UR-FUNNY
AUROC1
minimize OT
WMA
0,708
0.705
0.681
0.661
0.686
0.643
0.633
0.628
0.624
0.674
0,858
0.703
0.099
0.674
0.000
MOSEI
MAEL
malumizz OT
0.824
0.711
(1,850
0.676
0.629
0.531
0.623
0.707
WMA
0.615
0.676
0.824
0.664
0.619
0.813
0.623
0.691
Table 14: Results of our proposed method with different distance metrics all datasots under late fusion. The bold numbers mean the best performance. The bigger AUROC and F1 and smaller MAE refer to better performance.
normal Best
test with modalitt poise
Sost with maisting mociaity
Dittasets
Metric
Methods
normaa
Laage/ video nouse
text noise
audoo Doise
Imagee/ milsiig
tex: missing
modio missing
avessage
Hateful Metnes
AUROC1
mnmmmnrr OT
0.719
0.064
0.866
0.678
0.832
0.672
WMA
0.728
0.671
0.667
0.602
0.663
0.684
MM IMDb
FL1
ininimize OT
0,803
0.463
0.506
0.554
0.455
0.502
WMA
0.610
0.489
0.510
0.559
0.426
0.519
UR-PUNNY
AUROC1
mainiiize OT
0.604
0,452
0.811
0.585
0.542
0.588
0.501
0.575
WMA
0.710
0.060
0.698
0.702
0.680
0.672
0
0.084
MOSEI
MAEL
mamiiniz OT
0.825
0.675
0.810
0.029
0.525
0.518
1.089)
WMA
0.608
0.643
0.807
0.615
0.812
0.801
0.609
0.671
20

---

Published in Transactions on Machine Learning Research (01/2025)
Table 15: Results of our proposed method with different distance metrics on all datasets and test cases for -arlarly. The bold numbers mean the best performance. The bigger AUROC and and smaller MAE refer to better performance.
sormal less.
Fest roodullity
with missing modality
Untasets
Metese
Methods
notaaal
Imsage Video modse
tost: noise
nidlio moise
Image: Video miissing
text intislin
nurão unisilng
average
XTT-early-CS
0.742
0.575
0.674
0.656
0.672
0.664
Hateful Mienos
AUROCT
MT--arly-JS
0.745
0.500
0.673
0.635
0.605
0.856
MT WT-eaaly-WMA
0.748
0.583
0,642
0.643
0.600
0.083
3I early
0.551
012251
162
0,384
0,412
0.125
ATT-cadl--CS
0.569
0.473
0.451
MM IMD6
F11
KTT-oarl-JS
0.588
0.395
3.470
0.476
0.417
0445
MT-eady-TV
0.565
0.480
0.466
MT-aarly-MA
0.563
0.355
0.471
0.477
0.437
31T early
0.700
0.515
0.618
0.635
0.612
0.070
0.600
0.362
MT-eaady-CS
UR FUNNY
AUROC1
MI early
0.711
0,900
0.064
0.639
0.594
0.883
0.705
0.062
MT LiT-caryyTV
0.708
0.965
0.685
0.530
01.22
0.680
0.70M
0.671
MT-aadyyWMA
KTT-onlly
0.633
0.801
0.834
0.672
0.651
01926
0.636
0.706
MT-earty--C
0.624
0.677
0.805
0.671
0.651
0.821
0.633
0.607
MOSEI
MAE.
MTT-eall-TV
0.612
0.725
0.821
0.9G8
0.632
01822
0.636
0.702
MT early WMA
0.615
0.676
0.824
0.684
0.619
0.691
Table 16: Results of our proposed method with different distance metrics on all datasets and test cases for MT-late. The bold numbers mean the best performance. The bigger AUROC and F1 and smaller MAE refer better performance.
modialit mode
missing modality
Datnorts
Metrie
Methorks
neemna
Image//idiin molse
leict Inloe
Audlo noise
Imago/ Vidio missing
Last miissnng
audio missing
MT--lae
0.718
0.652
0.622
0.003
MTT-late-CS
0.728
0.675
0.687
0.621
0.578
Hateful Memes
AUROC1
Jate J8
0.721
0.666
0.674
0.661
0.639
0.673
MT WTT-ltee-'V
0.724
0.665
0.000
0.633
0.075
MT Jate WMA
0.728
0.671
0.067
0.682
0.663
0.684
MT--aaee
0.602
6.460
0.458
0.555
0.414
0.505
MAG-MDb
F11
MT-late-JS
0.608
0.475
0.500
0,550
0.421
0.511
MT Ia---ate--P TV
0.609
0.470
0.450
0.549
0.419
0.507
MTT-lats-WM.A
0.610
0.489
0.559
0.426
0.519
MTT-late
0.700
0.651
0.086
0UB93
0.654
0.661
0.076
0.672
MT-late-CS
0.702
0.661
0.050
0.676
UR-FUNNY
AUROCH
MT-late-JS
0.710
0.654
0.692
0.608
0.640
0.662
0.677
0.676
MTT-ltt--TV
0.708
6.654
0.683
0.601
0.651
0.656
0.678
0.676
MTT-late
0.638
0.665
0.806
0.642
0.635
0.822
0.640
0.206
Jate CS
0.612
0.658
0.822
0.614
0,821
0.816
0.681
MOSEI
MAEL
MT-late-JS
0.613
0.703
0.615
0.620
0.814
0.615
0.685
NIT Jate TV
0.613
0.645
0,509
0.615
0.619
0.809
0.620
0.676
MT--le WVT-Jate-VV WMA
0.608
0.643
0.615
0.801
0.671
21


Reviews:
[{"review_id": "V5PHS0YzkH", "reviewer": "Reviewer_k9gC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n\n1. As multimodal large foundation models are becoming popular, improving the performance of multimodal Transformers is an important topic. This paper is timely in addressing these challenges with a new and efficient alignment method.\n\n2. The paper focuses on alignment during the fine-tuning stage, which is crucial since fine-tuning pre-trained models is widely used in practical applications. This scenario requires a more lightweight and efficient approach for multimodal learning compared to the resource-intensive pre-training phase.\n\n### Weakness\n1. The most recent dataset mentioned in Section 4.1 is from 2020. It would be beneficial to include newer datasets, such as LLaVA instruct, to justify the model's effectiveness on more current, large-scale benchmarks relevant to the evolving field of multimodal learning.\n\n2. This paper does not discuss too much on the computational issue. Since the proposed method involves solving an OT problem,  it would be valuable to include an analysis of the additional time complexity or a real-time computation comparison with baseline methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Wasserstein Modality Alignment (WMA), a method to improve multimodal transformer models' robustness and performance by aligning modality representations using the optimal transport distance. WMA has the following advantages: \n\n1. This alignment does not add trainable parameters, making it lightweight and efficient. \n\n2. The approach regularizes the transformer during the fine-tuning stage to maintain an optimal modality distance. WMA adapts the alignment based on task requirements, rather than minimizing the distance between modalities outright. \n\n3. The proposed method works for both early and late fusion settings in multimodal learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the **Weakness**.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EzBs2AePIV", "reviewer": "Reviewer_B5qq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe proposal of WMA as a novel method for implicit regularization in multimodal transformers is innovative and addresses a significant problem in the field. The concept of task-dependent modality alignment is a strong contribution, offering flexibility and adaptability that is crucial for real-world applications.\n\nWeaknesses:\n\n1. While the empirical results are strong, the paper could benefit from a deeper theoretical analysis of why WMA works and under what conditions it is most effective.\n\n2. The paper focuses on a limited set of modalities (text, image, audio). It would be beneficial to understand how WMA generalizes to other types of modalities or data.\n\n3. The paper does not discuss the computational overhead introduced by WMA. Analyzing the efficiency and scalability, especially for larger models or datasets, would be important.\n\n4. While the paper compares WMA with several baselines, it could strengthen its claims by comparing against the latest state-of-the-art methods in multimodal learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose WMA, an implicit regularization method to align the Wasserstein distance between different modalities within a multimodal transformer. This method does not introduce any additional trainable parameters. Instead of directly minimizing the optimal transport (OT) distance between modalities, WMA aligns modalities based on the task requirements. This is achieved through two hyperparameters, allowing for a search for the optimal alignment. The paper provides insights into the degree of modality alignment required for different tasks, showing that task-dependent alignment is more effective than simply minimizing the OT distance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness.", "broader_impact_concerns": "Multimodal systems can sometimes amplify biases present in the training data. If WMA improves the performance of such systems, it might also enhance the impact of any biases, leading to unfair or discriminatory outcomes in applications like surveillance, hiring, or credit scoring. The use of implicit regularization through WMA might make the model's decision process less transparent. This could be a concern in high-stakes applications where explainability is crucial.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "b4y3vQ7DqU", "reviewer": "Reviewer_TJU2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe fusion of multi-modal data is an important next step for foundation models. The paper points out an important problem of current multi-modal transformer architectures for multi-modal fusion, which is that some modality dominates the prediction and other modalities are actually not used. This is because a common learning paradigm tends to perform lazy and greedy learning and learn the easiest way to solve the problem.\n\nThe paper plots the contributions of different modalities to the final prediction, which shows that the contribution distribution is very skewed. This further verifies the claim in the challenge.\n\nThe proposed modality alignment loss is a plug-and-play method, which could be used in varieties of transformer architectures.\n\nExperimental results show that using the proposed WMA method improves the multi-modal prediction performance. Also, the authors compare the performance between directly minimizing the optimal-transport distance and minimizing the difference to a reference distance. The results support the claim that minimizing the difference to a reference distance is a better choice.\n\n\nWeakness:\n\nAlthough empirically effective, the reason why aligning the features of different modalities could address the problem of skewed contribution of modalities. This is obviously a regularization loss but how it functions still needs some explanation.\n\nThe reference distance is decided as the average of the first batch. This is a quite random choice, which is severely influenced by the random seed. Could the authors demonstrate that the performance is robust to the choice of the batch of data?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to address the multi-modal fusion problem in the multi-modal transformer architecture when processing multi-modal data.\n\nThe paper mainly addresses the challenge that prior multi-modal fusion directly fuses all the modalities without alignment, where the final prediction may only exploit the easiest modality for prediction. The paper shows the contributions of different input modalities to demonstrate this claim.\n\nThe paper proposes an optimal-transport-based Wasserstein Modality Alignment (WMA) loss to align features of different modalities. The loss minimizes the difference between the optimal-transport distance of different modalities and a reference distance.\n\nThe paper conducts experiments on four datasets including modalities text, image, audio and video to verify the effectiveness of the ideas.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Could the authors address the problem in the weakness?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall I believe that this is strong work that I recommend should be accepted, as did the two reviewers who provided their final recommendations.

There are only two small additions I would like to see for the minor revisions:
- A brief discussion on the relevance of the approach to generative tasks.  I think it is fine that the method is currently limited to discriminative tasks, but it needs to make this restriction clearer and it would be good to have a small bit of discussion about if and how it might be generalised to generative settings (or and explanation for why it can't if this is the case).
- As noted by one of the reviewers, multimodal systems can amplify biases in the data and its possible the approach could have impact (positive or negative) on this.  I think the paper would thus benefit from a brief discussion on broader impacts relating to how the approach might impact on bias and fairness.
Decision: Accept (poster)