Robust Multimodal Learning via Cross-Modal Proxy Tokens
Authors: Md Kaykobad Reza
Ameya Patil
Mashhour Solh
Salman Asif
Venue: Accepted by TMLR ()

Abstract: Multimodal models often experience a significant performance drop when one or more modalities are missing during inference. To address this challenge, we propose a simple yet effective approach that enhances robustness to missing modalities while maintaining strong performance when all modalities are available. Our method introduces cross-modal proxy tokens (CMPTs), which approximate the class token of a missing modality by attending only to the tokens of the available modality without requiring explicit modality generation or auxiliary networks. To efficiently learn these approximations with minimal computational overhead, we employ low-rank adapters in frozen unimodal encoders and jointly optimize an alignment loss with a task-specific loss. Extensive experiments on five multimodal datasets show that our method outperforms state-of-the-art baselines across various missing rates while achieving competitive results in complete-modality settings. Overall, our method offers a flexible and efficient solution for robust multimodal learning. The code for this paper is available at: https://github.com/CSIPlab/Cross-Modal-Proxy-Tokens.

Published in Transactions on Machine Learning Research (10/2025)
Robust Multimodal Learning via Cross-Modal Proxy Tokens
Md Kaykobad Reza
University of California Riverside
mreza025eucr. edu
Ameya Patil Amazon
ameyapat@anazon,com com
Mashhour Solh
Amazon
mashhour@amazon.com com
M. Salman Asif University of California Riverside
sasif@ucr.
Reviewed on OpenReview: https: openreview. net/ forum? id= ctcweeYJJ
Abstract
Multimodal models often experience a significant performance drop when one or more modal- itics are missing during inference. To address this challenge, we propose simple yet effective approach that enhances robustness to missing modalities while maintaining strong perfor- mance when all modalities are available. Our method introduces cross-modal proxy tokens (CMPTss) which approximate the class token of a missing modality by attending only to the tokens of the available modality without requiring explicit modality generation or auxiliary networks. To efficiently learn these approximations with minimal computational overhead, we employ low-rank adapters in frozen unimodal encoders and jointly optimize an alignment loss with task-specific loss. Extensive experiments on five multinodal datasets show that our method outperforms state-of-thie tht---rttt-- baselines across various missing rates while achiev- ing competitive results in complcte-modality settings. Overall, our method offers flexible and efficient solution for robust multimodal learning. The code for this paper is available Saapllaaarrosss.e Cross-Modal Pooodal-rrxy--Tooi
at: https:/
1 Introduction
Multimodal learning (Ngiam al,, 2011; Xu al., 2023) integrates information from multiple (diverse) input sources to improve the performance downstream tasks. For example, examining movic poster image along with short text synopsis can provide greater insight into the movie's type and genre. In recent years, a number of methods have been proposed to effectively combine information from multiple modalities, including early-stage fusion (Mo & Morgado, 2024), token-level fusion (Wang et al., 2022b), channel exchange (Wang et al., 2020b), and bottleneck mid-fusion (Nagrani et al., 2021a). Furthermore, specialized architectures have been proposed for specific multimodal applications, such as vision-language tasks (Kim et al., 2021; Li et al., 2019; Lu et al., 2019), action recognition (Woo et al., 2023; Zhou et al., 2022), and segmentation (Zhang al., 2023a; Reza et al., 2024b). Most of these approaches assume that all modalities are available for every sample during both training and inference, yielding better performance than unimodal baselines. However, their performance drops sharply when one or more modalities get missing (Reza et al., 2024a; Ma et al., 2022).
In practical applications,, input modalities can be missing during training and/or inference for various reasons; such as lack of synchronous data capture (limited paired data during training) and sensor failure or privacy concerns (modalities missing during inference). Ma et al. (2022) observed that multimodal transformers such ViLT (Kim et al., 2021) exhibit drastic performance drop when one more modalities missing

---

Published in Transactions on Machine Learning Research (10/2025)
Einbedding Layer
Emsgge
Different Missing Svenarios
CMPT
Tikies
Eneoder
1000% Teat
Que
Modality my Insgg
modalty
Alignnenn Uing
Chanifer
Prodiction
missing
alrags
0% Insge 100% Text
65% Image
655% Text
Mixce
Eacbodding Layer
Yeu
Image 100%
long
X
100 medilly avuiiabiity
Caepion
Garing Module
2023)
(CVVS
Modality my: Text
Aue (CV2R 2022)
Kin Kim (BCCV 2024)
Kin al- MILTCCLL 2021)
CMPT (Cus))
(a) Illustration of our proposed method.
(b) Performance comparison.
Figure 1: (a) We introduce Cross-Modal Proxy Tokens (CMPTs), a novel approach address missing modality challenges. CMPTs effectively learn to approximate missing modality class tokens by adapting pretrained encoders through a joint optimization alignment and task-specific objectives. Our approach accommodates both complete and missing modalities during training and inference, thereby enhancing robustness across varying missing modality scenarios. (b) CMPIss achieve state-of-the-art performance, consistently outperforming recent baseline methods in both complete and missing modality scenarios. Following the experimental setup of Lee et al. (2023); Kim & Kim (2021), the radar plot illustrates Fl-macro scores on the MM-IMDb dataset across varying modality availability.
during inference as shown in Figure 1b (yellow curve). They developed an optimal fusion strategy to recover the performance loss of multimodal transformers in the missing modality scenarios (purple curve). Subscquently, Lee ot al. (2023) developed a prompt tuning technique (pink and blue curves), and Kim & Kim (2024) combined the prompt-based approach with Variance Covariance Invariance regularization (VICReg) (brown curve) to further improve the robustness of multimodal models missing modalities. Despite these efforts, achieving consistently strong performance under varying degrecs of missing modalities remains open challenge. In this work, aim to improve robustness across wide range of missing modality conditions while maintaining strong performance when all modalities are available.
In this paper, we introduce a simple yet effective framework to approzimate missing modalities from the avail- able ones for robust multimodal learning. Specifically, we introduce cross-modal proxy tokens (CMPTs), which learn to approximate the class tokens of missing modalities by attending to all tokens of the available modality, enabling effective handling of various missing modality scenarios. As illustrated in Figure la, our framework integrates pre-trained unimodal encoders into multimodal learning setting while introducing CMPTss to partially bridge the performance gap in missing modality scenarios. To ensure efficient learning of CMPTs with minimal additional learnable parameters, we employ low-rank adapters within the frozen unimodal encoders and jointly optimize both an alignment loss and a task-specific loss. The radar plot in Figure 1b demonstrates the superior performance of our approach compared to five existing methods (Kim et al., 2021; Lee et al., 2023; Kim & Kim, 2024; Ma et al., 2022) across six different complete and missing modality scenarios on the MM-IMDb dataset. Our CMPT-basod model consistently outperforms five multinodal datasets, com- paring approach against existing state-of-the-art baselines. Results show that our framework achieves state-of-the-art performance in both missing modality (Section 4.3) and complete modality (Soction 4.4) scenarios, while maintaining a minimal number of learnable parameters. Ablation studies further confirm that our method significantly improves performance even in severe missing-modality scenarios and provides performance enhancement across most the classes, leading to an overall performance boost (Section 4.6). Main contributions of our work are as follows.
all baselines across all the scenarios. We conduct extensive experiments
We propose CMPTs to effectively learn an approrimation of the missing modality from the available modality with minimal adaption of pre-trained unimodal encoders via alignment loss.
2

---

Published in Transactions on Machine Learning Research (10/2025)
Our approach outperforms 12 recent baseline methods across four multimodal datasets in various missing- modality scenarios, demonstrating state-of-the-art performance (Section 4.3).
Across five datascts, our method achieves on-par or superior performance compared to state-of-the-art techniques when all modalities are present during training and inference (Section 4.4).
We demonstrate that CMPTs generalize effectively to unseen missing rates (Section 4.5) and boost per- formance in most of the classes (Section 4.6).
2 Related Work
Recent works (Ma et al., 2022; Hazarika et al., 2022; Reza et al., 2024a) show that performance can degrade significantly when modalities are missing, as multimodal models are not inherently robust to such scenarios. Several approaches have been proposed to address this issue, In this section, we primarily discuss works where the main focus is on making multimodal models robust to missing modalities. More broader related work on multimodal learning is discussed in appendix Section A1.
Robust model design is one of the approaches for handling missing modalities. Rocently, Shin et al. (2024) developed a robust. framework utilizing modality masking and knowledge distillation. Wang et al. (2023) proposed ShaSpec to impute missing features through modality-speeifiic and modality-shared feature learning. Wang et al. (2022a) proposed TokenFusion, which dynamically replacos uninformative tokens to enhance robustness. Additionally, different fusion strategies (Choi & Lee, 2019; Fan et al., 2023a; Lin et al., 2023) have been developed to improve robustness across various missing-modali,, scenarios. However, many of these approaches are task-specific, limiting their generalization to other multinodal tasks. In our work, we focus on reusing pro-trained unimodal encoders, making it flexible for extension to different input modalities and multinodal tasks.
Robust training and model adaptation can enhance missing modality robustness. Recently, Reza et al. (2024a) proposed a parameter-ccffiient adaptation framework for different multimodal tasks. Shi et al. (2024) introduced a modality-aware low-rank adaptation method (MoRA) that uses a shared down- projection layer to map inputs to a low-dimensional space, and then applies modality-specific up-projections for missing modality adaptation. Nezakati et al. (2024) trained a single model to infer missing information by projecting available modalities. Ma et al. (2021) introduced SMIL, a Bayesian meta-learning method for flexible handling of missing modalities during both training and testing. Ma et al. (2022) proposed multi- task optimization approach with differentiable algorithm for optimal fusion strategy selection. Maheshwari et al. (2024) developed semi-supervised framework that leverages unlabeled data to improve robustness, while Wei et al. (2023) used a teacher network to transfer multimodal information for improved missing modality robustness. These methods, however, cither require extensive adaptation or specialized training procedures. In our approach, we employ lightweight adaptation of unimodal encoders (using Rank-1 LoRA adapters) and a single extra loss term to explicitly learn mooss- relationships. Hence, the overall training flow is mostly left unchanged.
Prompt tuning methods improve missing modality performance by incorporating learnable prompts across multiple layers of the model. Lee et al. (2023) introduced missing-aware prompts that learns separate set of prompts for each modality combination. Later, Jang et al. (2024) simplified the approach, showing that a single set of prompts per modality can achieve similar performance. More recently, Dai et al. (2024) proposed a multi-step adaptive prompt learning method through modality alignment, while Kim & Kim (2024) introduced a read-only prompt tuning approach with Variance Covariance Invariance regularization. Although effective, these methods require learning and storing a large nunber of prompts, typically 16 prompts (Lee et al., 2023; Jang et al., 2024) per modality per layer, for each missing modality scenario. In contrast, our method uses a single cross-modal prozy token per modality to estimate the missing modality feature from the available modality, thereby reducing the complexity.
We carry out extensive experimental comparisons with all the above prompt-based techniques to demonstrate superior performance across various missing modality scenarios. Furthermore, our approach efficient, uses few learnable parameters, and is robust to missing modalities in both training and testing which we discuss in Section 4.
3

---

Published in Transactions on Machine Learning Research (10/2025)
3 Proposed Approach
In this paper, we propose new framework to train multimodal systems such that they retain strong performance under different scenarios of missing modalities during inference. Furthermore, we seek to develop a simple and efficient framework that can be used for different multimodal tasks and datasets without any specialized (task-specific) changes. To achieve these objectives, we propose cross-modal proxy tokens (CMPTs), which effectively learn approximate the class tokens of missing modalities by adapting pretrained, modality-speeiffc (lransformer-baseel) encoders through joint optimization of alignment and task-specific objectives, as illustrated in Figure la.
3.1 Multimodal embedding tokens
Let consider multimodal dataset containing two modalities M {m m2)- We denote the available dataset D = (De, Dm. Pmzh, which can be further divided into three subsets: D. (Tmyy 33 where both modalities are present with corresponding output labels Dm. = {Im,, R.v) where only modality mi (Fma) is available with corresponding labels y; Dm, = {0, xm V) where only modality m2 (Fma) is available with corresponding labels y. Here, Ø denotes a placeholder for the missing modality (e.g., empty string for text, zeros for image, audio, or video) following prior works (Lee et al., 2023; Reza al., 2024a).
For each input modality, we assume that transformer- based encoder, pre-trained on some large dataset, is available. This is common and effective practice in modern multimodal learning, particularly given the wide variely of high-performing, pre-trained models now readily accessible (Devlin el al., 2019; Dosovitskiy et al., 2021; Gong et al., 2021). Leveraging these modality-specific encoders trained on extensive data leads to improved performance and faster convergence (Han et al., 2023; Xu et al., 2023). Specifically, each modality m E .M is processed through a pre-trained embedding layer, EmbeddingLayer m with parameters 2e.m. The input Fm for each modality is passed through its corresponding embedding layer generate modality-speoifie tokens as
Tm EmbeddingLayer.. (#mm Oe,m):
(1)
where Tm E Nxd denotes N tokens, each of dimension d, for modality m. During training, we keep the pretrained parameters 2.m frozen.
3.2 Cross-Modal Proxy Tokens (CMPTs)
To compensate for the missing modalitics, we introduce cross-modal proxy tokens (CMPTs) that approzimate the missing modality by allending only to the tokens of the available modality. For each modality m E M, we concatenate one CMPT (TeMPT,m E Rixd) with the class token (CI.S) (TeLs. E R.Id)) and all the modality specific tokens Tm generate the modified embedding tokens Tm [
f. Concat((TcMPr., TCLS.m. Tm))).
(2)
Note that the proposed CMPT serves as an extra token, similar to the class token (Dosovitskiy et al., 2021; Devlin al., 2019; Gong et al., 2021), but with a complementary purpose. Class token attends all the tokens of a given modality to aggregate information and learn a rich feature representation of that modality. In contrast, CMPT attends to all the tokens of given modality to learn approzimation of the class token of the other modality (e.g., TCMPT,mg approxinates Ters,mg while altending to Tm
3.3 Learn CMPTs by alignment and adaptation
For each modality, assume that pretrained transformer encoder fm with parameters θy.m is available. The encoder for each modality needs to be updated so that they can learn the corresponding CMPTs. To keep the number of trainable parameters minimal, we use low-rank adaptation (LoRA) (Hu et al., 2022) to learn a small number of parameters A The modified embedding tokens for modality m, Tma are then passed through the adapted encoder for modality m to get the final output tokens as
+m Smm(Tm: f. A.m.)
(3)
4

---

Published in Transactions on Machine Learning Research (10/2025)
where Fm ( represents the CMPT, CLS, and remaining tokens from the last encoder layer. Note that the pretrained encoder parameters f.m remain frozen while we only learn A f.m-
We aim to train the CMPTss such that they act as a proxy of the class token of the missing modality at the inference time. In particular, we want the CMPT TCMPT.m, of modality mı to approximate the class token mCLs,mg of modality m2, and vice versa. To achieve this, we use an alignment loss Laligu: defined as
Σ LMSE (MsE((cMPTT. TCI...m2
falign
(4)
ND. sED:
LUSE (TCMPT,m2 TCLS.m)),
where MSE denotes the mean squared error loss and ND. denotes the number of samples in the training data where both modalities are present.
3.4 Gating module and fusion
After computing the final output tokens Tm extract the class token (TCLS,m) and the CMPT (fCMPT. for both modalities and pass them through the gating module. If both modalities are available, the gating module returns the class tokens from cach modality. If one modality is missing, the gating module substitutes
the missing class Loken with the CMPT from the available modality. The returned lokens are then fused using additive fusion to produce the final fused feature token T E IR 1xd as
TolS,m, + TCLs.
mi; M2 available,
T
Tors m2 + TCMPT.m2>
my is missing, ma missing.
(5)
FeLS.m, + TCMPT,
The fused feature token T is then passed through a linear classifier head to make a prediction & as
I(T;0)).
(6)
O₁ represents the parameters of the classifier head that learned during training.
Finally, we learn the low-rank factors for each encoder f(A), and classifier head (0x) by minimizing the task-specific loss Lrask, which for our purposes is standard cross-entropy loss for classification tasks, combined with the alignment loss:
Liotal Ltask + ALaligns
(7)
where À is a hyperparameter that we set to À = 0.20 in all our experiments. As discussed in Section 4.6.3, this value provides a balance in performance across different scenarios.
3.5 Handling missing modalities
Our model can handle missing modalities during both training and inference. During training complete modality data, we apply random modality dropout, where we either use both modalities randomly drop one in each iteration. This way, the model learns to make correct predictions in missing modality scenarios using the CMPTs from the available modality. When training data contains missing modalities, we learn the CMPTs by optimizing the alignment loss using only the samples with complete modalities, as shown in equation 4. For missing modalitics, we employ placcholders (c.g., an empty string for text and zeros for images, audio, and video) following standard practices (Lee et al., 2023; Reza al., 2024a), and use the CMPT from the available modality as approximation to make predictions, described in equation 5.
To enhance training efficiency, we leverage LoRA (Hu et al., 2022) with rank of to fine-tune the encoders in a parameter-efficien. manner. LoRA layers are inserted after the query, key, value, and output layers in every attention blocks within the encoders. Our experiments demonstrate that LoRA with rank 1 sufficient achieve better performance compared cxisting stato-of-the-art methods, as discussed in Scction 4.4 and Section A4. Our method is also compatible with newer parameter-offeien adaptation techniques such DoRA (Liu et al., 2024) and VeRA (Kopiezko et al., 2024), discussed in Section A10.
5

---

Published in Transactions on Machine Learning Research (10/2025)
Table Performance comparison when modality gets missing during both training and inference. We train and evaluate our model with same modality ratio following existing baselines (Lee et al., 2023; Jang et al., 2024; Kim & Kim, 2024) and report the mean and standard deviation (SD) of three independent runs with different seeds.
% availability
ViLT
MAP (2023)
MSP
VisualBERT
Ma et al. Kim & Kim (2024)
MuAP (2024)
CMPT (Ours)
Datasets
Image Text
(2021)
Input
Attn.
(2024)
(2019)
(2022)
Head
Cross
Mean
30%
100%
37.61
46.30
44.74
47.45
38.62
46.63
56.03
47.21
46.73
60.21 = 0.40
MM-IMDb
(FI-Macro)
65%
65%
36.30
42.66
41.56
42.03
37.23
41.28
49.24
42.57
43.92
54.04 = 0.23
100%
30%
34.71
39.22
38. 16
38.34
36.41
38.65
43.21
41.37
39.88
52.61 = 0.12
UPMC Food-101
30%
100%
76.93
86.18
86.05
86.34
77.41
86.38
87.11
86.90
86.59
87.48 = 0.13
65%
65%
69.03
79.08
78.09
78.89
71.06
78.58
82.67
77.87
78.95
82.83
:: 0.25
(Accuracy)
100%
30%
66.29
74.53
72.57
73.77
67.78
73.41
78.81
74.61
74.60
80.37 : 0.18
Table 2: Performance comparison when a complete modality gets missing during inference. We report % accuracy for all the datasets. Our method shows overall better performance compared recent state-of-the-art methods.
FILM
BiGated (2018)
OGM-GE
QMF (2023b)
MLA (2024)
CMPT (Ours)
Datasets
Available Modality
Missing Modality
(2018)
(2022)
Audio
Video
53.89
51.49
53.76
59.41
59.27
67.20
CREMA-D
Video
Audio
18.67
17.34
28.09
39.11
64.91
76.21
Audio Video
60.07
59.21
68.14
63.71
79.70
88.84
Audio
Video
48.6
49.96
48.87
51.57
54.67
68.27
KS
Video
Audio
23.15
23.77
29.73
32.19
51.03
85.77
Audio Video
63.33
63.72
65.74
65.78
71.35
91.21
Image
Text Image
4.68
14.20
22.35
45.74
69.60
75.66
UPMC Food-101
Text
85.84
85.79
85.17
84.13
86.47
85.31
Image Text
87.21
88.87
87.54
92.87
93.33
94.47
4 Experiments and Results
4.1 Datasets
We evaluate our approach on five popular multimodal datasets across different tasks. A brief description of each dataset provided here, with further details in Section A2 the appendix.
multimodal classification dataset consisting of 101 food classes. It contains 90,704 image-text pairs, divided into training set of 67,988 pairs and Lest set of 22,716 pairs.
UPMC Food-101 (Wang et al., 2015)
MM-IMDb (Arevalo et al., 2017) has 25,959 samples with image and text modalities for multi-label movie genre classification. It is split into 15,552 training, 2,608 validation, and 7,799 test samples across 23 classes.
Kinetics-Sound (KS) (Arandjelovic & Zisserman, 2017) is a subset of Kinetics400 dataset (Kay et al., 2017) for human action recognition using audio and video as input modalities. It has 31 classes, with 14,739 samples in the training set and 2,594 samples in the test set.
Audio- Audio-Visual Event (AVE) dataset (Tian et al., 2018) contains 4,143 10-second videos for multimodal event localization. It is divided into vaal/estt sets containing 3,339/402//44 samples across 28 categories.
CREMA-D dataset (Cao et al., 2014) includes 7,442 short video clips from 91 actors expressing six emotion categories. It contains 6,698 training and 744 test samples for multimodal emotion recognition.
4.2 Implementation details
Langua-Lang datascts. We use pretrained ViT-B (Dosovitskiy et al., 2021) from CLIP model (Radford et al., 2021) image encoder and BERT-base-uneasee model (Lu et al., 2019) as text encoder. The max token length is set. to 40 for UPMC Food-101 and 256 for MM-IMDb dataset, respectively.

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: Performance comparison with both modalities available during training and inference Image-Texx datasets. We report % accuracy for UPMC Food-101 and F1-Macro/ /F-Mrro score for MM-IMDb dataset. Here, ' indicate results that are not reported corre- sponding papers.
Table 4: Performance (% accuracy) comparison with both modalities available during training and inference on Audio AudiooViiee Video dalasets. Here, denoles results gen- crated using existing code.
Parameters (M)
Method
KS
AVE.
CREMA-D
Total Learnable
Paremeters (M)
UPMC
Methods
MM-IMDb
G-Blend (2020s)
22.36
22.36
62.20
65.50
58.70
Total Learnable
Fond-101
FILM (2018)
22.35
22.35
63.33
60.07
ViLT (2021)
112.63
112.65 63
92.00
55.30
64.70
BiGated (2018)
22.35
22.35
63.72
59.21
MBT (2021b)
196.00
196.00
93.56
59.60
64.81
OGM-GE (2022)
22.35
22.35
65.74
76.90
68.14
MMBT (2019)
170.00
170.00
94.10
60.80
66.10
QMF (2023h)
22.36
22.36
65.78
08.16'
63.71
MMLoRA (2023)
196.10
196. 10
93.70
61.70
67.20
PMR (2023b)
22.36
22.36
66.40
61.80
PMP (2023)
198.43
2.54
91.51
58,77
64.51
MLA (2024)
22.37
22.37
71.35
61.68"
79.70
PMF-L (2023)
643.95
4.44
91.68
61.66
66.72
MBT (2019)
172.00
172.00
85.00
77.80
MLA (2024)
218.26
218.26
93.33
MMLoRA (2023)
172.19
172.19
91.40
96,20
88.60
CMPT (Ours)
195.51
0.27
94.47
63.58
69.23
CMPT (Ours)
172.16
0.17
91.21
96.77
88.84
Audio- Video datasets. We use AST model (Gong et al., 2021) pretrained on AudioSet dataset (Gemmeke et al., 2017) as audio encoder and ViT-B (Dosovitskiy et al., 2021) from CLIP model (Radford et al., 2021) as video encoder. Raw audio is converted to a 128- bin mel spectrogram at 16 kHz, with a maximum length of 1024 frames. For video, we sample 3 frames randomly while training and uniformly while testing. Other pre-processing steps are similar to (Zhang et al., 2024; Peng et al., 2022; Du al., 2023).
Hyperparameter settings. We use Python 3.8.19 and Pylorch 2.2.2 for training and evaluating our models. All the models are trained using two NVIDIA RTX 2080Ti GPUs. For vision-language datascts, we set the learning rate to 10-3 and train the models for 10 epochs with a batch size of 8. For audio-video datasets, the learning rate is set to 5 x 10-5 and models are trained for 100 epochs with a batch size of 4. We use AdamW (Loshchilov Hutter, 2019) optimizer with e = 10-8 and weight decay = 0.02. While training, we utilize cross entropy loss, polynomial learning rate scheduler with power=0.9 and set the first 5 epoch warm-up. The learning rate is set to 0.1 times the original learning rate during warm-up epochs. We set LoRA (Hu et al., 2022) rank = 1 and insert them after query, key, value and output layers of each transformer block. Further details with all the hyperparameters can be found in Section A3 in the appendix.
For each task and dataset, we compare against state-offthe-art published work. For this reason, the set of baselines varies across experiments based on the availability of published results. In each table, the best and second best results are bold and underlined, respectively.
4.3
Performance when modalities are missing
Existing literature considers missing modalities at both training and inference or only at inference. Our method achieves state-of-the-art performance in both scenarios, as detailed in the next. two sections.
4.3.1 Missing modality during training and inference
In the first setup, a modality is missing at a fixed rate during both training and inference, as studied by Lee et al. (2023); Jang et al. (2024); Kim & Kim (2024). We follow their experiment setup and compare our method against seven state-of-the-art approaches on MM-IMDb and UPMC Food-101 datasets. As shown in Table 1, our method consistently outperforms all baselines across various missing modality configurations, Specifically, compared to the most recent approach by Kim & Kim (2024), we achieve 4.18%-9.400% improvement in Fl-macro score MM-IMDb dataset and 0.16%-1.55 1.56% improvement in accuracy UPMC Food-101 datasct across different missing modality scenarios. It is worth noting that while the performance of Kim & Kim (2024) on UPMC Food-101 dataset is comparable to ours in two scenarios, our method still performs best in every configuration. These results also suggest that the CMPTs can provide good proxy for the missing class tokens, which significantly improves the model robustness. To ensure a reliable evaluation, we run our method with three different random seeds and report the mean and standard deviation (SD) for various missing scenarios. The low SD values indicate the stability and reliability of our
7

---

Published in Transactions on Machine Learning Research (10/2025)
70
95
92.5
94.5
61.4
63.6
88.6
91.9
92.0
92.0
F1-Macro
60
47.0
52.
65.6
50.8
52.0
58.6
53.8
$6.9
$1.8
55.3
59.3
Acouracy
85
77.5
81.0
82.6
82.1
84.8
85.8
8321
80.7
88.2
42.9
46.6
75
77.5
40
41.8
45.1
Kim et. VVLT (ICML 2021)
73.3
73.3
et VVLT (ICML 2021)
37.3
38.9
al. (CVPR
Ma et all (CVPR 2022)
65.9
Kim Kim (ECCV 2024)
31.2
Kim & Kim (ECCV 2024)
CMPT (Ours)
23.1
CMPT (Ours)
55
58
10%
30%
60%
70%
90%
100%
10%
30%
50%
70%
90%
100%
Available Text
Available Text
(a) Dataset: MM-IMDb
(b) Dataset: UPMC Food-101
Figure 2: Generalization to varying missing rates during inference. Models are trained with 100% image + 100% text and evaluated with 100% image + =% text following Ma et al. (2022); Kim & Kim (2024). Our approach demonstrates better generalization, particularly under severe modality loss.
method. We also assessod the performance of the models with different distributions of modality ratios at training and testing, for which the results are presented in Section A5 of the appendix.
4.3.2 Missing modality during inference only
In the second setup; following the work of Zhang et al. (2024), we evaluate our model when all modalities are available during training, but one modality completely missing during inference. As shown in Table 2, our method outperforms existing baselines in most cases. For instance, when the audio modality missing, all methods except MLA (Zhang et al., 2024) experience significant performance drop on the CREMA- D and Kinetics-Sound (KS) datasets. On the CREMA-D dataset, our method achieves 7.93% and 11.30% improvements over the most recent MLA approach, when video and audio are missing, respectively. Similarly, on the KS dataset, our method outperforms MLA with 13.60% and 34.74% improvements when video and audio are missing, respectively. While MLA performs slightly better when the image modality is missing on the UPMC Food-101 datasct, suffers a larger performance drop when text is missing. In contrast, our method maintains a balanced performance across different missing modalities. Furthermorre, across all datasets, our method achieves the best performance when all modalities available and demonstrates superior robustness compared to other approaches. These results highlight the effectiveness of our method in handling scenarios where modality is entirely absent during inference.
4.4 Performance when all modalities are available
A method designed to handle missing modalities should not compromise performance when all modalities are present. In this subsection, we demonstrate that our approsch also achieves comparable or superior performance to state-of-the-ae the-art models across five datasets when all modalities are available.
To evaluate this, we train and test our method when all modalitics are available during training and inference. Results in Table demonstrate that our proposed method achieves state-of-the-arl performance on both UPMC Food-10 101 and MM-IMDb datasets. Our method surpasses the most recent MLA approach (Zhang et al., 2024) by 1.14% on UPMC Food-101. Our method achieves 1.88% and 2.03% improvement in F1-macro and F1-micro scores, respectively, over second-best performing MMLoRA (Du et al., 2023) on MM-IMDb.
We present the results for audio-video datasets in Table 4, where we compare our method against nine state- of-tho-art methods. Our approach consistently achieves comparable or superior performance across all three datasets, while using the fewest number of learnable parameters. These results highlights that incorporating CMPTss does not roduee any drawback when all modalities remain available.
8

---

Published in Transactions on Machine Learning Research (10/2025)
Ft-Mtoro
54
64
10.82
16
56.94
57.80
58.00
51.94
06
60.35
65
60
80
61:25
59
54.31
57.10
02.04
60.21
63.68
63.16
F1-Macre
49
59
64
54
-46
97
40.71
50
42
50.16
53.94
57.18
$2.00
02.55
$9.07
60.05
01.44
60 50
63.58
63.1
44
44.07
49.47
44
41:42
40.80
34
20.96
with Modally Dropout
Baseline
39
35.3
38.50
38077
Baseling
Modally Drapout
24.20
with Modeliy Dropout and CMPT
34
91
with Modalty Drapout and CMPT
24
29
0%
10%
20%
30%
40%
50%
80%
70%
80%
90%
100%
0%
10%
20%
30%
40%
50%
80%
70%
80%
90%
100%
Avallable Image
Available Text
(a) Available: Text, Missing: Image.
(b) Available: Image, Missing: Text.
Figure 3: Effectiveness of CMPTs on the MM-IMDb dataset. All models are trained with 100% image and 100% text data, then evaluated under varying amount of missing modalities. CMPTs achieve significant performance improvement, especially under severe missing modality scenarios.
without CMPT
with CMPT
without CMPT
with CMPT
100
100
76
76
F1 F1-Spora
F1 Soore
50
80
3
Homor Aninalion
Miuiical
Action
Fantaay
History
Comn
Mystery
Biggrpplty
Reyton
Fanily
Comedd
Sport
Drana
Dosumentayy Short
Thriter
Aution
Adventure
Wastem
Havor
Cranta
Cometty
Animation
Momc Fattesy
Hisooy
Decumensee Sport
Faln Si
Broggphly
Miyeerry
War Droms
Short
Fiim
Class Name
Class Name
(a) Available: Image, Missing: Text.
(b) Available: Text, Missing: Image.
Figure 4: CMPTss improve performance across most of the classes when modalities are missing, However, it fails to accurately predict certain classes (e.g. short /documentary) in the absence of text, likely due to the image modality's insufficient information for those classes. Classes are sorted based on the amount of performance improvement.
4.5 Generalization to different missing rates
In real-world scenarios, the missing rates for modalities during inference can be unpredictable. Any method designed to handle missing modalities needs to generalize effectively across varying levels of missing modality rates during inference. To evaluate this capability, we trained our models on complete multimodal data and assessed their performance under varying levels of missing modalities during inference. Specifically, we follow the experiment setup similar to Ma et. al. (2022); Kim & Kim (2024) where models are trained with 100% image + 100% text data and evaluated with 100% image + x% text.
As illustrated in Figure 2, our method consistently outperforms all baselines across different missing scenarios, demonstrating superior robustness. We observe a decline in performance of all methods as the amount of available text decreascs, but our approach maintains stronger resilience compared to other methods. For instance, on the MM-IMDb dataset, our method achieves a 5.8% higher F1-Macro score than the most recent method by Kim & Kim (2024) and a 25.6% improvement over ViLT (Kim et al., 2021) when only 10% of text is available. A similar trend is observed on the UPMC Food-101 dataset, where our method consistently yields superior performance, confirming its robustness to missing modalities at inference.
Note that published work by Ma et al. (2022); Kim & Kim (2024); Kim et al. (2021) did not report results for different missing rates of images, which prevents us from reporting a direct comparison that sotting. Nev- ertheless, our results highlight the effectiveness of CMPTs in handling different missing modality scenarios, making it more generalizable solution for real-world applications.
9

---

Published in Transactions on Machine Learning Research (10/2025)
Table Ablation study on the aligument loss weight X. We report performance on the UPMC Food-101 and AVE
datasets across both complete and missing modality scenarios.
Dataset
Available Modality
Missing Modality
|| A == 0.0
= 0.1
À
0.2
À
0.3
À
0.4
À = 0.5
Image
Text Image
73.12
73.48
75.66
73.72
74.39
74.46
UPMC Food-101
Text
81.31
84.79
85.31
84.51
84.81
81.88
Image Text
93.96
94.14
94.47
94.13
94.34
94.21
Andio
Video Audio
81.82
84.33
85.21
84.82
84.33
81.82
AVE
Video
82.83
83.08
84.58
85.32
81.84
84.83
Audio Video
95.77
96.02
96.77
96.52
96.52
96.27
4.6
Ablation studies
4.6.1
Effectiveness of CMPTs
To evaluate the effectiveness of CMPTs in handling missing modalities, compare method against two baselines: (i) standard model trained without modality dropout or CMPTs (baseline) and (ii) a model trained with modality dropout, where modalities are randomly removed during training to improve robustness to missing data. For a fair comparison, all models were finc-tuned on the downstream task using LoRA with a rank of
Figure 3 presents the main results for MM-IMDb dataset, where the baseline model experiences a sharp performance drop as the proportion of missing modalities increases. While modality dropout improves ro- bustness, the performance gap remains large. Incorporating CMPTs further enhance performance, achieving 32.06% improvement over the baseline and 11.55% over modality dropout when the image modality is en- tirely missing as shown in Figure 3a. A similar trend is observed when text is missing as shown in Figure 3b. Notably, CMPTs become increasingly effective as the amount of missing data increases, demonstrating its strong ability to substantially mitigate performance degradation in the presence of severe modality loss. Similar trend appears in UPMC food-101 datasct, which we discuss in Sec. A6 in the appendix.
4.6.2 Effects of CMPTs on different classes
When analyzing the effectiveness of CMPTs, a key question whether the overall performance gains are driven by just a few classes if CMPTs consistently improves performance across most classes, To address this, we conducted a per-class performance analysis with and without CMPTs.
As shown in Figure 4, CMPTss improve performance across nearly all classes in both text-missing and image-missing scenarios. Notably, it significantly mitigates performance degradation in classes. For example, when text is missing, classes like Horror, Animation, and War experience severe drops in performance, but CMPTs effectively recover much of the loss (Figure 4a). Similarly, when images are missing, CMPTs provide substantial gains in Thriller, Action, and Romance (Figure 4b). A slight drop is observed in Short and Documentary classes when text is missing, likely due insufficient information in the image modality alone for those classes. These results demonstrate that CMPTs enhance performance across most classes, leading to a strong overall performance improvement. A similar trend is observed on the UPMC Food-101 dataset which is discussed in Section A7 of the appendix.
4.6.3 Determining the Optimal Alignment Loss Weight
To determine the optimal value for the alignment loss weight X, we performed ablation study the UPMC Food-101 and AVE datasets, varying À from 0.0 to 0.5. The results, shown in Table 5, indicate that the choice of À is crucial for performance in different scenarios. For our experiments, X = 0.20 provides an overall better performance compared to other values. On the UPMC Food-101 dataset, setting À = 0.20 consistently achioves the best performance across all conditions: with missing text, missing image, and with both modalities available. Using 0.0, which is equivalent to removing the alignment loss, results in a significant. performance drop, highlighting the importance of the alignment objective.
10

---

Published in Transactions on Machine Learning Research (10/2025)
Available: Image, Missing: Text
Available: Text, Missing: Image
Multimodal without CMPTS
gres
pancakes cakes
frozen yogurt
sushi
guacamole
beignets
Figure 5: (-SNE plots of the fused feature tokens T for classes under different modality settings. The I-SNE plots show that fused features without CMPTs (red) deviate siguificantly from the complete multimodal features (green). Incorporating CMPTs (blue) provides embeddings that closely align with the complete multimodal features, indicating improved semantic alignment and robustness.
On the AVE dataset, the results show similar trend. A weight of X = 0.20 yields the best performance when video is missing and when both modalities are available. 0.30 shows a slight performance improvement in the video-only scenario. Given that = 0.20 demonstrates superior or near-optimal performance consistently across both datascts, we set À 0.20 for all our experiments, as provides a balance in performance across different modality conditions.
4.6.4 Qualitative Analysis of CMPT Effectiveness
To gain deeper insights into how CMP'ss enhance robustness in missing modality scenarios, we conducted a qualitative analysis on the UPMC Food-101 dataset using both fused feature projections and attention visualizations. Figure 5 presents 1-SNE plots of the fused feature for ten representative food classes where CMPTss yield notable performance gains. We compare three settings: (i) complete multimodal input (green), (ii) missing modality without CMPTs (red), and (iii) missing modality with CMPTs (bluc). The t-SNE plots show that fused features without CMPTs (red) deviate significantly from the complete multimodal features (green) in missing modality scenarios. Incorporating CMPTs (blue) provides embeddings that closely align with the complete multimodal features, indicating improved semantic alignment and robustness.
We also analyze the attention maps from the last layer of the vision encoder for the CLS token and CMPTs in the text-missing scenario, as shown in Figure We selected random examples from six classes where fused features without CMPTss provide incorrect predictions, but including the CMPTs leads to correct predictions (approximately 15% samples in the dataset show this behavior). The attention maps show that CLS tokens attend to semanticall--irrelee irrelevant regions, which likely leads incorrect predictions. In contrast, CMPTs attend to cemantically-releaan regions for the respective classes and lead to accurate predictions. These visual lizations demonstrate that CMPTs not only act as proxies for the missing modality but also reshape the model's attention to reflect modality-relevan attention maps for some cases when the fused features with and without CMPTs provide correct predictions (approximately 60% samples show this pattern) and when the fused features with CMPTs lead to incorrect predictions (approximately 3% samples show this pattern) in Section A8.
cues. For the sake of completeness, we have also included
11

---

Published in Transactions on Machine Learning Research (10/2025)
Input Image
CLS Token Attention
CMPT Attention
Input Image
CLS Token Attention
CMPT Attention
GT: Guacamole
Pred: Edamame
Pred: Guacamole
GT: Pancakes
Pred: French Teast
Pred: Pancakes
GT: Fole Gras
Pred: French Toast
Pred: Fole Gras
GT: Clam Chowder
Pred: French Onion Soup
Pred: Clam Chowder
GT: Frozen Yogurt
Pred: Ice Cream
Pred: Frozen Yogurt
GT: Beignets
Pred: Donuts
Pred: Beignets
Figure 6: Attention map visualizations from the last layer of the vision encoder for the CLS token and CMPTs in the text-missing scenario on UPMC Food-101 dataset. The attention maps show that CLS tokens attend ammantically-irreeeve regions, which likely leads to incorrect predictions. In contrast, CMPTss attend to semantically- relevant regions for the respective classes and lead accurate predictions.
5 Limitations and Future Directions
While our approach demonstrates consistent performance improvements over existing methods in different missing modality scenarios, the remaining performance especially in extreme cases indicates room for further improvement. Additionally, we observe that certain classes are heavily dependent on a specific modality, which leads to significant performance drop in those classes when that specific modality is missing, A natural question arises as how can we design a better training strategy that encourages more uniform reliance on different modalities while maintaining overall performance both complete and missing modality scenarios? Future work can also explore developing general framework for explaining modality contribution and expected performance drop under different missing modality scenarios. Finally, this paper primarily focused on two-modality settings due to the availability of standard benchmarks and basclines that enable direct and fair comparisons. Extending the CMPT framework is natural next step, and we are actively investigating this direction as part of our ongoing work.
6 Conclusion
We presented a simple yet effective framework for utilizing pre-trained unimodal encoders for robust mul- timodal learning. Our proposed framework enables training of multimodal systems that offer robust per- formance under different scenarios of missing modalities during training and inference. To compensate for missing modalities, we propose to learn cross-modal proxy tokens (CMPTs) to approximate the missing modality class lokens from the available modality using an aligument loss during training. Instead of training the unimodal encoders from scratch, we only learn rank-1 adapters along with task-specific classifier heads. Our extensive evaluations across diverse tasks and datascts demonstrate the robustness and adaptability of our approach in both complete and missing modality scenarios. Our analysis also reveals that CMPTs can generalize well to different missing scenarios and enhance performance across most of the classes to provide
12

---

Published in Transactions on Machine Learning Research (10/2025)
overall performance boost. Overall, our approach provides a robust and efficient solution for multimodal learning that maintains high performance in different real-world scenarios.
Acknowledgment
This work is supported in part by NSF awards 2046293 and 2406199 and Amazon Gift awards.
References
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE international conference computer vision, pp. 609 617, 2017. 6,
John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multimodal units for information fusion. arXiv preprint arXix: 01992, 2017. 6,
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer vision-FCCC 2014: 19th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 18, pp. 446-461. Springer, 2014. 1
Houwei Cao, David Cooper, Michael Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema- d: Crowd-sourced emotional multimodal actors dataset. TEEE transactions on affective computing, 5(4): 377 390, 2014. 6,
Jun-Ho Choi and Jong-Scok Lec. EmbraceNet: A robust deep lcarning architecture for multimodal classifi- 270, cation. Information Pusion, 51:259 -2:09--770, 2019. doi: i000f//.inffuss.201.02...00 3
Ruiting Dai, Yuqiao Tan, Lisi Mo, Tao He, Ke Qin, and Shuang Liang. Muap: Multi-step adaptive prompt learning for vision-language model with missing modality. arXiv preprint arXiv:2409.0044 2024. 3, 6
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Papers), pp. 4171-4186, June 2019. doi: 10.18653/ 10.18653/v1/N194 11/N19-1412.. 4,
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. 4, 6, 7, 10, 11
Chenzhuang Du, Yue Zhao, Chonghua Liao, Jiacheng You, Jie Fu, and Hang Zhao. Improving discriminative multi-modal learning with large-scale pre-trained models. arXiv preprint arXto:2310.051900 051193, 2023. 7, 8, 1
Siqi Fan, Zhe Wang, Yan Wang, and Jingjing Liu. SpiderMesh: Spatial-aware demand-guided recursive meshing for RGB-T semantic segmentation. arXiw:2303.088 08692, 2023a. 3
Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20029 20038, 2023b. 7,
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 TEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776-780, 2017. doi: AAL100//CAASSPP2211.9222220
Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Proc. Interspeech 2021, pp. 571-575, 2021. doi: 10.21437 htt433///ntsppeech222. 2021-669.. 4,
13

---

Published in Transactions on Machine Learning Research (10/2025)
Xue Han, Yi-Tong Wang, Jun-Lan Feng, Chao Deng, Zhan-Heng Chen, Yu-An Huang, Hui Su, Lum Hu, and Peng-Wet Hu. A survey of transformer-based multimodal pre-trained modals. Nearocompuling, 515: 89 106, 2023. ISSN 0925-2312. doi: uttps://doi.org/10.. 10..01//.....eoom..mmmmmmmm 2022.09.136. 4
Devamanyu Hazarika, Yingting Li, Bo Cheng, Shuai Zhao, Roger Zimmermann, and Soujanya Poria. Ana- lyzing modality robustness in multimodal sentiment analysis. In Proceedings of the 2022 Conference of the North Americon Chapter of the Association for Compatational Linguistics: Human Language Technologies, pp. 685-696, July 2022. doi: 10.18653, vv 10.18653/v1/2022..0 ma...m.namm
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. 4, 5, 10
Jaehyuk Jang, Yooseung Wang, and Changick Kim. Towards robust multimodal prompting with miss- ing modalities. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8070 8074. IEFE, 2024. 3, 6, 7,4
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arxiv preprinl Xiv: 060500 2017. 6,
Douwe Kiela, Edouard Grave, Armand Joulin, and Tomas Mikolov. Efficient large-scale multi-modal classi- fication. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018. 6, 7
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. Supervised multimodal bitrans- formers for classifying images and text. Xi preprint arXiv:1909.0 02950, 2019. 7,
Donggeun Kim and Taesup Kim. Missing modality prediction for unpaired multimodal learning via joint embedding of unimodal models. In European Conference on Computer Vision, pp. 171-187. Springer, 2024. 2, 3, 6, 8,
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International conference machine learning, pp. 5583 5594.. PMLR, 2021. 1, 2, 6, 7, 9, 4
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adap- tation. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview. (llpenreriev.net/ffrrfftt httt//frrmmm.ddjjjM.......
5, 10
Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen-Yu Lee. Multimodal prompting with missing modal- ities for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14943 14952, 2023. 2, 3, 4, 5, 6,
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant bascline for vision and language. arXiv preprint arXio: 03557, 2019. 1, 6
Yaowei Li, Ruijic Quan, Linchao Zhu, and Yi Yang. Efficient multimodal fusion via interactive prompting. In Proceedings of the IEEE/CVI conference on compuler vision and pallern recognition, pp. 2604 2613, 2023.
Baihong Lin, Zengrong Lin, Yulan Guo, Yulan Zhang, Jianxiao Zou, and Shicai Fan. Variational probabilistic fusion network for RGB-T semantic segmentation. arXiv preprint arXiv:2307.088 005366, 2023. 3
Shih- Yang Liu, Chien-Yo Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Tine Cheng, and Min-Hung Chen. Dora: deight-decompseed low-rank adaptation. In Forty-first International Con- ference on Machine Learning, 2024. 5, 10
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. 7
14

---

Published in Transactions on Machine Learning Research (10/2025)
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lec. Vilbert: Pretraining task-agnostic visiolinguistic representation ions for vision-and-languuae tasks. Aduances in neural informalion processing systems, 32, 2019. 1, 6
Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learn- ing with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2302 2310, 2021.
Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust missing modality? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 18177-18186, 2022. 1, 2, 3, 6, 8,
Harsh Maheshwari, Yen-Cheng Liu, and Zsolt Kira. Missing modality robustness in semi-supervised multi- modal semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1009-1019. IEEE Computer Society, 2024. doi: ALL00/WWCV57770.202222 3
Shentong Mo and Pedro Morgado. Unveiling the power of visial early fusion transformers with dense interactions through masked modeling. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 271962 2024.
Arsha Nagrani, Shan Yang, Amurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances in newral information processing systems, 34:14200-14213, 2021a.
Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances in neural information processing systems, 34:14200-142U1
2021b. 7,
Niki Nezakati, Md Kaykobad Reza, Ameya Patil, Mashhour Solh, and M. Salman Asif. Mmp: Towards robust multi-modal learning with masked modality projection. arXie preprint or Xivv 10.03010, 2024. 3
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, Andrew Y Ng, et al. Multimodal deep learning. In ICML, volume 11, pp. 689 6966 2011.
Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on- the-fly gradient modulation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 8238 8388-247, 2022. 6, 7,
Ethan Perez, Florian Strub, Harm De Vries, Vincent, Dumoulin, and Aaron Courville. Film: Visual rea- soning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. 6,7 7
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021. 6, 7
Md Kaykobad Reza, Ashley
and M. Salman Asif. Robust multimodal learning with miss- ing modalities via perfiener-lfficient adaptation. IEEE Transactions on Pallern Analysis and Machine Intelligence, pp. 1-13, 2024a. doi: ALLL09/TTAMM2222..3333.7
1, 3, 4,5
Md Kaykobad Reza, Ashley
and M. Salman Asif. Mmsformer: Multimodal transformer for material and semantic segmentation. IEFE Open Journal of Signal Processing, 5:599-610, -100,660,--0 2024b. doi: /00.0////JSP.2243338811111
Zhiyi Shi, Junsik Kim, Wanhua Li, Yicong Li, and Hanspeter Pfister. Mora: Lora guided multi-modal disease diagnosis with missing modality. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 273 282. Springer, 2024. 3
15

---

Published in Transactions on Machine Learning Research (10/2025)
Ukcheol Shin, Kyunghyun Lee, In So Kweon, and Jean Oh. Complementary random masking for geb-thermal semantic segmentation. In 2024 IEEE International Conference on Robotics and Automation (TCRA), pp. 11110 11117. IEEE, 2024. 3
Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained vidcos. In Proceedings of the European conference computer vision (ECCV), pp. 247- 263, 2018. 6,
Yao-Hung Hubert Tsai, Shaojic Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multinodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 0558-6568) July 2019. doi: 10.18653/ 110/5633/11191166666
Hu Wang, Yuanhong Chen, Congbo Ma, Jodic Avery, Louise Hull, and Gustavo Carnciro. Multi-modal learning with missing modality via shared-specific feature modelling. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pp. 15878-15887 2023. 3
Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-moda classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12695-12705, 2020a. 7,
Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso. Recipe recognition with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp- 1 1-6. TEFE, 2015. 6,
Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, and Junzhou Huang. Deep multimodal fusion by channel exchanging. Advances in neural information processing systems, 33:4835 4845,-484,44 2020b.
Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, and Yunhe Wang. Multimodal token fusion for vision transformers. In Proceedings of the IEEE/CVP conference compuler vision and paltern recognition, pp. 12186-12195, 2022a. 3
Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, and Yunhe Wang. Multimodal token fusion for vision transformers. In Proceedings of the IEFE/CV /EF conference on computer vision and pattern recognition, pp. 12186 12195, 2022b. 1
Shicai Wei, Chunbo Luo, and Yang Luo. Mmanet: Margin-aware distillation and modality-aware regular- ization for incomplete multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 20039 20049, 2023. 3
Sangmin Woo, Sumin Lee, Yeonju Park, Muhammad Adi Nugroho, and Changick Kim. Towards good practices for missing modality robust action recognition. Tn Proceedings of the AAAT Conference on Artificial Intelligence, volume 37, pp. 27642784, 2023.
Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12113-122 12132, 2023. 1, 4
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVT international conference on compuler vision, pp. 11975 11986, 2023. 10, 11
Jiaming Zhang, Ruiping Liu, Hao Shi, Kailun Yang, Simon Reiß, Kunyu Peng, Haodong Fu, Kaiwei Wang, and Rainer Stiefelhagen. Delivering arbitrary-modal semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1136 1147, 2023a.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Jocy Tianyi Zhou, and Xi Peng. Provable dynamic fusion for low-quality multimodal data. In International conference on machine learning. pp. 41753 41 469. PMLR, 2023b. 6, 7,
16

---

Published in Transactions on Machine Learning Research (10/2025)
Xiaohui Zhang, Jachong Yoon, Mohit Bansal, and Huaxiu Yao. Multimodal representation learning by alternating unimodal adaptation. In Proceedings of the IERE/CVE Conference on Compuler Vision and Pattern Recognition, pp. 27456 27466, 2024 6, 7, 8,
Benjia Zhou, Pichao Wang, Jun Wan, Yanyan Liang, Fan Wang, Du Zhang, Zhen Lei, Hao Li, and Rong Jin. Decoupling and recoupling spatiotemporal representation for rgb-d-based motion recognition. In Proceedings of the IBEE/CVP Conference on Compuler Vision and Pallern Recognition, pp.
2022. 1
17

---

Published in Transactions on Machine Learning Research (10/2025)
A Appendix
A1 Related Work on Multimodal Learning
In this section, we provide a general overview of multimodal learning, which involves integrating information from different input sources to improve performance on downstream tasks. We can broadly divide multimodal learning approaches into two main categories: designing specialized models and adapting existing models.
Designing specialized models to effectively fusee multiple input sources is a common strategy in multi- modal learning. For vision-language tasks, models such as ViLT (Kim et al., 2021), ViLBERT (Lu et al., 2019), and Visual-BERT (Li et al., 2019) process image patches and text tokens directly within trans- former architecture for tasks like image captioning and visual question answering. MMBT (Kiela et al., 2019) integrates both text and images within a BERT-based framework (Devlin et al., 2019) to model cross- modal dependencies. MBT (Nagrani et al., 2021b) leverages fusion bottlenecks for efficient modality-speciffi interaction, particularly in audio-video tasks. Specialized multimodal models have also been proposed for other applications, including action recognition (Woo et al., 2023; Zhou et al., 2022), segmentation (Zhang et al., 2023a; Wang et al., 2022b; Reza et al., 2024b), and sentiment analysis (Woo et al., 2023; Tsai et al., 2019). While these models excel in specific tasks, extending them to new modalities or domains often proves to be challenging and resource-intensive, as it requires retraining or significant modification of the model architecture.
Adapting existing models offers an alternative approach to multimodal learning, focusing on the use of pre-trained unimodal models and adapting them for multimodal tasks. One such strategy is gradient modulation, where techniques like G-Blend (Wang et al., 2020a) modulate gradients to combine unimodal and multimodal knowledge. OGM-GE (Peng et al., 2022) improves upon this by emphasizing modality- specific information, enhancing the model's ability to leverage unique features from cach input source. PMR (Fan et al., 2023b) further refines this approach by modulating parameter groups based on the relevance of each modality, while QMF (Zhang et al., 2023b) introduces quadratic modulation for adaptive fusion. More recently, MLA (Zhang et al., 2024) proposed alternating adaptation, which toggles between unimodal adap- enable more efficient cross-modal learning. Another notable approach,
tation and gradient modulation MMLoRA (Du et al., 2023), uses a two-phase strategy: unimodal fine-tuning followed by multimodal adap- tation, which helps achieve optimal performance across a variety of tasks. However, while these methods allow for effective cross-modal learning, they often require significant computational resources and lcarning a large number of parameters to achieve optimal performance on downstream lasks.
A2 Detailed Dataset Description
UPMC Food-101 dataset (Wang et al., 2015) is a popular and challenging multimodal classífication dataset. It has two input modalities: image and text. The dataset has 90,704 samples divided into training and test sets having 67,988 and 22,716 samples, respectively. It contains 101 classes, which are the same as those in the ETHZ Food-101 dataset (Bossard et al., 2014). The samples are noisy, each category contains around 5% irrelevant images as they were collected in an uncontrolled environment without any human human intervention during the data collection process.
MM-IMDb dataset (Arevalo et al., 2017) is a widely used multimodal dataset for multi-label movie genre classification task. It consists of 25,959 samples, cach containing both image and text as input modality. The dataset is divided into three subsets: 15,552 samples for training, 2,608 for validation, and 7,799 for testing. It has 23 classes and each sample can have one or more genre. This dataset serves as standard benchmark for evaluating multimodal models in the multi-label classification task.
Kinetics-Sound (KS) dataset (Arandjelovic & Zisserman, 2017) is a subset of the Kinetics-400 dataset (Kay et al., 2017). It is used for human action recognition using audio and video as input modalities. This datasct includes 31 different action classes that span a variety of everyday human activitics, making it suitable for evaluating multimodal models. The dataset is split into two subsets: a training set containing 14, 1,,739 samples and a test set containing 2,594 samples.

---

Published in Transactions on Machine Learning Research (10/2025)
Table List of all the hyperparameters used in our experiments.
Image-Tert Text Datasets
Audio-Video Datasets
Hyperparameters
Learning Rate
10 10-3
x 10-5
Scheduler
Polynomial
Polynomial
Power
0.9
0.9
Num Epochs
10
100
Warmup Epochs
5
5
Optimizer
AdamW
AdamW
10 8
10-8
Weight Decay
0.02
0.02
0.20
0.20
LoRA Rank
LoRA Alpha
LoRA Dropout
0.1
0.1
Image Encoder
ViT-B
ViT-B
Text Encoder
BERT-Base
Audio Encoder
AST
Image Resolution
224 x 224
224 224
Patch Size
16
16 16
Batch Size
8
Audio Sampling Rate
16 kHz 128
Frequency Bins
Maximum Length
1024 Frames
Frames Per Video
3
Python Version
3.8.19
3.8.19
PyTorch Version
2.2.2
2.2.2
GPU Model
RTX 2080Ti 2
RTX 2080Ti
Num GPUs
Audio- Visual Event (AVE) Localization datasct (Tian et al., 2018) is benchmark dataset used for multimodal event localization task. It contains 4,143 10-second videos. The dataset is divided into training, validation, and test sets, with 3,339 videos for training, 402 for validation, and 402 for testing. The dataset encompasses 28 different event categories providing a broad range of scenarios where both audio and visual signals are crucial for accurate event detection. All vidcos in the AVE dataset are collected from YouTube.
CREMA-D dataset (Cao et al., 2014) is used for multimodal emotion recognition. Tt has audio and video input modalities. The dataset contains 7,442 short video clips performed by 91 actors. The clips cover six emotions: angry, happy, sad, neutral, disgust, and fear. Emotion labels were obtained from 2,443 crowd- sourced raters to ensure diverse evaluations. The dataset is divided into 6,698 samples for training and 744 samples for testing.
A3 Implementation Details
In Section 4.2 of the main paper, we brielly discussed the hyperparameter settings used in our experiments. Here, we provide a complete list of all the hyperparameters and their values in Table 6. The "Image- Text Datascts' column includes the hyperparameters for UPMC Food-101 and MM-IMDb datascts, while the "Audio-Video Datasets' column lists the hyperparameters for Kinotics-Sound, AVE, and CREMA-D datasets.
2

---

Published in Transactions on Machine Learning Research (10/2025)
Table Performance comparison with different LoRA ranks (r). The models are trained on complete multimodal data and tested on both complete and missing modality scenarios. The best Accuracy (Acc.) for each scenario is highlighted bold. Params (M) indicates the number of learnable parameters millions.
1
2
4
Datasets
Available Modality
Missing Modality
Params (M)
Acc. (%)
Params (M)
Acc. (%)
Params (M)
Acc. (%)
Image
Text
75.66
75.62
70.38
UPMC Food-101
Text
Image
0.271
85.31
0.376
85.61
0.671
83.92
Image
Text
94.47
94.64
93.30
Audio
Video Audio
85.21
85.07
84.33
AVE
Video
0.173
84.58
0.319
84.30
0.614
80.84
Audio Video
96.77
96.77
96.27
Andio
Video Audio
67.20
65.32
63.46
CREMA-D
Video
0.157
76.21
0.303
75.54
0.598
76.61
Andio Video
88.84
87.65
87.77
Audio
Video Audio
68.27
69.93
69.54
KS
Video
0.176
85.77
0.322
85.20
0.617
86.43
Audio
Video
91.21
91.06
90.59
A4
Determining the Optimal LoRA Rank
To determine the optimal LoRA rank, we conducted an ablation study evaluating ranks r E (1,2, 4). For this analysis, we trained our models on modality-complee data and evaluated their performance in both complete and missing modality scenarios, with the results presented in Table
Our analysis indicates that increasing the rank, and thereby the number of learnable parameters, does not guarantee consistent improvement in performance. For instance, the AVE dataset, rank of consistently achieves the highest accuracy. On the UPMC Food-101 dataset, performance between == =1 and r= 2 is comparable. While r = offers a marginal improvement in two scenarios, it requires a 38.7% increase in learnable parameters. In contrast, using r = consistently resulted degraded performance in both of these datasets compared to r = 1. A similar trend is observed for the CREMA-D dataset, where = performs best in two of the three scenarios. Conversely, the KS dataset presented mixed results, with different ranks yielding optimal performance in different scenarios.
Overall, rank of r demonstrates competitive, and often superior performance across a majority of the tested scenarios. The performance degradation at higher ranks, such as r = 4, suggest that larger number of learnable parameters may not be necessary for these tasks and could introduce a risk of overfitting. This aligns with the common characteristic of LoRA, where a low rank is often sufficient to capture essential lask-specific information. Based on these observations, we use LoRA with r = for all our experiments as it provides an optimal balance, delivering robust performance with the greatest parameter efficiency.
A5 Different Modality Ratios During Training and Inference
A robust multimodal learning method should maintain strong performance even when the missing modality ratio during inference differs from that during training. To evaluate the robustness of our approach, we follow the experimental setup of Jang et al. (2024); Kim & Kim (2024) and train and test our model with varying modality ratios. Let 77 denote the missing rate. We consider three scenarios: missing-text, missing-image. and missing-both. In the first two cases, 17% of the samples contain only images or only text, while (1 (1-1) 7)% retain both modalities. For the missing- both case, 5% of the samples contain only texts, 3% contain only images, and (1 - 7)% include both modalitios. We set 70 throughout our experiments, following existing baselines (Jang et al., 2024; Kim & Kim, 2024).
As summarized Table 8, our method consistently outperforms existing state-of the--artttte approaches in most scenarios. While the method proposed by Kim & Kim (2024) performs well in certain cases, our approach demonstrates superior performance across both the MM-IMDb and UPMC Food-101 datasets. To ensure rigorous evaluation, we report the mean and standard deviation (SD) over three independent runs with
3

---

Published in Transactions on Machine Learning Research (10/2025)
Table Performance comparison when modality gets missing during both training and inference. We train and evaluate our model with different modality ratio following existing baselines (Jang et al., 2024; Kim & Kim, 2024) and report the mean and standard deviation (SD) three independent rums with different seeds.
Missing Rate (17
Train
Inference
ViLT
MAP
MSP
Kim & Kim (2024)
CMPT (Ours)
Datasets
Image
Text
Image
Text
(2021)
(2023)
(2024)
(Mean SD)
30%
100%
37.63
47.18
47.45
56.03
60.21 = 0.40
30%
100%
65%
65%
34.47
37.25
42.17
49.81
50.05 0.53
100%
30%
30.00
24.31
34.32
43.07
36.10 + 0.19
MM-IMDb
30%
100%
36.48
42.72
44.81
55.26
57.45
0.21
70%
(F1-Macro)
65%
65%
65%
65%
36.54
40.84
42.03
49.24
54.04 = 0.23
100%
30%
33.76
37.80
37.30
42.46
50.05 = 0.29
30%
100%
27.25
20.17
34.11
54.67
44.72 +: 0.46
100%
30%
65%
65%
32.51
29.79
36.97
49.09
49.26
0.42
100%
30%
34.26
36.89
38.34
43.21
52.61
30%
100%
76.40
86.34
86.34
87.11
87.48
0.13
30%
100%
65%
65%
59.03
57.92
71.88
81.22
77.52
0.29
100%
30%
41.94
29.71
56.87
75.41
67.60 + 0.26
UPMC Food-101
30%
100%
73.62
85.55
85.91
87.35
87.36
70%
(Accuracy)
65%
65%
65%
65%
68.60
78.49
78.89
82.67
82.83
0.25
100%
30%
64.25
71.17
71.58
78.13
78.40 = 0.09
30%
100%
43.67
27.46
57.22
86.90
85.34 + 0.15
100%
30%
65%
65%
54.75
50.79
65.13
82.35
82.78 = 0.22
100%
30%
66.01
73.71
73.77
78.81
80.37 = 0.18
96
95
93.55
90.00
Acsuraty (%6
85.31
86.27
8521
88.36
86:20
80.00
90
87.47
90
88:68
68:77
90.14
92
47C16
93.24
Accurssy
85
75
90
70
7/40
-241
75.4
81.04
81:15
84.79
71.85
80.74
76.14
30.10
93.24
94.46
04.75
36.06
Baseline
85
66.700
08.76
69.33
57.59
Baseline
63.32
with Modality Dropout
54.60
with Modality Dropout
8034-6114
Modalily Dropout
50.07
with Modality Dropout and CMPT
80
50
0%
10%
20%
30%
40%
50%
60%
70% 80%
90%
100%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Available Image
Available
(a) Available: Text, Missing: Image.
(b) Available: Image, Missing: Text.
Figure 7: Effectiveness of CMPTs in handling missing modalities on UPMC Food-101 101 dataset. CMPTs significantly
improves performance when modalities are severely missing. Models are trained on 100% image 100% and tested with varying levels of missing modalities.
different seeds. The low SD values further indicate the stability and reliability of our method. These results substantiate our claim that CMPTs can effectively estimate missing modality features, enhancing robustness to varying missing modality conditions. Other methods reported in Table do not report results for this specific experimental setup, and thus we are unable to compare with them.
A6 Effectiveness of CMPTs
We extend the study from Section 4.6.1 1 to evaluate the effectiveness of CMPTs handling missing modalities on the UPMC Food-101 dataset. We compare it against two basclines: (i) standard model trained without modality dropout CMPTs (baseline) and (ii) model trained with modality dropout, where modalities
4

---

Published in Transactions on Machine Learning Research (10/2025)
without CMPT
with CMPT
100
75
Acouracy (%)
60
25
guacamole pancaken fole gran clam chooder
frazen, yogurt
Enticchett fried calamann aprle pio
club sandwich ictister bisquee
chocollate moines
lacos
broad purking
susti
alonk
lasarpon
Oyeterg
Fanadia
wolffo
caroaocio
fried rice
roto sunverd eated
panna calta
poking dlick and sour soup
(ho huiry back rbs
pad thail rachon
davied aggo
lokoyati
hummun
inco seup
cmbb. cakes
garle broad
tuna tortare
fish. and chips
cannoll
raman
cakou
ricitto
gyora Shortc-
baicieva beet aolaad
oggs
beef
strawberry.
hot
Class Name
without CMPT
with CMPT
100
75
Accuracy (%)
80
25
carrot, cake gnocchi
spagheti belogn...
grited salmon
chures
creme. bruleo ravaoll beef tartarg hot dog
huevas rancheros
breakled burrit
gecanjots
chicken. curry cevicha
prime. nb
macatora croszlate cake
red. veivel cake
pulled pork card.... macaroni and on.
chicken quasadilla
salad
intatel
poutine
chosecate
scallops
omelette
edamama
chicken wimps
franch fries
framiiu
dumplnggs
canser salad
pizza
onton rings
frenchh Loast
grank salad
shrupp and gnts
pork chop
crqque mademme
ica. cream
donuts french. onton soup cheesi plate
Inbater roll eand. bibimbapp sachimi
gritad chesse caprese
Class Name
Figure 8: CMPTss improve performance across most of the classes when text modality is missing on UPMC Food- 101 dataset. However, it fails to accurately predict certain classes (e.g., bibimbap, bibimbaapsashhimi). sashimi), likely due to the image modality's insufficient information for those classes. Classes are sorted based on the amount of performance improve- ment.
are randomly removed during training to improve robustness to missing data. For a fair comparison, all models were fine-tuned on the downstream task using LoRA with a rank of
As illustrated in Figure 7, the baseline model experiences a significant performance drop as the proportion of missing modalities increases. While modality dropout improves robustness, it fails to fully recover the lost performance, Incorporating CMPTs further enhances performance, achieving 5.07% improvement over the bascline and 3.81% over modality dropout when the image modality is entirely missing as shown in Figure 7a. A similar trend is observed when text is missing as shown in Figure 7b, where CMPTs achieves 25.59% and 11.96% higher accuracy compared to the baseline and modality dropout, respectively, Consistent with our previous findings, CMPTs becomes increasingly effective as the proportion of missing data increases, demonstrating its strong capability to substantially mitigate performance degradation in the presence of severe modality loss.
5

---

Published in Transactions on Machine Learning Research (10/2025)
without CMPT
with CMPT
100
75
Accuracy (%)
60
26
pazza oup susti cakes
donune condie
cofo retutte
pun that
french shorts...
devied appoe pho fole grau
lotster. blsque
lakoyot french onion, soup gritad, chemse aie capreise dakee!
and chips cream frenn fe escarporo
wafffen
she ch, dick snkking conoti Corte
dl Pintta moch
heote hot, beel cornane diee and sortre
cromte
straeberry
macaroni.
Class Name
without CMPT
with CMPT
100
75
Accuracy (%)
80
25
parna colta shrimp and grtte Garnest
frozan yogurt hamburger
liremion
poutins ringo wo chicken
sookk huevon, sercin sole
eid carho to mer
sour
faldtal
conetto
chocolate mruuss
pannrraa
mussalss
nacher
calerr solegn Gollnto pork, chup
ramen puacamelo St saled breakfast burrito hol dog carrol cako bognbap tortare groza quesantillaa
peahimi taby back ribs red, valver, cake fiet. migronn
lobster roll sand. cheese piato
onion
crab
lune
spaghett
chicken
Class Name
Figure 9: CMPTs improve performance across most of the classes when image modality is missing on UPMC Food-101 dataset. However, it fails to accurately predict certain classes (e.g., lobster roll _anddwichh cheese _pcsss)_________________ likely due to the text modality's 's insufficient information for those classes. Classes are sorted based on the amount of performance improvement.
A7 Do CMPTs Benefit All Classes?
To further evaluate CMPT's effectiveness, we conduct a per-class performance analysis on the UPMC Food- 101 dataset, following the approach from Section 4.6.2.
As shown in Figure 8 and 9, CMPT's improve or maintain performance across most of the classes in both text- missing and image-missing scenarios. Notably, it mitigates performance degradation in modality-sensitiyy classes. For instance, when text is missing, classes such as guacomole, pancakes, and foie__rrreee experience severe drops in performance, but CMPTs effectively recovers much of the loss as shown in Figure 7b. Similarly, in image-missing scenarios, CMPTs provide substantial gains for sushi, cup cakes, and pizza as shown in Figure 3a. However, a slight performance drop is observed in certain classes when either text or image is absent. For instance, CMPT fails to recover performance for classes such as bibimbap and sashimi in the absence of text, and for classes like lobster_role Baarr__ooo_sssddsiieai sandwich and cheese_plate when the image modality is missing. This is likely due insufficient information in the available modality alone for those classes
6

---

Published in Transactions on Machine Learning Research (10/2025)
Available: Image, Missing: Text
Available: Text, Missing: Image
Multimodal
with CMPTs
X omelette
pork pork_chop
sashimi
falafel
bibimbap
phetse_plate
poutine
lobster yoll sandwich
lasagna
Figure 10: I-SNE plots the fused feature lokens T for ten classes where performance gains from CMPTs are limited. In these cases, the fused feature embbeddings with CMPTs (blue) remain diffuse and overlapping, and exhibit weak alignment with complete multimodal features (green). Although CMPTs show improvement over the baseline without CMPTs (red), the alignment remains insufficient to form consistently discriminative clusters, indicating limited or no improvement missing modality robustness in these classes.
Overall, these results reinforce our previous findings, demonstrating that CMPTss enhance performance across most classes, leading to strong overall performance improvement.
A8 Limitations and Failure Case Analysis
While CMPTs demonstrate notable improvements in robustness, it is essential to understand their limita- tions. Figure 10 presents t-SNE plots of the fused feature tokens T for ten classes the UPMC Food-101 dataset where the performance gains from CMPTs are limited. In these cases, the fused feature embed- dings with CMPIs (blue) remain diffuse and overlapping, and exhibit weak alignment with the complete multimodal features (green). This is particularly evident for visually ambiguous classes or classes with high intra-class variance the feature embedding space. Although CMPTs show improvernntt over the base- line without CMPTs (red), the alignment remains insufficient form consistently discriminative clusters, indicating limited or improvement in missing modality robustness.
We also analyze the attention maps from the last layer of the vision encoder for the CLS token and CMPTs in the text-missing scenario, as shown in Figure 11. The left half of the figure includes examples when the fused features with and without CMPTs provide correct predictions (approximately 60% samples show this pattern). For these cases, the CLS token already attends to semantically--eleeaatt regions and make the correct prodiction. CMPTs further refine the attention but do not change the model's prediction, as is already correct. Thus, the inclusion of CMPTss does not necessarily improve performance. The right half of the figure shows examples when the fused features with CMPTs lead to incorrect predictions (approximately 3% samples show this pattern). We observe that CMPTss attend to semantically-relevand regions of the image but still lead to incorrect predictions. For example, in the French Onion Soup sample, the CMPT focuses heavily on the melted cheese surface a visually salient but ambiguous feature leading to misclassificatiin as Shrimp and Grits. Similarly, for the Lobster Roll Sandwich, the CMPT fails to recover from the original misclassification and predicts Foie Gras based on misloading visual cues. These examples highlight scenarios where either the available modality is insufficient the learned cross-modal associations are misleading, resulting in CMPTs contributing incorrect predictions rather than correcting them.

---

Published in Transactions on Machine Learning Research (10/2025)
Input Image
Attention
Input Image
CLS Attention
CMPT Attention
GT: Donuts
Pred: Donuts
Pred: Donuts
GT: French Onion Soup
Pred: French Onion Soup
Pred: Shrimp Grits
H+
HH!
GT: Cheese Plate
Pred: Cheese Plate
Pred: Cheese Plate
GT: Sashimi
Pred: Sashimi
Pred: Tuna Tartare
Bibimbap
Pred: Bibimbap
Bibimbap
GT: Lobster Roll Sandwich
Pred: Croque Madame
Pred: Fole Gras
Figure 11: Attention map visualizations from the last layer of the vision encoder for the CLS token and CMPTss in the text-missing scenario the UPMC food-101 dataset. Left: Cases where the fused features with and without CMPTs provide correct predictions. Here, CLS tokens already attend to semantically-relevant regions and CMPTs provide further refinement, but do not change the prediction. Right: Cases where CMPTs atlend semantically- relevant regions but lead to incorrect predictions. The last example illustrates scenario where both the fused features with and without CMPTs result in incorrect predictions. These examples highlight limitations of CMPTs when the available modality lacks sufficient discriminative information when learned tross-modal associations are imprecise.
Image Only
Text Only
Image-Text
100
25
Drama Comedy Ronarco
Thiillr
Grono
Action Adventure
Horror Cocumantary
Myatorr
Fantavy
Begraphy
War
History
Mustic
Animation
Misical
Weatur
Sport
Shart
Film-Nol
Class Name
Figure 12: Per class performance analysis on MM-IMDb dataset. Certain classes show strong dependence on a specific modality.
A9 Analysis of Modality Dependence
We observe that the impact of missing modalities varies significantly across classes. Certain classes show strong dependence on specific modality. In this experiment, we train our model on modality-complete data and evaluate it on both modality-complete and modality-incomplete test samples.
Figure 12 presents per-class performance on the MM-IMDb datasct, revealing heterogencous effects of miss- ing modalities. For instance, the Drama and Thriller classes maintain consistent performance regardless of missing modalities. In contrast, the Biography and Documenlary classes experience significant decline in
8

---

Published in Transactions on Machine Learning Research (10/2025)
Image Only
Text Only
Image-Text
100
Accuracy (%)
50
con eor
Spe ootenn
owi staaip
safling
they cou
Louthur jay
couat
Dingu
guing pomponut
adbapors
four suneeoue spntrns
doup sadd Aouida
Stud
apcettand
yooo qua
coyee
Buntee (onns) dou poudit
prddyya cittings
Busedd
pas lay
qnp
sunyy
p0u Tumaesuu
100
Accuracy (%)
80
70
60
50
oor paly
ond stii-
Sine prcoun e piito auued
co cowo- tio
acon
cen suggont
Dato jospa tou
Bu pisin
enuzaiig uep
oo cxtas Cxtats
Ai Wind
me band
60 Loverugge ueel Pilm Jeccont
aqu Xcri) baboy ecpoo
Dipituung eddlo
sbb Guide eueuccue
samms
holrnn singe aad
sxoapheito bolognass til
Capron sialad
deviled egge
Class Name
Figure 13: Per class performance analysis UPMC Food-101 dataset. Certain classes show strong dependence a specific modality.
performance when text is missing, even if images are available. Interestingly, certain classes like Animation show slightly higher performance with single-modality inputs, particularly in image-only scenario. Similar patterns emerge in Sport and Music categories, where text-only samples performs slightly better than com- plete modality. This discrepancy may stem from the simple sum-fusion strategy we use, which might. not effectively model complex modality interactions. These findings indicate that the effect of missing modalities is not uniform across classes, with some being more modality-ddpendeent than others.
Extending this analysis to UPMC Food-101, we observe that multinodal (inage-text) data consistently outperforms unimodal inputs across all classes as shown in Figure 13. However, the reliance on specific modalities varies widely among food categories.
Classes such as dumplings, Frozen Frozen_yogurt,, and Tee cream depend heavily on text, showing strong perfor- mance with text-only inputs but experiencing significant drops when text missing. Conversoly, categories like tuna tarlare, beef carpaccio, and beel salad rely more on image information, with performance dete- riorating notably when images are absent. This observation suggests that some classes rely more texts, while others depend heavily on images for accurate classification.
Interestingly, some classes such as beef lartare, chicken quesadillo, and croque madame exhibit poor perfor- mance with unimodal inputs but improve significantly with both image and text, reinforcing the importance of multimodal data. Performance on these classes improve significantly when we utilize both image and text. Overall, these findings confirm that while multimodal information universally enhances performance, the degree of modality dependence remains heterogeneous across classes.
9

---

Published in Transactions on Machine Learning Research (10/2025)
Table Performance comparison with three different parameter efficient adaptation methods. We used rank-1
for all the methods. The best Accuracy (Acc.) for each scenario is highlighted in bold. Params (M) indicates the number of learnable parameters millions.
LoRA
DoRA
VeRA
Datasets
Available Modality
Missing Modality
Params (M)
Acc. (%)
Params (M) Acc. (%)
Params (M)
Ace. (%)
Image
Text Image
75.66
69.03
70.82
UPMC Food-101
Text
0.271
85.31
0.302
83.52
0.155
82.85
Image:
Text
94.47
92.59
92.34
Andio
Video Audio
85.21
86.37
89.05
AVE
Video
0.173
84.58
0.246
91.79
0.098
90.05
Audio Video
96.77
96.27
96.27
Andio
Video Audio
67.20
67.07
66.13
CREMA-D
Video
0.157
76.21
0.229
73.
0.082
71.64
Andio Video
88.84
88.04
86.02
Audio
Video Audio
68.27
68.77
69.93
KS
Video
0.176
95.77
0.248
85.97
0.101
83.69
Audio
Video
91.21
91.32
90.71
A10
Generalizability to Other Parameter-Efficien Adaptation Methods
To demonstrate that CMPTs are not restricted to a particular parameter- adaptation method, we extend our analysis beyond LoRA (Hu et al., 2022), which is used in our primary experiments. Specifically, we evaluate CMPTs with two recent alternatives: DoRA (Liu al., 2024) and VeRA (Kopiczko et al., 2024). For fair comparison, we fix the rank across all methods. We train the models on modality-complete data and evaluate them both complety and modality-incopplet scenarios. The results are summarized in Table 9.
We find that the number of learnable parameters differs across methods, with VeRA being the most parameter-efficien and DoRA the least. In terms of performance, no single method consistently outper- forms the others across all datasets. LoRA achieves the best overall results on the UPMC Food-101 and CREMA-D datascts, while DoRA shows superior performance on the Kinetics-Sound dataset and remains highly competitive AVE. VeRA, while using substantially fewer parameters, delivers strong performance but is generally slightly below LoRA and DoRA, suggesting trade-off between parameter efficiency and maximal performance.
Most importantly, our framework maintains robust performance missing-modality scenarios regardless of the underlying adaptation mothod. The relative stability observed between complete and missing-modalife settings is consistent across all three adaptation techniques. These findings confirm that CMPTss function as modular approach whose effectiveness is not tied to any specific adaptation strategy, underscoring their scalability and versatility across different parameter-efficient methods.
A11 Applicability to SigLIP-Based Models
To assess the architectural flexibility of our method, we replaced the VIT encoder (Dosovitskiy et al., 2021) with a pretrained SigLIP model (Zhai et al., 2023), which employs Global Attention Pooling (GAP) instead of a CLS token. For audio-vidco datasots, we retained the AST model as the audio encoder, and for imago-text datasets, we used BERT as the text encoder. In this setup, CMPT tokens must perform cross-architectural reconstruction: inferring missing audio or text features from SigLIP's vision embedding, and vice versa.
We train the models on modality-complete data and evaluate them on both modality-complete and modality- incomplete scenarios. The results in Table 10 show that our method is not restricted to a single encoder type. On UPMC Food-101 dataset, the SigLIP- model consistently outperforms the ViT-based model, demonstrating that CMPTs can be effectively applied to GAP-basod vision backbones for vasion-language tasks. In contrast, for audio-video datasets, SigLIP exhibits sharp performance degradation when audio is missing. Here, CMPTs struggle to recover audio features from global video embeddings, likely because
10

---

Published in Transactions on Machine Learning Research (10/2025)
Table Performance comparison our method with VIT versus SigLIP model as the vision backbone. The models are trained on complete multimodal data and tested on both complete and missing modality scenarios. We report % accuracy as performance metric, and highest score for each scenario is highlighted bold.
ViT
SigLIP (Zhai et al. (2023))
Datasets
Available Modality
Missing Modality
(Dosovitskiy et al. (2021))
Image
Text
75.66
77.50
UPMC Food-101
Text
Image
85.31
85.58
Image
Text
94.47
94.58
Audio
Video Audio
85.21
86.32
AVE
Video
84.58
30.60
Audio Video
96.77
94.78
Audio
Video Audio
67.20
65.73
CREMA-D
Video
76.21
35.35
Audio Video
88.84
84.68
Audio
Video
68.27
69.66
KS
Video
Audio
85.77
30.76
Audio
Video
91.21
89.44
Table 11: Performance comparison of method using Base Large VIT and BERT models. The models
are trained on complete multimodal data and tested on both complete and missing modality scenarios. We report % accuracy for UPMC Food-101 and F1-Macro learnable parameters in millions. The best performance for each scenario is highlighted in bold.
for MM-IMDb dataset. Params (M) indicates the number of
Available Modality
Missing Modality
ViT-Base and BERT-Base Performance
ViT-Large and BERT-Large Performance
Datasets
Params (M)
Params (M)
Image Text
Text Image
46.97
53.42
MM-IMDb (F1-Macro)
0.211
56.32
0.540
59.92
Image Text
63.58
64.00
UPMC
Image Text
Text
75.66
77.17
Food-101
Image
0.271
85.31
0.501
83.91
(Accuracy)
Image Text
94.47
93.74
GAP discards the fine-grained temporal information that is critical for inferring a corresponding andio representation. The reverse process, however, is more successful. When the video modality is missing, AST's audio features provide a strong basis for reconstructing SigLIP's global video representation. This suggests that AST's feature space more compatible with SigLIP than vice versa.
Overall, these findings indicate that CMPTs remain applicable to GAP-based models like SigLIP, but their effectiveness depends the modality. While GAP facilitates efficient vision embeddings, it limits the rep- resentational richness required for audio reconstruction, constraining CMPTs in certain cases. Nonetheless, the results confirm that CMPTs are not inherently incompatible with SigLIP, though their utility is shaped by architectural and modality-specific factors.
A12 Scalability with Larger Vision and Text Encoders
To assess the scalability of our framework with larger encoders, we replaced the ViT-Base and BERT- Base encoders used in our primary experiments with ViT-Large and BERT-Large. We train the models on modality-complett data and evaluate them on both modality-conplele and modality-incomplete scenarios on the UPMC Food-101 and MM-IMDb datasets.
The results, summarized in Table 11, show that our framework can leverage the increased capacity of larger backbones. On MM-IMDb, scaling yields consistent improvements in F1-Macro across all scenarios.
11

---

Published in Transactions on Machine Learning Research (10/2025)
The gains particularly notable in missing-modalle cases, with +6.45% increase when text modality is missing and +3.60% when image modality is missing. These results suggest. that stronger backbone representations improve CMPT's ability to reconstruct missing features and maintain robust predictions.
On UPMC Food-101, the larger backbone improves image-only accuracy by +1.51% but shows a decline of -1.41% for text-only performance. With both modalities available, performance remains comparable (94.47 vs. 93.74). TTiss indicates a potential trade-off, where the added capacity may overfit to the text modality, reducing generalization when the image is absent.
In summary, CMPT scales effectively with larger backbones, particularly on complex datasets such as MM- IMDb. At the same time, the results highlight that the benefits of scaling depend on dataset and modality characteristics, underscoring the need for careful backbone selection to balance capacity and generalization.
12


Reviews:
[{"review_id": "ysJHFEhVSU", "reviewer": "Reviewer_ZCWM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-\tThe method is simple and effective.\n-\tThe paper is well written and easy to follow.\n\nWeaknesses:\n\n-\tThe method uses pretrained encoders, creating a new classifier based on the CLS tokens of the modalities. How would this method work if there is already a multimodal classifier trained?\n-\tThe experiments are performed using ViT-B from CLIP. How does the method work on bigger architecture? How about architectures such as SigLIP that does not have a CLS token?\n-\tThere is a missing related work that is very similar (applied on different domain) to this work, MoRA [A] that was not discussed in the related work.\n-\tWhy was LoRA chosen when there are many other low-rank adaptation methods in the literature (eg. DoRA, VeRA etc) \n-\tHow would this method work with more modalities such as 3?\n-\tSince the method is straightforward, more experiments are needed to improve the relevance of the paper.\n\n[A] MoRA: LoRA Guided Multi-Modal Disease Diagnosis with Missing Modality, MICCAI 2024", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method for low-adaption multimodal classification under missing modalities. The method proposes to use a new token CMPT for each modality that can learn to predict CLS token of the missing modality. The encoder parameters are updated using LoRA. The paper presents SOTA results on different datasets including audiovisual and text-images.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "see Weaknesses above", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "drgC5nacSG", "reviewer": "Reviewer_Th7k", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths  \nPaper is well written and explanations are clear. While the proposed method does not seem extremely novel, its simplicity and elegance are appreciated (as well as its strong performance). The experimental validation seems solid: extensive experiments across multiple datasets and scenarios, as well as reporting mean & stdev (though it seems this is only reported in Table 1?).\n\n\nWeaknesses  \n- The choice of lambda = 0.2 in Sec 3.4 is not justified at all. This warrants more context.  \n- While I understand the adoption of benchmarks used by previous work, the selection of AudioVisual datasets seems a bit weak, with AVE and CREMA-D being pretty small, and KS not much larger. It'd be great if authors included an additional dataset in their study, eg the larger and more recent VGGSound.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on multimodal learning in scenarios of missing modalities. It proposes cross-modal proxy tokens (CMPTs) to learn an approximation of the missing modality class tokens from the available modality using an alignment MSE loss. The paper applies LoRA to frozen unimodal encoders to learn the CMPTs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical for acceptance:  \n0- please justify the usage of  lambda = 0.2 in Sec 3.4. Ideally, authors would include a Table sweeping lambda in [0, some value) for one of the basic comparisons. Ideally, this would be carried out for 2 datasets, to show if the choice of lambda is critical and/or very dataset-dependent.\n\n\nOther optional comments to strengthen the work:  \n1- As mentioned in Weaknesses, it would be a service to the community if authors paved the way for more comprehensive evaluations. Specifically, a) reporting results for different missing rates of images in Sec 4.5.; b) IT'd be great if authors included an additional dataset in at least one of their studies, eg the larger and more recent VGGSound. For these two cases, authors cannot compare to previous work, but authors can set a baseline for future works.  \n2- it would be greatly appreciated to report results using mean & stdev (as in Table 1) in at least some of the other Tables too.  \n3- in Fig 1, consider inverting the arrows feeding the alignment loss such that they reflect CMPT_1 vs CLS_2 and vice versa  \n4- seeing as the considered tasks are eminently classification, perhaps this can be mentioned earlier in the text, eg Sec 3.4, or even in the Introduction.  \n5- authors state: \"LoRA with rank 1 is sufficient to achieve better performance compared to existing state-of-the-art methods\". Have authors considered other ranks? Are further boosts observable? Please provide more context.  \n6- in 4.3.1 it would be prudent to explicitly acknowledge the fact that the proposed system works comparable to that of Kim & Kim when reporting UPMC Food results (in two conditions).  \n7- in 4.3.2, maybe first two sentences are redundant?", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IvyxJMipDt", "reviewer": "Reviewer_SQkA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[S1] The idea of a proxy token as a stand-in for missing modality representations is easy to implement, avoiding the need for complex imputation.\n\n[S2] By employing frozen encoders and rank-1 LoRA adapters, the method keeps the learnable parameter overhead very low.\n\n[S3] The experimental section is extensive, featuring comparisons across different missing configurations (training/inference), strong baselines, generalization tests, and per-class impact analysis.\n\n[S4] The method maintains strong predictive performance even as modality availability degrades.\n\n---\n\n[W1] While the empirical results are well-demonstrated, the paper lacks a deeper theoretical or conceptual analysis of why the CMPTs effectively capture the semantics of missing modalities. Are they learning structured embeddings, redundancy, or co-occurrence priors?\n\n[W2] Scalability beyond two modalities is unclear. All experiments are limited to bimodal settings. Extending the method to three or more modalities (e.g., image, audio, text) introduces ambiguity in token alignment, gating, and fusion, yet this is not addressed.\n\n[W3] While dependence on pretrained encoders is efficient, this dependence also limits flexibility. The frozen encoder assumption may not hold in niche or low-resource modalities/domains where pretrained models are unavailable or poorly aligned.\n\n[W4] Despite overall improvements, certain classes (e.g., documentary, short) degrade significantly in the absence of text. This suggests that CMPTs struggle to extrapolate for semantically sparse or modality-specific categories.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Cross-Modal Proxy Tokens (CMPTs) as a parameter-efficient mechanism to enhance the robustness of multimodal models in scenarios involving missing modalities. CMPTs serve as surrogates for missing modality class tokens, learned via attention over available modality tokens and supervised through an alignment loss. The framework builds on pretrained unimodal transformers, adapted using lightweight low-rank adapters (LoRA), and integrates a gating mechanism for selective fusion. The authors evaluate the method across five datasets spanning image-text and audio-video tasks and benchmark against a comprehensive set of state-of-the-art methods. The results demonstrate consistent gains under both complete and partial modality settings.\n\nThe key contributions include:\n- A simple yet effective method for missing modality compensation via proxy token learning.\n- Robust performance across datasets and tasks, with empirical superiority in both full and partial input scenarios.\n- Extensive experiments including ablation, class-wise analysis, and generalization under varying missing ratios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add qualitative analyses such as attention visualizations or embedding projections to show what CMPTs “attend to” and how closely they resemble the original CLS tokens of the missing modality, and show token-level dynamics in varying input conditions (e.g., which modality dominates proxy behavior).\n2. Evaluate in multi-modal (>2) settings. Add at least one tri-modal experiment (e.g., audio-video-text), even if synthetic or partial, to test scalability and fusion behavior.\n3. Include a comparison against partially fine-tuned models (e.g., unfrozen last layers of the encoder) to clarify how much of CMPT’s gain comes from structure vs. just adaptation strategy.\n4. Discuss failure modes more explicitly where performance drops on certain classes or modality losses are not recoverable, including a dedicated discussion or error analysis table to help practitioners make informed deployment decisions.", "broader_impact_concerns": "If training data embeds biased correlations (e.g., gendered visual-textual links such as associating certain professions with specific genders or linking emotional expression to particular demographics), CMPTs may propagate or amplify these, especially in missing modality cases where the full context is unavailable to correct such biases.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)