Deep Correlated Prompting for Visual Recognition with Missing Modalities
Authors: Lianyu Hu
Tongkai Shi
Wei Feng
Fanhua Shang
Liang Wan
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal prompting
Mutltimodal models
Missing modalities
Primary area: machine_vision
TL;DR: A deep correlated promp learning paradigm for visual recognition for missing-modality scenarios with minimal computational costs

Abstract: Large-scale multimodal models have shown excellent performance over a series of tasks powered by the large corpus of paired multimodal training data. Generally, they are always assumed to receive modality-complete inputs. However, this simple assumption may not always hold in the real world due to privacy constraints or collection difficulty, where models pretrained on modality-complete data easily demonstrate degraded performance on missing-modality cases. To handle this issue, we refer to prompt learning to adapt large pretrained multimodal models to handle missing-modality scenarios by regarding different missing cases as different types of input. Instead of only prepending independent prompts to the intermediate layers, we present to leverage the correlations between prompts and input features and excavate the relationships between different layers of prompts to carefully design the instructions. We also incorporate the complementary semantics of different modalities to guide the prompting design for each modality. Extensive experiments on three commonly-used datasets consistently demonstrate the superiority of our method compared to the previous approaches upon different missing scenarios. Plentiful ablations are further given to show the generalizability and reliability of our method upon different modality-missing ratios and types.

Deep Correlated Prompting for Visual Recognition with Missing Modalities
Lianyu Hu,Tongken Shi, Wei Feng, Fanhua Shang, Liang Wan" College of Intelligence and Computing. Tianjin University
[hly2021, stk, www.g, 1tan) i hwtp////wlaaayyyyy/ppp./oalaaaae
Abstract
Large-scale multimodal models have shown excellent performance over a series tasks powered by the large corpus of paired multimodal training data. Generally, they are always assumed to receive modality-complest inputs. However, this simple assumption may not always hold in the real world due to privacy constraints collection difficulty, where models pretrained on modality-complete data easily demonstrate degraded performance we refer to prompt learning to adapt large pretrained multimodal models to handle wwsssing-mddlite scenarios by regarding different missing cases as different types of input. Instead of only prepending independent prompts to the intermediate layers, we present leverage the correlations between prompts and input features and excavate the relationships between different layers of prompts to carefully design the instructions. We also incorporate the complementary semanties of different modalities to guide the prompting design for each modality. Extensive experiments on three commonly-used datasets consistently demonstrate the superiority of our method compared to the previous approaches upon different missing scenarios. Plentiful ablations are further given to show the generalizability and reliability our method upon different modality-missing ratios and types.
missiig-modality cases. To handle this issue,
Introduction
Our human beings typically perceive information of multiple modalities such as visual, linguistic and audio signals to understand the world, where different signals drawn from various perspectives are inherently complementary. Thus, modeling and coordinating multimodal information is of great value for large-scale models to reason about real-world scenarios. Recently, multimodal
models [34] [] 22 29 have developed fast powered by the large collection of multimodal data pairs and evolution of model architectures (e-g., Transformer 31), which demonstrate promising performance across a series of downstream tasks, such model retrieval g [8] 29 [9] image
captioning [24] and image/video generation (33 [3] [22] Supported by their large capacity and general
knowledge acquired by training upon web-scale data, these models have been more and more applied
to daily work our life (e.g., GPT-4 used by millions of people).
However, there exist two major concerns that may hinder these impressive models from broader applications. First, a common assumption previous methods is that the input data is modality- complete, which may not always hold in the real world due to privacy considerations, collection difficulty and security issues [27] 19 When an input modality is missing in general real-world conditions, the performance of these models usually degrades lot (regardless of training or testing settings) 277] which is easily influenced by the input completeness. Second, these large models
are usually parameter-abundan [ 6 21 and require heavy computations 30 45 to pretrain and
Cooresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

finetune on downstream tasks, whose computational demands may not always be available in most real-world applications due to limited computing resources. It's necessary develop a new method to efficiently adapt these powerful methods to perform robustly against missing- modallty scenarios.
19 of multimodal learning have considerably explored the missing- modality issues. Earlier works [2 42 32 28) mostly reconstruct the absent information of missing modalities or use other modalities augment the missing modalities, MMP [4 has first introduced prompt learning to handle missing-modality scenarios by regarding different missing cases as different types of input. The multimodal backbone is kept frozen and only the newly introduced prompts are updated in the fine-tuning process, thus only incurring a few extra computations. However, MMP [99] simply inserts independent prompts into each layer, and overlooks the relationships among prompts and input features. The prompts across different layers lack cooperation to aggregate beneficial information to well guide the model predictions. The fixed prompts are used for different input samples which fail to consider the characteristics of various inputs. The complementary multimodal information of different input modalities overlooked in the fine-tuning process.
Previous works [27
28
To better adapt large multimodal models to missing-modaling scenarios, we introduce deep correlated prompting (DCP) by capturing different types of correlations between prompts and input features. Specifically, to leverage the hierarchical semantics of different layers, we propose correlated prompts by perceiving beneficial information from preceding layers to instruct the features of the current layer. We further propose to dynamically generate the prompts according to the input features to better fit the characteristics of different inputs. To leverage the complementary information of multimodal inputs, we decompose the prompts into modal-common and modal-specific parts to guide each encoder to focus on its unique features. The proposed prompts are concatenated and prepended to the input and intermediate features of the multimodal backbone to instruct the model to alleviate the performance drop caused by the missing modality. The multimodal backbone keeps frozen during training and only the learnable prompts are tuned, thus offering high training efficiency. Extensive experiments on three widely-used datasets including MM-IMDb 22, UPMC Food-101 33 and Hateful Memes [17 demonstrate consistently superior performance compared to other methods across all benchmarks, which verify the effectiveness of our proposed method. Ablation studies are further given to verify the generalizability and reliability of our method upon different
types and ratios.
2 Related Work
2.1 Missing-Modality for Multimodal Learning.
Earlier works of missing modalities for multimodal learning mostly generate the missing modality based other modalities [42 or align features of latent representations for multiple modalities
to help recognition [3 15] Recently, multimodal transformers [I] 22 23 emerge as an effective
tool to model information from various modalities and process them into a robust representation. which have shown impressive performance over series of tasks [g] 18 29 g [1] However, these methods typically assume that the inputs are in real-world scenarios. When a modality is missing, these methods usually demonstrate degraded accuracy and lead unstable performance 27
which may not always hold
To deal with missing modalities in multimodal transformers, MMIN predicts the intermediate features of the missing modality based on other available modalities, by learning a common multi- modal representation. SMIL 28 proposes a Bayesian meta-learning framework to estimate the latent features of the modolity-incomplete data. It further explores the effects faced with severe modality- incomplete samples (e.g.. 90% missing ratio). Ma et al. 277 test the robustness of multimodal transformers to missing modalities and their missing types, and propose a multitask optimization framework to improve it via modality fusion. Zeng et al. [40] propose a tag-assisted transformer Encoder network handle the problem of missing uncertain modalities. ShaSpec [32] employs a shared head across different tasks to aggregate information from various input samples, which complements the features for missing modalities. However, these methods mostly assume that fixed modality is missing, which is known in advance. Besides, these methods still require updating most parameters of the model, which consume heavy computations in downstream tasks. MMP [9] first introduced missing-aware prompts to handle scenarios with missing modalities with minimal extra computational costs. However, it simply inserts independent prompts into the intermediate features of the multimodal backbone, without considering the relationships between prompts of different

---

layers as well as the correlations between prompts and input features. In contrast, we carefully design the prompts by exploring the correlations between prompts and input features, and achieves much superior performance across all missing-modalty scenarios upon three commonly-used datasets.
2.2
Prompt Learning
Prompt Learning is first explored in neural language processing (NLP), which adopts a "prompt" to modify the input text to instruct the pre- pre-trained model for downstream tasks. Earlier works [ usually adopt manually designed prompts to improve the generalizability of large models over downstream tasks. Later, prompt tuning methods [26
20 25 emerge by prepending learnable prompts to the input features in the training phase to automate the optimization process. Recently,
prompt learning also introduced into computer vision tasks 4 13 37 and multimodal learning
tasks [36 35
[6. 43 CoOp 44 is first proposed to insert learnable soft prompts besides input images adapt vision-language models to various vision tasks. CoCoOp [43) generates an image-conditional prompt to utilize the power of input features. ProGrad 46 only updates the prompts whose gradients aligned to the "general knowledge" generated by the original prompts.
KgCoOp (36] tries to align the output embeddings of the text encoder with those of the pretrained CLIP to preserve beneficial information. MaPLe [16] injects deep learnable soft prompts into both the image and text branches to enable prompt collaboration. DualPrompt [35] extends the prompts to learn different task information conditionally in continual learning. DePT 44 decouples base- specific knowledge from feature channels into an isolated feature space during prompt tuning. These works have demonstrated the effectiveness of prompt learning to adapt large-scale vision- Sanguage models for downstream tasks with minimal extra computation costs. In this paper, we introduce prompt learning into multimodal models to increase their robustness to missing-modality scenarios, via attaching different types of prompts according to various missing cases.
3 Method
3.1
Overall framework
Problem definition. We first provide a succinct overview of the missing- scenario addressed in this paper. Without loss of generalizability, mi and m2 (c.g.. text and image). which is naturally extensible to more modalities. Specifically, with multimodal dataset D (Ds, D D ), we denote D==(z"",, x==', y) as the modality case, where x denotes the input and represents the label. We let Dmi= y) and D>"=="="" v) denote the modality-incompplte cases, wherein modality is absent (e.g., missing text and missing image). Some examples of modality-compler and
explore multimodal inputs with M 2 modalities
inputs are depicted
in the lower part of Fig.
Recap MMP 19]. MMP 19 first introduced missing-aware prompts to handle missing-modality cases. Specifically, it assigns 2d - types of prompts for tasks involving M modalities (e.g.. 3 prompts for vision-language tasks, including one prompt for modality-compleee case, one prompt for image-only case, and one prompt for text-only case). Given an input sample, it first selects the prompts corresponding to the missing case and then prepends the prompts to the input and intermediate features of the multimodal backbone, instructing the pretrained model to perform prediction. In this procedure, only the learnable prompts are updated and the multimodal backbone keeps frozen.
3.2 Overall framework
Though MMP [19] has achieved notable progress in improving the robustness of multimodal models upon missing-modalit cases by incurring minimal computational costs, only assigns independent prompts to the input and intermediate features, which (1) fails to consider the relationships between prompts of different layers, and (2) lacks the correlations between prompts and input features, and (3) overlooks the complementarity of multimodal inputs. To better adapt the pretrained multimodal model for missing-modality scenarios, we propose to design three types of missing- aware prompts by capturing the relationships between prompts and inputs. First, we generate missing-aware prompts by leveraging the correlations of preceding prompts across multiple layers and various modalities. Second, we dynamically generate the prompts for each input sample to fit its characteristics. Third,
3

---

Multimedal data with missing modality
Tulips
Huippospit 1501
Text enender (fixed)
Learnable Frozzn
in
various
shades
Layer
Layer N-1
Layer N
of
Camplete (x""
Oorelced
Select prompts
Concaa
layer
Comriaad
Pyrank:
Image encoder (fixed)
Task
Test-only: 0gg 1
target
Layer
Layer N-1
Layer
N
Imugeanlyy:
The overview of our proposed framework. We first select the prompt PT and pl with E fc, m1, m2} for the text encoder and image encoder according to the missing case (e.g.. complete, text-only, image-only) of the multimodal inputs (x""l x 2²2) The prompt pT (Pi is composed of three types of missing-aware prompts including the correlated prompts PT (P..R), dynamic prompts P (PI.P) and modal-common prompts PT.C (PI,C). Then we prepend the prompts the inputs and intermediate features of both encoders instruct the model to fit the missing case. Finally, we concatenate the task-related token of both encoders as the final representation, and pass through fully-connected layer for class prediction. the whole procedure, only the fully-connected (fc) layer and deep correlated prompts are updated while others keep frozen.
Figure 1:
m
m
decompose multimodal prompts into modal-common parts and modal-specific parts, which fuses beneficial information from other modalities and enables each encoder to focus its unique features.
Fig. [ shows the framework overview of our proposed method. Specifically, without loss of gen- cralizability, we adopt the widespread two-stream multimodal method CLIP (29) as our backbone. Given the input text and image, first employ pretrained text and image embedding layers from the pretrained CLIP (2 to convert them into token sequences. For each input sample, we select the corresponding prompt PT and P/A for the text encoder and image encoder, respectively, given the type missing modality with m E {c, m1, m2). The prompt for each encoder is composed of three types of missing-aware prompts including the correlated prompts pT. (PI."), dynamic prompts PI.D (PI.P) and modal-common prompts pft, (PI.C). We concatenate the missing-aware whole sequence to process. Finally,
prompts and prepend them the input tokens = (22 we concatenate the task-related token of both encoders as the final output representation, and pass through fully-connected layer for task prediction. In this procedure, only the parameters of the fully-connected layer and the newly introduced deep correlated prompts are updated in the training procedure, and the backbone (i.e., the text embedding layer, image embedding layer, text encoder and image encoder) is kept frozen. The illustration of our proposed prompts is given in Fig. 2 We next introduce our prompting designs in detail.
3.3
Deep Correlated Prompt Learning
3.3.1
Correlated Prompts
append independent prompts to the input and intermediate features of the multimodal backbone to guide model predictions. Though could theoretically provide enough guidance for features in each layer, the prompts across each layer lack synergy, which fails to cooperate with representations of each layer that various semantics. We argue that prompts across consecutive layers should be closely correlated to receive necessary semantics from preceding layers to instruct
MMP 19

---

Text cnander
Text snorede
Clootfer
Clasche
Inuie dnowder
Jmage cleckar
(a) Baseline
(b) MMP
encoder
Testi
P0
Clocsierr
Phooifer
proppe
Inage
Imsgee stooka
Correlated prompting
(d) Dynamic prompting
(c) Modal-commonn prompting
Figure 2: (1) Baseline, which simply uses fixed image encoder and text encoder and only finetunes the classifier handle downstream tasks. (2) MMP, which inserts independent prompts each layer to guide the model to handle missing-modality cases. (3) Correlated prompts, which generate the prompts of the next layer based on the prompts of both modalities in the current layer enable cooperation of prompts from both modalities. (4) Dynamic prompts, which dynamically computes the prompts based on different input features to better guide the behavior of the model, avoiding using fixed prompts for different inputs. (5) Modal-common prompts, which store the shared information across different modalities and facilitate the model to encode modal-specific information to better handle the missing scenarios in each modality.
the current layer handle missing modalities. Thus, we propose generate the prompts of the current layer based on the observation of the preceding prompts. Besides, the features of different modalities usually contain complementary information. Leveraging the complementary information from various modalities could further help the model predictions. We propose to incorporate the beneficial information from multimodal inputs to help guide the outputs of each modality.
Specifically, we prepend the prompts to the input features and the intermediate features of the multimodal backbone up to depth of J. Taking the image encoder an example, the calculation process of the ich (i E [0, [O....,.J 1)) layer V; could be expressed as:
x/| V.[[PLR
(I)
Here, X is the features of the ish layer in the image encoder and pl.R is the newly introduced correlated prompts of the 1)th layer. After the Jnn layer, the correlated prompt retained from the previous layer, and the calculation process for the inh (i E [J...., N 1)) layer V, can be presented as:
I m.i x = W((PLR I x/-1)
where N denotes the length of all layers.
To leverage the correlations of prompts between different layers, we generate the prompts of the iu E [1,...,J-- 1)) layer V; by observing prompts of its preceding layer V to inject beneficial information, as:
pl. =
(P
where F-)(()) denotes the prompt generation function for the (i-1) 1)th layer in the image encoder. To decrease the required parameters and add non-linearity to F(.), we adopt F(-) bottleneck

---

MLP with the GELU activation [0] in between, followed by LayerNorm (LN) function
3
as:
F() IN(F(GGIL((F(()))
The intermediate feature dimension of F-) (-) is r (usually r = + times of its imput channel dimension. For the correlated prompts the input level, we leave it randomly initialized without the generation procedure.
To leverage complementary information from different modalities, we incorporate prompts from both encoders to fuse their distinct semantics for instructions. Taking the image encoder as an example, we generate the prompts of the ie (i E [J.....J J 1)) layer V. based on the prompts of the layer Vi-1 in the image encoder and the prompts of layer T--1 in the text encoder as:
P (Concat( m,;-11 - P P))
Here, Concat denotes the concatenation operation. Instead of leaving prompts uncorrelated across different modalities, our design improves mutual synergy between different modalities to help adapt to various missing scenarios.
3.3.2 Dynamic prompts
Different input samples usually contain information of various semantics. Using fixed prompts for different inputs may not well fit their distinct features, and thus can't offer enough guidance for model to fit different mossiiggmddality scenarios. Thus, we propose to dynamically generate the prompts based on the input features to adjust the instructions to fit the characteristics of different inputs.
Specifically, taking the image encoder as an example, we generate the dynamic prompts PI.D based on the input features X as:
D'
(6)
where D' denotes the dynamic prompt generation function for the image encoder. To deal with the varying length of tokens for different inputs, we instantiate DD as self-attention layer [31, whose architecture can be expressed as :
D(-) = IN(MLP(N((HHA()))
Here, MHA is the multi-head attention mechanism in transformers (31). We set the number of heads as I for simplicity. The dynamic prompts are only inserted at the input level. For the subsequent layers, the prompts are retained from the previous layers and updated together with intermediate features following Eq. 2
3.3.3 Modal-common prompts
Multimodal inputs usually contain both modal-specific information and modal-common information across different modalities. By disentangling the modal-common features from the modal-specific features, each modality could aggregate beneficial information from the shared features across various input modalities and build more powerful representations based on its unique characteristics. We decompose the multimodal missing- aware prompts into modal-commor parts and modal-specific parts to store common characteristics across different modalities and model-specific features for each modality, respectively.
Specifically, we introduce modal-common prompt PC for multimodal inputs to embody the mutual features across different modalities, and accordingly encourage the proposed correlated prompts and dynamic prompts to embed modal-specific instructions. To cooperate with the features of different modalities, we project the modal-common prompt PC to the image and text space to obtain pT,C and PI. via projection layers, respectively, as:
PI.C FF (PC) pI. = Gd (P
(8)
Here, G and G are the projection layers for the text modality and the image modality, respectively, which are both instantiated as MLP with the intermediate feature reduction factor r=16. We only insert modal-common prompts the input level and leave the prompt in the intermediate layers reserved and updated following Eq.

---

4 Experiments
4.1 Experimental Setup
Datasets. We follow the previous works 19
27 to evaluate our methods:
MM-IMDb 2 is currently the largest publicly available multimodal dataset for genre prediction on movie genre classification. It is notated with both image and text modalities for 25959 movies. As movie might have several genres, the genre prediction task is thus multi-label classification task.
UPMC Food-101 33] is a large multimedia dataset consisting of noisy image-text pairs collected from Google Image Search with 101 food categories. It has identical categories to the largest publicly available ETHZ Food-101 dataset [5], used to classify the categories of different foods.
Hateful Memes 17 is a multimodal dataset for hateful meme detection (image + text) that contains 10,000+ new multimodal examples created by Facebook AI. To prevent the model from relying on a single modality, it is constructed to make unimodal models more likely to fail.
Implementation details. We use CLIP 299 as our multimodal backbone, with ViT-B/16 7 as the image encoder. For the image input, we follow CLIP [29] to resize input images into 224 224. The patch size set as 16 for the multimodal transformer. For the text modality, we use the tokenizer from pretrained CLIP to tokenize the text input. The maximum length of text inputs is 77. We freeze all the parameters of both the image encoder and text encoder, and only tune the parameters of deep correlated prompts and the fc layer (for task target). We set the length Lp of learnable prompts as 36 and prepend them to the features of M 6 layers. We use Adam optimizer with initial learning rate of 1e-2 and weight decay 2e-2. The learning rate is warmed up for 10% of the total training steps and then is decayed linearly to zero. We perform our experiments with batch size of on a 3090 GPU. For the missing modality, we stop feeding the inputs into the corresponding encoder and use zero-filled tensor as the output instead.
Metrics. We use corresponding proper metrics for each dataset to evaluate our method. For MM- IMDb 21, we adopt F1-Macro to measure the multi-label classification performance. For UPMC Food-101 101 B3 we employ the top-1 classification accuracy to evaluate the recognition performance. [17]. we use Area Underthe Receiver Operating Characteristic Curve (AUROC).
For Hateful Memes
Setting of Missing Modality. In this paper, we focus on the general realistic scenarios in real life, where any modality may appear in both training and testing phases. To stimulate this condition, follow MMP [19] to define the missing rate 1) as the proportion of modality-incomplee data to the entire dataset. For lssoua-anguuae tasks, there exist three types of missing cases: missing-both (text and image), missing-tea and missing-imaas are composed of 3 text-only data, Y image-only data and (I-7)) missing-text and missing-image cases, the training and testing data are composed of 1) image-only (text-only) data and (1-7)) modality-complett data. This definition could be naturally extended to data with more modalities by using 2²=22 podality-incomples data for each missing case and (1-7) complete data. In our experiments, we use 7770% by default.
For the missing- both case, the training and testing data
data. For the
4.2 Experimental Results
Effectiveness. We first verify the effectiveness of our proposed components across different missing cases including missing-image, missing-text and missing-both. We conduct the experiments upon the MM-IMDb [2] dataset with missing rates ranging from 0% to 100%. The methods used for comparison include (I) baseline, which directly sets the features as zeros when a modality is missing; (2) Ours (A), which only equips the correlated prompts; (3) Ours (B), which equips both the correlated prompts and the dynamic prompts; (4) Ours, which uses all the three proposed prompts. The only difference between our method and the baseline is inserting learnable prompts at each layer which only bring few extra parameters. The results are shown in Fig. 3
It's first noticed that compared to the baseline, all our three variants could notably promote the perfor- mance across all missing rates for various missing-modality cases, demonstrating strong robustness to different missing-modality scenarios. Using the correlated prompts boosts the performance most, and equipping the dynamic prompts or modal-common prompts offers a similar performance boost. Using all three proposed prompts achieves the best performance, which verifies the effectiveness of
7

---

Missing-both Casc (MM-IMDb)
Missing-image Case (MM-IMDb)
Missing-text Case (MM-IMD6)
60
60
58
(1-Mbro
57
54
50
51
51
46
10
30
50
70
90
100
10
30
50
2
6
Missing rall (%)
Missing rute (15)
Missing rall (%6
Bascline -urrurs (A) -ar Ours
Figure 3: Comparison of our final model (Ours) with (1) baseline, which directly drops the features when a modality is missing; (2) Ours (A), which only equips the correlated prompts; (3) Ours (B), which equips both the correlated prompts and the dynamic prompts. The experiments conducted on the val set MM-IMDb dataset (2) across different missing rates (0-100%) upon three different missing-modllity scenarios (missing-bood missing- -mage and missing-text).
capturing correlations between prompts and input features. An interesting observation is that the performance degradation is much smaller when input images are missing compared to missing- both and missing-text which shows that text is more important for the task target on this dataset.
Ablation study. We test the configurations for our proposed three prompts on the val set of MM- IMDb dataset in Tab. [ Tab. 2add Tab. 3 respectively. In the upper part of Tab we first explore how to leverage the correlations between prompts of different layers. "No projection' " denotes using independent prompts for each layer and "Fe" denotes using fully connected layer for projection. It's observed that compared to using independent prompts, either using a fc or a MLP offers a notable performance boost. The MLP gives better performance for its non-linearity and we use it by default. We then explore the prompt depth J (how many layers to insert prompts) in the middle of Tab. [ We notice that the performance reaches peak when J=6, and either decreasing it or increasing it would degrade the performance. Finally, we verify the efficacy of incorporating multimodal prompts for instruction in the bottom part of Tab. [ It's observed that introducing bi-modal prompts generate the prompts of the current layer gives better performance, which shows that different modalities can offer complementary information.
We test the configurations for the dynamic prompts Tab. 2 We first explore the prompt depth for the dynamic prompts. It's observed that the performance reaches peak when the prompt depth equals I, and continues to decrease when the prompt depth increases. We thus set the prompt depth as 1. We then explore the generation functions for the dynamic prompts. Besides the default attention mechanism, we test other alternatives which use a maximum, minimum or average pooling layer followed by fe layer to generate the prompts. It's observed that all approaches could give notable performance boost, and the attention function achieves the best performance.
Table 1: Ablations for the cor- related prompts.
Table 2: Ablations for the configu- Tablle Table 3: Ablations for the modal-
rations of dynamic prompts. common prompts.
Configurations Fl F1-Macro (%)
Configurations Fl-Macro(%%
Configurations FI F1-Macro(%)
No projection
52.54
Depth
54.24
Depth
54.24
Fc
53.76
Depth 2
54.08
Depth 2
54.16
MLP (6=16)
54.24
Depth
53.76
Depth =3
54.42
Depth = 3
53.74
Depth =
53.52
Depth =
54.52
Depth 6
54.24
Attention
54.24
Fc
53.76
Depth- = 12
53.86
Max & Projection
53.12
MIP (r=)
54.36
uni-modal
53.52
Min & Projection
53.35
MLP (r=
54.12
bi-modal
54.24
Avg & Projection 53.42
MLP (r=
54.24
We explore the configurations for the modal-comman prompts in Tab. We first examine the prompt depth for the modal-common prompts. It's observed that as the prompt depth ranges from to our model achieves similar results and reaches peak when the prompt depth equals 6. Considering the accuracy-computation trade-off, we set the prompt depth as by default. We then explore the projection functions for the modal-common prompts. Generally, observe that using MLP for
8

---

60
00
35
53
FLMaND
50
-55
39
Beselize
Misiing both 30%
Beneline
Missing- 30%
40
SeseLine
Mininn both 30%
Missingg both
Musiing both 70%
32
Masing
50%
Missing
35
Missing both
Missing- image
25
Musting
30
50
70
30
100
10
30
50
70
90
100
10
30
50
70
90
100
Missing (9)
Missing rate
Missingg rate (%%
(a) Test with missing- both
(b) Test with missing-image
(c) Test with missing-text
Figure 4: Ablations on the generalizability to different testing scenarios across various missing rates on the val set of evaluated on missing- mhishnng-both both cases with different missing rates. (b) Models are trained on missing-bothh or missing- -image cases, and evaluated on missing-iman -magee cases with different missing rates. (c) Models are trained on missing-both or missing- cases, and evaluated on missing- cases with different missing rates.
dataset [2]. (a) All models are trained on missing-bott cases, and
projection performs better than using fc for projection. Using different channel reduction factors including 4, and 16 achieves similar performance. We thus use a MLP with 1=16 by default.
Generalizability. We conduct experiments to verify the generalizability of our method when trained and tested upon different missing cases. We evaluate models trained on missing-both cases or missing specific modality, and test them on the missing-both, missing-image and missing text cases in Fig.Faa) aa). (b) and (c), respectively. It's first noticed that all our variants outperform the baseline by a large margin across various missing cases and missing rates. Models trained with a certain modality (assing-image) usually perform slightly better than models trained upon missing-both cases when tested with certain modality missing. Models trained with missing both modalities perform robustly to cases with any missing type (e.g. missing-bott missing-text and -mmann.. We further observe the model trained with higher missing rates (e.g.. 70%). is more robust to high-rate missing cases (70%-100%) during testing than models trained with lower missing rates (e.g.. 30% and 50%). It is concluded that models trained on missing-bott cases are robust to various missing cases, which demonstrate strong robustness compared to training with one modality.
missing (e-g., missing-text.
Comparison with other methods. We compare our method with recent approaches upon three commonly-use datasets, i.e., MM-IMDb 22. UPMC Food- 101 (33 and Hateful Memes [17), to missing-mdallity
verify its effectiveness upon different missing modality scenarios. We include the following methods for comparison (1) baseline, which directly drops the features when a modality is missing; (2) CoOp 44). which only prepends prompts at the input level; (3) MMP 19, which inserts independent prompts for the input and intermediate features of the multimodal backbone; (4) MaPLe 16 which generates the prompts in the image encoder based on those of the text encoder; (5) DePT
which
decouples sase-specific knowledge from feature channels into an isolated feature space during prompt tuning. MMP [19] is originally based on the ViLT backbone and we reimplement it following the same protocol as ours. We compare these methods across three different missing rates including 7=50%, 177% and 1799% upon three various missing- moddalty cases including missing- -imag,, missing-text and missing-bott both in Table. 4 Our method largely outperforms other methods on three datasets across different missing rates, which verifies its robustness to different missing scenarios. Compared to MMP [9], our method exceeds it over all missing cases by a large margin, which demonstrates the superiority of leveraging the correlations between prompts and input features.
An interesting observation is that on most datasets (c.g., MM-IMDb 2 and Food101 [33]), missing text input usually has higher effects on the performance than missing image input. We figure that texts are more crucial for the tasks by providing more detailed explanations and precise captions than images on these two datasets. Instead, on the Hateful Memes dataset [17], it's observed that missing image input influences the results more than missing text or missing both. This may reflect the different focus when constructing various datasets.
Efficiency. Compared with only finetuning the last fc layer to fit downstream tasks, our deep correlated prompts only introduce extra 4.0M parameters, which just consume 2.4% of the entire model (151 M), but could notably improve the performance by a large margin over different missing cases and missing rates.

---

Table 4: Comparison with CoOp 44
MMP
19
MaPLe 16 and DePT 41 on the MM-IMDb (2).
UPMC Food- 101 [33. and Hateful Memes 17 datasets under various missing- modality cases with
different missing rates. The bold number indicates the best performance.
Missing
Teain/Test
Validation set
Testing set
Datasets
rate
Image
Text
CoOp
MMP
MaPLe
DePT
Ours
CoOp
MMP
MaPLe
DePT
Ours
100%
50%
51.23
52.07
52.76
53.87
55.23
48.06
48.88
49.58
50.64
52.13
50%
50%
100%
53.04
54.52
55.26
56.04
57.32
49.89
51.46
52.32
52.78
54.32
75%
75%
51.46
52.12
52.87
54.02
55.45
48.37
49.32
49.56
50.87
52.32
100%
30%
47.26
48.23
48.75
49.87
51.35
44.13
45.64
45.52
46.38
48.52
MM-IMDb
70%
30%
100%
52.32
53.21
53.98
55.04
56.21
48.82
50.52
50.64
52.13
53.14
(FI-Macro)
65%
65%
50.22
51.34
52.31
53.17
54.24
46.84
48.12
49.16
50.32
51.42
100%
10%
47.86
48.84
50.12
50.98
52.36
44.76
45.32
46.84
47.56
49.26
90%
10%
100%
51.65
52.36
53.14
54.12
55.42
48.32
49.12
50.13
50.88
52.22
55%
55%
47.44
48.04
48.82
49.98
51.26
44.12
44.87
45.12
46.54
48.04
100%
50%
77.36
78.24
79.87
80.24
82.33
77.89 77.45
79.64
80.
82.11
50%
50%
100%
86.98
87.12
87.48
87.85
89.23
87.02
87.16
87.35
82.14
89.12
75%
75%
81.76
81.98
82.58
83.26
85.25
81.24
81.72
82.34
83.12
85.24
100%
30%
76.65
76.74
76.87
76.87
79.18
76.34
76.52
77.02
77.34 78.87
Food101
70%
30%
100%
85.21
86.12
86.36
86.52
87.53
84.78
85.64
85.89
86.12 87.32
(Accuracy)
65%
65%
79.14
79.56
80.06
81.85
82.38
78.87
79.12
79.84
81.46 81.87
100%
10%
72.65
73.74
73.25
74.22
75.54
71.87
73.14
73.46
74.12
75.26
90%
10%
100%
82.16
82.78
83.42
84.02
86.26
81.67
82.14
83.12
83.56
85.78
55%
55%
77.78 77.36
78.26
78.66
80.39
76.46
76.58
77.85
78.12
79.87
100%
50%
58.32
58.56
58.78
59.31
60.24
60.56
60.31
60.87
61.87
62.32
50%
50%
100%
60.34
61.12
61.34
61.78
62.34
62.41
62.35
63.13
63.88
64.46
75%
75%
62,34
62.87
63.14
63.24
63.78
64.87
65,84
65,46
65.86
66.02
Hateful
100%
30%
58.54
59.02
59.36
60.02
60.56
60.74
61.12
61.26
61.56
62.82
Memes
70%
30%
100%
60.12
60.78
61.32
61.54
62.32
62.74
63.24
63.14
63.48
64.12
(AUROC)
65%
65%
62.34
62.56
63.12
63.32
63.78
64.82
65.04
65.23
65.48
66.08
100%
10%
58.02
57.34
58.32
59.02
60.34
60.03
57.21
60.74
61.14
62.08
90%
10%
100%
59.02
59.32
60.21
60.56
61.34
61.46
61.52
61.87
62.42
63.87
55%
55%
62.32
62.56
63.24
63.78
64.34
64.32 63.34
64.85
65.37
66.78
Performance v.s. MMP when owning comparable parameters. We compare our method with MMP by allowing them to own comparable parameters Tab. Specially, either when we decrease
the required parameters of our method to Table 5: Performance v.s. MMP when owning com- 0.2M (0.2% of the entire model) by reduc-parable parameters on the val set of MM-IMDb (2). ing the channel dimension, or we simulta-
neously increase the required parameters of Parameters 0.2M 1.6M 2.8M 4.0M 5.2M 6.4M both methods, our method consistently outper- MMP(%) 51.34 50.74 50.82 50.72 50.64 50.46 forms MMP, which verifies its effectiveness. Ours(%) 53.14 53.56 53.88 54.24 54.12 53.98
Limitations and broader impacts. The limitations include (1) we only test the effectiveness upon two commonly-used two-stream multimodal models, and don't apply our method to other popular multimodal models. (2) we only include two modalities in the experiments, and will incorporate more modalities in the future. The broader impacts include (1) The missing- missing-modaliin modality cases happen at times in real life. This paper proposes novel method by adapting large multimodal models towards missing- moddalty scenarios, which increases the robustness of large multimodal models. (2) Our methods notably decrease the required computations compared to previous methods upon missing-modality modality learning, which achieves a better sccuracy-computttion trade-off in real life.
missing-
5 Conclusion
In this paper, we tackle two main challenges in multimodal learning, including (1) any modality may be missing in any learning phase (e.g., training, testing or both), and (2) how to efficiently adapt large-scale multimodal transformers to midsing- cases. We propose deep correlated prompting by leveraging the correlations between prompts of different layers and the relationships between prompts and input features. Results on three diverse datasets across various missing types and missing ratios verify the effectiveness of our proposed method.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by National Key Research and Development Program of China FC10072)))2 and National Natural Science Foundation of China (Project No. 62072334 and No. 62276182).
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems,
35:23716-237366 2022.
[2] John Arevalo. Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multimodal units for information fusion. arXiv preprint arXiv:1702.0199) 2017.
[3] Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
arXiv:1607.06433 16..60000645.. 2016.
[4] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Modifyynn pixel space to adapt pre-trained models. arXiv preprint arXiv:2203. 222200...774 1:
2022.
[5] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101d-101-1 mining discriminative
components with random forests. In European Conference on Computer Vision, 2014.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melamie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint fttv:20100119223 2020.
[8] Maksim Dzabraev, Maksim Kalashnikov, Stepan Komkov, and Aleksandr Petiushko. Mdmmt:
Multidomain multimodal transformer for video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3354-3363, 2021.
[9] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In Computer Vision-ECCY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 214-229. Springer, 2020.
[10] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
1rXiv:1606.088 08415, 2016.
(11) Lianyu Hu, Liqing Gao, Zekang Liu, and Wei Feng. Continuous sign language recognition with correlation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2529-2539, 2023.
[12] Lianyu Hu, Liqing Gao, Zekang Liu, and Wei Feng. Scalable frame resolution for efficient continuous sign language recognition. Pattern Recognition, 145: 109903, 2024. [13] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
[14] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
[15] Mengmeng Jing, Jingjing Li, Lei Zhu, Ke Lu, Yang Yang, and Zi Huang. Incomplete cross- modal retrieval with dual-aligned variational autoencoders. In Proceedings of the 28th ACM International Conference on Multimedia, pages 3283-3291, 2020.
[16] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113-19122. 2023.
[17] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, 33:2611 1-2624,, 2020.
[18] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: dision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning, pages 5583-5594. PMLR, 2021.
[19] Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen- Yu Lee. Multimodal prompting with missing modalities for visual recognition. In Proceedings of the IEEE/CVF Conference
11

---

Computer Vision and Pattern Recognition, pages 14943-14952, 2023.
[20]
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, arXiv preprint 104v:2200408691,, 2021.
[21]
Chunyuan Li, Zhe Gan. Zhengyuan Yang. Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao. Multimodal foundation models: From specialists to general-purposs assistants. arXiv preprint arXiv:2309. 10020. 1, 2023.
[22] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint
WxXi:2231.12559) 2023.
[23] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-tninnig for unified vision- language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR. 2022.
[24] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages (0965-10975) 2022.
[25] Xiang Lisa Li Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation.
arXiv preprint arXiv:2101.00 101000190, 2021.
[26] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv: 10.077602, 2021.
[27] Mengmeng Ma. Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust to missing modality? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18177-18186, 2022.
[28] Mengmeng Ma, Jian Ren, Long Zhao. Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1302-2310 2021.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8766 PMLR, 2021.
[30] Teven Le Scao, Thomas Wang. Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al. What language model to train you have one million gpu hours? arXiv preprint arXiv:22100 10.2200..5424 15424, 2022.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, J Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[32] Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carnciro. Multi- modal learning with missing modality via -hared-specific feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15878- 15887, 2023.
[33] Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso. Recipe recognition with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pages 1-6. IEEE, 2015.
[34] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint ar arXiv:2108. 10904,
2021.
[35] Zifeng Wang, Zizhao Zhang. Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pages 631 1-48. Springer, 2022.
[36] Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge- guided context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6757-6767, 2023.
[37] Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2109.1 / 1797, 2021.
[38] Qiyue Yin, Shu Wu, and Liang Wang. Unified subspace learning for incomplete and unlabeled wiew data. Pattern Recognition, 67:313-327, 2017.
[39] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Trans. Mach. Learn. Res.,
12

---

2022, 2022.
[40] Jiandian Zeng. Tianyi Liu, and Jiantao Zhou. Tag-assisted multimodal sentiment analysis under uncertain missing modalities. In Proceedings of the 45th Interaational ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1545-1554, 2022.
[41]
Zhang, Shihan Wu, Lianli Gao, Hengtao Shen, and Jingkuan Song. Dept: Decoupled prompt tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[42] Jinming Zhao, Ruichen Li, and Qin Jin. Missing modality imagination network for emotion recognition with uncertain missing modalities. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the I 1th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages
2021.
[43] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[44] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9))337722344 222233777288,,,, 2022. [45] Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining n-modality by language- semantic alignment. arXiv preprint arXiv:2310.01852. 2023. [46] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15659-15669, 2023.
13

---

A Appendix supplemental material
Compatibility with complete data. In this paper, we have previously assumed that there is no guarantee modality-complees data can be entirely collected due to various factors in real-world scenarios. However, these still exist publicly-availbll modality-complete datasets which are carefully collected and annotated for model training. Thus, perform additional experiments to compare our method with the baseline and input-level prompting by training on modalityycommpee stimulate the mrising-modality case during testing, we randomly select data with different modality- missing cases for each data pair (i.e., text-only, image-only, or complete) during training. Note that, different from other experimental settings introduced in the paper, here one data pair be in various missing-modality cases at different training epochs. We plot the results in Fig. [ is observed that our method consistently outperforms the baseline and input-level prompting with better performance across different missing ratios.
data. To
Train: Modality-complete; Test: Missing-both
Baseline
MMP
Ours
59
F1 1--Macro
53
47
41
35
30
50
Missing rate (%)
Figure 5: All models are trained with modality-complete data, where each data pair can be randomly assigned with different missing modalities (i.e., text-only, image-only, and modality-complete at different training epochs to mimic the possible missing modalities during testing. Evaluation is on missing-both cases with different missing rates.
Deployment on other backbones. We deploy our proposed method upon other commonly vision-language models like CoCa (39 and ViLT [18], to verify its flexibility. CoCa is trained jointly with contrastive loss and captioning loss, which subsumes model capabilities from both contrastive approaches like CLIP [29] and generative methods like Sim VLM (34). ViLT is single-stream multimodal backbone which directly concatenates the input text features and image features and feed them into common transformer for processing. We adopt ViT-B/32 7 the image encoder for CoCa, and list the results in Table. 6 and Table 7 respectively. bring notable performance boost across three datasets compared to the baseline, which verifies its flexibility across various large-scale vision- lssiunlanguuae models.
observed that our method can
Table 6: Quantitative results with CoCa 39 and Hateful Memes datasets with missing rate 17 = 70% under various modality-missiie scenarios. The bold number indicates the best performance.
as our backbone on the MM-IMDB, UPMC Food-101.
Datasets
Image
Text
Baseline
Ours
100%
30%
38.96
49.34
MM-IMDb
30%
100%
45.78
55.12
(F1-Macro)
65%
65%
42.35
51.24
100%
30%
73.41
77.56
Food101
30%%
100%
82.34
86.26
(Accuracy)
65%
65%
77.24
80.46
100%
30%
54.87
58.36
Hateful Memes
30%
100%
56.46
60.34
(AUROC)
65%
65%
57.82
61.83
14

---

Table 7: Quantitative results with ViLT [8] as our backbone on the MM-IMDB, UPMC Food-101,
and Hateful Memes datasets with missing rate = 70% under various modality-missing scenarios. The bold number indicates the best performance.
Datasets
Image
Text
MMP
Ours
100%
30%
39.22
45.26
MM-IMDb
30%
100%
46.30
51.24
(F1-Macro)
65%
65%
42.66
48.45
100%
30%
74.53
78.85
Food101
30%
100%
86.18
86.76
(Accuracy)
65%
65%
79.08
80.85
100%
30%
59.11
61.24
Hateful Memes
30%
100%
63.06
64.12
(AUROC) 65%
65%
66.07
66.68
Table Ablations for prompt length on the MM-IMDB dataset with missing rate rj = 70%. The bold number indicates the best performance.
Prompt length 12
24
36
48
60
FI-Macro(%) 53.56
53.88
54.24
54.12
53.98
Ablations for the prompt length. We ablate the prompt length for our DCP on the MM-IMDB dataset with missing rate 1 = 70% in Tab. [8] It's observed that the performance continues to increase when the prompt depth ranges from 12 to 36, and reaches peak when the prompt depth equals 36. Further increasing thr prompt depth would degrade the performance. We thus set the prompt depth as 36.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: They are discussed at the end of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
15

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specing well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
They are given in the implemention details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
16

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code is released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/
gewss CCodeSoodieeeeeee ssi mollicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
17

---

Answer: [Yes]
Justification:
They are given in the implemention details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported due to the high computational costs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma eгror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
They are included in the implemention details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
18

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httlgh/ttppp//oorrrrccccoo/////t
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
They are discussed at the end of the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
19

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
20

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "TzCE7UHORQ", "reviewer": "Reviewer_nnhg", "summary": "This paper proposes a new method to handle missing modalities in visual and language recognition systems.\nThe paper proposes a very similar method to the one proposed by MMP [17] but using different way of getting the prompts to feed them into the transformer layers. \nComparison with other works show that the method seems to be effective and some ablations studies are performed to study the different design choices. The method is validated using the most common datasets for this task.", "questions": "- Please explain well the mechanism of the different types of prompts, input, output at train and test time for each one of them. It could have been done easily with a figure, but at least with a few sentences it could become clearer. \n- What makes a \"dynamic\" prompt \"dynamic\"?\n- What does baseline mean and how was implemented?\n- How was MMP implemented on your framework?\n- What if using ViLT instead of CLIP, would still your method be better than MMP?\n- What is the zero-shot performance of CLIP on these datasets? it is important since this might be a robust method that does not suffer from missing modality. It can be implemented using nearest neighbor to each of the class embedding using either modality, and combining them when both are present.", "limitations": "Limitations have been addressed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The method seems to work when compared with other state-of-the-art models.\n- The paper presents results on several datasets and with different settings of the model.", "weaknesses": "- The main weakness of the paper is clarity. There are three different sets of prompts that are appended to the intermediate representations. However, the only difference between them seems to be the type of architecture the method uses to compute them. The explanation is very limited and Figure 1 does not illustrate where do these prompts come from. Without the clarity of this explanation it becomes really hard to understand how the motivation of each type of prompt fits the design. What are exactly correlated prompts, dynamic prompts, and modal-common prompts? What make them correlated, dynamic and modal-common? This is not clear in the paper at all. \n\n- It is not clear what is baseline. What does dropping features when modality is missing? The input sequence become shorter and coming from only a single modality? If that's the case, what is trainable and what is not? \nPlease explain well this part. I would expect that this baseline is: training with the same number of parameters as the base method, by simply adding learnable prompts at each layer and training using mod-drop (dropping modalities randomly when training, dropping modalities can be done by inputting noise instead of tokens, the average of the modality tokens, zeroes, or not passing the missing modality at the input, it is a design choice that needs to be explained). If it is not what I'm thinking, please explain well, since this is a key experiment.\n\n- When comparing with MMP, how did the authors do it? Please explain exactly how was this re-implementation. Also, to be fair, the authors should have applied their method using ViLT instead of CLIP, in that way there is no doubt that this method is better than the very similar MMP. \n\n- What is the zero-shot performance of CLIP on these datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. The mechanism of different prompts.\n\n  Many thanks for your question. We have plotted a figure to further illustrate our proposed prompts by comparing them with our baseline and MMP[17], which can be found in the pdf file of Author Rebuttal. The baseline simply uses fixed image encoder and text encoder and only finetunes the classifier to handle downstream tasks. To well adapt to missing modalities, MMP[17] inserts learnable tensors, i.e., prompts at each layer which still keeping the image encoder and text encoder fixed to guide the model to fit missing-modality cases. However, the inserted prompts of MMP across different layers are independent, while we believe that the prompts across different layers and various modalities can provide beneficial information for each other to better fit missing-modality cases. Thus, we propose correlated prompts which generate the prompts of the next layer based on the prompts of both modalities in the current layer. For dynamic prompts, our intuition is that the prompts proposed by MMP are fixed for different inputs during inference and fail to fit the missing cases of different inputs, and we thus propose to dynamically compute the prompts based on different input features to better guide the behavior of the model. This procedure is implemented by a self-attention layer with a randomly initialized tensor as a query and the input features as keys and values. Besides, we propose modal-common prompts which store the shared information across different modalities, which can complement the model with common information across different modalities and facilitate the model to encode modal-specific information to better handle the missing scenarios in each modality. \n\n2. What makes a \"dynamic\" prompt \"dynamic\"?\n\nWe propose to dynamically compute the prompts based on different input features, which avoids employing fixed prompts for different input features during training and inference. This procedure is implemented by a self-attention layer with a randomly initialized tensor as a query and the input features as keys and values. You can view the illustration in the figure of Author Rebuttal.pdf.\n\n3. The baseline setting.\n  \nThe baseline setting is using the text encoder and image encoder to encode the input texts and images, whose output features are fed into the classifier for recognition. In this procedure, only the classifier is updated and other model components including the text encoder and the image encoder are kept fixed. The only difference between our method and the baseline is inserting learnable prompts at each layer which only bring few extra parameters. Specifically, when a modality is missing, we simply don’t feed the corresponding features into the modality encoder for processing and set the outputs of this modality encoder as zeros. Overall, we have tried three baseline settings, including (1) inputting zeros for the modality encoder when a modality is missing, (2) inputting the average of the modality tokens for the modality encoder when a modality is missing, and (3) not feeding input features for the modality encoder when a modality is missing and setting the outputs of this modality encoder as zeros (Default). As shown below, out experiments on the MMIMDb dataset with η=70% show the last choice gives best performance, and thus we adopt it as a stronger baseline.\n\n| Baseline Configurations | Acc(%) |\n| --- | :---: | \n| Inputting zeros | 47.65 | \n| Inputting averaged tokens | 46.24| \n| Default| **49.21**| \n\n\n4. The re-implementation setting of MMP.\n  \nMMP adds learnable prompts with a length of 16 at the bottom 6 layers based on the ViLT backbone. we re-implement it based on the CLIP backbone by adding learnable prompts with a length of 16 at the bottom 6 layers in the image encoder and text encoder, respectively. Our method owns the same number of inserted layers and prompt length with MMP. The only difference is that we insert three proposed prompts into the model.\n\n5. Comparison with MMP using the ViLT backbone \n  \nWe compare our method with MMP using the ViLT backbone and show the results on three datasets with η=70% upon the missing-both setting as below. It’s observed that our method shows superior performance than MMP. It’s also noticed that using ViLT as the backbone achieves inferior performance than CLIP, and thus we adopt CLIP as the default backbone.\n\n| Dataset | MMP| Ours |\n| --- | :---: | :---: |\n| MMIMDb | 42.66% | **48.45%** |\n| Food101 | 79.08%| **80.85%** |\n| Hateful Memes | 66.07% | **66.68%** |\n\n6. Zero-shot performance of CLIP.\n  \n  We test the performance of zero-shot CLIP on the MMIMDb, Food101 and Hateful Memes datasets. We first calculate and store the averaged embeddings of all classes of both modalities on each dataset. When the input modalities are complete, we calculate the similarities between the output features and the pre-computed class embeddings for each modality encoder. We select the class with the highest similarity as the recognition output within both modalities. When a modality is missing, we select the class whose embedding owns the highest similarity with the output features of the available input modality as the recognition output. We compare the performance of zero-shot CLIP, finetuned CLIP (our baseline, which sets output features as zeros when a modality is missing) and our method as below. The zero-shot CLIP achieves inferior performance than the other methods, and ours perform best. We suppose that the zero-shot CLIP not finetuned on the downstream datasets fails to well adapt to downstream setting. Finetuning the CLIP could notably increase the performance, and ours further boost the performance by injecting different kinds of learnable prompts.\n  \n| Datasets | Zero-shot CLIP | Finetuned CLIP | Ours |\n| --- | :---: | :---: | :---: | \n| MMIMDb | 34.52% | 49.21% | **54.24%**|\n| Food101 | 57.02% | 77.74% | **82.38%** |\n| Hateful Memes | 55.23% | 62.58% | **66.08%** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AiQepvzBYF", "reviewer": "Reviewer_xVBv", "summary": "This paper proposes to address the missing modality problem for the multimodal recognition model (i.e. the multi-modal data could be incomplete). There are three techniques of prompting being proposed (while the recognition model, i.e. two-stream multimodal method CLIP in this paper, is kept fixed), including: 1) correlated prompts, where a part of the prompts in the input-level are firstly selected according to the missing scenario (e.g. complete, text-only, or image-only), then the prompt in each of the following network layers are predicted from the multimodal prompt of its preceding layer; 2) dynamic prompts, the input-level prompts contain a portion generated according to the input sample; 3) modal-common prompts, where the rest of the input-level prompts is stemmed from a common component shared across modalities. The combination of the aforementioned three techniques experimentally shows better performance in comparison to various baselines (mainly the SOTA method from MMP [17]).", "questions": "Although currently the proposed method seems to provide superior performance with respect to various baselines and its proposed techniques (i.e. correlated prompts, dynamic prompts, modal-common prompts) are experimentally shown to benefit the model performance, there are concerns regarding limited novelty (where only the correlated prompts are considered to be related to missing modality while the other two techniques, i.e. dynamic and modal-common prompts, are not) and detailed analysis for the number of learnable parameters versus model performance, (as listed in the weaknesses), in which the the authors are highly encouraged to make the corresponding clarifications in the rebuttal.", "limitations": "no potential negative societal impact is found.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The proposed method provides superior performance with respect to various baselines and its proposed techniques (i.e. correlated prompts, dynamic prompts, modal-common prompts) are experimentally shown to benefit the model performance.\n+ The extensive experiments are conducted on multiple dataset with various experimental settings.\n+ The presentation is clear and easy to follow.", "weaknesses": "- The modal-common prompts and the dynamic prompts actually are not directly connected to the missing modality problem (or being irrelevant to different cases of missing modality). While excluding these two prompting techniques from the proposed method (in which such variant becomes \"Ours (A)\" in Figure 2), the improvement with respect to the state-of-the-art approach of handling missing modality (i.e. MMP[17]) would become marginal (please include MMP[17] into the ablation study shown in Figure 2 or directly provide the tabular quantitative results for the ablation study). Similarly, while we only consider the technique of correlated prompts as the manner in the proposed to tackle the missing modality, it becomes the only difference in the proposed method compared to MMP [17] (in terms of methodology), thus leading to the concern of limited novelty. Furthermore, there should be a baseline of integrating the modal-common prompts (acting as a basic component of prompt) and dynamic prompts into MMP[17] to better highlight the contribution of the proposed correlated prompting technique (which is the main technique in the proposed method to be connected with the missing modality challenge). Moreover, as modal-common prompts and the dynamic prompts introduce additional learnable parameters (in comparison the correlated prompts), there should be further detailed analysis/comparison in terms of number of learnable parameters versus model performance.\n- Though the proposed dynamic prompts do experimentally shown to improve the overall performance under various missing modality cases, such prompting technique is actually not new, where we can see its similar application in various research problems (e.g. Wu et al., IDPG, NAACL'22; Lu et al., PromptPG, ICLR'23; Qiu et al., FedTPG, FL@FM-NeurIPS’23).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Proposed prompts not directly connected to the missing modality problem.\n\nSorry for the mis-clarification in the manuscript to mislead you. In line 146-150 of our manuscript, we state that we set different prompts for various missing modalities. Specifically, for correlated prompts, we independently set the initial prompt of the first layer for different missing-modality settings, which will be transformed via a projection function to generate the prompts for the next layer. For the dynamic prompts, we initialize different queries for different missing modalities, which are then used to compute the dynamic prompts based on input features in a self-attention manner. For the modal-common prompts, we initialize different modal-common prompts for different missing-modality settings, which will be transformed into embedding space via a projection function to provide prompts. Sorry for missing the expressions for the dynamic prompts and modal-common prompts upon different missing-modality settings. We will update them in the manuscript to give better clarifications. \n \n2. Effectiveness regarding the correlated prompts compared to MMP.\n  \nBased on the same CLIP backbone, we compare the performance of MMP to our proposed correlated prompts on the MMIMDb dataset with different missing ratios under the missing-both case as below. It’s observed that our proposed correlated prompts consistently offer superior performance than MMP under the same setting. \n\n| Settings| η=10% | η=30%| η=50% | η=70% | η=90% |\n| --- | --- | --- | --- | ---| ---|\n| MMP | 56.54%| 53.64% |52.12%| 51.34% | 48.04%|\n| Correlated prompts | **57.91%** | **54.84%** | **53.78%** | **52.12%** | **48.52%** |\n\n3. Comparison between the integration of MMP and ours.\n\n  We integrate the proposed dynamic prompts and modal-common prompts into MMP based on the CLIP backbone to form a method termed as MMP*, and compare our method with it on the MMIMDb dataset with different missing ratios under the missing-both case as below. It’s observed that our method notably outperforms MMP* with different η.\n\n| Settings| η=10% | η=30%| η=50% | η=70% | η=90% |\n| --- | --- | --- | --- | ---| ---|\n| MMP* | 58.58%| 55.46% |54.16%| 52.96% | 50.04%|\n| Ours | **59.82%** | **56.87%** | **55.45%** | **54.24%** | **51.26%** |\n\n4. Analysis between parameters v.s. performance\n\n  We first give an ablation for the parameters of each proposed prompt. We give the performance with η=70% on the MMIMDb dataset upon the missing-both setting. As shown below, as we insert each proposed prompt, the performance consistently increases. The proposed prompts overall bring extra 4.0M trainable parameters, which is only 2.4% of the overall framework.\n\n | Configurations| Brought parameters | F1-Macro|\n| --- | --- | --- | \n| Baseline | - | 49.21% |\n| +Correlated prompts | +1.2M| 52.12% |\n| +Dynamic prompts | +1.8M | 53.22% |\n| +Modal-common prompts | +1.0M | 54.24% | \n\n  Besides, we test the relationships between the brought extra parameters and performance, and show the results as below. It’s observed that the performance continues to increase when the prompt depth ranges from 12 to 36, and reaches a peak when the prompt depth equals 36. The parameters consistently increase as the prompt length grows.\n\n| Brought parameters | 1.6M | 2.8M | 4.0M | 5.2M | 6.4M| \n| --- | --- | --- | --- | --- | --- |\n | F1-Macro | 53.56% | 53.88% | **54.24%** | 54.12% | 53.98% |  \n\n Finally, we compare our method with MMP with the same parameters. It’s observed that with the same parameters, our method consistently achieves better performance.\n\n| Parameters| 0.2M | 1.6M | 2.8M | 4.0M | 5.2M | 6.4M |\n | --- | :---: | :---: | :---: | :---: | :---: | :---: |\n | MMP | 51.34%| 50.74% | 50.82% | 50.72% | 50.64% | 50.46%| \n | Ours | **53.14%** | **53.56%** | **53.88%** | **54.24%** | **54.12%** | **53.98%** |  \n\n5. The dynamic prompting technique is not new.\n\n  Many thanks for your question. The dynamic prompts have been previous investigated in other methods. However, dynamic prompts in the missing-modality scenarios have not been studied. It’s worth exploring whether generating various prompts according to different missing cases and input features can well guide the model to fit the missing-modality settings. Our manuscript verifies the efficacy of this design.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jwxK1jL8N0", "reviewer": "Reviewer_64zT", "summary": "This paper addresses the challenge of generalized missing modalities in multimodal learning, where a modality can be absent during any learning phase (e.g., training, testing, or both). he authors investigate prompt learning with missing modalities and propose deep correlated prompts designed to capture various types of correlations between prompts and input features across different modalities. Specifically, the proposed prompts include mechanisms for perceiving beneficial information from preceding layers, dynamically generating prompts based on input characteristics, and leveraging the complementary information from multimodal inputs. These designs improve the robustness of large multimodal models (e.g., CLIP) to missing modalities. Extensive experiments and ablation studies demonstrate consistently superior performance and verify the effectiveness of the proposed method.", "questions": "1. In Table 4, I noticed that some values of the related work MMP are the same as the figures recorded in the paper. For example, the settings with:\n   - missing rate = 70% (100% image and 30% text) in MM-IMDb,\n   - missing rate = 70% (30% image and 100% text) in Hateful Memes,\n   - missing rate = 70% (65% image and 65% text) in Hateful Memes.\n\n   As far as I know, the MMP backbone model is the multimodal transformer ViLT. The authors state they re-implemented MMP on their setting (i.e., CLIP) for a fair comparison. It seems that the numbers should not be the same since they use different backbone models. Can the authors clarify why the values are identical despite using different backbone models?\n\n2.\tAccording to the design of prompts, it seems that the proposed method is not limited to two-stream models (e.g., it could be applied to single-stream models without using Eq. (5)). Generalizing the method to single-stream models and comparing it with related works could be helpful in verifying the generalizability of the proposed method. Have the authors tried it for single-stream models? If so, what were the results?\n3.\tI am willing to revise my rating if the authors also address the concerns mentioned in the weaknesses.", "limitations": "One limitation is that the proposed method requires modality-specific deep correlated prompts for each available modality, which could be challenging to extend to more modalities (e.g., five or more modalities).", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper addresses a more challenging missing modality setting, where modalities may be absent during both training and testing phases, making it highly practical and essential for real-world applications.\n2.\tThe paper is well-motivated. The authors highlight the weaknesses of prior work and propose several designs (e.g., deep correlated prompts, dynamic prompts, common prompts) to improve robustness.\n3.\tThe paper explores various types of correlations between prompts and input features across different modalities, and the proposed designs for each are technically sound.\n4.\tExtensive experiments show great improvement on the baseline and consistently superior performance compared to other methods across all benchmarks.\n5.\tComprehensive ablation studies are conducted to validate the effectiveness of each proposed component.", "weaknesses": "1.\tThe paper lacks a detailed explanation or discussion on the efficacy of different prompt designs. In Figure 2, it shows that sequentially adding different designs improves the baselines, but it does not discuss the individual improvement gains for each design. Additional discussion on each design could help validate whether the increasing gains from sequentially adding designs are not merely due to more learnable parameters.\n2.\tThe paper lacks visualization of each learnable prompt (e.g., deep correlated prompts, dynamic prompts, and common prompts). Visualizations could help validate whether the different components work as expected. For example, do dynamic prompts genuinely capture the different characteristics of inputs, or do they merely distinguish between different missing cases, which might be easier to learn due to the obvious absence of a modality?\n3.\tFor each available modality, it seems there are a total of $(3*(2^M-1))$ prompts for each missing modality case. This could lead to an exponential increase and redundant prompts as more modalities are considered (i.e., M>2). For example, in a vision-and-language task, in the case of complete and missing-image, the text modality is available for both cases. However, it requires two separate prompt sets for the text encoder, which may actually learn the prompts for the same “text-available” case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Efficacy of each proposed prompt.\n \n  We place each proposed prompt upon the baseline method, and show the results as below on the MMIMDb dataset upon the missing-both setting with η=70%. It’s observed that each proposed prompt could notably boost the performance.\n  | Configurations| Extra brought parameters | F1-Macro|\n| --- | --- | --- | \n| Baseline | - | 49.21% |\n| +Correlated prompts | +1.2M| 52.12% |\n| +Dynamic prompts | +1.8M | 51.04% |\n| +Modal-common prompts | +1.0M | 51.26% | \n\n2. Visualization of each learnable prompt.\n  \n  We offer visualizations for the dynamic prompts using the T-SNE method on the Food101 dataset upon the missing-both setting with η=70% and η=50%. As Food101 have 101 classes which is redundant for visualization, we select a subset with 11 classes to give better visualization results. It’s observed that the dynamic prompts could roughly categorize the prompts into distinct classes. Besides, for different missing settings with η=70% and η=50%, the distribution of dynamic prompts with respect to the same inputs are different, which show that dynamic prompts can learn to generate various expressions for each input upon different missing settings.\n\n3. Exponential increased prompts when more modalities are introduced.\n\n For M input modalities, it requires $2^M-1$ types of prompts for all missing-modality cases. This may be redundant, but we expect a specific prompt to handle each missing-modality case to better guide the model to handle different missing scenarios. Besides, the overall extra consumed parameters are quite few as the modalities increase as the projection functions are shared across different modalities. Thus as the modalities increase, we only introduce new learnable prompts which occupy quite few parameters. For example, for each missing-modality case, the newly introduced prompts own 768*36=27648≈0.03M parameters with a prompt length of 36 for each modality encoder.  \n\n4. Identical values in Table 4 compared with MMP\n  \nSorry for the errors. We wrongly set the values when organizing the tables from different resources. The correct values should be 48.23 for missing rate = 70% (100% image and 30% text) in MM-IMDb, 61.12 for missing rate = 70% (100% image and 30% text) in Hateful Memes, and 63.24 for missing rate = 70% (30% image and 100% text) in Hateful Memes. We will current them in the manuscript.\n\n5. Results on single-stream backbones\n\n  We compare our method with MMP based on the ViLT backbone. We don’t use features of two modalities to generate prompts of the next layer as Eq.5. The results on three datasets with η=70% upon different missing settings are shown below. It’s observed that our method shows superior performance than MMP. \n\n\n|  Datasets               | Image | Text  | MMP     | Ours    |\n|-------------------------|-------|-------|---------|---------|\n|   MM-IMDb (F1-Macro)   | 100%  | 30%   | 39.22   |  **45.26**  |\n | MM-IMDb (F1-Macro) | 30%                     | 100%  | 46.30 |  **51.24**  |\n| MM-IMDb (F1-Macro)  | 65%                     | 65%   | 42.66 |  **48.45**  |\n|  Food101 (Accuracy)  | 100%  | 30%   | 74.53   |  **78.85** |\n| Food101 (Accuracy)    | 30%                     | 100%  | 86.18 | **86.76**   |\n|Food101 (Accuracy)  | 65%                     | 65%   | 79.08 | **80.85**   |\n| Hateful Memes (AUROC)  | 100%  | 30%   | 59.11   |  **61.24**  |\n| Hateful Memes (AUROC)  | 30%                     | 100%  | 63.06 | **64.12**   |\n| Hateful Memes (AUROC)| 65%                     | 65%   | 66.07 | **66.68**   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nd9opw6Cn8", "reviewer": "Reviewer_BZuF", "summary": "The model proposes prompting strategy where both modalities (image and text) are prompted, and the prompt for both modalities are correlated. The strategy is to use multiple prompts, namely correlated prompts, dynamic prompts, and modal-common prompts. As the backbone itself is multimodal (CLIP), it is a good idea to consider synchronized multi-modal prompts to fully harness the model capabilities when prompting it. The model surpasses multiple multimodal SoTAs on multiple datasets and also has proven to be effective in handling missing modalities in training and inference.", "questions": "1. Please elaborate further on how modal-common features are disentangled.\n2. If possible, show the layer J-th in Figure 1 (framework overview)\n3. Minor suggestion: The phrase \"abundant ablations\" in the introduction is a bit overboard, I suggest to write it as just \"Ablation studies are further given...\"", "limitations": "The limitations are discussed in the appendix, including that only text and visual modalities are tested with this model and the number of tested models.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The strategy of using multiple types of multimodal prompts, along with the correlation strategy, is logically sound as the multimodal backbone itself is trained to understand the relationship between image and text modalities.\n\n2. The modal surpasses multiple SoTAs on multiple benchmarks with considerable score improvement.\n\n3. The ablation studies are sufficient to understand the justification of the network design.", "weaknesses": "1. Ablation studies regarding the multimodal backbone, e.g. using other model than CLIP or use dedicated unimodal encoders for each modality, highly recommended to increase paper quality.\n2. In table 4, what are the performances when either image or text modalities are completely missing?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Ablation studies by using other multimodal backbones\n\nWe provide the results by comparing our method with the baseline method upon the single-stream ViLT backbone, and also comparing them upon the two-stream CoCa backbone as below.\n\nWe first provide the results on the ViLT backbone. Our method large outperforms the baseline across different settings on three datasets.\n\n|  Datasets               | Image | Text  | MMP     | Ours    |\n|:---|:---:|:---:|:---:|:---:|\n|   MM-IMDb (F1-Macro)   | 100%  | 30%   | 39.22   |  **45.26**  |\n | MM-IMDb (F1-Macro) | 30%                     | 100%  | 46.30 |  **51.24**  |\n| MM-IMDb (F1-Macro)  | 65%                     | 65%   | 42.66 |  **48.45**  |\n|  Food101 (Accuracy)  | 100%  | 30%   | 74.53   |  **78.85** |\n| Food101 (Accuracy)    | 30%                     | 100%  | 86.18 | **86.76**   |\n|Food101 (Accuracy)  | 65%                     | 65%   | 79.08 | **80.85**   |\n| Hateful Memes (AUROC)  | 100%  | 30%   | 59.11   |  **61.24**  |\n| Hateful Memes (AUROC)  | 30%                     | 100%  | 63.06 | **64.12**   |\n| Hateful Memes (AUROC)| 65%                     | 65%   | 66.07 | **66.68**   |\n\nWe offer the results on the CoCa backbone. We can find that our method achieves superior performance than the baseline.\n\n|  Datasets               | Image | Text  | Baseline | Ours    |\n|:---|:---:|:---:|:---:|:---:|\n|  MM-IMDb (F1-Macro)     | 100%  | 30%   | 38.96    |  **49.34**  |\n|  MM-IMDb (F1-Macro) | 30%                     | 100%  | 45.78 |  **55.12**   |\n|  MM-IMDb (F1-Macro) | 65%                     | 65%   | 42.35 |  **51.24**   |\n|  Food101 (Accuracy)     | 100%  | 30%   | 73.41    |  **77.56**  |\n|  Food101 (Accuracy) | 30%                     | 100%  | 82.34 | **86.26**    |\n|  Food101 (Accuracy) | 65%                     | 65%   | 77.24 | **80.46**    |\n|  Hateful Memes (AUROC)  | 100%  | 30%   | 54.87    |  **58.36**  |\n|  Hateful Memes (AUROC) | 30%                     | 100%  | 56.46 | **60.34**    |\n|  Hateful Memes (AUROC) | 65%                     | 65%   | 57.82 | **61.83**    |\n\n2. Performance when either image or text modalities are completely missing in Table 4\n\nWe provide the results as follows.\n|  Datasets | Image | Text  |  CoOp  | MMP   | MaPLe | DePT  | Ours  |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|  MM-IMDb (F1-Macro)     | 100%  | 0%    | 45.24  | 46.32 | 46.85 | 48.02 | **50.64** |\n|  MM-IMDb (F1-Macro) | 0%  | 100%  | 51.23 | 52.21  | 52.76 | 54.13 | **55.45** |\n|  Food101 (Accuracy)     | 100%  | 0%    | 70.34  | 71.52 | 71.18 | 72.25 | **73.87** |\n|  Food101 (Accuracy) | 0% | 100%  | 80.76 | 81.52  | 82.13 | 83.24 | **85.64** |\n|  Hateful Memes (AUROC)  | 100%  | 0%    | 59.56  | 60.14 | 60.25 | 61.32 | **62.12** |\n|  Hateful Memes (AUROC) | 0%   | 100%  | 60.87 | 61.32  | 61.54 | 62.12 | **63.24** |\n\n3. How modal-common features are disentangled \n\n Specifically, we introduce a shared learnable prompt across different modality encoders which embeds modal-common information for different modalities. To transform it into the feature space of various modalities, we introduce an independent projection function for each modality to change the channels of the shared learnable prompt. It explicitly embeds modal-common information and inversely encourages other prompts to provide modal-specific information to guide the model to handle different missing-modality cases.\n \n4. Show the layer J-th in Figure 1 (framework overview)\n\nThanks for your comments. We will update the J-th layer in our manuscript. \n\n5. Phrase correction.\n\nMany thanks for your advice, we will update the expressions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d2qZle7PHJ", "reviewer": "Reviewer_YNgk", "summary": "The paper proposes a prompt optimization approach to the missing modality issues in multimodal learning. Inspired by the missing-aware prompt (MMP), this paper adds more prompts, including correlated, dynamic and modal-common prompts, to each encoder to improve the performance. The experiment on three datasets shows the effectiveness of the proposed method.", "questions": "What is the specific contribution of this paper compared with MMP, other than adding more parameters and functions?\n\n\nWill MMP's performance be better or comparable when MMP uses the same number of parameters as the proposed method?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The missing modality issue in multimodal learning is a practical challenge. \n\nThe designed method is clearly presented.", "weaknesses": "1. The novelty of the proposed method is limited since the MMP has proposed the prompt optimization approach to solving the missing modality issue. Compared with MMP, this paper adds more parameters in the form of prompt tokens from different inputs and functions. \n\n2. The empirical comparison with MMP is probably not quite fair as the proposed method uses more additional parameters compared with MMP. According to Line 337, this method adds 2.4% additional parameters, while MMP only adds 0.2%.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reviewer# YNgk\n1. Novelty compared to MMP\n\n  Many thanks for your question. MMP has first introduced prompt learning to handle the missing-modality setting. It inserts learnable tensors, i.e., prompts at each layer which still keeping the image encoder and text encoder fixed to guide the model to fit missing-modality cases. However, the inserted prompts of MMP across different layers are independent, while we believe that the prompts across different layers and various modalities can provide beneficial information for each other to better fit missing-modality cases. Thus, we propose correlated prompts which generate the prompts of the next layer based on the prompts of both modalities in the current layer. For dynamic prompts, our intuition is that the prompts proposed by MMP are fixed for different inputs during inference and fail to fit the missing cases of different inputs, and we thus propose to dynamically compute the prompts based on different input features to better guide the behavior of the model. This procedure is implemented by a self-attention layer with a randomly initialized tensor as a query and the input features as keys and values. Besides, we propose modal-common prompts which store the shared information across different modalities, which can complement the model with common information across different modalities and facilitate the model to encode modal-specific information to better handle the missing scenarios in each modality.\n\n2. Performance v.s. MMP when owning comparable parameters\n\nWe compare our method with MMP by allowing them to own comparable parameters. Specially, either when we decrease the required parameters of our method to 0.2M (0.2% of the entire model) by reducing the channel dimension, or we simultaneously increase the required parameters of both methods, our method consistently outperforms MMP, which verifies its effectiveness. \n\n| Parameters| 0.2M | 1.6M | 2.8M | 4.0M | 5.2M | 6.4M |\n | :---:|:---:|:---:|:---:|:---:|:---:|:---:|\n | MMP | 51.34%| 50.74% | 50.82% | 50.72% | 50.64% | 50.46%| \n | **Ours** | **53.14%** | **53.56%** | **53.88%** | **54.24%** | **54.12%** | **53.98%** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We provide (1) a figure to further illustrate out proposed three prompts by comparing them with our baseline and MMP[17]. (2) Visualizations for the dynamic prompts using the T-SNE method on the Food101 dataset upon the missing-both setting with η=70% and η=50%.

Meta-review: The paper receives four positive and one negative ratings. Overall, the reviews initially have concerns about technical clarity (e.g., prompt designs, baseline setting) and experimental results (e.g., results using ViLT, contribution of each designed prompt, results of completely missing one modality). After the rebuttal, most reviewers find the major issues resolved (three reviewers upgraded the rating). The remaining concern from reviewer xVBv and YNgk is on the technical novelty. The AC took a close look at the paper and finds that designing these prompts is of great interest in the missing-modality situations, and hence recommends the acceptance decision. It is highly recommended that the authors should incorporate reviewers' suggestion in the paper (especially the illustration of prompt design, more experimental results) and release the code for fostering this important research area.
Decision: Accept (poster)