Deep Correlated Prompting for Visual Recognition with Missing Modalities

Authors: lianyu  hu
tongkai  shi
wei  feng
fanhua  shang
liang  wan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 05:15:52

Abstract: Large-scale multimodal models have shown excellent performance over a series of tasks powered by the large corpus of paired multimodal training data. Generally, they are always assumed to receive modality-complete inputs. However, this simple assumption may not always hold in the real world due to privacy constraints or collection difficulty, where models pretrained on modality-complete data easily demonstrate degraded performance on missing-modality cases. To handle this issue, we refer to prompt learning to adapt large pretrained multimodal models to handle missing-modality scenarios by regarding different missing cases as different types of input. Instead of only prepending independent prompts to the intermediate layers, we present to leverage the correlations between prompts and input features and excavate the relationships between different layers of prompts to carefully design the instructions. We also incorporate the complementary semantics of different modalities to guide the prompting design for each modality. Extensive experiments on three commonly-used datasets consistently demonstrate the superiority of our method compared to the previous approaches upon different missing scenarios. Plentiful ablations are further given to show the generalizability and reliability of our method upon different modality-missing ratios and types.

Main paper: 
 introduction: 
 Our human beings typically perceive information of multiple modalities such as visual, linguistic and audio signals to understand the world, where different signals drawn from various perspectives are inherently complementary. Thus, modeling and coordinating multimodal information is of great value for large-scale models to reason about real-world scenarios. Recently, multimodal models [34, 1, 22, 29] have developed fast powered by the large collection of multimodal data pairs and evolution of model architectures (e.g., Transformer [31] ), which demonstrate promising performance across a series of downstream tasks, such as cross-model retrieval [8, 18, 29, 9] , image captioning [24, 1] and image/video generation [23, 22] . Supported by their large capacity and general knowledge acquired by training upon web-scale data, these models have been more and more applied to daily work in our life (e.g., GPT-4 is used by millions of people).However, there exist two major concerns that may hinder these impressive models from broader applications. First, a common assumption of previous methods is that the input data is modalitycomplete, which may not always hold in the real world due to privacy considerations, collection difficulty and security issues [27, 19] . When an input modality is missing in general real-world conditions, the performance of these models usually degrades a lot (regardless of training or testing settings) [27] , which is easily influenced by the input completeness. Second, these large models are usually parameter-abundant [1, 6, 21] and require heavy computations [30, 45] to pretrain and finetune on downstream tasks, whose computational demands may not always be available in most real-world applications due to limited computing resources. It's necessary to develop a new method to efficiently adapt these powerful methods to perform robustly against missing-modality scenarios.Previous works [27, 42, 32, 28, 19] of multimodal learning have considerably explored the missingmodality issues. Earlier works [27, 42, 32, 28] mostly reconstruct the absent information of missing modalities or use other modalities to augment the missing modalities. MMP [42] has first introduced prompt learning to handle missing-modality scenarios by regarding different missing cases as different types of input. The multimodal backbone is kept frozen and only the newly introduced prompts are updated in the fine-tuning process, thus only incurring a few extra computations. However, MMP [19] simply inserts independent prompts into each layer, and overlooks the relationships among prompts and input features. The prompts across different layers lack cooperation to aggregate beneficial information to well guide the model predictions. The fixed prompts are used for different input samples which fail to consider the characteristics of various inputs. The complementary multimodal information of different input modalities is overlooked in the fine-tuning process.To better adapt large multimodal models to missing-modality scenarios, we introduce deep correlated prompting (DCP) by capturing different types of correlations between prompts and input features. Specifically, to leverage the hierarchical semantics of different layers, we propose correlated prompts by perceiving beneficial information from preceding layers to instruct the features of the current layer. We further propose to dynamically generate the prompts according to the input features to better fit the characteristics of different inputs. To leverage the complementary information of multimodal inputs, we decompose the prompts into modal-common and modal-specific parts to guide each encoder to focus on its unique features. The proposed prompts are concatenated and prepended to the input and intermediate features of the multimodal backbone to instruct the model to alleviate the performance drop caused by the missing modality. The multimodal backbone keeps frozen during training and only the learnable prompts are tuned, thus offering high training efficiency. Extensive experiments on three widely-used datasets including MM-IMDb [2] , UPMC Food-101 [33] and Hateful Memes [17] demonstrate consistently superior performance compared to other methods across all benchmarks, which verify the effectiveness of our proposed method. Ablation studies are further given to verify the generalizability and reliability of our method upon different missing-modality types and ratios. Earlier works of missing modalities for multimodal learning mostly generate the missing modality based on other modalities [42] , or align features of latent representations for multiple modalities to help recognition [38, 15] . Recently, multimodal transformers [1, 22, 23] emerge as an effective tool to model information from various modalities and process them into a robust representation, which have shown impressive performance over a series of tasks [8, 18, 29, 9, 24, 11, 12] . However, these methods typically assume that the inputs are modality-complete, which may not always hold in real-world scenarios. When a modality is missing, these methods usually demonstrate degraded accuracy and lead to unstable performance [27] .To deal with missing modalities in multimodal transformers, MMIN [42] predicts the intermediate features of the missing modality based on other available modalities, by learning a common multimodal representation. SMIL [28] proposes a Bayesian meta-learning framework to estimate the latent features of the modality-incomplete data. It further explores the effects faced with severe modalityincomplete samples (e.g., 90% missing ratio). Ma et al. [27] test the robustness of multimodal transformers to missing modalities and their missing types, and propose a multitask optimization framework to improve it via modality fusion. Zeng et al. [40] propose a tag-assisted transformer Encoder network to handle the problem of missing uncertain modalities. ShaSpec [32] employs a shared head across different tasks to aggregate information from various input samples, which complements the features for missing modalities. However, these methods mostly assume that a fixed modality is missing, which is known in advance. Besides, these methods still require updating most parameters of the model, which consume heavy computations in downstream tasks. MMP [19] first introduced missing-aware prompts to handle scenarios with missing modalities with minimal extra computational costs. However, it simply inserts independent prompts into the intermediate features of the multimodal backbone, without considering the relationships between prompts of different layers as well as the correlations between prompts and input features. In contrast, we carefully design the prompts by exploring the correlations between prompts and input features, and achieves much superior performance across all missing-modality scenarios upon three commonly-used datasets. 
prompt learning: 
 Prompt Learning is first explored in neural language processing (NLP), which adopts a "prompt" to modify the input text to instruct the pre-trained model for downstream tasks. Earlier works [6, 14] usually adopt manually designed prompts to improve the generalizability of large models over downstream tasks. Later, prompt tuning methods [26, 20, 25] emerge by prepending learnable prompts to the input features in the training phase to automate the optimization process. Recently, prompt learning is also introduced into computer vision tasks [4, 13, 37] and multimodal learning tasks [36, 44, 35, 16, 43] . CoOp [44] is first proposed to insert learnable soft prompts besides input images to adapt vision-language models to various vision tasks. CoCoOp [43] generates an image-conditional prompt to utilize the power of input features. ProGrad [46] only updates the prompts whose gradients are aligned to the "general knowledge" generated by the original prompts. KgCoOp [36] tries to align the output embeddings of the text encoder with those of the pretrained CLIP to preserve beneficial information. MaPLe [16] injects deep learnable soft prompts into both the image and text branches to enable prompt collaboration. DualPrompt [35] extends the prompts to learn different task information conditionally in continual learning. DePT [41] decouples basespecific knowledge from feature channels into an isolated feature space during prompt tuning. These works have demonstrated the effectiveness of prompt learning to adapt large-scale vision-language models for downstream tasks with minimal extra computation costs. In this paper, we introduce prompt learning into multimodal models to increase their robustness to missing-modality scenarios, via attaching different types of prompts according to various missing cases. 
overall framework: 
 Problem definition. We first provide a succinct overview of the missing-modality scenario addressed in this paper. Without loss of generalizability, we explore multimodal inputs with M = 2 modalities m 1 and m 2 (e.g., text and image), which is naturally extensible to more modalities. Specifically, with a multimodal dataset D = {D c , D m1 , D m2 }, we denote D c ={x m1 , x m2 , y} as the modality-complete case, where x denotes the input and y represents the label. We let D m1 ={x m1 , y} and D m2 ={x m2 , y} denote the modality-incomplete cases, wherein one modality is absent (e.g., missing text and missing image). Some examples of modality-complete and modality-incomplete inputs are depicted in the lower part of Fig. 1 .Recap MMP [19] . MMP [19] first introduced missing-aware prompts to handle missing-modality cases. Specifically, it assigns 2 M -1 types of prompts for tasks involving M modalities (e.g., 3 prompts for vision-language tasks, including one prompt for modality-complete case, one prompt for image-only case, and one prompt for text-only case). Given an input sample, it first selects the prompts corresponding to the missing case and then prepends the prompts to the input and intermediate features of the multimodal backbone, instructing the pretrained model to perform prediction. In this procedure, only the learnable prompts are updated and the multimodal backbone keeps frozen.Though MMP [19] has achieved notable progress in improving the robustness of multimodal models upon missing-modality cases by incurring minimal computational costs, it only assigns independent prompts to the input and intermediate features, which (1) fails to consider the relationships between prompts of different layers, and (2) lacks the correlations between prompts and input features, and (3) overlooks the complementarity of multimodal inputs. To better adapt the pretrained multimodal model for missing-modality scenarios, we propose to design three types of missing-aware prompts by capturing the relationships between prompts and inputs. First, we generate missing-aware prompts by leveraging the correlations of preceding prompts across multiple layers and various modalities. Second, we dynamically generate the prompts for each input sample to fit its characteristics. Third, ). Then we prepend the prompts to the inputs and intermediate features of both encoders to instruct the model to fit the missing case. Finally, we concatenate the task-related token of both encoders as the final representation, and pass it through a fully-connected layer for class prediction. In the whole procedure, only the fully-connected (fc) layer and deep correlated prompts are updated while others keep frozen.we decompose multimodal prompts into modal-common parts and modal-specific parts, which fuses beneficial information from other modalities and enables each encoder to focus its unique features. Fig. 1 shows the framework overview of our proposed method. Specifically, without loss of generalizability, we adopt the widespread two-stream multimodal method CLIP [29] as our backbone. Given the input text and image, we first employ pretrained text and image embedding layers from the pretrained CLIP [29] to convert them into token sequences. For each input sample, we select the corresponding prompt P T m and P I m for the text encoder and image encoder, respectively, given the type of missing modality with m ∈ {c, m 1 , m 2 }. The prompt for each encoder is composed of three types of missing-aware prompts including the correlated prompts P T,R m (P I,R m ), dynamic prompts P T,D m (P I,D m ) and modal-common prompts P T,C m (P I,C m ). We concatenate the missing-aware prompts and prepend them to the input tokens x m1 (x m2 ) as a whole sequence to process. Finally, we concatenate the task-related token of both encoders as the final output representation, and pass it through a fully-connected layer for task prediction. In this procedure, only the parameters of the fully-connected layer and the newly introduced deep correlated prompts are updated in the training procedure, and the backbone (i.e., the text embedding layer, image embedding layer, text encoder and image encoder) is kept frozen. The illustration of our proposed prompts is given in Fig. 2 . We next introduce our prompting designs in detail.MMP [19] append independent prompts to the input and intermediate features of the multimodal backbone to guide model predictions. Though it could theoretically provide enough guidance for features in each layer, the prompts across each layer lack synergy, which fails to cooperate with the representations of each layer that various semantics. We argue that prompts across consecutive layers should be closely correlated to receive necessary semantics from preceding layers to instruct the current layer to handle missing modalities. Thus, we propose to generate the prompts of the current layer based on the observation of the preceding prompts. Besides, the features of different modalities usually contain complementary information. Leveraging the complementary information from various modalities could further help the model predictions. We propose to incorporate the beneficial information from multimodal inputs to help guide the outputs of each modality.Specifically, we prepend the prompts to the input features and the intermediate features of the multimodal backbone up to a depth of J. Taking the image encoder as an example, the calculation process of the i th (i ∈ [0, . . . , J -1]) layer V i could be expressed as:EQUATIONHere, X I i is the features of the i th layer in the image encoder and P I,R m,i-1 is the newly introduced correlated prompts of the (i -1) th layer. After the J th layer, the correlated prompt is retained from the previous layer, and the calculation process for the i th (i ∈ [J, . . . , N -1]) layer V i can be presented as:EQUATIONwhere N denotes the length of all layers.To leverage the correlations of prompts between different layers, we generate the prompts of the i th (i ∈ [1, . . . , J -1]) layer V i by observing prompts of its preceding layer V i-1 to inject beneficial information, as:EQUATIONwhere F I,R i-1 (•) denotes the prompt generation function for the (i -1) th layer in the image encoder. To decrease the required parameters and add non-linearity to F(•), we adopt F(•) as a bottleneck MLP with the GELU activation [10] in between, followed by a LayerNorm (LN) function [3] as:EQUATIONThe intermediate feature dimension of F(•) is r (usually r = 1 16 ) times of its input channel dimension. For the correlated prompts at the input level, we leave it randomly initialized without the generation procedure.To leverage complementary information from different modalities, we incorporate prompts from both encoders to fuse their distinct semantics for instructions. Taking the image encoder as an example, we generate the prompts of the i th (i ∈ [1, . . . , J -1]) layer V i based on the prompts of the layer V i-1 in the image encoder and the prompts of layer T i-1 in the text encoder as:EQUATIONHere, Concat denotes the concatenation operation. Instead of leaving prompts uncorrelated across different modalities, our design improves mutual synergy between different modalities to help adapt to various missing scenarios.Different input samples usually contain information of various semantics. Using fixed prompts for different inputs may not well fit their distinct features, and thus can't offer enough guidance for the model to fit different missing-modality scenarios. Thus, we propose to dynamically generate the prompts based on the input features to adjust the instructions to fit the characteristics of different inputs.Specifically, taking the image encoder as an example, we generate the dynamic prompts P I,D m based on the input features X I 0 as:EQUATIONwhere D I denotes the dynamic prompt generation function for the image encoder. To deal with the varying length of tokens for different inputs, we instantiate D I as a self-attention layer [31] , whose architecture can be expressed as :EQUATIONHere, MHA is the multi-head attention mechanism in transformers [31] . We set the number of heads as 1 for simplicity. The dynamic prompts are only inserted at the input level. For the subsequent layers, the prompts are retained from the previous layers and updated together with intermediate features following Eq. 2.Multimodal inputs usually contain both modal-specific information and modal-common information across different modalities. By disentangling the modal-common features from the modal-specific features, each modality could aggregate beneficial information from the shared features across various input modalities and build more powerful representations based on its unique characteristics. We decompose the multimodal missing-aware prompts into modal-common parts and modal-specific parts to store common characteristics across different modalities and model-specific features for each modality, respectively.Specifically, we introduce a modal-common prompt P C m for multimodal inputs to embody the mutual features across different modalities, and accordingly encourage the proposed correlated prompts and dynamic prompts to embed modal-specific instructions. To cooperate with the features of different modalities, we project the modal-common prompt P C m to the image and text space to obtain P T,C m and P I,C m via projection layers, respectively, as:EQUATIONHere, G T and G I are the projection layers for the text modality and the image modality, respectively, which are both instantiated as a MLP with the intermediate feature reduction factor r=16. We only insert modal-common prompts at the input level and leave the prompt in the intermediate layers reserved and updated following Eq. 2.4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental setup: 
 Datasets. We follow the previous works [19, 27] to evaluate our methods: MM-IMDb [2] is currently the largest publicly available multimodal dataset for genre prediction on movie genre classification. It is notated with both image and text modalities for 25959 movies. As a movie might have several genres, the genre prediction task is thus a multi-label classification task.UPMC Food-101 [33] is a large multimedia dataset consisting of noisy image-text pairs collected from Google Image Search with 101 food categories. It has identical categories to the largest publicly available ETHZ Food-101 dataset [5] , used to classify the categories of different foods.Hateful Memes [17] is a multimodal dataset for hateful meme detection (image + text) that contains 10,000+ new multimodal examples created by Facebook AI. To prevent the model from relying on a single modality, it is constructed to make unimodal models more likely to fail.Implementation details. We use CLIP [29] as our multimodal backbone, with ViT-B/16 [7] as the image encoder. For the image input, we follow CLIP [29] to resize input images into 224×224. The patch size is set as 16 for the multimodal transformer. For the text modality, we use the tokenizer from pretrained CLIP to tokenize the text input. The maximum length of text inputs is 77. We freeze all the parameters of both the image encoder and text encoder, and only tune the parameters of deep correlated prompts and the fc layer (for task target). We set the length L p of learnable prompts as 36 and prepend them to the features of M = 6 layers. We use Adam optimizer with initial learning rate of 1e-2 and weight decay 2e-2. The learning rate is warmed up for 10% of the total training steps and then is decayed linearly to zero. We perform our experiments with batch size of 4 on a 3090 GPU. For the missing modality, we stop feeding the inputs into the corresponding encoder and use a zero-filled tensor as the output instead.Metrics. We use corresponding proper metrics for each dataset to evaluate our method. For MM-IMDb [2] , we adopt F1-Macro to measure the multi-label classification performance. For UPMC Food-101 [33] , we employ the top-1 classification accuracy to evaluate the recognition performance. For Hateful Memes [17] , we use Area Underthe Receiver Operating Characteristic Curve (AUROC).Setting of Missing Modality. In this paper, we focus on the general realistic scenarios in real life, where any modality may appear in both training and testing phases. To stimulate this condition, we follow MMP [19] to define the missing rate η as the proportion of modality-incomplete data to the entire dataset. For vision-language tasks, there exist three types of missing cases: missing-both (text and image), missing-text and missing-image. For the missing-both case, the training and testing data are composed of η 2 text-only data, η 2 image-only data and (1-η) modality-complete data. For the missing-text and missing-image cases, the training and testing data are composed of η image-only (text-only) data and (1-η) modality-complete data. This definition could be naturally extended to data with more modalities by using ( η M 2 -2 ) modality-incomplete data for each missing case and (1-η) complete data. In our experiments, we use η=70% by default.Effectiveness. We first verify the effectiveness of our proposed components across different missing cases including missing-image, missing-text and missing-both. We conduct the experiments upon the MM-IMDb [2] dataset with missing rates ranging from 0% to 100%. The methods used for comparison include (1) baseline, which directly sets the features as zeros when a modality is missing;(2) Ours (A), which only equips the correlated prompts; (3) Ours (B), which equips both the correlated prompts and the dynamic prompts; (4) Ours, which uses all the three proposed prompts. The only difference between our method and the baseline is inserting learnable prompts at each layer which only bring few extra parameters. The results are shown in Fig. 3 .It's first noticed that compared to the baseline, all our three variants could notably promote the performance across all missing rates for various missing-modality cases, demonstrating strong robustness to different missing-modality scenarios. Using the correlated prompts boosts the performance most, and equipping the dynamic prompts or modal-common prompts offers a similar performance boost. Using all three proposed prompts achieves the best performance, which verifies the effectiveness of capturing correlations between prompts and input features. An interesting observation is that the performance degradation is much smaller when input images are missing compared to missing-both and missing-text, which shows that text is more important for the task target on this dataset.Ablation study. We test the configurations for our proposed three prompts on the val set of MM-IMDb dataset in Tab. 1, Tab. 2 and Tab. 3, respectively. In the upper part of Tab. 1, we first explore how to leverage the correlations between prompts of different layers. "No projection" denotes using independent prompts for each layer and "Fc" denotes using a fully connected layer for projection. It's observed that compared to using independent prompts, either using a fc or a MLP offers a notable performance boost. The MLP gives better performance for its non-linearity and we use it by default.We then explore the prompt depth J (how many layers to insert prompts) in the middle of Tab. 1. We notice that the performance reaches a peak when J=6, and either decreasing it or increasing it would degrade the performance. Finally, we verify the efficacy of incorporating multimodal prompts for instruction in the bottom part of Tab. 1. It's observed that introducing bi-modal prompts to generate the prompts of the current layer gives better performance, which shows that different modalities can offer complementary information.We test the configurations for the dynamic prompts in Tab. 2. We first explore the prompt depth for the dynamic prompts. It's observed that the performance reaches a peak when the prompt depth equals 1, and continues to decrease when the prompt depth increases. We thus set the prompt depth as 1. We then explore the generation functions for the dynamic prompts. Besides the default attention mechanism, we test other alternatives which use a maximum, minimum or average pooling layer followed by a fc layer to generate the prompts. It's observed that all approaches could give a notable performance boost, and the attention function achieves the best performance. We explore the configurations for the modal-common prompts in Tab. 3. We first examine the prompt depth for the modal-common prompts. It's observed that as the prompt depth ranges from 1 to 6, our model achieves similar results and reaches a peak when the prompt depth equals 6. Considering the accuracy-computation trade-off, we set the prompt depth as 1 by default. We then explore the projection functions for the modal-common prompts. Generally, we observe that using a MLP for projection performs better than using a fc for projection. Using different channel reduction factors r including 4, 8 and 16 achieves similar performance. We thus use a MLP with r=16 by default.Generalizability. We conduct experiments to verify the generalizability of our method when trained and tested upon different missing cases. We evaluate models trained on missing-both cases or missing a specific modality, and test them on the missing-both, missing-image and missing text cases in Fig. 4 (a), (b) and (c), respectively. It's first noticed that all our variants outperform the baseline by a large margin across various missing cases and missing rates. Models trained with a certain modality missing (e.g., missing-text, missing-image), usually perform slightly better than models trained upon missing-both cases when tested with a certain modality missing. Models trained with missing both modalities perform robustly to cases with any missing type (e.g. missing-both, missing-text and missing-image). We further observe the model trained with higher missing rates (e.g., 70%), is more robust to high-rate missing cases (70%-100%) during testing than models trained with lower missing rates (e.g., 30% and 50%). It is concluded that models trained on missing-both cases are robust to various missing cases, which demonstrate strong robustness compared to training with one modality.Comparison with other methods. We compare our method with recent approaches upon three commonly-used datasets, i.e., MM-IMDb [2] , UPMC Food-101 [33] and Hateful Memes [17] , to verify its effectiveness upon different missing-modality scenarios. We include the following methods for comparison (1) baseline, which directly drops the features when a modality is missing; (2) CoOp [44] , which only prepends prompts at the input level; (3) MMP [19] , which inserts independent prompts for the input and intermediate features of the multimodal backbone; (4) MaPLe [16] , which generates the prompts in the image encoder based on those of the text encoder; (5) DePT [41] , which decouples base-specific knowledge from feature channels into an isolated feature space during prompt tuning. MMP [19] is originally based on the ViLT backbone and we reimplement it following the same protocol as ours. We compare these methods across three different missing rates including η=50%, η=70% and η=90% upon three various missing-modality cases including missing-image, missing-text and missing-both in Table . 4. Our method largely outperforms other methods on three datasets across different missing rates, which verifies its robustness to different missing scenarios. Compared to MMP [19] , our method exceeds it over all missing cases by a large margin, which demonstrates the superiority of leveraging the correlations between prompts and input features.An interesting observation is that on most datasets (e.g., MM-IMDb [2] and Food101 [33] ), missing text input usually has higher effects on the performance than missing image input. We figure that texts are more crucial for the tasks by providing more detailed explanations and precise captions than images on these two datasets. Instead, on the Hateful Memes dataset [17] , it's observed that missing image input influences the results more than missing text or missing both. This may reflect the different focus when constructing various datasets.Efficiency. Compared with only finetuning the last fc layer to fit downstream tasks, our deep correlated prompts only introduce extra 4.0M parameters, which just consume 2.4% of the entire model (151M), but could notably improve the performance by a large margin over different missing cases and missing rates.Table 4 : Comparison with CoOp [44] , MMP [19] , MaPLe [16] and DePT [41] on the MM-IMDb [2] , UPMC Food-101 [33] , and Hateful Memes [17] Performance v.s. MMP when owning comparable parameters. We compare our method with MMP by allowing them to own comparable parameters in Tab. 5. Specially, either when we decrease the required parameters of our method to 0.2M (0.2% of the entire model) by reducing the channel dimension, or we simultaneously increase the required parameters of both methods, our method consistently outperforms MMP, which verifies its effectiveness. Limitations and broader impacts. The limitations include (1) we only test the effectiveness upon two commonly-used two-stream multimodal models, and don't apply our method to other popular multimodal models. (2) we only include two modalities in the experiments, and will incorporate more modalities in the future. The broader impacts include (1) The missing-modality cases happen at times in real life. This paper proposes a novel method by adapting large multimodal models towards missing-modality scenarios, which increases the robustness of large multimodal models. (2) Our methods can notably decrease the required computations compared to previous methods upon missing-modality learning, which achieves a better accuracy-computation trade-off in real life.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: They are given in the implemention details. 
conclusion: 
 In this paper, we tackle two main challenges in multimodal learning, including (1) any modality may be missing in any learning phase (e.g., training, testing or both), and (2) how to efficiently adapt large-scale multimodal transformers to fit missing-modality cases. We propose deep correlated prompting by leveraging the correlations between prompts of different layers and the relationships between prompts and input features. Results on three diverse datasets across various missing types and missing ratios verify the effectiveness of our proposed method.A Appendix / supplemental material Compatibility with complete data. In this paper, we have previously assumed that there is no guarantee modality-complete data can be entirely collected due to various factors in real-world scenarios. However, these still exist publicly-available modality-complete datasets which are carefully collected and annotated for model training. Thus, we perform additional experiments to compare our method with the baseline and input-level prompting by training on modality-complete data. To stimulate the missing-modality case during testing, we randomly select data with different modalitymissing cases for each data pair (i.e., text-only, image-only, or complete) during training. Note that, different from other experimental settings introduced in the paper, here one data pair can be in various missing-modality cases at different training epochs. We plot the results in Fig. 5 . It is observed that our method consistently outperforms the baseline and input-level prompting with better performance across different missing ratios. Deployment on other backbones. We deploy our proposed method upon other commonly-used vision-language models like CoCa [39] and ViLT [18] , to verify its flexibility. CoCa is trained jointly with contrastive loss and captioning loss, which subsumes model capabilities from both contrastive approaches like CLIP [29] and generative methods like SimVLM [34] . ViLT is a single-stream multimodal backbone which directly concatenates the input text features and image features and feed them into a common transformer for processing. We adopt ViT-B/32 [7] as the image encoder for CoCa, and list the results in Table . 6 and Table 7 , respectively. It is observed that our method can bring notable performance boost across three datasets compared to the baseline, which verifies its flexibility across various large-scale vision-language models. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Error bars are not reported due to the high computational costs.Guidelines:• The answer NA means that the paper does not include experiments. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: [Yes] Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Justification: They are given in the implemention details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 


Decision: The paper receives four positive and one negative ratings. Overall, the reviews initially have concerns about technical clarity (e.g., prompt designs, baseline setting) and experimental results (e.g., results using ViLT, contribution of each designed prompt, results of completely missing one modality). After the rebuttal, most reviewers find the major issues resolved (three reviewers upgraded the rating). The remaining concern from reviewer xVBv and YNgk is on the technical novelty. The AC took a close look at the paper and finds that designing these prompts is of great interest in the missing-modality situations, and hence recommends the acceptance decision. It is highly recommended that the authors should incorporate reviewers' suggestion in the paper (especially the illustration of prompt design, more experimental results) and release the code for fostering this important research area.

Reviews: review: paper_summary: paper_summary: This paper proposes a new method to handle missing modalities in visual and language recognition systems.
The paper proposes a very similar method to the one proposed by MMP [17] but using different way of getting the prompts to feed them into the transformer layers. 
Comparison with other works show that the method seems to be effective and some ablations studies are performed to study the different design choices. The method is validated using the most common datasets for this task.
strength_weakness: strength_weakness: - The main weakness of the paper is clarity. There are three different sets of prompts that are appended to the intermediate representations. However, the only difference between them seems to be the type of architecture the method uses to compute them. The explanation is very limited and Figure 1 does not illustrate where do these prompts come from. Without the clarity of this explanation it becomes really hard to understand how the motivation of each type of prompt fits the design. What are exactly correlated prompts, dynamic prompts, and modal-common prompts? What make them correlated, dynamic and modal-common? This is not clear in the paper at all. 

- It is not clear what is baseline. What does dropping features when modality is missing? The input sequence become shorter and coming from only a single modality? If that's the case, what is trainable and what is not? 
Please explain well this part. I would expect that this baseline is: training with the same number of parameters as the base method, by simply adding learnable prompts at each layer and training using mod-drop (dropping modalities randomly when training, dropping modalities can be done by inputting noise instead of tokens, the average of the modality tokens, zeroes, or not passing the missing modality at the input, it is a design choice that needs to be explained). If it is not what I'm thinking, please explain well, since this is a key experiment.

- When comparing with MMP, how did the authors do it? Please explain exactly how was this re-implementation. Also, to be fair, the authors should have applied their method using ViLT instead of CLIP, in that way there is no doubt that this method is better than the very similar MMP. 

- What is the zero-shot performance of CLIP on these datasets?
questions: questions: - Please explain well the mechanism of the different types of prompts, input, output at train and test time for each one of them. It could have been done easily with a figure, but at least with a few sentences it could become clearer. 
- What makes a "dynamic" prompt "dynamic"?
- What does baseline mean and how was implemented?
- How was MMP implemented on your framework?
- What if using ViLT instead of CLIP, would still your method be better than MMP?
- What is the zero-shot performance of CLIP on these datasets? it is important since this might be a robust method that does not suffer from missing modality. It can be implemented using nearest neighbor to each of the class embedding using either modality, and combining them when both are present.
limitations: limitations: Limitations have been addressed.
paper_summary: paper_summary: This paper proposes to address the missing modality problem for the multimodal recognition model (i.e. the multi-modal data could be incomplete). There are three techniques of prompting being proposed (while the recognition model, i.e. two-stream multimodal method CLIP in this paper, is kept fixed), including: 1) correlated prompts, where a part of the prompts in the input-level are firstly selected according to the missing scenario (e.g. complete, text-only, or image-only), then the prompt in each of the following network layers are predicted from the multimodal prompt of its preceding layer; 2) dynamic prompts, the input-level prompts contain a portion generated according to the input sample; 3) modal-common prompts, where the rest of the input-level prompts is stemmed from a common component shared across modalities. The combination of the aforementioned three techniques experimentally shows better performance in comparison to various baselines (mainly the SOTA method from MMP [17]).
strength_weakness: strength_weakness: - The modal-common prompts and the dynamic prompts actually are not directly connected to the missing modality problem (or being irrelevant to different cases of missing modality). While excluding these two prompting techniques from the proposed method (in which such variant becomes "Ours (A)" in Figure 2), the improvement with respect to the state-of-the-art approach of handling missing modality (i.e. MMP[17]) would become marginal (please include MMP[17] into the ablation study shown in Figure 2 or directly provide the tabular quantitative results for the ablation study). Similarly, while we only consider the technique of correlated prompts as the manner in the proposed to tackle the missing modality, it becomes the only difference in the proposed method compared to MMP [17] (in terms of methodology), thus leading to the concern of limited novelty. Furthermore, there should be a baseline of integrating the modal-common prompts (acting as a basic component of prompt) and dynamic prompts into MMP[17] to better highlight the contribution of the proposed correlated prompting technique (which is the main technique in the proposed method to be connected with the missing modality challenge). Moreover, as modal-common prompts and the dynamic prompts introduce additional learnable parameters (in comparison the correlated prompts), there should be further detailed analysis/comparison in terms of number of learnable parameters versus model performance.
- Though the proposed dynamic prompts do experimentally shown to improve the overall performance under various missing modality cases, such prompting technique is actually not new, where we can see its similar application in various research problems (e.g. Wu et al., IDPG, NAACL'22; Lu et al., PromptPG, ICLR'23; Qiu et al., FedTPG, FL@FM-NeurIPS’23).
questions: questions: Although currently the proposed method seems to provide superior performance with respect to various baselines and its proposed techniques (i.e. correlated prompts, dynamic prompts, modal-common prompts) are experimentally shown to benefit the model performance, there are concerns regarding limited novelty (where only the correlated prompts are considered to be related to missing modality while the other two techniques, i.e. dynamic and modal-common prompts, are not) and detailed analysis for the number of learnable parameters versus model performance, (as listed in the weaknesses), in which the the authors are highly encouraged to make the corresponding clarifications in the rebuttal.
limitations: limitations: no potential negative societal impact is found.
paper_summary: paper_summary: This paper addresses the challenge of generalized missing modalities in multimodal learning, where a modality can be absent during any learning phase (e.g., training, testing, or both). he authors investigate prompt learning with missing modalities and propose deep correlated prompts designed to capture various types of correlations between prompts and input features across different modalities. Specifically, the proposed prompts include mechanisms for perceiving beneficial information from preceding layers, dynamically generating prompts based on input characteristics, and leveraging the complementary information from multimodal inputs. These designs improve the robustness of large multimodal models (e.g., CLIP) to missing modalities. Extensive experiments and ablation studies demonstrate consistently superior performance and verify the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1.	The paper lacks a detailed explanation or discussion on the efficacy of different prompt designs. In Figure 2, it shows that sequentially adding different designs improves the baselines, but it does not discuss the individual improvement gains for each design. Additional discussion on each design could help validate whether the increasing gains from sequentially adding designs are not merely due to more learnable parameters.
2.	The paper lacks visualization of each learnable prompt (e.g., deep correlated prompts, dynamic prompts, and common prompts). Visualizations could help validate whether the different components work as expected. For example, do dynamic prompts genuinely capture the different characteristics of inputs, or do they merely distinguish between different missing cases, which might be easier to learn due to the obvious absence of a modality?
3.	For each available modality, it seems there are a total of $(3*(2^M-1))$ prompts for each missing modality case. This could lead to an exponential increase and redundant prompts as more modalities are considered (i.e., M>2). For example, in a vision-and-language task, in the case of complete and missing-image, the text modality is available for both cases. However, it requires two separate prompt sets for the text encoder, which may actually learn the prompts for the same “text-available” case.
questions: questions: 1. In Table 4, I noticed that some values of the related work MMP are the same as the figures recorded in the paper. For example, the settings with:
   - missing rate = 70% (100% image and 30% text) in MM-IMDb,
   - missing rate = 70% (30% image and 100% text) in Hateful Memes,
   - missing rate = 70% (65% image and 65% text) in Hateful Memes.

   As far as I know, the MMP backbone model is the multimodal transformer ViLT. The authors state they re-implemented MMP on their setting (i.e., CLIP) for a fair comparison. It seems that the numbers should not be the same since they use different backbone models. Can the authors clarify why the values are identical despite using different backbone models?

2.	According to the design of prompts, it seems that the proposed method is not limited to two-stream models (e.g., it could be applied to single-stream models without using Eq. (5)). Generalizing the method to single-stream models and comparing it with related works could be helpful in verifying the generalizability of the proposed method. Have the authors tried it for single-stream models? If so, what were the results?
3.	I am willing to revise my rating if the authors also address the concerns mentioned in the weaknesses.
limitations: limitations: One limitation is that the proposed method requires modality-specific deep correlated prompts for each available modality, which could be challenging to extend to more modalities (e.g., five or more modalities).
paper_summary: paper_summary: The model proposes prompting strategy where both modalities (image and text) are prompted, and the prompt for both modalities are correlated. The strategy is to use multiple prompts, namely correlated prompts, dynamic prompts, and modal-common prompts. As the backbone itself is multimodal (CLIP), it is a good idea to consider synchronized multi-modal prompts to fully harness the model capabilities when prompting it. The model surpasses multiple multimodal SoTAs on multiple datasets and also has proven to be effective in handling missing modalities in training and inference.
strength_weakness: strength_weakness: 1. Ablation studies regarding the multimodal backbone, e.g. using other model than CLIP or use dedicated unimodal encoders for each modality, highly recommended to increase paper quality.
2. In table 4, what are the performances when either image or text modalities are completely missing?
questions: questions: 1. Please elaborate further on how modal-common features are disentangled.
2. If possible, show the layer J-th in Figure 1 (framework overview)
3. Minor suggestion: The phrase "abundant ablations" in the introduction is a bit overboard, I suggest to write it as just "Ablation studies are further given..."
limitations: limitations: The limitations are discussed in the appendix, including that only text and visual modalities are tested with this model and the number of tested models.
paper_summary: paper_summary: The paper proposes a prompt optimization approach to the missing modality issues in multimodal learning. Inspired by the missing-aware prompt (MMP), this paper adds more prompts, including correlated, dynamic and modal-common prompts, to each encoder to improve the performance. The experiment on three datasets shows the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The novelty of the proposed method is limited since the MMP has proposed the prompt optimization approach to solving the missing modality issue. Compared with MMP, this paper adds more parameters in the form of prompt tokens from different inputs and functions. 

2. The empirical comparison with MMP is probably not quite fair as the proposed method uses more additional parameters compared with MMP. According to Line 337, this method adds 2.4% additional parameters, while MMP only adds 0.2%.
questions: questions: What is the specific contribution of this paper compared with MMP, other than adding more parameters and functions?


Will MMP's performance be better or comparable when MMP uses the same number of parameters as the proposed method?
limitations: limitations: Yes.

Comments: title: Response for Reviewer 64zT
Response for Reviewer xVBv
Response for Reviewer nnhg
Response for Reviewer nnhg
comment: Dear reviewer, thanks for your time and efforts in reviewing our manusript. We have provided a point-to-point response regarning your concerns and we are looking forward to receiving your valuable feedback on the points we addressed in the response. If you have further concerns, place let us know and we will respond to you as soon as possible. Thank you for your dedication to the review process.

Sincerely,
Authors
Dear reviewer, thanks for your time and efforts in reviewing our manusript. We have provided a point-to-point response regarning your concerns and we are looking forward to receiving your valuable feedback on the points we addressed in the response. If you have further concerns, place let us know and we will respond to you as soon as possible.Thank you for your dedication to the review process.

Sincerely,
Authors
Thanks for the response. My concern about the experimental comparison is addressed, so I will increase the score to 5. The novelty concern still holds but not a major flaw as the novelty concern is possibly not quite objective as it always is, so I wouldn’t fight for rejection.
Thank you to the authors for addressing the questions. The authors have addressed most of my concerns, and I hope the answer can be further added in the final version of the paper either in the main paper or in the appendix. Overall, I stand with my initial rating of Weak Accept as the authors have demonstrated that their proposed method is competitive with logical explanation and proof to back it up.
Many thanks for your careful comments. We will incorporate detailed explanations in the manuscript to clarify the improvements of our method compared to the baseline and existing methods. We will also make the code publicly available.
Authors have addressed most of my concerns. I still think the paper could be clearer at explaining the different model components, and be more thorough when explaining the different setups for baselines and other methods. The included experiments in the rebuttal helped clarifying most of the doubts. However, the paper will need a substantial effort at explaining better all the unclear points that I and other reviewers have raised. I will increase my score to 6, and I hope the authors work on the manuscript to improve its clarity and especially its reproducibility in terms of experiments, experimental setup, baseline definition and definition of model components. Thank you for your great work in this rebuttal.
Many thanks for your reply. We will try to save space to include the figures in our manuscript. For the baseline, the Fine-tuned CLIP in the last Table in the rebuttal is the same baseline model that we used for Figure 2 and Figure 3 in the main paper. The baseline is trained with the same missing modalities at training time to keep fair comparison with other methods. We also keep the identical training settings for other methods to make a fair comparison.
Thanks to the authors for the thorough reply. I think the figure makes the contribution of the paper way more clear, and I suggest that it is included in the main manuscript.

It is still not 100% clear to me what is baseline for you. I apologize for not being clear with my question in the first place. 
**Fine-tuned CLIP** in the last Table that you showed in the rebuttal is the same baseline model that you used for Figure 2 and Figure 3 in the main paper? Was this baseline trained with missing modalities at train time (as a sort of "augmentation"), i.e. modality dropping at train time? 

That's my only remaining question, in order for me to give my final evaluation on the paper.