Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation

Authors: Duy A. Nguyen, Do Huu Quan, Khoa D Doan, Minh N. Do
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multimodal learning, Uncertainty estimation

Abstract:
Multimodal learning has demonstrated incredible successes by integrating diverse data sources, yet it often relies on the availability of all modalities - an assumption that rarely holds in real-world applications. Pretrained multimodal models, while effective, struggle when confronted with small-scale and incomplete datasets (i.e., missing modalities), limiting their practical applicability. Previous studies on reconstructing missing modalities have overlooked the reconstruction's potential unreliability, which could compromise the quality of the final outputs. We present **SURE** (Scalable Uncertainty and Reconstruction Estimation), a novel framework that extends the capabilities of pretrained multimodal models by introducing latent space reconstruction and uncertainty estimation for both reconstructed modalities and downstream tasks.  Our method is architecture-agnostic, reconstructs missing modalities, and delivers reliable uncertainty estimates, improving both interpretability and performance. SURE introduces a unique Pearson Correlation-based loss and applies statistical error propagation in deep networks for the first time, allowing precise quantification of uncertainties from missing data and model predictions. Extensive experiments across tasks such as sentiment analysis, genre classification, and action recognition show that SURE consistently achieves state-of-the-art performance, ensuring robust predictions even in the presence of incomplete data.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1729400700664

thread_type:
official_review

<thread_object>
creator: reviewer_dpnx

<content>
summary:
This paper introduces latent space reconstruction and uncertainty estimation to address missing modality issues. Experiments validate the effectiveness of the proposed method.

strengths:
1. The proposed method achieves good results on multiple datasets.

weaknesses:
1. The authors leverage a pre-trained model and only fine-tune the reconstruction module and the prediction head. This pipeline is similar to prompt-based methods [1,2]. Therefore, these baselines should be incorporated in the experiments.
2. The authors claim that existing methods overlook the unreliability of reconstruction features which may damage the quality of the final outputs. However, in [2], they also perform a reconstruction process in the latent space and design prompts to enhance the reliability of the reconstructed features.  Meanwhile, [2] also leverages a pre-trained model. Therefore, the differences should be explained to highlight your contributions.
3. Why do you choose Pearson correlation instead of other correlation coefficient? Have you tried to use other nonlinear correlation coefficient?
4. Why not train the reconstruction module and classifier together?
5. The authors should report the results of all missing cases in Table 1-3. Besides, performances of different missing ratios can be added to enhance the paper.

[1] multimodal prompting with missing modalities for visual recognition, cvpr2023

[2] multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition, acl 2024

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730567603861

thread_type:
official_review

<thread_object>
creator: reviewer_dtc9

<content>
summary:
This paper focuses on an important challenge in multimodal learning—handling missing modalities, which is common in practical applications. The proposed SURE framework (Scalable Uncertainty and Reconstruction Estimation) aims to enhance pretrained multimodal models by incorporating latent-space reconstruction and uncertainty estimation. The inclusion of uncertainty estimates for both reconstruction and final predictions is valuable, as it provides insights into the model's confidence, especially when modalities are incomplete.

However, while the paper presents extensive experiments, its motivation and novelty are somewhat ambiguous. Moreover, claims regarding “scalability”, “distribution-free” and other aspects lack sufficient justification.

strengths:
1. The paper brings up an important issue in multimodal learning—handling missing modalities, which is common in real-world applications where all data sources are not consistently available.
2. The framework provides both reconstruction and output uncertainty estimates, which help assess the reliability of predictions.
3. The paper includes extensive experiments across diverse tasks (sentiment analysis, genre classification, and action recognition) and datasets.

weaknesses:
1. The motivation behind the paper is ambiguous. Although the introduction discusses the benefits of adapting pretrained multimodal models for downstream tasks, the rest of this paper primarily centers on uncertainty estimation. The authors could consider aligning the main content more closely with the stated motivation to improve coherence.
2. Figure 1 suggests that pretraining alone may be sufficient for handling incomplete inputs. Therefore, it remains unclear how missing modalities or incomplete inputs impact the adaptation of pretrained multimodal models.
3. The novelty of this paper may be limited, given that latent-space reconstruction and Bayesian uncertainty estimation are well-established techniques [1][2][3]. The authors should elaborate on how these approaches uniquely enhance the adaptation of pretrained multimodal models for downstream tasks, as their current combination appears somewhat trivial.
4. The authors describe the proposed Pearson Correlation Coefficient (PCC) Loss as a 'distribution-free' loss function. However, the notation in equation (4) suggests that the PCC loss still relies on the Gaussian assumption. Additionally, the uncertainty constraint for PCC loss also appears to be derived from this Gaussian assumption. The authors should further clarify the “distribution-free” property of the proposed PCC loss.

[1] Abdar, Moloud, et al. "A review of uncertainty quantification in deep learning: Techniques, applications and challenges." Information fusion 76 (2021): 243-297.

[2] Lian, Zheng, et al. "GCNet: Graph completion network for incomplete multimodal learning in conversation." IEEE Transactions on pattern analysis and machine intelligence 45.7 (2023): 8419-8432.

[3] Ma, Mengmeng, et al. "Smil: Multimodal learning with severely missing modality." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 3. 2021.

questions:
1. What does MMML stand for?
2. According to Table 3 and 3, it seems that pretraining contributes to the majority of the performance gain of SURE. For example, the performance of the proposed SURE without pretrained weights in Table 4 seems to be worse than the performance of previous methods in Table 3. However, with pretrained weights, the preformance is significantly increased and surpasses previous methods.
3. The integration of latent reconstructors and multiple uncertainty estimations could add computational overhead. A quantitative analysis of the computational costs compared to other methods would help in understanding SURE’s efficiency and scalability.
4. The authors claim that the proposed approach is scalable (**Scalable** Uncertainty and Reconstruction Estimation). However, no justification or experimental evidence is provided to support this claim.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730582101767

thread_type:
official_review

<thread_object>
creator: reviewer_5fya

<content>
summary:
This paper introduces SURE (Scalable Uncertainty and Reconstruction Estimation), a framework designed to improve pretrained multimodal models when modalities are missing during training or evaluation. SURE focuses on two key areas: reconstructing missing modalities within the latent space and quantifying uncertainty for both these reconstructions and subsequent predictions. A novel Pearson Correlation Coefficient loss function and statistical error propagation are used to estimate uncertainties, improving the reliability of the reconstructed inputs and final predictions. SURE is architecture-agnostic and can adapt to various pretrained multimodal models, achieving robust and interpretable results in experiments on sentiment analysis, genre classification, and action recognition.

Overall, this paper introduces an interesting perspective for enhancing pretrained multimodal models when modalities are missing. The framework integrates techniques for estimating uncertainty and is adaptable across various architectures. However, it requires clearer theoretical justification, especially regarding how uncertainty estimation contributes to improved predictive performance. Additionally, inconsistencies in benchmark results and limited evaluation of uncertainty estimation indicate areas that need refinement. Addressing these issues would enhance both the impact and comprehensibility of the framework.

strengths:
1. The proposed approach introduces uncertainty estimation to adapt pretrained multimodal models for downstream tasks with missing modalities. This method not only enhances overall performance on downstream tasks but also enables the estimation and interpretation of uncertainty arising from both the model's inherent stochasticity and incomplete inputs.

2. The proposed approach is validated across multiple benchmarks and compared against previous methods.

3. The approach is designed to be compatible with a wide range of pretrained multimodal models, making it adaptable and applicable across diverse architectures and tasks.

weaknesses:
1. It is unclear how and why uncertainty estimation enhances overall performance. While Section 2.3 suggests that incorporating Bayesian learning can help model the aleatoric uncertainty, there is a lack of theoretical explanation for how this contributes to improvements in predictive performance, such as accuracy.

2. In section 3.2, the authors mentioned that the Gaussian assumption allows for a closed-form solution to uncertainty estimation, suggesting that a strong alignment or correlation between uncertainty and prediction error leads to better performance. However, if the underlying distribution does not follow the Gaussian assumption, is this correlation still necessary for achieving an optimal solution? If not, optimizing the PCC loss to align uncertainty with prediction error may need further justification.

3. The performance of IMDer on the CMU-MOSI dataset seems to be inconsistent to the results reported in the original paper. For example, the F1 score for full inputs reported in the original IMDer paper is 85.1 v.s. 62.0 reported in this paper. The authors may need to clarify the difference.

4. The evaluation of uncertainty estimation in the paper is limited. Currently, the paper assesses uncertainty estimation by calculating the correlation between uncertainty and prediction error. However, it does not intuitively explain why a high correlation between these metrics is desirable or how it contributes to the model’s reliability.

questions:
1. A detailed caption is missing for Figure 3, making it difficult to understand the symbols and the flow of the proposed pipeline since their descriptions are spread across sections 2.2 to 2.4.

2. How does the number of reconstructors \( r_i \) scale with the number of modalities? Figure 3 suggests that a large number of reconstructors may be necessary to handle all possible missing-modality scenarios, which could affect the model’s scalability. A detailed explanation would clarify how the framework manages this growth efficiently.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644393022

thread_type:
official_review

<thread_object>
creator: reviewer_z2fy

<content>
summary:
This paper presents several key contributions through the development of the SURE (Scalable Uncertainty and Reconstruction Estimation) framework. First, SURE introduces a novel approach for handling missing modalities by incorporating latent space reconstruction
techniques into pretrained multimodal models. Second, SURE addresses the critical challenge of evaluating the reliability of both reconstructed inputs and final outputs by integrating uncertainty estimation. This is achieved through (1) a Pearson Correlation-based loss function and (2) the first application of error propagation in deep network training, allowing SURE to effectively quantify
uncertainties from multiple sources. Third, SURE is adaptable to a wide range of pretrained multimodal networks, demonstrating its robustness across various datasets and tasks, ultimately enhancing both interpretability and performance in multimodal learning scenarios.

strengths:
* The theoretical proofs and methodology of this article are solid and valid.

* The Error Propagation and Uncertainty Estimation proposed by the authors is interesting and unique.

* The authors implemented diverse experiments on multimodal datasets from different domains.

weaknesses:
* A great deal of multimodal works [1,2,3,4,5] related to modal absence has been neglected, leading to results and limitations that are difficult to measure.
These multimodal efforts are the main current SOTA in the field of studying modal absence across different modeling paradigms, including learning robust joint representations and reconstructions. It would be useful for the authors to clarify and compare the differences with these works.

* The writing of the article is not good enough. The author does not discuss and explain in depth many of the concepts and structures introduced. For example, there is no clear explanation of what exactly constitutes the reconstruction module described in the methods section.

* The authors consider a limited number of modal absence scenarios, and the study of modal absence should encompass both intramodal absence and intermodal absence obeying real-world scenarios. Then, the authors seem to follow a specific setup, leading to a lack of generalization of the experimental results.

[1] Li, Mingcheng, et al. "Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In AAAI.

[3] Zheng Lian, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. Gcnet: graph completion network for incomplete multimodal learning in conversation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

[4]Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabas P ´ oczos. Found in translation: Learn- ´
ing robust joint representations by cyclic translations between modalities. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 6892–6899, 2019.

[5] Li, Mingcheng, et al. "A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 9. 2024.

questions:
* Why use MMML specifically for the introductory introductory presentation? The reader cannot understand where the representation is. Please explain the MMML framework in detail. Would it still be similar if it was replaced with another model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732462629778

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Message from Authors to Reviewers

comment:
Dear Reviewers,

Thank you for your valuable comments on our article.

We have carefully revised the manuscript and addressed your feedback in detail. We understand your time is limited, but we sincerely hope you can review our responses. We believe we have resolved your concerns to the best of our ability during this discussion period. If you have any further questions, please feel free to contact us.

We would greatly appreciate it if you could reconsider your evaluation and kindly improve your score if you find our revisions satisfactory.

Thank you again for your time and consideration.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733310997081

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of authors' responses during rebuttal phase and message from the authors

comment:
We thank you for the helpful comments and support with our submission. With this summary, we want to summarize the major contributions, as well as highlight what we have done to fully address all the Reviewers' concerns and questions during this Discussion period. 

Regarding our contributions:

- Using a latent reconstruction technique as a means, we can apply pretrained multimodal frameworks to small-scale, multimodal datasets with missing modalities.
- Our work is the first in quantifying and analyzing the uncertainty of a pre-trained multimodal model's output, and its connection to the missing modalities (via the reconstruction process), besides the model' inherent nature themselves. The focus on uncertainty estimation and the use of Error Propagation are unique as acknowledged by Reviewer Z2fY and DTc9.
- We introduce a novel PCC loss to guide the uncertainty estimation process. This loss requires no bias regarding the underlying distribution of predictions but still enforces the desirable characteristic of uncertainty.

Regarding the work we have delivered during this Discussion period:
- We explained the reason behind our choice of baselines for the experiments. We provided additional experiments with more baselines and metrics (suggested by Reviewer 5FYA and dpnx). The result shows that our method produces strong performance and good uncertainty estimation, consistently outperforming all baselines over diverse experiments on multimodal datasets from different domains as acknowledged by Reviewer Z2fY and DTc9.

- We explained that the primary use of uncertainty estimation is a tool for assessing the reliability of results, while the reconstruction technique is a means for leveraging pretrained frameworks efficiently, in response to Reviewer DTc9. We added a method to select a suitable uncertainty threshold in the response to Reviewer 5FYA. We revised the Introduction and Method sections with these clarifications in the revised version.
- We explained that our method is a distribution-free technique that introduces no bias regarding the underlying distribution as shown in the response to Reviewer 5FYA and DTc9. We included this discussion in the Method section.

We believe that our responses have now sufficiently addressed all the concerns of the Reviewers. Our method, SURE, is a highly-performant technique that can be applied to various practical applications with missing data scenarios, especially in accuracy-critical domains such as healthcare or autonomous vehicles.


As we did not hear from all the Reviewers for most of the Rebuttal Phase, we sincerely hope that the Reviewers can take into account the significance and practicality of our work, and especially our responses and clarifications to the Reviewers’ concerns, and consider revising the ratings of our paper accordingly.

Regards,

The Authors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel approach for managing missing modalities by incorporating latent space reconstruction into pre-trained multimodal models, as well as uncertainty estimation for both reconstructed modalities and downstream tasks. This is performed adopting a Pearson Correlation-based loss function and applying statistical error propagation in deep model training.

As strong points, this work is recognized to present a sound and original methodology, adaptive to different architectures, provide theoretically insights, and having a good experimental analysis.

However, main concerns relate to missing works when discussing the state of the art, and corresponding lack of comparative analysis, leading to lack of generalization of the experimental results, the tests seem limited to scenarios in which only a selected number of modes is absent, still impeding to clearly assess the generalization of the method, and weak clarity, lack of details or justification of some parts of the methodology. Other issues regarding the experimental analysis and questions about the novelty are also reported.

This paper received all ratings below threshold (5, 5, 5, 5), which are maintained also after rebuttal. This paper was also discussed between the AC and the reviewers, who stuck in their original evaluation, not being fully satisfied of the authors' feedback.
The AC recognizes the good job made by the authors in the rebuttal, but also noted that this was not sufficient to eliminate reviewers' doubts who, unanimously, did not increase their original scores.

For these reasons, this paper is not considered acceptable for publication to ICLR 2025.