Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the contradiction in the existing multimodal robustness literatures. Intuitively, multimodal models are supposed to be more robust than unimodal model as extra redundancy is provided by multimodal data. However, some existing works find that multimodal integration may be more vulnerable to attack, noise or corruption. To understand this contradiction, the authors make a theoretical analysis using information theory. By formally giving the definition of complementary information, the authors establish the theoretical guarantee to quantify the different Bayes error rate of multimodal and miss-/noisy modality cases, and provide a metric to quantify the complementariness of multi-modalities. Experiments on synthetic and real world dataset show the value of measuring complementariness. 

--- post-rebuttal ---\
Thanks for the authors' response. I have read the response and other reviewers' comments. Both reviewer TcW7 and I have a similar concern about the influence of pre-trained model. The authors propose two possible ways of constraining the feature extraction part to be the same. While it is still unclear to me for the case when two to-be-compared multi-modal approaches have different feature extraction modules. Despite this, I still believe there is value of being a pioneering work exploring modality complementariness, but slightly lower my score by accounting the review confidence weight.
strength_and_weaknesses: Strength: 

-The studied problem of how modality complementariness affects multimodal robustness is interesting. This paper provides a new angle to look at the multimodal robustness. 

-The authors a thorough theoretical analysis of the interconnection of modality complementariess and robustness in multimodal setting.  

-The empirical results clearly demonstrate that the proposed metric is able to quantify how complementariess of each multimodal dataset. It provides an opportunity to make a fair comparison under similar complementariness for future studies. 

 

Questions/Weakness: 

-The proposed metric provides an opportunity to make a fairer comparison of multimodal robustness among the existing methods. I’d like to see an analysis of the existing popular multimodal robustness approaches by applying the proposed metric. 

-Although network architecture, training routines are out of this paper’s scope, the proposed complementariness metric is measured based on pretrained models for feature extraction. A more powerful base model usually extracts richer features. I am not clear on how to make a fair comparison if two multimodal models have different backbone feature extractors.  
clarity,_quality,_novelty_and_reproducibility: The paper is mostly well presented. The studied problem of the interconnection between modality complementariness and multimodal robustness is novel and interesting. The calculation of the metric is reproducible.
summary_of_the_review: The studied problem of how modality complementariness affects multimodal robustness is interesting. This paper provides a new angle to look at multimodal robustness. I have two general questions regarding the metric. Please see them in the weakness section.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper empirically studied the connection between the complementariness of modalities and multimodal model robustness. The authors pointed out that the multimodal models are more robust as the multimodal data are less complementary and otherwise. The conclusion is theoretically verified along with well-defined experiments. Moreover, the author proposed a new metric for measuring the complementary of multimodal data that may help improve the robustness of the multimodal model.

-------after rebbuttal
I read the resp from the authors, but only some concerns of mine have been addressed. Hence, I rise my score to 5. In fact,  if there is 4, I will only rise to it. 
strength_and_weaknesses: Strengths:
1.	This paper studied a good question about what affects multimodal robustness. The conclusion is well verified and may help improve model robustness.
2.	The paper is written clearly and is easy to follow.

Weaknesses:
1.	However, my major concern is that the contribution is insufficient. In general, the authors studied the connection between the complementary and the model robustness but without further studies on how to leverage such characteristics to improve model robustness. Even though this paper could be the first work to study this connection, the conclusion could be easily and intuitively obtained, i.e., when multimodal complementary is higher, the robustness is more delicate when one of the modalities is corrupted. Except for the analysis of the connection between complementary and robustness, it is expected to see more insightful findings or possible solutions.
2.	The proposed metric is calculated on the features extracted by some pre-trained models. So the pre-trained models are necessary for metric computing which is contradictory to that the metric is used to measure the multimodal data complementary. In addition, in my opinion, the metric is unreliable since the model participates in the metric calculation and will inevitably affect the calculation results.
3.	There are many factors that will affect the model's robustness. The multimodal data complementary is one of them. However, multimodal data complementary is not solely determined by the data itself. For example, classification on MS-COCO data is obviously less complementary than VQA on MS-COCO data. As mentioned by the author, the VQA task requires both modalities for question answering, accordingly the complementary is determined by the multimodal and the target task. However, I didn't see much further discussion about these possible factors.

clarity,_quality,_novelty_and_reproducibility: The background, the mathematical notations throughout the paper, and the proposed method are introduced clearly. The paper is well-written and organized in general.
summary_of_the_review: This paper studied the connection between the multimodal data complementary and the model robustness. The paper is well-written. However, the contribution of this paper is limited.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The article investigates the robustness of multi-modal learning by introducing the concept of modality complementariness. The analysis is based on information theory, presenting the increasing Bayes error in the condition of missing and noisy modality. Then, a dataset-wise metric is proposed to measure the complementariness of each dataset. Furthermore, a dataset with more complementary information is designed in this paper. More experiments are applied to support this theory with this metric.
strength_and_weaknesses: Strengths:
1. Although multi-modal complementariness has been focused on, this paper provides an intuitional solution (i.e. information theory) to explain the effect of this characteristic. 
2. The notation and proof of this article are clear.

Weakness:
1. This work only focuses on Bayes error on a fixed model and shows how complementariness affects robustness. That is a strong assumption and thus the application scenarios are limited. What if the model takes in noisy or incomplete multi-modal data when training?
2. The experiment setting seems weak and less supportive. Through Figure 3 and Table 1, it is disturbing whether the deterioration is caused by complementariness or other nature of the dataset itself.  
3. Details of these datasets had better be provided.
4. Some special meanings are best represented in the figure. For example, the meaning of $\Tau_Z$ when modality missing is quite confusing. Should it vary when data suffer from contamination?
5. This paper focused on two modalities. Then, how to define the metric in datasets with more than two modalities? 
6. The word Complementarity may be more commonly used to denote complementary properties than Complementariness.

clarity,_quality,_novelty_and_reproducibility: Good, but less novelty. 
summary_of_the_review: This paper examines the concept of Complementariness in multi-modal learning paradigm. Although the perspective is nice, some deeper insights had better to be proposed to enrich the theory.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper gives an information-theoretical abstraction to represent  the modality robustness through the proposed complementary metric. The deduction is based on the assumption that models are anticipated to be more robust due to the possible redundancy between modalities. The authors also do some controlled experiments and try to support their theoretical analysis.

strength_and_weaknesses: Pros:

1. This work presents a very detailed theoretical analysis of multi-modal robustness. The complementary is estimated through a litter modification on MINE with a novel information metric.
2. The organization of the paper is good in general.

Cons:

1. My main concern with this paper is its usability. To analyze the complementariness, the methods need the label information beforehand. However, as we have known the label information, one could directly train the multi-modal network and percept the robustness intuitively. I think the use of label information is an obvious drawback of this work.
2. Some works focus on the reliability of the modality [1], which could also learn the uncertainty (importance) of each modality, which is also a kind of robustness. In contrast, the proposed method could only analyze the robustness of the datasets, ignoring an important perspective, i.e., improving the representation of the network.
3. Many recent multi-view works about information theory are missing [2-5]. All these methods attempt to establish the relationship between the different modalities and the label information in an unsupervised manner, although these methods are based on some extra assumptions, e.g., mutual redundancy [3,5] or containing view-specific information [2]. Especially, [3,4] also discuss the usability of the missing modality from information theory. The paper could do a better job by citing these relevant works and clarifying the differences between this work and them.

[1] Trusted Multi-View Classification, ICLR'21

[2] Self-supervised Learning from a Multi-view Perspective, ICLR'21

[3] Dual Contrastive Prediction for Incomplete Multi-View Representation Learning, TPAMI'22

[4] COMPLETER: Incomplete Multi-view Clustering via Contrastive Prediction, CVPR'21

[5] Learning Robust Representations via Multi-View Information Bottleneck, ICLR'20
clarity,_quality,_novelty_and_reproducibility: Good quality and clarity.
summary_of_the_review: I recommend accepting the paper (rating 6). I liked the abstraction proposed by the authors and particularly liked the way authors make connections between the complementariness with missing modality and noisy modality. Although this paper has some drawbacks for practical application, the contribution is actually enough. Ratings can be improved further if the authors could solve my questions, especially the use of label information.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper received borderline ratings overall, with some initial divergence across reviewers. There was therefore additional discussion between the reviewers and myself after the reviewer-author discussions. In the additional discussions, there was support both for marginal acceptance and rejecting of the paper. However, reviewers had an overall lukewarm assessment of the paper, finding it to provide some value but generally limited in contribution and direct utility. The experimental validation was also considered to be weak, and there were some specific concerns about the approach such as the effect of different pre-training. Multiple reviewers stated that the author rebuttals partially addressed concerns but were not fully satisfactory. The reviewer who originally had the highest score also lowered their score. Considering the paper, and all discussions, I tend to agree that the weaknesses of the paper outweigh the contributions and do not recommend acceptance at this time.
justification_for_why_not_higher_score: The contribution is relatively limited and there are still significant unresolved weaknesses.
Decision: Reject