Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper tries to prove that learning with multiple modalities can achieve a smaller population risk than only using its subset of modalities. 
main_review: I am glad to see the work which aims to answer the question: Can multimodal learning provably performs better than unimodal? Under a specific setting, the authors provide theoretical results.

Although under a simple setting, the theoretical results are good. There is another work [1] that has provided some analysis about why multiple modalities are better. I think it is interesting to discuss the relation between the proposed one and [1]. 

Some minors:
“which points to an interesting direction that are worth further…” should be “which points to an interesting direction that is worth further…”

The authors claim “we provide the first theoretical analysis to shed light on what makes multimodal outperform unimodal…”, “this is the first theoretical treatment to capture important qualitative phenomena observed in real multimodal applications”. Since the CPM-Nets has also provided related theoretical results (from other perspective), it not proper to claim the work as the first one. 

[1] CPM-Nets: Cross Partial Multi-View Networks, NeurIPS 2019

The authors claim that “two common principles, consensus and complement, which are crucial in the theoretical analysis of multi-view learning, are not applicable in multimodal learning”. Since in machine learning, multimodal and Multiview learning are quite similar, so the claim is not reasonable. Furthermore, the description for the difference between multimodal learning and multiview learning is arguable and even misleading. 

Why choose the composite multimodal framework, since there are lots of multimodal learning methods and why the chosen one is widely used?

In the CPM-Net, the versatility also implies the advantages of using multi-view information, and the paper also provides analysis in terms of empirical error.  It will be better if the authors could provide a detailed discussion to clarify the relationship between them and advantages of the proposed one. Providing theoretical results is a good direction but the current results seem a little straightforward. 


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper aims to provide a theoretical explanation on whether multimodal learning can perform better than unimodal learning. They study a popular multimodal learning framework which firstly encodes features from different modalities into a common latent space before mapping the latent representations into a label. They prove that under certain conditions multimodal learning enables better estimation of the latent space which then enables better task performance. They also show some experiment results to back up their theory.
main_review: Strengths:
1. The problem tackled is an important one and an important shortcoming in current work.
2. There are some empirical results supporting their results.

Weaknesses:
1. There are some severe issues with the setup, results, and writing which makes the paper hard to follow and understand:

a. In line 38 you say you analyze 'a widely used composite multimodal framework [35]' but [35] was only proposed in 2020 so I'm not sure how it can be deemed as a 'widely used' framework.

b. Also [35] uses a SVM on each modality's (total M modalities) features to get them into a common latent space from which a single classifier is used from that space to the label, and each modality is used for predicting the label to get M labels, from which you take an ensemble to predict the final label. But in your setup you don't do this, and the example you give in line 124-128 talks about self-attention to combine the features. I'm not sure why there is inconsistency here.

c. In line 115 I'm not sure why noisy modalities are being introduced. If you want to analyze multimodal learning why don't you start by analyzing the case where all modalities are present. I'm also not sure what p'_M means as an extension of p_M - are you adding noisy modalities on top of noisy modalities? I'm not sure what is going on here.

2. There are many multimodal datasets that can be used for research, but this paper only uses 1 dataset and a synthetic dataset. You can refer to https://arxiv.org/abs/2107.07502 for a set of comprehensive benchmarks to test on.

3. There are severe clarity issues with the writing such that I have a hard time understanding the experimental results.

a. The experiments are on concatenating features which seems to be different from your analysis and also different from [35].

b. line 245: 'freeze the encoder gˆM obtained 246 through pretraining and then finetune a better classifier h.' I'm not sure what this means.

c. Many of the figure captions are not informative (e.g. figure 2) which makes it hard to understand.

d. Many details are missing in the synthetic data, including what exactly 'overlap' means and how the labels are generated.

4. It would be good to refer to 'What Makes Training Multi-Modal Classification Networks Hard? https://arxiv.org/abs/1905.12681' as an example of where simple multimodal learning (i.e. concatenating features) fails. I believe the theory proposed here is unable to explain this phenomenon. I'm also not sure if the assumptions made in this paper are too strict or too loose to reconcile these empirical findings.

Overall this paper would benefit from a significant revision. I would suggest carefully reading the multimodal literature (see https://arxiv.org/abs/2107.07502 and https://arxiv.org/abs/1705.09406) for comprehensive models, benchmarks, and surveys. It would also be good to see under what settings do multimodal models fail (like in https://arxiv.org/abs/1905.12681 or https://aclanthology.org/2020.emnlp-main.62.pdf) because just by looking at methods that work will make it hard to come up with an explanation that is consistent with both sides of the story. The writing and clarity of this paper can also be significantly improved. 
limitations_and_societal_impact: I think the authors can say a bit more about the societal impact of multimodal models, including their applications in robotics, healthcare, education etc.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work introduces a generalization bound for multimodal learning, which shows that, for a sufficiently large sample size, the model risk decreases with an increasing number of considered modalities.
The authors also discuss a novel principle to dictate whether new modalities can be beneficial as a function of the dataset size and considered function classes.
An empirical evaluation on both toy and real-world data validates the theory, underlining the benefits of learning from different modalities and laying the foundations for future research.
main_review: The paper introduces a formal definition of the multimodal learning problem, clearly stating the initial assumptions and introducing the concept of "latent representation quality" to show under which condition learning can benefit from multiple modalities.
The paper is clear and well-written, and all the main results are clearly stated and proved.
To the best of the reviewer's knowledge, the analysis reported in this paper is novel and relevant to the field of multimodal learning.

Main comments:

1) Can the authors further comment on the role of the intrinsic properties mentioned in the principle in lines 189-190 with respect to the modality selection procedure? What role does the change in the size of the possible representation space $\mathcal{G}_{\mathcal{N}}$ play?

2) The data generation process described in lines 277-281 is unclear. How is the 1-dimensional label defined? What procedure used to correlate the modalities? Further details would improve clarity and reproducibility.

3) Reducing the number of available modalities can be considered as a restriction of the size of the hypothesis class. How does the proposed analysis relate to classic PAC bounds in learning theory [1]?

Other Comments and general questions:

4) According to the definition, for a sufficiently large $\mathcal{H}$ the representation quality seems to be closely related to the amount of predictive information loss: as long as the representation mapping $g$ does not discard predictive information, we can find a task mapping $h$ so that $r(h \circ g) = r(h^* \circ g^*)$. In this sense, the difference of representation quality for $\mathcal{N}\subset\mathcal{M}$ could be interpreted as the amount of task-relevant information loss caused by removing informative modalities: $I({\bf y};p_{\mathcal{M}}({\bf x})) \ge I({\bf y};p_{\mathcal{N}}({\bf x}))$ for $\mathcal{N}\subseteq\mathcal{M}$. On the other hand, the addition of new modalities increases the rademacher complexity and the total amount of information $I({\bf x};p_{\mathcal{M}}({\bf x})) \ge I({\bf x};p_{\mathcal{N}}({\bf x}))$.
Can the authors underline any connection with the Information bottleneck principle? Can the addition of redundant modalities $I({\bf y};p_{\mathcal{M}}({\bf x})) = I({\bf y};p_{\mathcal{N}}({\bf x}))$ ever be favorable according to the proposed view-selection principle.

5) Line 159-160: The expression for the Rademacher complexity of $\mathcal{F}$ should contain $f(Z_i)$ instead of $g(Z_i)$.

References:
[1] Shalev-Shwartz, Shai, and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.

limitations_and_societal_impact: The paper has a strong theoretical focus and does not mention any limitation or societal impact of the proposed analysis.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper focuses on proving multimodal learning performs better than unimodal under the most popular multimodal learning framework. Specifically, they proved that in the linear case, learning with multiple modalities achieves a smaller population risk than only using its subset.
main_review: *Originality*: As far as I know, this is the first theoretical paper to try to prove multimodal learning performs better than unimodal. This is an important direction and lack of analysis before.

*Quality*: Authors conduct sufficient theoretical analysis and proof. I don’t find any technical flaws. Meanwhile, the author conducted experiments to validate the conjectures and theoretical results. 

*Clarity*: The paper is easy to follow. I only find one typo, e.g., “When the number of *simple* size” on line 190.

*Significance*: This paper could provide new insight for multimodal theoretical research.

*Comments*:  Although this paper theoretically proves that in the linear case multimodal learning better than single, many potential risks may reduce the performance of the multimodal model in practice. Therefore, I think the paper should discuss why the use of multimodal data in practice will reduce the performance of the model in some cases. For example, [1] gives an analysis from the perspective of different model convergence speed. [2][3] explores multi-modal learning from the perspective of uncertainty. [4] removes the bias in multimodal classifiers to avoid some modalities contribute more to the classification results than others.

[1] What makes training multi-modal classification networks hard? In CVPR 2020

[2] Uncertainty-Aware Audiovisual Activity Recognition Using Deep Bayesian Variational Inference? In ICCV 2019

[3] Trusted multi-view classification. In ICLR 2021

[4] Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies. In NeurIPS 2020

limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative social impact of their work.
review_title: null

Meta-review: The paper presents a formal theoretical analysis of the utility of observing multimodal information over unimodal information from the perspective of empirical risk minimisation. The analysis is followed up with some empirical evidence seeking to confirm theoretical findings.

The reviewers all agreed that the theoretical contributions were both sound and principled.

There were a few concerns about related work (eg. CPM-Nets) and making things clear and precise that the authors should address.  For example the characterisation of [35] as 'widely-used' is odd to say the least.
  
The main concern with this paper is with the experiments---specifically the design and the observed results.

It appears to be the case, as the authors themselves point out (l265-267), that the complexity of the function class has a pretty strong effect on the _significance_ of the results (cf. Table 3). The 'bold' values are only significant when the ratio of sample size is 1.0; in all other cases, they're within the reported standard deviations, and cannot be claimed to particularly significant.

Moreover, as reviewer VZFZ points out with relevant citations, in practice, multimodal learning tends to perform worse. In the discussion, this is somewhat hand-waved away as having to do with optimisation, but it would seem to be a pretty serious issue affecting the theory if it is something typically not observed in practice!
A proper discussion of what the cause for this apparent contradiction is and how it might relate to the complexity of the function class/sample size ratio in such cases would strengthen the paper.

I would also urge the authors to incorporate the clarifications and edits in discussion with all the reviewers, especially reviewer eSGN into the updated manuscript.

If there such a thing as a conditional accept, this paper would be a prime candidate as it does have relevant contributions, but also leaves a few things to be desired before it can be classed a complete piece of publishable work.
It would probably be reasonable to accept the paper on the merits of its theoretical contributions, but *strongly* encourage the authors to address the question of applicability, relation to contradictory results in practice, how this relates to the function-class complexity vs empirical observations trade-off they discuss (esp Sec 5.1).

Decision: Accept (Poster)