Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper deals with multi-modal joint training methods on deep neural networks.
Related to this topic, several works have reported that the best uni-modal networks outperform the multi-modal networks even though the multi-modal networks receive more information.
This work is focused on this topic, and the main contributions of this paper are two-fold: (1) to propose an improved methodology with experimental supports on audio-video data for sound classification (VGG-Sound), audio-video data for action recognition (UCF-101, Kinetics-400), front-rear view of object data (ModelNet40), (2) to investigate theoretical results on characteristics among uni-modal features (learned from uni-modal training), and paired features (only learned from cross-modal interaction).

strength_and_weaknesses: *Strength
- To identify interesting conceptualization on uni-modal features and paired features
- To provide experimental evidences (mostly shown in Appendices) to justify their arguments and choices on this work
- To explain theoretical aspects to characterize modality laziness

*Weaknesses
- It seems that some terms in the paper are NOT so clear: what is the exact definition of modality laziness? Why should we utilize new term? What are the discriminating points for it? 
- This paper is hard to follow due to lack of explanation for core parts such as model architectures of uni-modal teacher, loss functions with λ_task, λ_distill, λ_distill in Algorithm 1 or the citation of reference works, which also needs for reproducibility.
- The organization of this manuscript makes readers confusing. It seems that theoretical part (Section 3.4) and the others are totally separated. Uni-modal features and paired features also are redefined in Section 3.4. On the other hand, there is no experimental result related to Section 3.4 in main body of the manuscript (but in Appendices). 
- Even though related work Section is not bad, it would be better to compare one of the recent works [1], which deeply related to this work.

[1] Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably), ICML 2022 
clarity,_quality,_novelty_and_reproducibility: If all of details has no problems, and so all of claims are validated, I think that the idea is novel and the contributions are explicit, and this work is worth publishing only considering the achievements and results. 

However, some parts of details are missing as described Weaknesses above.
I think that it needs to re-organize and re-write the manuscript with respect to clarity and reproducibility.

summary_of_the_review: The main contribution, a method of combination of Uni-modal Ensemble and Uni-modal Teacher, seems practically utilizable and showing better performance for multi-modal joint training on deep neural networks. 
Also, I think it is interesting idea to leverage distillation for generalized setting of multi-modal joint training. 
However, the writing is confusing and not enough clear to be reproducible.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the insufficiency phenomenon of uni-modal feature learning in supervised multi-modal learning problem. The authors identify that the recent late-fusion method suffers from insufficient learning issue, called as Modality Laziness. To understand this phenomenon, the authors conduct a series of experiments and make a theoretical analysis. The authors propose two late fusion learning methods, Uni-Model Ensemble and Uni-Model Teacher, which work better in different cases according to the distribution of uni-modal and paired features. A simple guiding strategy is also provided. Experiments show the advantages of the proposed method in several multimodal datasets. 

--- post-rebuttal ---\
Thanks for the authors' response. After reading the response and other reviewers' comments, my concerns are addressed. I keep my score.
strength_and_weaknesses: Strength: 

-The paper discovers an interesting phenomenon of Modality Laziness, and provides an in-depth understanding in both empirical and theoretical aspects. 

-The paper is well motivated. The analytical experiment in Table 1 really shows the severeness of existing supervised multimodal learning. 

-I appreciate the efforts that the authors try to explain the key concepts in an intuitive way. For example, the figure 3 is helpful. 

-The results compared with several existing methods are competitive. The proposed solution is simple but effective. 

 

Weakness 

-There is one method called “bimodal deep autoencoder” [Ngiam et al.], which can also be applied for joint feature extraction, then used for supervised learning. The discussion along this line is missing. Since  the motivation of this work is address Modality Laziness, the auto-encoder used in [Ngiam et al.] also has the capability to mitigate the insufficient learning. A comparison with this work is strongly recommended. 

[Ngiam et al,] Jiquan Ngiam et al., Multimodal Deep Learning, ICML 2011 

-The term of “naive multimodal training” appears multiple times across the paper, and its suffers from Modality Laziness. While the formal definition of this term is unclear to me. 
clarity,_quality,_novelty_and_reproducibility: The paper is well-motivated and mostly well presented. It addresses an interesting phenomenon in supervised multimodal learning and provides a theoretical analysis. The proposed solution is simple, and easy to reproduce.
summary_of_the_review: Overall, I think this work is well executed, with a strong motivation and good theoretical and empirical analysis. In terms of existing work, I'd like to see how this work compares with multimodal deep learning [Ngiam et al.].
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Summary:
This paper proposes to care about the learning of unimodal features in the setting of multimodal fusion. They find out that one modality may strongly dominate the multimodal learning than other modalities on some tasks, like video classification and action recognition. This observation is consistent of previous related works, like greedy nature of multimodal DNN (Wu et al, 2022), under-optimized unimodal representations caused by strong dominated modality (Peng et al, 2022). They argue that previous works are complex to implement and are empirically inferior on some evaluation datasets (UCF101 and ModelNet40). Instead, this paper introduces Uni-Modal Teacher (UMT) to distill pretrained
unimodal features to help learn corresponding unimodal features counterparts in the multimodal model. Furthermore, a trick is used to decide when to use the joint multimodal training (Uni-Modal Teacher, UMT) or just use the simple averaging predictions of unimodal models alone (Uni-Modal Ensemble, UME).

strength_and_weaknesses: Strong & Weak:
S1: The writing is easy to follow. A reproducibility code is given. Better experimental results are achieved than the complex late-fusion models. 
S2: The architecture of Uni-Modal Teacher is straightforward and is empirically better than classic distillation methods including soft-label.

W1: Although this paper cares about the weakness of multimodal learning and focuses on the importance of unimodal feature learning, assuming that unimodal priors are important and meaningful like previous works (Wu et al, 2022; Peng et al, 2022), it may be better to go further to design a unified, principal framework which can also subsume the settings where unimodal priors are not essential, for example on the visual question answer task. In other words, such a unified framework is agnostic to the unimodal priors and works well for both essential priors (including video classification and action recognition tasks) and meaningless priors (including visual question answer task). Perfectly, one does not need to choose which style of multimodal learning should be adopted to solve their own interest tasks at hand.

clarity,_quality,_novelty_and_reproducibility: See Strong&Weak
summary_of_the_review: See Strong&Weak
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tackled the modality laziness in current multi-modal fusion methods and suggested leveraging the uni-modal feature in two ways. Depending on the importance of paired features (maybe multi-modal information), simple averaging uni-modal features or knowledge distillation from strong uni-modal models. Also, the generalization degradation is described in theoretical perspective. 
strength_and_weaknesses: S1) a good motivation for the features yielded from multi-modal fusion modules and the motivation is shown experimentally.
S2) Mathematical description for the generalization degradation

W1) the detail of the uni-modal model is missed in UMT. If the uni-modal model is such a large network, UMT has benefits in terms of generalization. 
W2) Fewer baselines. Are there no works or techniques to preserve uni-modal information?
W3) The author insists that the existing fusion modules delete the important uni-modal information somewhat. In UMT, the knowledge distillation is performed before the fusion module. Following the author's insistence, the distilled knowledge can also be corrupted after fusion. What is the reason for the design choice?
W4) To select UMT or UME, the testing performance is required. It makes the proposed methods less mature. Is there any automatic rule for this? 

clarity,_quality,_novelty_and_reproducibility: The writing is fine, and the description is clear.
summary_of_the_review: Please handle the weakness in the author's response
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary

The paper investigates a problem in supervised deep neural network training, where sometimes a joint multi-modal (e.g. audio-visual) model fails to outperform well-trained uni-modal baselines followed by late fusion, despite having access to more information from cross-modal interactions. The paper makes two contributions: (1) authors provide a theoretical analysis of characteristics among features derived from uni-modal features inputs, and "paired" features, which can only be learned from cross-modal interaction. Authors find "modality laziness" when one modality dominates and prevents multimodal learning. This leads authors to propose (2) an improved methodology consisting of either a Uni-Modal Teacher (UMT) to distill from or just using simple averaging predictions of unimodal models (Uni-Modal Ensemble, UME), together with decision heuristics. Authors provide experimental validation on sound classification (VGG-Sound), audio-video data for action recognition (UCF-101, Kinetics-400), and front-rear view of object data (ModelNet40), outperforming baseline methods. 

Strengths

- Multimodal learning is an important area, and the authors make a theoretical and empirical contribution to advance the field
- Authors provide detailed experimental results and ablations to support their main arguments
- The proposed method is quite straightforward and yet results are competitive
- Experiments should be largely reproducible and the paper's approach is principled and motivated.

Weaknesses

- The presentation of the work can be improved in places, as reviewers point out. For example, some terms are not clearly defined, tables don't have sufficient captions (mAP? top-1 accuracy?), and some information for reproducibility is missing. While authors have made updates to the paper, it still feels like they could sometimes have made more thorough edits in order to better capture the spirit of the reviewers' comments.
- Related work is missing (as pointed out by reviewers), and should be added to the discussion and experimental comparison in more depth, not just cited (e.g. [Ngiam et al]); even after the update, information is split across multiple sections and appendix. I would for example find it more intuitive to move results on vgg (the main dataset used in this work) out of the appendix section on "different datasets" into the main text
- While the proposed method is simple and effective, a unified framework, agnostic to the unimodal priors and working well for both essential priors (including video classification and action recognition tasks) and meaningless priors (including visual question answer task), would arguably be a more significant contribution with more novelty and more impact. The proposed approach ultimately relies on heuristics.

justification_for_why_not_higher_score: The paper's readability needs to be improved, even after the updates. The authors introduce many terms that are unclear and not well defined, e.g.:
  - "modality laziness" could for example be formally defined as linear-probing-accuracy(one-arm-of-multimodal-encoder) < linear-probing-accuracy(corresponding-unimodal-encoder), rather than simply saying "insufficient representation"
  - similarly "naive multimodal training" -- what type of model do the authors mean here: early fusion? late fusion? other?
  - it is not immediately clear that "features" are learned representations in the authors' parlance, rather than inputs, as a reader could intuitively assume. it becomes clear eventually that these 'features' are learned, but this is an unnecessary barrier on reading comprehension
  - it would be helpful to provide explanations provided to the reviewers (e.g. about the practicality of the method) also to the readers of the paper, and formulate them more clearly. e.g. in a discussion about the "practicality" of the method, focus on the number of hyper parameters that are needed and how difficult it may be to estimate them reliably, no need to repeat that the proposed method outperforms others, but rather explain why that may be the case
Additional experimental comparisons as suggested by the reviewers could significantly strengthen the paper and lift it over the bar. Also, the core idea of mono-modal distillation to improve multi-modal learning has been published before, in Multimodal Learning with Incomplete Modalities by Knowledge Distillation; Qi Wang, Liang Zhan, Paul Thompson, Jiayu Zhou; KDD 2020. The use of this approach to improve performance even when modalities are not missing remains novel, but this work should be cited.

justification_for_why_not_lower_score: n/a
Decision: Reject