Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigates the modality greediness of learning in multi-modal deep neural networks (DNNs). The authors hypothesize that multi-modal DNNs learn to rely on one of input modalities (it could learn faster), which is expressed as a “greedy" behavior. 
The greediness is quantified with the difference between conditional utilization rate between modalities.
They claim the followings: (1) conditional utilization rate can be surrogated with conditional learning speed, (2) the strong regularization in the training process encourages the greediness, and (3) the greedy behavior often lowers generalization performances. 
As a remedy of performance decreasing phenomena, they propose balanced multi-modal learning methods, which control the training speeds of all of modalities to be within certain criterion via giving more steps considering the log-ratio of sum of the magnitudes of gradients of loss functions between of fusion model parameters and of unimodal model parameters.
The paper provides experimental results to validate their claims and the effectiveness of the proposed methods.

main_review: * Strengths
  - The greedy behaviors among modalities are frequently observed in practical learning process of multi-modal DNNs. The research topic of this paper is interesting and worthy of investigation.
  - The paper is easy to follow.

* Weaknesses
  - The main claims related to worse generalization performance from the greedy nature is not so clearly validated with experiments. It is critical issue since it is connected with the reasons of necessity of the proposed methods. Table 1 shows that the experimental results between *vanilla* and *guided* are not statistically significantly different in ModelNet40, NVGesture-scratch and NVGesture-pretrained. Even though those of Colored-and-gray-MNIST dataset seem to demonstrate big differences, I think the dataset is intentionally designed to be biased with color and labels and not compatible with standard multi-modal setting. So, only the results with Colored-and-gray-MNIST dataset are not enough to support the main claims.
   - The other claims on are also not so clear to accept. For Section 5.2, if the values of | d_util | are high, it seems to mean that there are differences of conditional utilization rates, not confirm that the standard multi-modal learning process encourages the model to rely on one modality and ignore the other one. In other words, I think the experimental design and their evidences are weak to validate their claims.


* Minors
  - Section 5.3 in page 8, obverse -> observe

summary_of_the_review: The research topic is interesting and it seems that the main answer points of the research questions are appropriate.
However, main claims in the paper are not so clearly validated with respect to both of theories and experiments.
I think that the authors should reconsider whether the differences on conditional utilization rate and those of conditional learning speed really affect the generalization performances, or they should try to establish other strategies to find evidences to support main claims.
From the reason, I recommend this paper as reject.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper talks about efficient multi-modal training. They hypothesize that multi-modal training is greedy and perform experiments to empirically show the same.
They propose metrics such as conditional utilization rate and conditional learning speed which serves as a proxy for the former to measure the imbalance in utilization between modalities while training.
Based on conditional learning speed, they propose a multi-modal training algorithm that mitigates the imbalance in modality utilization.
Through experiments, the authors empirically show that it is true and their training paradigm performs better than vanilla multi-modal training.
main_review: 1. The paper focuses on a relevant problem in multi-modal learning on how to utilize information from both modalities efficiently.
The strength of the paper is that they follow a structured and scientific way of identifying the problem, proposing a metric to measure the extent of the problem and perform experiments
to validate their hypothesis. They also show through experiments that the proposed training procedure leads to more efficient modality utilization and shows improved accuracy. The empirical results are strong.
2. Weakness: In a MTMM architecture, Theta_0 (red blocks in Fig 1) would also be receiving updates based on the modality m1 as information is being passed through the fusion blocks denoted by parameters (theta_0)'(green and red-green blocks).
In that case, I am not sure if definition 4.1 is a valid way to define the conditional learning speeds as both theta_0 and theta_0' would see the information flow from modality m1. It would have been valid if theta_0 did not se information from m1 at all (like the way conditional utilization rate was defined in 2.1). So, I am not clearly convinced with the statement in Section 4.1 that "If the greedy learner hypothesis is true, the conditional learning speed would serve as a proxy for the conditional utilization rate. dspeed(f;t) would predict dutil(f) at the end of training." Authors have tried to explain how to derive uni-modal outputs from MTMM in section A1 but it isn't clear to me
3. Re-balancing step is not clear as explained in Appendix A2. Lets say we want to balance the information flow from modality m0 in a MTMM architecture in Fig 1.
In that case, does it mean the top branch takes information from just m0 (i.e. the information flow from green and green-red blocks is stopped) to generate prediction y0_hat, while the bottom branch takes information from both m0 and m1 to generate the prediction y1_hat.
summary_of_the_review: This paper is easy to read and addresses the shortcomings of vanilla multi-modal training scenarios. Authors have done a good job of explaining the problem followed by laying out their hypotheses clearly. They have done experiments which enforces the hypotheses empirically. Authors should explain the re-balancing step (in section A2) and how to derive uni-modal predictions from multi-modal DNNs (in section A1) more clearly with consistent notations as used throughout the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper hypothesizes that due to the greedy nature of deep learning, these models tend to rely on just one modality while under-utilizing the other modalities. The authors empirically observe this phenomenon on several dataset with a proposed metric. They propose an algorithm to achieve balanced training, which forcing the model to update only one of the uni-modal branches when the two uni-modal branches are imbalanced. This method is applied to three datasets and gain some progress.
main_review: Strength:
This paper focuses on a very interesting problem in multi-modal learning: the imbalanced training between modalities. Researches about this issue are meaningful and the idea of controlling the training process specific to modality is an intuitively promising way.

Weakness:
1.	It seems the experiments verifies the greedy learner hypothesis, but no proofs or theories are provided to support this hypothesis, either detailed analysis why the training speeds are different.
2.	The re-balancing step method, which is used to make multi-modal learning less greedy, is not well described and analysis in the full text. The author should have a well explanation that why using the average feature of previous samples can stop the training of the uni-modal branch. 
3.	Lack comparation with other methods that solve imbalance problem, including [1]. The discussion in related work section is also inadequate. For example, related work [2-3] are not discussed. 
4.	The author set different imbalance parameter for different dataset, but did not provided the ablation study for this crucial parameter. Does it impact the model performance greatly?
5.	All datasets only contain visual modality. More natural modalities, like audio and visual, should be included for better evaluation.

[1] Weiyao Wang, Du Tran, and Matt Feiszli. What makes train- 1055 ing multi-modal classification networks hard? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12695–12705, 2020.
[2] Thomas Winterbottom, Sarah Xiao, Alistair McLean, and Noura Al Moubayed. On modality bias in the tvqa dataset. BMVC, 2020.
[3] Ya Sun, Sijie Mai, and Haifeng Hu. Learning to balance the learning rates between various modalities via adaptive tracking factor. IEEE Signal Processing Letters, 28:1650– 1654, 2021.




summary_of_the_review: This paper focuses on an interesting phenomenon in the multi-modal learning: the imbalance problem in the multi-modal training. A training algorithm, balanced multi-modal learning, is proposed to solve this problem. According to the provided experiment results, this method addresses the issue of greedy learning. However, the author did not provide solid theory analysis about not only the greedy learning problem but also the proposed method. Related works are also not well discussed and compared.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the multi-modal interaction problem, that multi-modal models tend to rely on just one modality while under-utilizing the other modalities. Since conditional utilization rate cannot be computed efficiently during training, they introduce an efficient proxy based on the pace at which a DNN learns from each modality, which we refer to as conditional learning speed. So they propose a training algorithm, balanced multi-modal learning, and demonstrate that it indeed addresses the issue of greedy learning. The proposed algorithm is found to improve the model’s generalization.
main_review: 1.	The topic of modal balance is interesting and has attracted much attention. The proposed method is direct and easy to follow.
2.	Experiments on several datasets verify the effectiveness.


weakness
1.	How to set the \alpha at the bottom of page five? More parameter analyses are expected.
2.	As introduced in the introduction and abstract, many related works about this topic have been proposed, e.g., “Rubi: Reducing unimodal biases in visual question answering”, “Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies”. What are the improvements to the proposed methods? Why not compare these methods? The reviewer believes that the experiments are very lacking in comparison with the SOTA methods.


summary_of_the_review: This paper focuses on the multi-modal interaction problem, that multi-modal models tend to rely on just one modality while under-utilizing the other modalities. Since conditional utilization rate cannot be computed efficiently during training, they introduce an efficient proxy based on the pace at which a DNN learns from each modality, which we refer to as conditional learning speed. So they propose a training algorithm, balanced multi-modal learning, and demonstrate that it indeed addresses the issue of greedy learning. The proposed algorithm is found to improve the model’s generalization. So I recommend weak accept.
review_title: null

Meta-review: PAPER: This paper presents analysis of cross-modal interactions in multimodal models and propose a method to help balance the multimodal learning process. The cross-modal analysis is based on measures related to conditional utilization rate and the proposed approach is related to conditional learning speed.
DISCUSSION: The reviewers showed support for this line of research, as a way to better understand the learning process for multimodal models. The discussion allowed to identify points that needed to be clarify and concerns about the experimental results. The authors addressed many of these issues in their response. All reviewers took the time to read these responses as well as other reviews. There are many reviewers who are still expressing concerns with the experimental results. 
SUMMARY: This is an important line of research, and the authors should continue this research endeavor. While the paper presents some interesting research hypotheses about multimodal learning, it seems that more experiments are needed to properly address these hypotheses. In its current form, the paper may not yet be ready for publication.
Decision: Reject