Recognizing and overcoming the greedy nature of learning in multi-modal deep neural networks
Authors: Nan Wu
Stanislaw Kamil Jastrzebski
Kyunghyun Cho
Krzysztof J. Geras
Venue: ICLR 2022 Submitted (2022)
Keywords: multi-modal learning
deep neural networks
multi-view learning

Abstract: We hypothesize that due to the greedy nature of learning in multi-modal deep neural networks (DNNs), these models tend to rely on just one modality while under-utilizing the other modalities. We observe empirically that such behavior hurts its overall generalization. We validate our hypothesis by estimating the gain on the accuracy when the model has access to an additional modality. We refer to this gain as the conditional utilization rate of the modality. In the experiments, we consistently observe an imbalance in conditional utilization rate between modalities, across multiple tasks and architectures. Since conditional utilization rate cannot be computed efficiently during training, we introduce an efficient proxy based on the pace at which a DNN learns from each modality, which we refer to as conditional learning speed. We thus propose a training algorithm, balanced multi-modal learning, and demonstrate that it indeed addresses the issue of greedy learning. The proposed algorithm is found to improve the model’s generalization on three datasets: Colored MNIST (Kim et al., 2019), Princeton ModelNet40 (Wu et al., 2015), and NVIDIA Dynamic Hand Gesture Dataset (Molchanov et al., 2016).

Under review as a conference paper at ICLR 2022
RECOGNIZING AND OVERCOMING THE GREEDY NA- TURE OF LEARNING IN MULTI-MODAL DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We hypothesize that due to the greedy nature of learning in multi-modal deep neural networks, these models tend rely on just one modality while under-utilizing the other modalities. We observe empirically that such behavior hurts their overall generalization. To estimate the model's modality-wise dependency, we compute the gain on the accuracy when the model has access to an additional modality and refer this gain as the conditional utilization rate of the added modality. the experiments, we consistently observe an imbalance in conditional utilization rates between modalities, across multiple tasks and architectures. Since conditional utilization rate cannot be computed efficiently during training, introduce an proxy for based on the pace at which the model learns from each modality, which refer to conditional learning speed. We propose algorithm to balance the conditional learning speed between modalities during training and demonstrate that it indeed addresses the issue of greedy learning. The proposed algorithm improves the model's generalization on three datasets: Colored MNIST (Kim et al.) (019), Princeton ModelNet40 (Wu et al. 2015), and NVIDIA Dynamic Hand Gesture Dataset (Molchanov et al. 2016).
INTRODUCTION
In real-world problems, each instance frequently has multiple modalities associated with it. For example, we detect cancer in both X-ray and ultrasound images. We seek clues from images to answer questions given in text. We are naturally interested in training deep neural networks (DNNs) end-to-end to learn from all available input modalities. We refer to such a training regime as multi-modal learning process and DNNs resulting from this process as multi-modal DNNs.
Several recent studies have reported unsatisfactory performance of multi-modal DNNs various tasks (Wang et al. 22200a] Wu et al. 2020 Gat et al.) 2020; Cadene al. 2019; Agrawal et al. 2016 Hessel & Lee 2020: Han et al.] 2021). For example, in Visual Question Answering, muuti--odal an image) and exploit statistical regularities shared between the textual modality (presented as a question) and the answer alone, resulting poor generalization (Cadene et al., 2019 Gat et al. [2000] Agrawal et al.) (016).. Similarly, in Human Action Recognition, multi-modal DNNs trained on images and audio were observed to under-perform uni-modal DNNs trained on images only (Wang al. 2020a).
DNNs were found ignore the visual modality (presented
These earlier negative findings compel us to ask, what prevents multi-modal DNNs from achieving better performance? In order answer this question. we first diagnose these DNNs as lacking utilization of all modalities by analyzing their conditional utilization rates. For a multi-modal DNN trained with two modalities, me and mi, the conditional utilization rate of m, given mg. denoted by ra (mm mo)) measures how important it is to use m., given the presence of mg- It is computed as the relative difference in accuracy between two derived models from the DNN, one using both modalities and the other using only one modality. In several multi-modal learning tasks, we consistently observe a significant imbalance in conditional utilization rates between modalities. For example, we observe 0.63 for DNN trained identify gestures in videos
(RGB) depth) 0.01 and u (depth/RGB) using the NVIDIA Dynamic Hand Gesture Dataset (NVGesture) (Molchanov et al. (2016). indicates that the DNN relies the depth modality make predictions and does not pay attention to the RGB

---

Under review as a conference paper at ICLR 2022
modality. These observations lead to conjecture that the multi-modal learning process often results in models that under-utilize some of the input modalities.
As the focal point of this work, we put forward the greedy learner hypothesis. The greedy learner hypothesis states that a multi-modal DNN learns to rely on one of the input modalities, based on which it could learn faster, and does not continue learn to use the other modalities. This greediness leads to an imbalance in conditional utilization rates between modalities. In other words, it prevents multi-modal DNN from learning to exploit all available modalities and often results in worse generalization. It explains the challenge in training multi-modal DNNs and motivates us to design a better modal learning algorithm.
According to the greedy learner hypothesis, it is the diverged speed at which a multi-modal DNN learns from different modalities that leads to an imbalance in conditional utilization rate. we intervene in the training process to adjust these speeds, we may be able to prevent the hurtful imbalance across input modalities. We analyze the learning dynamics of model components and propose metric named by conditional learning speed using the gradient norm and weight norm of models' parameters. measures the relative learning speed at which the model learns from one modality against the other modality. We empirically show that it is reasonable proxy for conditional utilization rate. We introduce a training algorithm, balanced multi-modal learning which guides the model to learn intentionally from one of the modalities according to the conditional learning speeds. We show that models trained with this algorithm learn to use all modalities appropriately and achieve stronger generalization on three multi-modal datasets: Colored MNIST dataset (Kim al. (019). ModelNet40 dataset of 3D objects (Su et al. (2015) and NVGesture Dataset (Molchanov et al.
2015
2 PROBLEM SETUP
We consider two input modalities, referred to as mo and mi without loss of generality towards tasks with more than two modalities. We denote multi-modal dataset by D {(x', } where (img- Tmy)) ). We partition the dataset D into training. validation and test sets, denoted by Darain Dval, and Diest, respectively. The goal is then to use this data set to train a multi-mode DNN such that it accurately predicts y from x.
Figure I: The multi-modal DNN with in- termediate fusion that we study this work. Blue and red blocks represent the two uni-modal networks learning from modalities mg and mi. The fusion lay- ers are marked with green, green-blue and green-red. The predictions from the two uni-modal branches are denoted by Jn and ý1. We denote the average of the two by ý which is the model's prediction.
So
Multi-modal DNNS We use multi-modal DNN that has two uni-modal branches, each taking one modality as input. The two uni-modal branches are interconnected by layer-wise fusion modules. These fusion modules enable information flow from one branch to another. According to categoriza- tion of modli fusion strategies in the deep learning literature, this is intermediate fusion (Ngiam) jet al. 2011] Atrey et al. 20100 Baltrušaitis et al., (2188). has demonstrated competitive performance against multi-modal DNNs with late fusion in many tasks (Perez et al. 2018 Joze et al. 2020
Anderson et al., 2018 Wang et al.) 2020b).
We implement every fusion module by a multi-modas transfer module (MMTM) (Joze et al. 2020). Each MMTM connects two layers from the two uni-modal branches. There is first the global average pooling applied over spatial dimensions to transform feature maps into vector. We concatenate the two vectors and apply linear transformation. We refer to its output as context representation. Next, for each uni- mni-modal branch, we implement a fully connected layer on the context representation and get a vector with a dimension of the number of feature maps. Feature maps are re-scaled by this vector before passing to the next layer of the uni-modal branch.
We train a multi-modal DNN f on Dirain Let ý be the prediction of f for an input x:
ÿ f( Fmyy)
2

---

Under review as a conference paper at ICLR 2022
As shown in Figure | y 2 (0u + f/),, where do and are the outputs of the two uni-modal branches. During training, the parameters of the modtl DNN are updated by stochastic gradient descent (SGD) to minimize the loss: refer to each of the cross-entropy losses as a modality-speciff loss. We train the model until / for all samples in Dirsin and take the checkpoint of it when reaches the highest accuracy on Dval
CE Bo) + CE(y, 3n), where CE stands for cross-entropy. We
During training, each uni-modal branch largely focuses on its associate input modality. The fusion modules generate context representation using all modalities and feed such information to the uni- modal branches. Both to and őI depend on information from both modalities. We end up with two functions, fo and f1, corresponding the two uni-modal branches:
yo Jo (a Bmy)
t/ Si [3 Bimy)
To evaluate the multi- modal DNN's ' ability in using single modality, we disable information sharing between branches ad derive Sú and fí, each taking only one of the input modalities as input. We describe how we derive them in detail in Appendix A.1 We Vo and v' to refer to their outputs:
% aa
In summary, we derive four models from f and we define the ability of each model as its accuracy on Drest We group the four accuracies into two pairs: A(f)), A(f)), and (A(fi), A(H)) and
define the conditional utilization rates as bellow:
Definition 2.1 (Conditional utilization rate) For multi-modal DNN, f. taking two modalities mo and my as inputs, its conditional utilization rates for my and ma are defined as u(milma) = A(f))-A(f) and u(mo m.) ((1)-A(J1) respectively. A(fa)
A(f)
The conditional utilization rate is the relative change accuracy between the two models within each pair. The conditional utilization rate of mo. u(ma m1). measures how important it is for f' to use mo together with mj. other words, it estimates the marginal contribution that mo has in increasing the accuracy of ý1 depending m..
A multi-modal learning process Given DNN's architecture and dataset, every time we run multi-modal learning process, we perform the following steps: (I) sample a value of learning rate randomly from given range, (2) initialize the DNN's parameters randomly, (3) update them to fit Dtrain with SGD, and (4) select the model checkpoint with the highest accuracy on Dval
3 THE GREEDY LEARNER HYPOTHESIS
In this section, we introduce the greedy learner hypothesis to explain challenges observed in training multi-modal DNNs. Before describing our hypothesis, discuss some assumptions the multi- modal data and preliminary observations made in the multi-modal learning literature.
3.1 ASSUMPTIONS MULTI-MODAL LEARNING
First, in multi-modal learning literature B Blum Mitchell 1998 Sridharan & Kakade 2008). both modalitics are assumed to be predictive of the target, i.c., I(Y X mg) 0 and ( X > 0. where denotes the mutual information. In order to minimize one of the modality-speciie losses, e.g. CE(y, šo), one either update the parameters of the fusion layers that pass information from m. to Do- or the parameters of the uni-moda branch taking my as input, or both.
Second, modalities observed that when training DNNs on each modality separately, they usually do not reach the same performance (Wang et al. 2020b Joze et al. 2020). In addition, multi-modal DNNs exhibit varying performance when being trained on different subsets of modalities present for the task (Weng et al.
predictive of the target at different degrees. For example, it has been
2021 Pérez-Rúa et al. 2019 Liu al. 2018).
Third, modalities are learned by the model at different speeds. This has been observed in both uni-modal DNNs and Multi-modal DNNs (Wang al. 2020a Wu al. 2020).
3

---

Under review as a conference paper at ICLR 2022
3.2 MULTI-MODAL LEARNING PROCESS IS GREEDY
Before formalizing our greedy learner hypothesis that explains the behaviour of multi-modal DNNs, we first demonstrate that we can tell whether the model utilizes all modalities by analyzing its conditional utilization rates.
Given a multi-modal DNN, f. with two imput modalities, my and my. we compute the difference between conditional utilization rates, u(m; (m)) and u(
duaa (/) = (m)) mm)))m1))
If we observe high (f)), we say that f exhibits imbalance in utilization between modalities.
The difference between conditional utilization rates bounded between and When it is close to 1 or 1, the model benefits only from one of the modalities given the other but not vice versa. This implies that the model's ability to predict do and ú1 comes only from one of the modalities.
We now introduce the greedy learner hypothesis:
We call a multi-modal learning process greedy, when it trains models rely on only one of the available modalities. A greedy multi-modal learning process cannot avoid producing models that exhibit a high degree of imbalance in utilization between modalities. We hypothesize that multi-modal learning process, in which a multi-modal DNN is trained to minimize the sum of modality-specific losses, greedy.
Let [[dut]] denote the expectation of dutil over the empirical distribution of models resulting from multi-modal learning process (as defined in $2 The absolute value of E[duu]] is associated with the greediness of the process. The higher the E duuall is, the greedier the learning process is.
We propose the following conjectures consistent with the greedy learner hypothesis:
1. There exists properties from $3.1 Otherwise, duui distributed symmetrically around zero and E|dutn]
0, s.t. [B[duan]] > e, long the multi-modal dataset has the last two
2. The stronger the regularization we apply to the DNNs parameters in training, the higher the [E[[mma]]] is.
We expect the first scenario in the first conjecture to be true for real-world multi-modal tasks. To test the second scenario in the first conjecture, we construct datasets of two identical modalities.
We suspect that the implicit regularization of DNNs is factor impacting the greediness of learning in multi-modal DNNs. The implicit regularization of DNNs commonly refers to consensus that DNNs have the tendency to lean towards simplicity during training and it explains their strong generalization (Valle- Perle- al. 2019 Zhang al. 2017 2021 Smith et al. 202I). Since it is difficult to precisely measure and control such implicit regulanization, alternatively, y, we construct the second conjecture using explicit regularization applied to the DNNs' parameters.
In order to validate the above hypothesis and conjectures empirically, we conduct experiments several multi- modal datasets using different network architectures (cf. $5.1p. We present observations in $5.2 and $5.3 We show that the multi-modas learning process cannot avoid producing models that exhibit high degree of imbalance in utilization between modalities. Further, in the next section, we propose metric for measuring the speeds at which the model learns from different modalities and we show the imbalance in such speeds is correlated with the imbalance in utilization.
MAKING MULTI-MODAL LEARNING LESS GREEDY
We aim to make multi-mode learning less greedy by controlling the speed at which a multi-modal DNN learns rely each modality. To this end, define conditional learning speed measure the speed at which the DNN learns from one modality. It serves as an efficient proxy to the conditional utilization rate of the corresponding modality, shown empirically in $5.2 and $5.3 We then propose the balanced multi-modal learning algorithm, which controls the difference in conditional learning speed between modalities that the model exhibits during training.
4

---

Under review as a conference paper at ICLR 2022
4.1 CONDITIONAL LEARNING SPEED
As demonstrated in 3.2 the imbalance in conditional utilization rates is sign of the model exploiting the connection between the target and only one of the input modalities, ignoring cross- modal information. However, conditional utilization rates are measured after training is done, making expensive to use them in real-time during training. We instead derive proxy metric, called conditional learning speed, that captures relative learning speed between modalities during training.
Let us revisit the architecture of the multi-modal DNN. We denote the parameters each uni-modal branch taking mg and my inputs, by 0g and 0I. Layers of the fusion modules marked with green and green/blue in Figure 1 are part of the function mapping æ to ÿo- Let 0 refer to their parameters. Analogously, the layers marked with green and green/red in Figure are part of the function mapping æ to U1. We denote their parameters by 0j. In this way, we divide the model's parameters into two pairs: (Ou, 0b) and (02, 01)
For parameters 0, let ((-11) and (3) denote its value before and after the gradient descent step i, and aL 0..- We define the model's conditional learning speed as:
G
Definition 4.1 (Conditional learning speed) Given malti-modal DNN. f. with two input modali- lies, mg and my, the conditional learning speeds of these modalities after I. training steps, are
Σ 1= 11 (():))
Σ 1 ((((;)))
8 sm1|mm; t) log
s(mglmm: =log
Ei= (();;))
Σ 11;1)
where u(0; 1 quantifies the change of f that comes from updating O at the ith step, which can be also interpreted as the effective update on O.
This definition of i) is inspired by discussion on the effective update of parameters in the literature (Van Laarhoven, 2017 Zhang et al., 2019 Brock et al.! 2021 Hoffer al., 2018). When normalization techniques, such as batch normalization (Ioffe & Szegedy 2015). are applied to the DNNs, the key property of the weight vector, is its direction, i.e., 0/| Thus, we measure the update on θ using the norm of its gradient normalized by its norm.
The conditional learning speed, s (m, Img t) (and analogously s(mg) m= t)), is the log-ratio between the learning speed of 0% and that of 0g. Because 6/ carries information from to go and information from my to jo is carried by on. s(m |ma; t) reflects how fast the model learns from my relative to mo, after the first & steps.
We further compute the difference between s(m=|mo; t) and (m [0] t) as:
dspeed (f:t)= = s(m) |mo; t) - smmolmm1
analogous to (f). For each model, we report dapeed as clspeed (f) where the model takes T' steps until reaching the highest accuracy on Dvai, We anticipate that the conditional learning speed would serve as a proxy for the conditional utilization rate and we say dispeed would
predict duun (ff) at the end of training. In 5.2 and 5.3 side-by-side validate this.
we show the distributions of depeed and dutil
4.2 BALANCED MULTI-MODAL LEARNING
Because depord (J,t) is predictive of the imbalanced utilization between modalities, we can take advantage of depora (f, () to balance conditional utilization on-the-fly. addition to training the network normally with both modalities, we accelerate the model to learn from either modality alternately to balance the conditional learning speeds of them. See Algorithm [ for an overall description of the algorithm.
We refer to the training steps at which we perform forward and backward passes normally as regular steps. We introduce re- re-balancing steps which we update of the uni-modal branches intentionally in order to accelerate the model to learn from the corresponding modality. See Appendix A.2 for the full explanation of the re-balancing step.
Some parameters both in 0 and 0/.
5

---

Under review as a conference paper at ICLR 2022
To up the model, we perform only regular steps in the first training epoch. Then we switch from regular steps to re-balancing steps if dapeed (t) > a, where a is a hyperparameter, referred to as the imbalance tolerance parameter. The training takes Q re- o--balancing steps before returning to regular mode. We refer to the hyperparameter Q as the re-balancing window size.
Algorithm 1: Balanced multi-modal learning
Input: Q. re-balancing window size;
for 4- Ti T do if regular_ -ooe then
imbalance tolerance parameter
T, # of training steps;
Take a regular step; Update Me,, Mog Me. Mo;
Tı. # of steps in the 1th training
epoch
Compute dspeed;
Let M denote the accumulated
dapeeal > a then
effective update (1)) in regular steps
1 = 0; regular mode = False end
Ma. Mo; = 0,0g 0;
Me, Me; = 0, 0;
else
for ( <- 1 to 7i do
- 9 q = Q then regular_ eeglar_mode True; Take a re- re-balancing step to accelerate learning from mi if dlppedd > 0 else mi;
Take regular step:
Update Me.. Mo; Me. Me;
end
regular mode = True
end
end
EXPERIMENTS AND RESULTS
5.1 DATASETS, TASKS AND BASELINES
-aaa--aann--gg------- Kim et al. 2019) is synthetic dataset based on MNIST (LeCun et al. 1998). In the training set of 60,000 examples, each example has two images, a gray-scale image and monochromatic image, with color strongly correlated with its digit label. For the validation set of 10,000 examples, each example also has a seale image and a corresponding monochromatic image however with a low correlation between the color and its label. We consider the monochromatic image as the first modality my and the gray-scale one as the second modality m.- We use a neural network with four convolutional layers as the uni-modal branch and employ three MMTMs connect them. The corresponding uni-modal DNNs trained on the monochromatic images and the gray-scale images achieve the validation accuracies of 43% and 98%, respectively. We use this synthetic dataset mainly to demonstrate the proposed greedy learner hypothesis.
ModelNet40 is one of the Princeton ModelNet datasets (Wu et al., 2015) with 3D objects of 40 categories (9,483 training samples and 2,468 test samples). We use the task that is to classify a 3D object based on the 2D views of its front and back (Su et al. 2015). Each example is a 3D object rendered as 2D images of 224 224 pixels. For the uni-modal branches, we use ResNet18 (He et al. 2016) and apply MMTMS the three final residual blocks. The uni-modal DNNs achieve 91.6% and 93.1% accuracy, when learning from the front view (mo) and the rear view (m1)) respectively.
NVIDIA Dynamic Hand Gesture Dataset (or NVGesture [Molchanvv et al. 2015)), consists of 1,532 video clips (1,050 training and 482 test ones) of hand gestures in 25 classes. We sample 20% training examples as the validation set and use depth and RGB as the two modalities. We adopt the data preparation steps used in Joze et al. (2020) and use the I3D architecture (Carreira & Zisserman 2017) as uni-modal branches and MMTMS as fusion modules the six final inception modules.
We provide examples of each dataset and details on data preprocessing in $ A.4 in Appendix.
5.2 VALIDATING THE GREEDY LEARNER HYPOTHESIS
In this section, run the conventional multi-modal learning process seven tasks with different input and output pairs to validate the first conjecture from 3.2 experimentally and the proposed greedy learner hypothesis.

---

Under review as a conference paper at ICLR 2022
Study design For each task introduced in $5.1 in addition to the original dataset, construct dataset with two identical input modalities by copying one of the modalities. For example, when using the colored-and soreed-andgray---ane -ray-MNISTT dataset, we predict the digit class using two identical gray-scale images. We train a multi-modal DNN on these dataset as explained below for each task:
train multi-modal DNNs using SGD with the momentum coefficient of 0.9 and a batch size of 128. We sample 20 learning rates at random from the interval [10-5 60-5,0 1] on a logarithmic scale. We train the model four times using each of the learning rate and random initialization of the parameters. In total, we train 80 models.
ModeINet40: use SGD without momentum and use minibatches of eight examples. We select nine learning rates from 10 33 to 1 and train model using each learning rate for three times. This ends up with 27 models.
NVGesture:
we use batch size of four, SGD with momentum of 0.9, and uniformly sample 20 learning rates from the interval [10-1, 10 10-1.5 on logarithmic scale, We train the model three times using each learning rate, resulting 60 models in total.
Figure 2: Histograms and estimated density functions of dutil and dspeed of models trained using ModelNet40 (top) or NVGesture (bottom). We mark zero and R[duiaa with dashed lines. Many models have high dutil- We see duti is distributed asymmet- rically around zero when using two different input modalities. When us- ing two identical input modalities, it is distributed symmetrically around zero. We see depeeed exhibits similar distributions dusil.
front & rear duplicaled front-view
dipased
dumi
front rear views
ouplicated fronbwiew
-0.1
-0.05
-0.2
-0.10
medels
number modills
1.0
0.6
RGB & Depth
RGB & Depth
Identical RGBs
dpeed
duss
0.0
-0.2
-0.5
-0.4
6 number models
number of models
Results We present the results for the experiments on ModelNet40 and NVGesture in Figure and on Solored-and-gray-MMNITT in Figure Appendix.
First, many models have high datil- This confirms that the multi-modal learning process encourages the model to rely on one modality and ignore the other one, which is consistent with our hypothesis. We make this observation across all tasks, confirming that the conventional multi-modal learning process is greedy regardless of network architectures and tasks.
Second, aut is distributed symmetrically around zero, and E[duaal is approximately 0.0, for all the experiments using two identical input modalities. On the other hand, if we use two distinct modalities, dum distributed asymmetrically, and observe [E[mma]] of approximately 0.3, 0. and 0.4 for waw-red-andggga-----a first conjecture $32
ModelNet40 and NVGesture, respectively. This validates the
Third, by observiig conditional learning speed depordd we can draw the same conclusions. In fact, the distributions of d'peed largely replicate the distributions of duaa- It validates our greedy learner hypothesis which blames the varying rate which the learner learns rely on each modality. moreover confirms depeed is an appropriate proxy to use to re-balance multi-modaa modal learning.
We analyze the model's 's behavior when using different learning rates. Interestingly, we see relatively balanced conditional utilization rates when using high learning rates, as shown in Figure 3 in Appendix. This observation indicates that high learning rates implicitly calibrate the learning pace between modalities. We leave this for future analysis.
5.3 STRONG REGULARIZATION ENCOURAGES GREEDINESS
We investigate LI regularization' 's impact on multi modil DNNs and demonstrate that, as the second conjecture in 4.2 says, strong regularization encourages greediness in multi-modal learning.
7

---

Under review as a conference paper at ICLR 2022
Study design We apply L1 regularization to the multi-modal DNNs. That is, we train the networks to optimize the loss L' = L + x||0||1, where L is the classification loss in $2. stands for all model parameters and À is the weight the regularizer.
We measure the effect
on the network| f by computing the fraction of its parameters smaller than 10-7. We denote this quantity by R(f)). Since L1 regularization encourages sparsity of the network's parameters, as shown in Appendix Figure 4 the larger the À we use, the higher the R(f)) we observe.
We conduct this study with ModelNet40, using the front and rear views. We choose ten values from an interval [10-9 10-$1 as X. We use SGD without momentum, set the learning rate to 0. and batch size eight. Using each combination of hyperparameters, we repeat training for three times with random initialization and get three models.
Results As shown in Figure 3 dutn (f)) increases along A, especially when log(A) 2 -5. We also see that (f) is positively correlated with R(f). In other words, the stronger the regularization is, the larger the imbalance in utilization between modalities we observe. We see that dspeeaa follows the same trend as ducil. Again, it supports our choice of using the conditional learning speed to predict the conditional utilization rate.
0.5
0.30
0.2
0.4
0.17
0.50
0.1
0.25
08
0.0
0.10
0.00
9.0-8.0-70--600--- 4.0-3.8-3.5533.
0.0
0.2
0.0
0.1
0.2
0.3
log(A)
R(N
R00
Figure 3: The observed dutil and [dspeed| for models trained with different weights (A) on the L1 regularizer. Left: dueal and dspeed) as a function of log(X) (A). Middle: | duaal increases along R()).
Right: The depped increases along R(f)) The multi-modal learning process exhibit higher greed if we introduce stronger regularization on the multi-modal DNNs.
5.4 BALANCED MULTI-MODAL LEARNING
Besides the proposed training algorithm (cf. $4.2 referred to as guided), we introduce a variant of it in Appendix A.3 referred to random. This algorithm is motivated by Modality Dropout (Neveroval ct al. 2015) but better suited to the multi-modal DNNs with intermediate fusion. We consider it stronger baseline that can also balance learning from inputs of different modalities.
Calibrated modality utilization We train modall DNNs as described in $5.2 using the guided, the random, and the conventional training algorithm (referred to as vanilla). For ModelNet40, we set the imbalance tolerance parameter to 0.01 and the re-balancing window size Q to 5. For NVGesture, we use a of 0.1 and Q of s Results are shown in Figure 4 Models trained with the guided algorithm have lower [dueil compared to the vanilla algorithm. On NVGesture, we obtain E(Quca) of approximately 0.3 and 0.4 for models trained with the guided and the vanilla algorithm. On ModelNer40, we obtain E(Quu) of approximately -0.0, and -0.1 for models trained with the guided and the vanilla algorithm.
The random algorithm also calibrates modality utilization effectively (see Figure 7 in Appendix). We will see in the next section that it helps less on generalization compared to the guided algorithm.
Improved generalization performance We compare the generalization ability of multi-moda DNNs trained by the three algorithms (guided, random and vanilla) and the RUBi learning strat- egy (Cadene et al. 2019). For each algorithm, we train each model three times with the same learning rate. We use 0.01, 0.1 and 0.01 as learning rate for folored-and-gray-e -NNIST, ModelNet40 and NVGesture respectively. We use a of 0.1 for Colored-and-gral -NNIT and NVGesture and 0.01 for ModelNet40. We set Q of 5 for all three datasets. For NVGesture, we add one experiment where we initialize the model with parameters pre-trained using the Kinetics dataset (Carreira & Zisserman 2017) in addition random initialization. We refer to this setting as "NVGesture preetttieeppetraieed" and to the other one as
scrracch.scratch"...
and We provide studies on the model's sensitivity to Q and a Figure and Figure 6 Appendix.
8

---

Under review as a conference paper at ICLR 2022
Figure 4: Histograms and esti- mated density functions of dutn of models trained using the guided and the vanilla algorithm. We use dashed lines to mark zero and E duaa] over the set of mod- els trained with each of the algo- rithm. The guided algorithm is less greedy than the vanilla one.
0.1
vanilla quided
vanilla quided
0.75
0.0
0.50
0.00
7.5
10.0
12.5
(a) ModelNet40
(b) NVGesture
We report means and standard deviations of the models' test accuracies in Table I RUBi does not show consistent improvement across tasks compared to the vanilla algorithm. The guided algorithm improves the models' generalization performance over all three other methods in all four cases.
Table I: Test accuracy of models trained with the vanilla, random and guided algorithms.
Colored-and -radddd--aa---IIS
ModelNet40
NVGesture-scratt
WVGesture-pretraingg
vanilla RUBi random 74.07=2.75
45.26-0.46
90.09+0.58
79.81+1.14
83.20+-0.21
44.79=0.62
90.45:10.58
79.95:10.12
81.60:11.28
91.36:10.10
79.8810.90
82.64.10.84
guided
91.01=1.20
91.37++0.28
00.22++0.73
83.82:+1.45
RELATED WORK
Previous works on multi-modal learning focus more on architectural designs of the DNNs (Ngiam
al. 2011 Tran et al., 2015 Lazaridou et al.) 2015 Wang et al./ 2020bb Pérez- Rúa al.
2019).
Recently, unbalanced learning dynamics and modality-wiss utilization in the end-to-end trained
multi-modal classifiers were discussed (Goyal et al. 2017: Han et al. 2021 Wang al.)
2020a
Gat et al. 2020 Hessel & Lee 2020 Collell & Moens| 2018 Winterbottom et al. 2020 Sun et al.
2021). For example, WWng et al. (2020b) design a parameter-free mudti-modal fusion framework that
dynamically exchanges channels between the uni-modal branches. Wang et al. (2020a) estimate the uni-modal branches' generalization and overfitting speeds and calibrate the learning through loss re-weighting. Their weight estimation slows down training and relies on representative subset of the training data. Finally Gat et al.) (2020) point out the bias in the multi-modal data as the cause of the issue. They propose to supply inputs with Gaussian perturbations to the model and regularize it by maximizing functional entropies. In this work, we explain this phenomenon by the greedy nature of learning in multi-modal DNNs and propose an algorithm based on the training dynamics of model. Our proposed method does not introduce any additional computation and only relies on the norm of the gradients and weights computed during training. does not require architectural changes of the multi-modal DNNs and can be easily modified to accommodate different fusion modules. In addition to the multi-modal classifiers, the unbalanced modality wwie utilization has been identified in the multi-modal pre-trained models (Li et al.) 2020 Cao et al. 2020). We expect that the insights we provide in this work are applicable to the self-supervised multi-modal training framework too.
CONCLUSIONS
Our work demonstrates that the end-to-end trained multi-modal DNNs rely on one of the input modalities to make predictions while leaving the other modalities underutilized. We hypothesize that it is due to the greedy nature of learning in multi-moda DNNs, and validated statements experimentally on three mudti- datasets. We illustrated that using the proposed algorithm to balance the models' learning from different modalities enhances generalization. This result emphasizes that the adequate modality utilization a desired property that a model should achieve in multi-modal learning.
Results on NVGesture are not directly comparable with numbers in other works since use 20% training samples the validation data.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
This work focuses on analyzing the challenges of training multi-modal deep neural networks. does not involve any human subjects. We do not anticipate any potentially harmful consequences for society due to the publication of this work. As we use publicly available datasets in our study, we do not anticipate privacy and security issues. We claim conflicts of interest.
REPRODUCIBILIIT STATEMENT
We provide complete descriptions of the used datasets and the applied processing steps in $5.1 and in Appendix $A.4 We state our experimental configurations for each study in $5.1 $5.2 and $5.3 We plan to publish the source code once the paper is released.
REFERENCES
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. EMNLP, 2016.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. CVPR, 2018.
Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb El Saddik, and Mohan S Kankanhalli. Multimodal fusion for multimedia analysis: survey. Multimedia systems, 2010.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 2018.
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. 1998.
Andrew Brock, Soham De, Samuel Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. ICML, 2021.
Remi Cadene, Corentin Dancette, Hedi Youne,,,s, Matthieu Cord, and Devi Parikh. Rubi: Reducing unimodal biases visual question answering. NIPS, 2019.
Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. Behind the scene: Revealing the secrets of pre-trained vision-and- -nd---na-l models. ECCV, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. CVPR, 2017.
Guillem Collell and Marie-Francine Moens. Do neural network cross-modal mappings really bridge modalities? ACL, 2018.
Itai Gat, Idan Schwartz, Alexander Schwing, and Tamir Hazan. Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies. NearIPS, 2020.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. CVPR, 2017.
Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification. ICLR, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Jack Hessel and Lillian Lee. Does my multimodal model lear cross-modal interactions? it's harder to tell than you might think! EMNLP, 2020.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. NeurIPS, 2018.
10

---

Under review as a conference paper at ICLR 2022
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML, 2015.
Hamid Reza Vaezi Joze, Amirreza Shaban, Michael L Iuzzolino, and Kazuhito Koishida. Mmtm: multimodal transfer module for enn fusion. CVPR, 2020.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. CVPR, 2019.
Angeliki Lazaridou, Marco Baroni, et al. Combining language and vision with a multimodal skip- gram model. HLT-NAACL, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
Linjie Li, Zhe Gan, and Jingjing Liu. A closer look at the robustness of vision-and-language pre-trained models. arXiv:20/2.08675 2020.
Kuan Liu, Yanen Li, Ning Xu, and Prem Natarajan. Learn to combine modalities in multimodal deep learning. arXiv:/8035 /730, 2018.
Pavlo Molchanov, Shalini Gupta, Kihwan Kim, and Jan Kautz, Hand gesture recognition with 3d convolutional neural networks. CVPR, 2015.
Pavlo Molchanov, Xiaodong Yang. Shalini Gupta, Kihwan Kim, Stephen Tyree, and Jan Kautz. Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network. CVPR, 2016.
Natalia Neverova, Christian Wolf, Graham Taylor, and Florian Nebout. Moddrop: Adaptive multi- modal gesture recognition. IEEE transactions on pattern analysis and machine intelligence, 2015.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multi- modal deep learning. ICML, 2011.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. AAAI, 2018.
Juan-Manuel Pérez-Rúa, Valentin Vielzeuf, Stéphane Pateux, Moez Baccouche, and Frédéric Jurie. Mfas: Multimodal fusion architecture search. CVPR, 2019.
Samuel L. Smith, Benoit Dherin, David Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. ICLR, 2021.
Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi-view learning. COLT, 2008
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learned-Miller. Multi-view convolu- tional neural networks for 3d shape recognition. ICCV, 2015.
Ya Sun, Sijie Mai, and Haifeng Hu. Learning to balance the learning rates between various modalities via adaptive tracking factor. IEEE Signal Processing Letters, 2021.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani. and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. ICCV, 2015.
Guillermo Perle,Perez, Chico Q Camargo, and Ard A. Louis. Deep learning generalizes because the parameter-funntion map is biased towards simple functions. ICLR, 2019.
Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv:1706.053500 2017.
Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? CVPR, 2020a.
11

---

Under review as a conference paper at ICLR 2022
Yikai Wang. Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, and Junzhou Huang. Deep multimodal fusion by channel exchanging. NeurIPS, 2020b.
Zejia Weng, Zuxuan Wu, Hengduo Li, and Yu-Gang Jiang, Hms: Hierarchical modality selectionfor efficient video recognition. arXiv:2104. 09760, 2021.
Thomas Winterbottom, Sarah Xiao, Alistair McLean, and Noura Al Moubayed. On modality bias in the tvqa dataset. BMVC, 2020.
Nan Wu, Stanislaw Jastrzębski, Jungkyu Park. Linda Moy, Kyunghyun Cho, and Krzysztof J. Geras. Improving the ability of deep networks to use information from multiple views in breast cancer screening. MIDI, 2020.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang. Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. CVPR, 2015.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 2021.
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. ICLR, 2019.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 DERIVING J0 AND Jí FROM THE MOLTI-MODAL DNNS
We denote the hth fusion module by 9k- For an input æ. let hi (x) and (xx denote the feature vectors from the layer in the uni-modal branches where Ok is implemented.
To derive fol (2m)) and (Ti,, ). we force each uni-modal branch make predictions based on the single modality. Precisely, for each uni-modal branch, we aggregate its feature vectors over the whole training set and compute their average:
mo h
Σ h (a'),
mi li
Σ hg (x').
Let w (a:) (x)) and w (a:) denote the context information gle gives:
W b (x)
mo
gl (h (1) (æ), lig ),
Wx 2i (a:) - gu (
h ma (x)).
In this way, we cut off the cross-modal information sharing between the uni-modal branches while limiting the distribution shift for both the fusion modules and the layers in the uni-modal branches.
In our modal DNN, we employ the fusion module Multi-Modal Transfer Module (MMTM). We adopt the annotation used by Joze et al. (2020). A MMTM takes two sets of tensors, A E R xNixx-XXM Nxx and B E IR Ma x.. Mr xC" each as feature maps of a convolutional layer of a uni- modal branch. Here, N; and M, represent the spatial dimensions of each feature map, and and C represent the number of feature maps.
In order to derive (xm)) and (a we have operations in MMTM written as:
Nerasning Σ
Nerasning
(c(c) Ntraining
Sa(c),
3c(() N Ntraining
()(()
1=1
ZA = 3n),,
Zn W WS, Ap
E WAZA bA
E/ Zn + bn
Á' = 2 x o (EA) @ A,
B' = 2 o (EEB) B
where [ represents the concatenation operation; W; and b, are parameters the fully connected layers; O is the channel-wise product operation; and a(.) is the sigmoid function.
A.2 THE RE- RE-BALANCING STEP
Suppose that we have K fusion modules and let w. (x) and wt (x) denote the output of the Atth fusion module. At the t training step in the balanced multi modal learning process, we have
1
w
(x'),
M1 K
Σ
w (x'),
N
N
D
D'D
pussed
prased
where Dt passed denotes all samples appeared in the previous regular training steps until now. For example, in re-balancing step on mo, we aim to accelerate the model's learning from my. and we have Úo = do (mmgg T7th
Mo )
i = Φi (amyy " m1 (a:), ... w (æ)),
where æ = (I Im1) is the input at this step, and @ and Φ1 denote the uni-modal branch take mo and m, inputs.
A.3 RANDOM VERSION OF THE BALANCED MULTI-MODAL LEARNING ALGORITIIM
We present a random version of the balanced multi-modal learning algorithm in Algorithm I Similar to the guided version, perform regular steps in the first epoch. Afterwards, at each time, let the model take the step that is randomly sampled from: regular step. re- balancing step on ma- and b-balanciig step m71
13

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Random version of the balanced multi-modal learning
Input: T, T.# # of training steps;
for 4 to 7i do
Ti, # of steps in the 4 training epoch;
Take regular step; end + do
Qstep regular step,
re- re-balancing step to accelerate learning from ma- re- re-balancing step to accelerate learning from m²)
for
Randomly sample a step from Qstepi
Take the sampled step:
end
A.4 DATA PREPARATION
We used three datasets in the paper: Colored MNIST dataset (Kim et al. (019), ModelNet40 dataset (Su et al. 2015) and NVGesture dataset (Molchanov et al. 2015), as illustrated in Figure
For Colored MNIST and ModelNet40, we did not perform any extra data pre-processing steps on the original datasets.
For the NVGesture dataset, each video has a resolution of 240x320 and a duration of 80 frames from action starting to ending. There three videos with unmatched starting indices between RGB and depth. We adopt the starting frame indice of RGB for all modalities. We randomly select 64 consecutive frames from the videos the dataset and if the video has less than 64 frames, zero-pad on both sides of it to obtain 64 frames. Frames are resized as x and are cropped into 224x224 inputs (we static cropping where we crop from the same location across times and modalities).
During training, we perform spatial augmentation the video, including flipping and random cropping. During inference on validation or test set, we perform center cropping on the video.
2
3
4
5
6
7
9
RGB Depth
0
4
5
6
7
9
4
5
8
2
5
(a) Colored MNIST
(b) ModelNet40
(c) NVGesture
Figure I: (a) The Colored MNIST dataset (Km et al. 2019). We consider the monochromatic image, and the gray-scale image as the two input modalities (b) The ModelNet40 dataset (Su et al. (015). The 2D representations (c)The NVGesture dataset (Molchanov et al.) 2015). We use depth and RGB channels as the two modalities.
geay-scale images rendered from 12 different viewpoints of the object.
A.5 MORE RESULTS
14

---

Under review as a conference paper at ICLR 2022
color graye
duplicated color
dutil
duplicated gray-scale
2
color & gray-scale
duplicated color
duplicated gray-scale
Speed
-2
15
10 number models
5
15
number of models
number of models
Figure 2: Histograms and estimated density functions of dutil and depeed of models trained for colored-and Bouddgray-Min graS-MNIST, using monochromatic and gray-scale images two modalities, using identical monochromatic images as two modalities and using identical gray-scale images as two modalities.
0.8
0.6
-0.10
0.4
0.2
-0.15
0.0
0.001
00003
0.01
0.03
0.7
1.0
3357
5.71
-3519
2,085
2001
1.939
1.732
161
41489
learning rate
leaming (logia]
(a) ModelNet40 (front and rear views)
(b) NVGesture (RGB and Depth)
Figure 3: The imbalance in utilization, measured by duuil for models trained using different learning rate. It appears that high learning rates help with mitigating the imbalance in utilization between modalities.
0.30
0.25
0.20
0.15
0.10
0.05
0.00
9.0-8.0-7.0-6.0
-4.3 -4.44.33.0000 -.0.5555 log(A)
3.8.333-3355333
Figure 4: The mean and standard deviation of R(f) for three model repetitions obtained by training the model with À the weight on the LI regularization. Generally, the larger the is, the higher the R()) is, i.e., the sparser the model's 's parameters are.
15

---

Under review as a conference paper at ICLR 2022
accuracy
91.5
Id
91.0
0.0
1.0
5.0 10.0
50.0
1.0
5.0
10.0
20.0 50.0
Q
Q
Figure 5: Models' behavior when using different values for the re-balancing window size Q in the balanced multi-modal training algorithm. We ModelNet40 (front and rear views) in the study. We fix the learning rate at 0.1 and the imbalance tolerance parameter at 0.01 while using Q of 1, 5, 10, 20 and 50. We can effectively control the imbalance in conditional utilization except for Q 50 (see dutil shown in the left panel). According to the accuracy the model reaches, we choose to use Q = 5.
0.15
accuracy
Iduel
0.10
0.05
90
0.001
0.01 0.025 0.05
0.1
0.001 0.01 0.025 0.05 0.1
Figure 6: Models' behavior when using different values for the imbalance tolerance parameter a in the balanced multi-modal training algorithm. We use ModelNet40 (front and rear views) in the study and fix the learning rate at 0.1 and the re-balancing window size at The values we use for are ratios of B(dyu) computed in the study in $5.2 Precisely. we use 0,0 ((duet)). 0. 1.1E((E (Eduat), (dat), (.5E(duta). E(duut). Based on the pattern of dati shown in the left panel, using less than 0.25 of (dduu))) gives desirable results. We choose use a (d = 0.01 according to the accuracy.
1.00
vanilla
vanilla random
random
0.75
0.0
dutil
0.50
-0.1
0.25
-0.2
0.00
2
10
15
number models
number of models
(a) ModelNet40 (front and views)
(b) NVGesture (RGB and Depth)
Figure 7: Histograms and estimated density functions of dutil for random version of the balanced multi-modal learning process and the vanilla process. Both versions of the balanced multi-modal learning process are less greedy than the vanilla one.
16


Reviews:
[{"review_id": "idTUktOofMb", "reviewer": "Reviewer_BHoa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.\tThe topic of modal balance is interesting and has attracted much attention. The proposed method is direct and easy to follow.\n2.\tExperiments on several datasets verify the effectiveness.\n\n\nweakness\n1.\tHow to set the \\alpha at the bottom of page five? More parameter analyses are expected.\n2.\tAs introduced in the introduction and abstract, many related works about this topic have been proposed, e.g., “Rubi: Reducing unimodal biases in visual question answering”, “Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies”. What are the improvements to the proposed methods? Why not compare these methods? The reviewer believes that the experiments are very lacking in comparison with the SOTA methods.\n\n", "summary_of_the_paper": "This paper focuses on the multi-modal interaction problem, that multi-modal models tend to rely on just one modality while under-utilizing the other modalities. Since conditional utilization rate cannot be computed efficiently during training, they introduce an efficient proxy based on the pace at which a DNN learns from each modality, which we refer to as conditional learning speed. So they propose a training algorithm, balanced multi-modal learning, and demonstrate that it indeed addresses the issue of greedy learning. The proposed algorithm is found to improve the model’s generalization.", "summary_of_the_review": "This paper focuses on the multi-modal interaction problem, that multi-modal models tend to rely on just one modality while under-utilizing the other modalities. Since conditional utilization rate cannot be computed efficiently during training, they introduce an efficient proxy based on the pace at which a DNN learns from each modality, which we refer to as conditional learning speed. So they propose a training algorithm, balanced multi-modal learning, and demonstrate that it indeed addresses the issue of greedy learning. The proposed algorithm is found to improve the model’s generalization. So I recommend weak accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HfdgvViCX0", "reviewer": "Reviewer_B4j4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper focuses on a relevant problem in multi-modal learning on how to utilize information from both modalities efficiently.\nThe strength of the paper is that they follow a structured and scientific way of identifying the problem, proposing a metric to measure the extent of the problem and perform experiments\nto validate their hypothesis. They also show through experiments that the proposed training procedure leads to more efficient modality utilization and shows improved accuracy. The empirical results are strong.\n2. Weakness: In a MTMM architecture, Theta_0 (red blocks in Fig 1) would also be receiving updates based on the modality m1 as information is being passed through the fusion blocks denoted by parameters (theta_0)'(green and red-green blocks).\nIn that case, I am not sure if definition 4.1 is a valid way to define the conditional learning speeds as both theta_0 and theta_0' would see the information flow from modality m1. It would have been valid if theta_0 did not se information from m1 at all (like the way conditional utilization rate was defined in 2.1). So, I am not clearly convinced with the statement in Section 4.1 that \"If the greedy learner hypothesis is true, the conditional learning speed would serve as a proxy for the conditional utilization rate. dspeed(f;t) would predict dutil(f) at the end of training.\" Authors have tried to explain how to derive uni-modal outputs from MTMM in section A1 but it isn't clear to me\n3. Re-balancing step is not clear as explained in Appendix A2. Lets say we want to balance the information flow from modality m0 in a MTMM architecture in Fig 1.\nIn that case, does it mean the top branch takes information from just m0 (i.e. the information flow from green and green-red blocks is stopped) to generate prediction y0_hat, while the bottom branch takes information from both m0 and m1 to generate the prediction y1_hat.", "summary_of_the_paper": "The paper talks about efficient multi-modal training. They hypothesize that multi-modal training is greedy and perform experiments to empirically show the same.\nThey propose metrics such as conditional utilization rate and conditional learning speed which serves as a proxy for the former to measure the imbalance in utilization between modalities while training.\nBased on conditional learning speed, they propose a multi-modal training algorithm that mitigates the imbalance in modality utilization.\nThrough experiments, the authors empirically show that it is true and their training paradigm performs better than vanilla multi-modal training.", "summary_of_the_review": "This paper is easy to read and addresses the shortcomings of vanilla multi-modal training scenarios. Authors have done a good job of explaining the problem followed by laying out their hypotheses clearly. They have done experiments which enforces the hypotheses empirically. Authors should explain the re-balancing step (in section A2) and how to derive uni-modal predictions from multi-modal DNNs (in section A1) more clearly with consistent notations as used throughout the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5ojlh5AFRfy", "reviewer": "Reviewer_ndMY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Strengths\n  - The greedy behaviors among modalities are frequently observed in practical learning process of multi-modal DNNs. The research topic of this paper is interesting and worthy of investigation.\n  - The paper is easy to follow.\n\n* Weaknesses\n  - The main claims related to worse generalization performance from the greedy nature is not so clearly validated with experiments. It is critical issue since it is connected with the reasons of necessity of the proposed methods. Table 1 shows that the experimental results between *vanilla* and *guided* are not statistically significantly different in ModelNet40, NVGesture-scratch and NVGesture-pretrained. Even though those of Colored-and-gray-MNIST dataset seem to demonstrate big differences, I think the dataset is intentionally designed to be biased with color and labels and not compatible with standard multi-modal setting. So, only the results with Colored-and-gray-MNIST dataset are not enough to support the main claims.\n   - The other claims on are also not so clear to accept. For Section 5.2, if the values of | d_util | are high, it seems to mean that there are differences of conditional utilization rates, not confirm that the standard multi-modal learning process encourages the model to rely on one modality and ignore the other one. In other words, I think the experimental design and their evidences are weak to validate their claims.\n\n\n* Minors\n  - Section 5.3 in page 8, obverse -> observe\n", "summary_of_the_paper": "This paper investigates the modality greediness of learning in multi-modal deep neural networks (DNNs). The authors hypothesize that multi-modal DNNs learn to rely on one of input modalities (it could learn faster), which is expressed as a “greedy\" behavior. \nThe greediness is quantified with the difference between conditional utilization rate between modalities.\nThey claim the followings: (1) conditional utilization rate can be surrogated with conditional learning speed, (2) the strong regularization in the training process encourages the greediness, and (3) the greedy behavior often lowers generalization performances. \nAs a remedy of performance decreasing phenomena, they propose balanced multi-modal learning methods, which control the training speeds of all of modalities to be within certain criterion via giving more steps considering the log-ratio of sum of the magnitudes of gradients of loss functions between of fusion model parameters and of unimodal model parameters.\nThe paper provides experimental results to validate their claims and the effectiveness of the proposed methods.\n", "summary_of_the_review": "The research topic is interesting and it seems that the main answer points of the research questions are appropriate.\nHowever, main claims in the paper are not so clearly validated with respect to both of theories and experiments.\nI think that the authors should reconsider whether the differences on conditional utilization rate and those of conditional learning speed really affect the generalization performances, or they should try to establish other strategies to find evidences to support main claims.\nFrom the reason, I recommend this paper as reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-E1Z_fcQ7pY", "reviewer": "Reviewer_PUm3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThis paper focuses on a very interesting problem in multi-modal learning: the imbalanced training between modalities. Researches about this issue are meaningful and the idea of controlling the training process specific to modality is an intuitively promising way.\n\nWeakness:\n1.\tIt seems the experiments verifies the greedy learner hypothesis, but no proofs or theories are provided to support this hypothesis, either detailed analysis why the training speeds are different.\n2.\tThe re-balancing step method, which is used to make multi-modal learning less greedy, is not well described and analysis in the full text. The author should have a well explanation that why using the average feature of previous samples can stop the training of the uni-modal branch. \n3.\tLack comparation with other methods that solve imbalance problem, including [1]. The discussion in related work section is also inadequate. For example, related work [2-3] are not discussed. \n4.\tThe author set different imbalance parameter for different dataset, but did not provided the ablation study for this crucial parameter. Does it impact the model performance greatly?\n5.\tAll datasets only contain visual modality. More natural modalities, like audio and visual, should be included for better evaluation.\n\n[1] Weiyao Wang, Du Tran, and Matt Feiszli. What makes train- 1055 ing multi-modal classification networks hard? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12695–12705, 2020.\n[2] Thomas Winterbottom, Sarah Xiao, Alistair McLean, and Noura Al Moubayed. On modality bias in the tvqa dataset. BMVC, 2020.\n[3] Ya Sun, Sijie Mai, and Haifeng Hu. Learning to balance the learning rates between various modalities via adaptive tracking factor. IEEE Signal Processing Letters, 28:1650– 1654, 2021.\n\n\n\n", "summary_of_the_paper": "This paper hypothesizes that due to the greedy nature of deep learning, these models tend to rely on just one modality while under-utilizing the other modalities. The authors empirically observe this phenomenon on several dataset with a proposed metric. They propose an algorithm to achieve balanced training, which forcing the model to update only one of the uni-modal branches when the two uni-modal branches are imbalanced. This method is applied to three datasets and gain some progress.", "summary_of_the_review": "This paper focuses on an interesting phenomenon in the multi-modal learning: the imbalance problem in the multi-modal training. A training algorithm, balanced multi-modal learning, is proposed to solve this problem. According to the provided experiment results, this method addresses the issue of greedy learning. However, the author did not provide solid theory analysis about not only the greedy learning problem but also the proposed method. Related works are also not well discussed and compared.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: PAPER: This paper presents analysis of cross-modal interactions in multimodal models and propose a method to help balance the multimodal learning process. The cross-modal analysis is based on measures related to conditional utilization rate and the proposed approach is related to conditional learning speed.
DISCUSSION: The reviewers showed support for this line of research, as a way to better understand the learning process for multimodal models. The discussion allowed to identify points that needed to be clarify and concerns about the experimental results. The authors addressed many of these issues in their response. All reviewers took the time to read these responses as well as other reviews. There are many reviewers who are still expressing concerns with the experimental results. 
SUMMARY: This is an important line of research, and the authors should continue this research endeavor. While the paper presents some interesting research hypotheses about multimodal learning, it seems that more experiments are needed to properly address these hypotheses. In its current form, the paper may not yet be ready for publication.
Decision: Reject