Year: 2021

Abstract:
We hypothesize that due to the greedy nature of learning in multi-modal deep neural networks, these models tend to rely on just one modality while under-utilizing the other modalities. We observe empirically that such behavior hurts their overall generalization. To estimate the model’s modality-wise dependency, we compute the gain on the accuracy when the model has access to an additional modality and refer to this gain as the conditional utilization rate of the added modality. In the experiments, we consistently observe an imbalance in conditional utilization rates between modalities, across multiple tasks and architectures. Since conditional utilization rate cannot be computed efficiently during training, we introduce an proxy for it based on the pace at which the model learns from each modality, which we refer to as conditional learning speed. We propose an algorithm to balance the conditional learning speed between modalities during training and demonstrate that it indeed addresses the issue of greedy learning. The proposed algorithm improves the model’s generalization on three datasets: Colored MNIST (Kim et al., 2019), Princeton ModelNet40 (Wu et al., 2015), and NVIDIA Dynamic Hand Gesture Dataset (Molchanov et al., 2016).

heading: 1 INTRODUCTION
text: In real-world problems, each instance frequently has multiple modalities associated with it. For example, we detect cancer in both X-ray and ultrasound images. We seek clues from images to answer questions given in text. We are naturally interested in training deep neural networks (DNNs) end-to-end to learn from all available input modalities. We refer to such a training regime as a multi-modal learning process and DNNs resulting from this process as multi-modal DNNs.
Several recent studies have reported unsatisfactory performance of multi-modal DNNs in various tasks (Wang et al., 2020a; Wu et al., 2020; Gat et al., 2020; Cadene et al., 2019; Agrawal et al., 2016; Hessel & Lee, 2020; Han et al., 2021). For example, in Visual Question Answering, multi-modal DNNs were found to ignore the visual modality (presented as an image) and exploit statistical regularities shared between the textual modality (presented as a question) and the answer alone, resulting in poor generalization (Cadene et al., 2019; Gat et al., 2020; Agrawal et al., 2016). Similarly, in Human Action Recognition, multi-modal DNNs trained on images and audio were observed to under-perform uni-modal DNNs trained on images only (Wang et al., 2020a).
These earlier negative findings compel us to ask, what prevents multi-modal DNNs from achieving better performance? In order to answer this question, we first diagnose these DNNs as lacking utilization of all modalities by analyzing their conditional utilization rates. For a multi-modal DNN trained with two modalities, m0 and m1, the conditional utilization rate of m1 given m0, denoted by u(m1|m0), measures how important it is to use m1, given the presence of m0. It is computed as the relative difference in accuracy between two derived models from the DNN, one using both modalities and the other using only one modality. In several multi-modal learning tasks, we consistently observe a significant imbalance in conditional utilization rates between modalities. For example, we observe u(RGB|depth) = 0.01 and u(depth|RGB) = 0.63 for a DNN trained to identify gestures in videos using the NVIDIA Dynamic Hand Gesture Dataset (NVGesture) (Molchanov et al., 2016). It indicates that the DNN relies on the depth modality to make predictions and does not pay attention to the RGB
modality. These observations lead to a conjecture that the multi-modal learning process often results in models that under-utilize some of the input modalities.
As the focal point of this work, we put forward the greedy learner hypothesis. The greedy learner hypothesis states that a multi-modal DNN learns to rely on one of the input modalities, based on which it could learn faster, and does not continue to learn to use the other modalities. This greediness leads to an imbalance in conditional utilization rates between modalities. In other words, it prevents a multi-modal DNN from learning to exploit all available modalities and often results in worse generalization. It explains the challenge in training multi-modal DNNs and motivates us to design a better multi-modal learning algorithm.
According to the greedy learner hypothesis, it is the diverged speed at which a multi-modal DNN learns from different modalities that leads to an imbalance in conditional utilization rate. If we intervene in the training process to adjust these speeds, we may be able to prevent the hurtful imbalance across input modalities. We analyze the learning dynamics of model components and propose a metric named by conditional learning speed using the gradient norm and weight norm of models’ parameters. It measures the relative learning speed at which the model learns from one modality against the other modality. We empirically show that it is a reasonable proxy for conditional utilization rate. We introduce a training algorithm, balanced multi-modal learning which guides the model to learn intentionally from one of the modalities according to the conditional learning speeds. We show that models trained with this algorithm learn to use all modalities appropriately and achieve stronger generalization on three multi-modal datasets: Colored MNIST dataset (Kim et al., 2019), ModelNet40 dataset of 3D objects (Su et al., 2015) and NVGesture Dataset (Molchanov et al., 2015).

heading: 2 PROBLEM SETUP
text: We consider two input modalities, referred to as m0 and m1, without loss of generality towards tasks with more than two modalities. We denote a multi-modal dataset by D = { (xi, yi) }N i=1
, where x = (xm0 ,xm1). We partition the dataset D into training, validation and test sets, denoted by Dtrain, Dval, and Dtest, respectively. The goal is then to use this data set to train a multi-modal DNN such that it accurately predicts y from x.
Multi-modal DNNs We use a multi-modal DNN that has two uni-modal branches, each taking one modality as input. The two uni-modal branches are interconnected by layer-wise fusion modules. These fusion modules enable information flow from one branch to another. According to categorization of multi-modal fusion strategies in the deep learning literature, this is intermediate fusion (Ngiam et al., 2011; Atrey et al., 2010; Baltrušaitis et al., 2018). It has demonstrated competitive performance against multi-modal DNNs with late fusion in many tasks (Perez et al., 2018; Joze et al., 2020; Anderson et al., 2018; Wang et al., 2020b).
We implement every fusion module by a multi-modal transfer module (MMTM) (Joze et al., 2020). Each MMTM connects two layers from the two uni-modal branches. There is first the global average pooling applied over spatial dimensions to transform feature maps into a vector. We concatenate the two vectors and apply linear transformation. We refer to its output as context representation. Next, for each uni-modal branch, we implement a fully connected layer on the context representation and get a vector with a dimension of the number of feature maps. Feature maps are re-scaled by this vector before passing to the next layer of the uni-modal branch.
We train a multi-modal DNN f on Dtrain. Let ŷ be the prediction of f for an input x: ŷ = f(xm0 ,xm1).
As shown in Figure 1, ŷ = 12 (ŷ0+ ŷ1), where ŷ0 and ŷ1 are the outputs of the two uni-modal branches. During training, the parameters of the multi-modal DNN are updated by stochastic gradient descent (SGD) to minimize the loss: L = CE(y, ŷ0) + CE(y, ŷ1), where CE stands for cross-entropy. We refer to each of the cross-entropy losses as a modality-specific loss. We train the model until ŷ = y for all samples in Dtrain and take the checkpoint of it when ŷ reaches the highest accuracy on Dval. During training, each uni-modal branch largely focuses on its associate input modality. The fusion modules generate context representation using all modalities and feed such information to the unimodal branches. Both ŷ0 and ŷ1 depend on information from both modalities. We end up with two functions, f0 and f1, corresponding to the two uni-modal branches:
ŷ0 = f0(xm0 ,xm1), ŷ1 = f1(xm0 ,xm1).
To evaluate the multi-modal DNN’s ability in using a single modality, we disable information sharing between branches ad derive f ′0 and f ′ 1, each taking only one of the input modalities as input. We describe how we derive them in detail in Appendix A.1. We use ŷ′0 and ŷ ′ 1 to refer to their outputs:
ŷ′0 = f ′ 0(xm0), ŷ ′ 1 = f ′ 1(xm1).
In summary, we derive four models from f and we define the ability of each model as its accuracy A on Dtest. We group the four accuracies into two pairs: (A(f0), A(f ′0)), and (A(f1), A(f ′1)) and define the conditional utilization rates as bellow:
Definition 2.1 (Conditional utilization rate) For a multi-modal DNN, f , taking two modalities m0 and m1 as inputs, its conditional utilization rates for m1 and m0 are defined as u(m1|m0) = A(f0)−A(f ′0)
A(f0) and u(m0|m1) = A(f1)−A(f ′ 1) A(f1) , respectively.
The conditional utilization rate is the relative change in accuracy between the two models within each pair. The conditional utilization rate of m0, u(m0|m1), measures how important it is for f to use m0 together with m1. In other words, it estimates the marginal contribution that m0 has in increasing the accuracy of ŷ1 depending on m1.
A multi-modal learning process Given a DNN’s architecture and a dataset, every time we run a multi-modal learning process, we perform the following steps: (1) sample a value of learning rate randomly from a given range, (2) initialize the DNN’s parameters randomly, (3) update them to fit Dtrain with SGD, and (4) select the model checkpoint with the highest accuracy on Dval.

heading: 3 THE GREEDY LEARNER HYPOTHESIS
text: In this section, we introduce the greedy learner hypothesis to explain challenges observed in training multi-modal DNNs. Before describing our hypothesis, we discuss some assumptions on the multimodal data and preliminary observations made in the multi-modal learning literature.

heading: 3.1 ASSUMPTIONS IN MULTI-MODAL LEARNING
text: First, in multi-modal learning literature (Blum & Mitchell, 1998; Sridharan & Kakade, 2008), both modalities are assumed to be predictive of the target, i.e., I(Y ,Xm0) > 0 and I(Y ,Xm1) > 0, where I denotes the mutual information. In order to minimize one of the modality-specific losses, e.g. CE(y, ŷ0), one can either update the parameters of the fusion layers that pass information from m1 to ŷ0, or the parameters of the uni-modal branch taking m0 as input, or both.
Second, modalities are predictive of the target at different degrees. For example, it has been observed that when training DNNs on each modality separately, they usually do not reach the same performance (Wang et al., 2020b; Joze et al., 2020). In addition, multi-modal DNNs exhibit varying performance when being trained on different subsets of modalities present for the task (Weng et al., 2021; Pérez-Rúa et al., 2019; Liu et al., 2018).
Third, modalities are learned by the model at different speeds. This has been observed in both uni-modal DNNs and Multi-modal DNNs (Wang et al., 2020a; Wu et al., 2020).

heading: 3.2 MULTI-MODAL LEARNING PROCESS IS GREEDY
text: Before formalizing our greedy learner hypothesis that explains the behaviour of multi-modal DNNs, we first demonstrate that we can tell whether the model utilizes all modalities by analyzing its conditional utilization rates.
Given a multi-modal DNN, f , with two input modalities, m0 and m1, we compute the difference between conditional utilization rates, u(m1|m0) and u(m0|m1) as
dutil(f) = u(m1|m0)− u(m0|m1).
If we observe high |dutil(f)|, we say that f exhibits imbalance in utilization between modalities. The difference between conditional utilization rates is bounded between −1 and 1. When it is close to −1 or 1, the model benefits only from one of the modalities given the other but not vice versa. This implies that the model’s ability to predict ŷ0 and ŷ1 comes only from one of the modalities.
We now introduce the greedy learner hypothesis:
We call a multi-modal learning process greedy, when it trains models to rely on only one of the available modalities. A greedy multi-modal learning process cannot avoid producing models that exhibit a high degree of imbalance in utilization between modalities. We hypothesize that a multi-modal learning process, in which a multi-modal DNN is trained to minimize the sum of modality-specific losses, is greedy.
Let E[d̂util] denote the expectation of d̂util over the empirical distribution of models resulting from a multi-modal learning process (as defined in §2). The absolute value of E[d̂util] is associated with the greediness of the process. The higher the |E[d̂util]| is, the greedier the learning process is. We propose the following conjectures consistent with the greedy learner hypothesis:
1. There exists an > 0, s.t. |E[d̂util]| > , as long the multi-modal dataset has the last two properties from §3.1. Otherwise, d̂util is distributed symmetrically around zero and E[d̂util] = 0.
2. The stronger the regularization we apply to the DNNs’ parameters in training, the higher the |E[d̂util]| is.
We expect the first scenario in the first conjecture to be true for all real-world multi-modal tasks. To test the second scenario in the first conjecture, we construct datasets of two identical modalities.
We suspect that the implicit regularization of DNNs is a factor impacting the greediness of learning in multi-modal DNNs. The implicit regularization of DNNs commonly refers to a consensus that DNNs have the tendency to lean towards simplicity during training and it explains their strong generalization (Valle-Perez et al., 2019; Zhang et al., 2017; 2021; Smith et al., 2021). Since it is difficult to precisely measure and control such implicit regularization, alternatively, we construct the second conjecture using explicit regularization applied to the DNNs’ parameters.
In order to validate the above hypothesis and conjectures empirically, we conduct experiments on several multi-modal datasets using different network architectures (cf. §5.1). We present our observations in §5.2 and §5.3. We show that the multi-modal learning process cannot avoid producing models that exhibit a high degree of imbalance in utilization between modalities. Further, in the next section, we propose a metric for measuring the speeds at which the model learns from different modalities and we show the imbalance in such speeds is correlated with the imbalance in utilization.

heading: 4 MAKING MULTI-MODAL LEARNING LESS GREEDY
text: We aim to make multi-modal learning less greedy by controlling the speed at which a multi-modal DNN learns to rely on each modality. To this end, we define conditional learning speed to measure the speed at which the DNN learns from one modality. It serves as an efficient proxy to the conditional utilization rate of the corresponding modality, as shown empirically in §5.2 and §5.3. We then propose the balanced multi-modal learning algorithm, which controls the difference in conditional learning speed between modalities that the model exhibits during training.

heading: 4.1 CONDITIONAL LEARNING SPEED
text: As demonstrated in §3.2, the imbalance in conditional utilization rates is a sign of the model exploiting the connection between the target and only one of the input modalities, ignoring crossmodal information. However, conditional utilization rates are measured after training is done, making it expensive to use them in real-time during training. We instead derive a proxy metric, called conditional learning speed, that captures relative learning speed between modalities during training.
Let us revisit the architecture of the multi-modal DNN. We denote the parameters of each uni-modal branch taking m0 and m1 as inputs, by θ0 and θ1. Layers of the fusion modules marked with green and green/blue in Figure 1, are part of the function mapping x to ŷ0. Let θ′0 refer to their parameters. Analogously, the layers marked with green and green/red in Figure 1 are part of the function mapping x to ŷ1. We denote their parameters by θ′1. In this way, we divide the model’s parameters into two pairs: (θ0,θ′0) and (θ1,θ ′ 1). 1
For parameters θ, let θ(i−1) and θ(i) denote its value before and after the gradient descent step i, and G = ∂L∂θ |θ(i−1) . We define the model’s conditional learning speed as:
Definition 4.1 (Conditional learning speed) Given a multi-modal DNN, f , with two input modalities, m0 and m1, the conditional learning speeds of these modalities after t training steps, are
s(m1|m0; t) = log ∑t i=1 µ(θ ′ 0; i)∑t
i=1 µ(θ0; i) , s(m0|m1; t) = log
∑t i=1 µ(θ
′ 1; i)∑t
i=1 µ(θ1; i) ,
where µ(θ; i) = ||G||22/||θ(i)||22 quantifies the change of f that comes from updating θ at the ith step, which can be also interpreted as the effective update on θ.
This definition of µ(θ; i) is inspired by discussion on the effective update of parameters in the literature (Van Laarhoven, 2017; Zhang et al., 2019; Brock et al., 2021; Hoffer et al., 2018). When normalization techniques, such as batch normalization (Ioffe & Szegedy, 2015), are applied to the DNNs, the key property of the weight vector, θ, is its direction, i.e., θ/||θ||22. Thus, we measure the update on θ using the norm of its gradient normalized by its norm.
The conditional learning speed, s(m1|m0; t) (and analogously s(m0|m1; t)), is the log-ratio between the learning speed of θ′0 and that of θ0. Because θ ′ 0 carries information fromm1 to ŷ0 and information from m0 to ŷ0 is carried by θ0, s(m1|m0; t) reflects how fast the model learns from m1 relative to m0, after the first t steps.
We further compute the difference between s(m1|m0; t) and s(m0|m1; t) as:
dspeed(f ; t) = s(m1|m0; t)− s(m0|m1; t),
analogous to dutil(f). For each model, we report dspeed(f ;T ) as dspeed(f) where the model takes T steps until reaching the highest accuracy on Dval. We anticipate that the conditional learning speed would serve as a proxy for the conditional utilization rate and we say dspeed(f ; t) would predict dutil(f) at the end of training. In §5.2 and §5.3, we show the distributions of d̂speed and d̂util side-by-side to validate this.

heading: 4.2 BALANCED MULTI-MODAL LEARNING
text: Because dspeed(f, t) is predictive of the imbalanced utilization between modalities, we can take advantage of dspeed(f, t) to balance conditional utilization on-the-fly. In addition to training the network normally with both modalities, we accelerate the model to learn from either modality alternately to balance the conditional learning speeds of them. See Algorithm 1 for an overall description of the algorithm.
We refer to the training steps at which we perform forward and backward passes normally as regular steps. We introduce re-balancing steps at which we update one of the uni-modal branches intentionally in order to accelerate the model to learn from the corresponding modality. See Appendix A.2 for the full explanation of the re-balancing step.
1Some parameters are both in θ′0 and θ′1.
To warm-up the model, we perform only regular steps in the first training epoch. Then we switch from regular steps to re-balancing steps if |dspeed(t)| > α, where α is a hyperparameter, referred to as the imbalance tolerance parameter. The training takes Q re-balancing steps before returning to regular mode. We refer to the hyperparameter Q as the re-balancing window size.
Algorithm 1: Balanced multi-modal learning Input: Q, re-balancing window size;
α, imbalance tolerance parameter T , # of training steps; T1, # of steps in the 1th training epoch
Let M denote the accumulated effective update (µ) in regular steps Mθ0 ,Mθ′0 = 0, 0; Mθ1 ,Mθ′1 = 0, 0; for t← 1 to T1 do
Take a regular step; Update Mθ0 ,Mθ′0 ,Mθ1 ,Mθ′1
end regular_mode = True
for t← T1 to T do if regular_mode then
Take a regular step; Update Mθ0 ,Mθ′0 ,Mθ1 ,Mθ′1 ; Compute dspeed; if |dspeed| > α then
q = 0; regular_mode = False end
else q ← q + 1; if q = Q then regular_mode = True; Take a re-balancing step to accelerate learning from m0 if dspeed > 0 else m1;
end end

heading: 5 EXPERIMENTS AND RESULTS

heading: 5.1 DATASETS, TASKS AND BASELINES
text: Colored-and-gray-MNIST (Kim et al., 2019) is a synthetic dataset based on MNIST (LeCun et al., 1998). In the training set of 60,000 examples, each example has two images, a gray-scale image and a monochromatic image, with color strongly correlated with its digit label. For the validation set of 10,000 examples, each example also has a gray-scale image and a corresponding monochromatic image however with a low correlation between the color and its label. We consider the monochromatic image as the first modality m0 and the gray-scale one as the second modality m1. We use a neural network with four convolutional layers as the uni-modal branch and employ three MMTMs to connect them. The corresponding uni-modal DNNs trained on the monochromatic images and the gray-scale images achieve the validation accuracies of 43% and 98%, respectively. We use this synthetic dataset mainly to demonstrate the proposed greedy learner hypothesis.
ModelNet40 is one of the Princeton ModelNet datasets (Wu et al., 2015) with 3D objects of 40 categories (9,483 training samples and 2,468 test samples). We use the task that is to classify a 3D object based on the 2D views of its front and back (Su et al., 2015). Each example is a 3D object rendered as 2D images of 224×224 pixels. For the uni-modal branches, we use ResNet18 (He et al., 2016) and apply MMTMs in the three final residual blocks. The uni-modal DNNs achieve 91.6% and 93.1% accuracy, when learning from the front view (m0) and the rear view (m1), respectively.
NVIDIA Dynamic Hand Gesture Dataset (or NVGesture (Molchanov et al., 2015)), consists of 1,532 video clips (1,050 training and 482 test ones) of hand gestures in 25 classes. We sample 20% training examples as the validation set and use depth and RGB as the two modalities. We adopt the data preparation steps used in Joze et al. (2020) and use the I3D architecture (Carreira & Zisserman, 2017) as uni-modal branches and MMTMs as fusion modules in the six final inception modules.
We provide examples of each dataset and details on data preprocessing in § A.4 in Appendix.

heading: 5.2 VALIDATING THE GREEDY LEARNER HYPOTHESIS
text: In this section, we run the conventional multi-modal learning process on seven tasks with different input and output pairs to validate the first conjecture from §3.2 experimentally and the proposed greedy learner hypothesis.
Study design For each task introduced in §5.1, in addition to the original dataset, we construct a dataset with two identical input modalities by copying one of the modalities. For example, when using the colored-and-gray-MNIST dataset, we predict the digit class using two identical gray-scale images. We train a multi-modal DNN on these dataset as explained below for each task:
• Colored-and-gray-MNIST: we train multi-modal DNNs using SGD with the momentum coefficient of 0.9 and a batch size of 128. We sample 20 learning rates at random from the interval [10−5, 1] on a logarithmic scale. We train the model four times using each of the learning rate and random initialization of the parameters. In total, we train 80 models.
• ModelNet40: we use SGD without momentum and use minibatches of eight examples. We select nine learning rates from 10−3 to 1 and train model using each learning rate for three times. This ends up with 27 models.
• NVGesture: we use a batch size of four, SGD with momentum of 0.9, and uniformly sample 20 learning rates from the interval [10−4, 10−1.5] on a logarithmic scale. We train the model three times using each learning rate, resulting in 60 models in total.
Results We present the results for the experiments on ModelNet40 and NVGesture in Figure 2 and on Colored-and-gray-MNIST in Figure 2 in Appendix.
First, many models have high |dutil|. This confirms that the multi-modal learning process encourages the model to rely on one modality and ignore the other one, which is consistent with our hypothesis. We make this observation across all tasks, confirming that the conventional multi-modal learning process is greedy regardless of network architectures and tasks.
Second, d̂util is distributed symmetrically around zero, and E[d̂util] is approximately 0.0, for all the experiments using two identical input modalities. On the other hand, if we use two distinct modalities, d̂util is distributed asymmetrically, and we observe |E[d̂util]| of approximately 0.3, 0.1 and 0.4 for colored-and-gray-MNIST, ModelNet40 and NVGesture, respectively. This validates the first conjecture in §3.2.
Third, by observing conditional learning speed d̂speed, we can draw the same conclusions. In fact, the distributions of d̂speed largely replicate the distributions of d̂util. It validates our greedy learner hypothesis which blames the varying rate at which the learner learns to rely on each modality. It moreover confirms dspeed is an appropriate proxy to use to re-balance multi-modal learning.
We analyze the model’s behavior when using different learning rates. Interestingly, we see relatively balanced conditional utilization rates when using high learning rates, as shown in Figure 3 in Appendix. This observation indicates that high learning rates implicitly calibrate the learning pace between modalities. We leave this for future analysis.

heading: 5.3 STRONG REGULARIZATION ENCOURAGES GREEDINESS
text: We investigate L1 regularization’s impact on multi-modal DNNs and demonstrate that, as the second conjecture in §3.2 says, strong regularization encourages greediness in multi-modal learning.
Study design We apply L1 regularization to the multi-modal DNNs. That is, we train the networks to optimize the loss L′ = L+ λ||θ||1, where L is the classification loss in §2, θ stands for all model parameters and λ is the weight on the regularizer.
We measure the effect of λ on the network f by computing the fraction of its parameters smaller than 10−7. We denote this quantity by R(f). Since L1 regularization encourages sparsity of the network’s parameters, as shown in Appendix Figure 4, the larger the λ we use, the higher the R(f) we observe.
We conduct this study with ModelNet40, using the front and rear views. We choose ten values from an interval [10−9, 10−3] as λ. We use SGD without momentum, set the learning rate to 0.1 and batch size to eight. Using each combination of hyperparameters, we repeat training for three times with random initialization and get three models.
Results As shown in Figure 3, |dutil(f)| increases along λ, especially when log(λ) ≥ −5. We also see that |dutil(f)| is positively correlated with R(f). In other words, the stronger the regularization is, the larger the imbalance in utilization between modalities we observe. We see that |dspeed| follows the same trend as |dutil|. Again, it supports our choice of using the conditional learning speed to predict the conditional utilization rate.

heading: 5.4 BALANCED MULTI-MODAL LEARNING
text: Besides the proposed training algorithm (cf. §4.2, referred to as guided), we introduce a variant of it in Appendix A.3, referred to as random. This algorithm is motivated by Modality Dropout (Neverova et al., 2015) but better suited to the multi-modal DNNs with intermediate fusion. We consider it a stronger baseline that can also balance learning from inputs of different modalities.
Calibrated modality utilization We train multi-modal DNNs as described in §5.2, using the guided, the random, and the conventional training algorithm (referred to as vanilla). For ModelNet40, we set the imbalance tolerance parameter α to 0.01 and the re-balancing window size Q to 5. For NVGesture, we use α of 0.1 and Q of 5.2 Results are shown in Figure 4. Models trained with the guided algorithm have lower |d̂util| compared to the vanilla algorithm. On NVGesture, we obtain E(d̂util) of approximately 0.3 and 0.4 for models trained with the guided and the vanilla algorithm. On ModelNet40, we obtain E(d̂util) of approximately -0.0, and -0.1 for models trained with the guided and the vanilla algorithm.
The random algorithm also calibrates modality utilization effectively (see Figure 7 in Appendix). We will see in the next section that it helps less on generalization compared to the guided algorithm.
Improved generalization performance We compare the generalization ability of multi-modal DNNs trained by the three algorithms (guided, random and vanilla) and the RUBi learning strategy (Cadene et al., 2019). For each algorithm, we train each model three times with the same learning rate. We use 0.01, 0.1 and 0.01 as learning rate for Colored-and-gray-MNIST, ModelNet40 and NVGesture respectively. We use α of 0.1 for Colored-and-gray-MNIST and NVGesture and 0.01 for ModelNet40. We set Q of 5 for all three datasets. For NVGesture, we add one experiment where we initialize the model with parameters pre-trained using the Kinetics dataset (Carreira & Zisserman, 2017) in addition to random initialization. We refer to this setting as “NVGesture-pretrained” and to the other one as “NVGesture-scratch”.
2We provide studies on the model’s sensitivity to Q and α in Figure 5 and Figure 6 in Appendix.
0 2 4 6 8 number of models
0.2
0.1
0.0 0.1 d u til vanilla guided
(a) ModelNet40
0.0 2.5 5.0 7.5 10.0 12.5 number of models
0.00
0.25
0.50
0.75
1.00
d u til
vanilla guided
(b) NVGesture
Figure 4: Histograms and estimated density functions of d̂util of models trained using the guided and the vanilla algorithm. We use dashed lines to mark zero and E[d̂util] over the set of models trained with each of the algorithm. The guided algorithm is less greedy than the vanilla one.
We report means and standard deviations of the models’ test accuracies in Table 1.3 RUBi does not show consistent improvement across tasks compared to the vanilla algorithm. The guided algorithm improves the models’ generalization performance over all three other methods in all four cases.

heading: 6 RELATED WORK
text: Previous works on multi-modal learning focus more on architectural designs of the DNNs (Ngiam et al., 2011; Tran et al., 2015; Lazaridou et al., 2015; Wang et al., 2020b; Pérez-Rúa et al., 2019). Recently, unbalanced learning dynamics and modality-wise utilization in the end-to-end trained multi-modal classifiers were discussed (Goyal et al., 2017; Han et al., 2021; Wang et al., 2020a; Gat et al., 2020; Hessel & Lee, 2020; Collell & Moens, 2018; Winterbottom et al., 2020; Sun et al., 2021). For example, Wang et al. (2020b) design a parameter-free multi-modal fusion framework that dynamically exchanges channels between the uni-modal branches. Wang et al. (2020a) estimate the uni-modal branches’ generalization and overfitting speeds and calibrate the learning through loss re-weighting. Their weight estimation slows down training and relies on a representative subset of the training data. Finally Gat et al. (2020) point out the bias in the multi-modal data as the cause of the issue. They propose to supply inputs with Gaussian perturbations to the model and regularize it by maximizing functional entropies. In this work, we explain this phenomenon by the greedy nature of learning in multi-modal DNNs and propose an algorithm based on the training dynamics of model. Our proposed method does not introduce any additional computation and only relies on the norm of the gradients and weights computed during training. It does not require architectural changes of the multi-modal DNNs and can be easily modified to accommodate different fusion modules. In addition to the multi-modal classifiers, the unbalanced modality-wise utilization has been identified in the multi-modal pre-trained models (Li et al., 2020; Cao et al., 2020). We expect that the insights we provide in this work are applicable to the self-supervised multi-modal training framework too.

heading: 7 CONCLUSIONS
text: Our work demonstrates that the end-to-end trained multi-modal DNNs rely on one of the input modalities to make predictions while leaving the other modalities underutilized. We hypothesize that it is due to the greedy nature of learning in multi-modal DNNs, and validated our statements experimentally on three multi-modal datasets. We illustrated that using the proposed algorithm to balance the models’ learning from different modalities enhances generalization. This result emphasizes that the adequate modality utilization is a desired property that a model should achieve in multi-modal learning.
3Results on NVGesture are not directly comparable with numbers in other works since we use 20% training samples as the validation data.
ETHICS STATEMENT
This work focuses on analyzing the challenges of training multi-modal deep neural networks. It does not involve any human subjects. We do not anticipate any potentially harmful consequences for society due to the publication of this work. As we use publicly available datasets in our study, we do not anticipate privacy and security issues. We claim no conflicts of interest.
REPRODUCIBILITY STATEMENT
We provide complete descriptions of the used datasets and the applied processing steps in §5.1 and in Appendix §A.4. We state our experimental configurations for each study in §5.1, §5.2, and §5.3. We plan to publish the source code once the paper is released.

heading: A APPENDIX
text: A.1 DERIVING f ′0 AND f ′ 1 FROM THE MULTI-MODAL DNNS
We denote the kth fusion module by gk. For an input x, let hm0k (x) and h m1 k (x) denote the feature vectors from the layer in the uni-modal branches where gk is implemented.
To derive f ′0(xm0) and f ′ 1(xm1), we force each uni-modal branch to make predictions based on the single modality. Precisely, for each uni-modal branch, we aggregate its feature vectors over the whole training set and compute their average:
h m0 k =
1
N ∑ x′ hm0k (x ′), h m1 k = 1 N ∑ x′ hm1k (x ′).
Let wm0k (x) and w m1 k (x) denote the context information gk gives:
wm0k (x) = gk(h m0 k (x), h m1 k ), w m1 k (x) = gk(h m0 k , h m1 k (x)).
In this way, we cut off the cross-modal information sharing between the uni-modal branches while limiting the distribution shift for both the fusion modules and the layers in the uni-modal branches.
In our multi-modal DNN, we employ the fusion module as a Multi-Modal Transfer Module (MMTM). We adopt the annotation used by Joze et al. (2020). A MMTM takes two sets of tensors, A ∈ RN1×···×NK×C and B ∈ RM1×···×ML×C′ , each as feature maps of a convolutional layer of a unimodal branch. Here, Ni and Mi represent the spatial dimensions of each feature map, and C and C ′ represent the number of feature maps.
In order to derive f ′0(xm0) and f ′ 1(xm1), we have operations in a MMTM written as:
SA(c) = 1
Ntraining Ntraining∑ i=1 SiA(c), SB(c) = 1 Ntraining Ntraining∑ i=1 SiB(c)
ZA = W [SA, SB ], ZB = W [SA, SB ] E′A = WAZA + bA, E ′ B = WBZB + bB Ã′ = 2× σ(E′A) A, B̃′ = 2× σ(E′B) B,
where [·, ·] represents the concatenation operation; Wi and bi are parameters of the fully connected layers; is the channel-wise product operation; and σ(·) is the sigmoid function.
A.2 THE RE-BALANCING STEP
Suppose that we have K fusion modules and let wm0k (x) and w m1 k (x) denote the output of the k th fusion module. At the t training step in the balanced multi-modal learning process, we have
wm0k = 1
Nt ∑ x′∈Dtpassed wm0k (x ′), wm1k = 1 Nt ∑ x′∈Dtpassed wm1k (x ′),
where Dtpassed denotes all samples appeared in the previous regular training steps until now. For example, in a re-balancing step on m0, we aim to accelerate the model’s learning from m0, and we have
ŷ0 = φ0(xm0 , w m0 1 , · · · , w m0 K ), ŷ1 = φ1(xm1 , w m1 1 (x), · · · , w m1 K (x)),
where x = (xm0 ,xm1) is the input at this step, and φ0 and φ1 denote the uni-modal branch take m0 and m1 as inputs.
A.3 RANDOM VERSION OF THE BALANCED MULTI-MODAL LEARNING ALGORITHM
We present a random version of the balanced multi-modal learning algorithm in Algorithm 1. Similar to the guided version, we perform regular steps in the first epoch. Afterwards, at each time, we let the model take the step that is randomly sampled from: regular step, re-balancing step on m0, and re-balancing step on m1.
Algorithm 1: Random version of the balanced multi-modal learning Input: T , # of training steps;
T1, # of steps in the 1th training epoch; Ωstep = { regular step,
re-balancing step to accelerate learning from m0, re-balancing step to accelerate learning from m1}
for t← 1 to T1 do Take a regular step; end for t← T1 to T do
Randomly sample a step from Ωstep; Take the sampled step;
end
A.4 DATA PREPARATION
We used three datasets in the paper: Colored MNIST dataset (Kim et al., 2019), ModelNet40 dataset (Su et al., 2015) and NVGesture dataset (Molchanov et al., 2015), as illustrated in Figure 1.
For Colored MNIST and ModelNet40, we did not perform any extra data pre-processing steps on the original datasets.
For the NVGesture dataset, each video has a resolution of 240×320 and a duration of 80 frames from action starting to ending. There are three videos with unmatched starting indices between RGB and depth. We adopt the starting frame indice of RGB for all modalities. We randomly select 64 consecutive frames from the videos in the dataset and if the video has less than 64 frames, we zero-pad on both sides of it to obtain 64 frames. Frames are resized as 256×256 and are cropped into 224×224 as inputs (we use static cropping where we crop from the same location across times and modalities).
During training, we perform spatial augmentation on the video, including flipping and random cropping. During inference on validation or test set, we perform center cropping on the video.
A.5 MORE RESULTS

Review summaries:
This paper focuses on the multi-modal interaction problem, that multi-modal models tend to rely on just one modality while under-utilizing the other modalities. Since conditional utilization rate cannot be computed efficiently during training, they introduce an efficient proxy based on the pace at which a DNN learns from each modality, which we refer to as conditional learning speed. So they propose a training algorithm, balanced multi-modal learning, and demonstrate that it indeed addresses the issue of greedy learning. The proposed algorithm is found to improve the model’s generalization.

This paper investigates the modality greediness of learning in multi-modal deep neural networks (DNNs). The authors hypothesize that multi-modal DNNs learn to rely on one of input modalities (it could learn faster), which is expressed as a “greedy" behavior.  The greediness is quantified with the difference between conditional utilization rate between modalities. They claim the followings: (1) conditional utilization rate can be surrogated with conditional learning speed, (2) the strong regularization in the training process encourages the greediness, and (3) the greedy behavior often lowers generalization performances.  As a remedy of performance decreasing phenomena, they propose balanced multi-modal learning methods, which control the training speeds of all of modalities to be within certain criterion via giving more steps considering the log-ratio of sum of the magnitudes of gradients of loss functions between of fusion model parameters and of unimodal model parameters. The paper provides experimental results to validate their claims and the effectiveness of the proposed methods. 

This paper hypothesizes that due to the greedy nature of deep learning, these models tend to rely on just one modality while under-utilizing the other modalities. The authors empirically observe this phenomenon on several dataset with a proposed metric. They propose an algorithm to achieve balanced training, which forcing the model to update only one of the uni-modal branches when the two uni-modal branches are imbalanced. This method is applied to three datasets and gain some progress.

The paper talks about efficient multi-modal training. They hypothesize that multi-modal training is greedy and perform experiments to empirically show the same. They propose metrics such as conditional utilization rate and conditional learning speed which serves as a proxy for the former to measure the imbalance in utilization between modalities while training. Based on conditional learning speed, they propose a multi-modal training algorithm that mitigates the imbalance in modality utilization. Through experiments, the authors empirically show that it is true and their training paradigm performs better than vanilla multi-modal training.