Recognizing and overcoming the greedy nature of learning in multi-modal deep neural networks

1 INTRODUCTION . In real-world problems , each instance frequently has multiple modalities associated with it . For example , we detect cancer in both X-ray and ultrasound images . We seek clues from images to answer questions given in text . We are naturally interested in training deep neural networks ( DNNs ) end-to-end to learn from all available input modalities . We refer to such a training regime as a multi-modal learning process and DNNs resulting from this process as multi-modal DNNs . Several recent studies have reported unsatisfactory performance of multi-modal DNNs in various tasks ( Wang et al. , 2020a ; Wu et al. , 2020 ; Gat et al. , 2020 ; Cadene et al. , 2019 ; Agrawal et al. , 2016 ; Hessel & Lee , 2020 ; Han et al. , 2021 ) . For example , in Visual Question Answering , multi-modal DNNs were found to ignore the visual modality ( presented as an image ) and exploit statistical regularities shared between the textual modality ( presented as a question ) and the answer alone , resulting in poor generalization ( Cadene et al. , 2019 ; Gat et al. , 2020 ; Agrawal et al. , 2016 ) . Similarly , in Human Action Recognition , multi-modal DNNs trained on images and audio were observed to under-perform uni-modal DNNs trained on images only ( Wang et al. , 2020a ) . These earlier negative findings compel us to ask , what prevents multi-modal DNNs from achieving better performance ? In order to answer this question , we first diagnose these DNNs as lacking utilization of all modalities by analyzing their conditional utilization rates . For a multi-modal DNN trained with two modalities , m0 and m1 , the conditional utilization rate of m1 given m0 , denoted by u ( m1|m0 ) , measures how important it is to use m1 , given the presence of m0 . It is computed as the relative difference in accuracy between two derived models from the DNN , one using both modalities and the other using only one modality . In several multi-modal learning tasks , we consistently observe a significant imbalance in conditional utilization rates between modalities . For example , we observe u ( RGB|depth ) = 0.01 and u ( depth|RGB ) = 0.63 for a DNN trained to identify gestures in videos using the NVIDIA Dynamic Hand Gesture Dataset ( NVGesture ) ( Molchanov et al. , 2016 ) . It indicates that the DNN relies on the depth modality to make predictions and does not pay attention to the RGB modality . These observations lead to a conjecture that the multi-modal learning process often results in models that under-utilize some of the input modalities . As the focal point of this work , we put forward the greedy learner hypothesis . The greedy learner hypothesis states that a multi-modal DNN learns to rely on one of the input modalities , based on which it could learn faster , and does not continue to learn to use the other modalities . This greediness leads to an imbalance in conditional utilization rates between modalities . In other words , it prevents a multi-modal DNN from learning to exploit all available modalities and often results in worse generalization . It explains the challenge in training multi-modal DNNs and motivates us to design a better multi-modal learning algorithm . According to the greedy learner hypothesis , it is the diverged speed at which a multi-modal DNN learns from different modalities that leads to an imbalance in conditional utilization rate . If we intervene in the training process to adjust these speeds , we may be able to prevent the hurtful imbalance across input modalities . We analyze the learning dynamics of model components and propose a metric named by conditional learning speed using the gradient norm and weight norm of models ’ parameters . It measures the relative learning speed at which the model learns from one modality against the other modality . We empirically show that it is a reasonable proxy for conditional utilization rate . We introduce a training algorithm , balanced multi-modal learning which guides the model to learn intentionally from one of the modalities according to the conditional learning speeds . We show that models trained with this algorithm learn to use all modalities appropriately and achieve stronger generalization on three multi-modal datasets : Colored MNIST dataset ( Kim et al. , 2019 ) , ModelNet40 dataset of 3D objects ( Su et al. , 2015 ) and NVGesture Dataset ( Molchanov et al. , 2015 ) . 2 PROBLEM SETUP . We consider two input modalities , referred to as m0 and m1 , without loss of generality towards tasks with more than two modalities . We denote a multi-modal dataset by D = { ( xi , yi ) } N i=1 , where x = ( xm0 , xm1 ) . We partition the dataset D into training , validation and test sets , denoted by Dtrain , Dval , and Dtest , respectively . The goal is then to use this data set to train a multi-modal DNN such that it accurately predicts y from x. Multi-modal DNNs We use a multi-modal DNN that has two uni-modal branches , each taking one modality as input . The two uni-modal branches are interconnected by layer-wise fusion modules . These fusion modules enable information flow from one branch to another . According to categorization of multi-modal fusion strategies in the deep learning literature , this is intermediate fusion ( Ngiam et al. , 2011 ; Atrey et al. , 2010 ; Baltrušaitis et al. , 2018 ) . It has demonstrated competitive performance against multi-modal DNNs with late fusion in many tasks ( Perez et al. , 2018 ; Joze et al. , 2020 ; Anderson et al. , 2018 ; Wang et al. , 2020b ) . We implement every fusion module by a multi-modal transfer module ( MMTM ) ( Joze et al. , 2020 ) . Each MMTM connects two layers from the two uni-modal branches . There is first the global average pooling applied over spatial dimensions to transform feature maps into a vector . We concatenate the two vectors and apply linear transformation . We refer to its output as context representation . Next , for each uni-modal branch , we implement a fully connected layer on the context representation and get a vector with a dimension of the number of feature maps . Feature maps are re-scaled by this vector before passing to the next layer of the uni-modal branch . We train a multi-modal DNN f on Dtrain . Let ŷ be the prediction of f for an input x : ŷ = f ( xm0 , xm1 ) . As shown in Figure 1 , ŷ = 12 ( ŷ0+ ŷ1 ) , where ŷ0 and ŷ1 are the outputs of the two uni-modal branches . During training , the parameters of the multi-modal DNN are updated by stochastic gradient descent ( SGD ) to minimize the loss : L = CE ( y , ŷ0 ) + CE ( y , ŷ1 ) , where CE stands for cross-entropy . We refer to each of the cross-entropy losses as a modality-specific loss . We train the model until ŷ = y for all samples in Dtrain and take the checkpoint of it when ŷ reaches the highest accuracy on Dval . During training , each uni-modal branch largely focuses on its associate input modality . The fusion modules generate context representation using all modalities and feed such information to the unimodal branches . Both ŷ0 and ŷ1 depend on information from both modalities . We end up with two functions , f0 and f1 , corresponding to the two uni-modal branches : ŷ0 = f0 ( xm0 , xm1 ) , ŷ1 = f1 ( xm0 , xm1 ) . To evaluate the multi-modal DNN ’ s ability in using a single modality , we disable information sharing between branches ad derive f ′0 and f ′ 1 , each taking only one of the input modalities as input . We describe how we derive them in detail in Appendix A.1 . We use ŷ′0 and ŷ ′ 1 to refer to their outputs : ŷ′0 = f ′ 0 ( xm0 ) , ŷ ′ 1 = f ′ 1 ( xm1 ) . In summary , we derive four models from f and we define the ability of each model as its accuracy A on Dtest . We group the four accuracies into two pairs : ( A ( f0 ) , A ( f ′0 ) ) , and ( A ( f1 ) , A ( f ′1 ) ) and define the conditional utilization rates as bellow : Definition 2.1 ( Conditional utilization rate ) For a multi-modal DNN , f , taking two modalities m0 and m1 as inputs , its conditional utilization rates for m1 and m0 are defined as u ( m1|m0 ) = A ( f0 ) −A ( f ′0 ) A ( f0 ) and u ( m0|m1 ) = A ( f1 ) −A ( f ′ 1 ) A ( f1 ) , respectively . The conditional utilization rate is the relative change in accuracy between the two models within each pair . The conditional utilization rate of m0 , u ( m0|m1 ) , measures how important it is for f to use m0 together with m1 . In other words , it estimates the marginal contribution that m0 has in increasing the accuracy of ŷ1 depending on m1 . A multi-modal learning process Given a DNN ’ s architecture and a dataset , every time we run a multi-modal learning process , we perform the following steps : ( 1 ) sample a value of learning rate randomly from a given range , ( 2 ) initialize the DNN ’ s parameters randomly , ( 3 ) update them to fit Dtrain with SGD , and ( 4 ) select the model checkpoint with the highest accuracy on Dval . 3 THE GREEDY LEARNER HYPOTHESIS . In this section , we introduce the greedy learner hypothesis to explain challenges observed in training multi-modal DNNs . Before describing our hypothesis , we discuss some assumptions on the multimodal data and preliminary observations made in the multi-modal learning literature . 3.1 ASSUMPTIONS IN MULTI-MODAL LEARNING . First , in multi-modal learning literature ( Blum & Mitchell , 1998 ; Sridharan & Kakade , 2008 ) , both modalities are assumed to be predictive of the target , i.e. , I ( Y , Xm0 ) > 0 and I ( Y , Xm1 ) > 0 , where I denotes the mutual information . In order to minimize one of the modality-specific losses , e.g . CE ( y , ŷ0 ) , one can either update the parameters of the fusion layers that pass information from m1 to ŷ0 , or the parameters of the uni-modal branch taking m0 as input , or both . Second , modalities are predictive of the target at different degrees . For example , it has been observed that when training DNNs on each modality separately , they usually do not reach the same performance ( Wang et al. , 2020b ; Joze et al. , 2020 ) . In addition , multi-modal DNNs exhibit varying performance when being trained on different subsets of modalities present for the task ( Weng et al. , 2021 ; Pérez-Rúa et al. , 2019 ; Liu et al. , 2018 ) . Third , modalities are learned by the model at different speeds . This has been observed in both uni-modal DNNs and Multi-modal DNNs ( Wang et al. , 2020a ; Wu et al. , 2020 ) .

Summary: This paper investigates the modality greediness of learning in multi-modal deep neural networks (DNNs). The authors hypothesize that multi-modal DNNs learn to rely on one of input modalities (it could learn faster), which is expressed as a “greedy" behavior.  The greediness is quantified with the difference between conditional utilization rate between modalities. They claim the followings: (1) conditional utilization rate can be surrogated with conditional learning speed, (2) the strong regularization in the training process encourages the greediness, and (3) the greedy behavior often lowers generalization performances.  As a remedy of performance decreasing phenomena, they propose balanced multi-modal learning methods, which control the training speeds of all of modalities to be within certain criterion via giving more steps considering the log-ratio of sum of the magnitudes of gradients of loss functions between of fusion model parameters and of unimodal model parameters. The paper provides experimental results to validate their claims and the effectiveness of the proposed methods.