Recognizing and overcoming the greedy nature of learning in multi-modal deep neural networks

1 INTRODUCTION . In real-world problems , each instance frequently has multiple modalities associated with it . For example , we detect cancer in both X-ray and ultrasound images . We seek clues from images to answer questions given in text . We are naturally interested in training deep neural networks ( DNNs ) end-to-end to learn from all available input modalities . We refer to such a training regime as a multi-modal learning process and DNNs resulting from this process as multi-modal DNNs . Several recent studies have reported unsatisfactory performance of multi-modal DNNs in various tasks ( Wang et al. , 2020a ; Wu et al. , 2020 ; Gat et al. , 2020 ; Cadene et al. , 2019 ; Agrawal et al. , 2016 ; Hessel & Lee , 2020 ; Han et al. , 2021 ) . For example , in Visual Question Answering , multi-modal DNNs were found to ignore the visual modality ( presented as an image ) and exploit statistical regularities shared between the textual modality ( presented as a question ) and the answer alone , resulting in poor generalization ( Cadene et al. , 2019 ; Gat et al. , 2020 ; Agrawal et al. , 2016 ) . Similarly , in Human Action Recognition , multi-modal DNNs trained on images and audio were observed to under-perform uni-modal DNNs trained on images only ( Wang et al. , 2020a ) . These earlier negative findings compel us to ask , what prevents multi-modal DNNs from achieving better performance ? In order to answer this question , we first diagnose these DNNs as lacking utilization of all modalities by analyzing their conditional utilization rates . For a multi-modal DNN trained with two modalities , m0 and m1 , the conditional utilization rate of m1 given m0 , denoted by u ( m1|m0 ) , measures how important it is to use m1 , given the presence of m0 . It is computed as the relative difference in accuracy between two derived models from the DNN , one using both modalities and the other using only one modality . In several multi-modal learning tasks , we consistently observe a significant imbalance in conditional utilization rates between modalities . For example , we observe u ( RGB|depth ) = 0.01 and u ( depth|RGB ) = 0.63 for a DNN trained to identify gestures in videos using the NVIDIA Dynamic Hand Gesture Dataset ( NVGesture ) ( Molchanov et al. , 2016 ) . It indicates that the DNN relies on the depth modality to make predictions and does not pay attention to the RGB modality . These observations lead to a conjecture that the multi-modal learning process often results in models that under-utilize some of the input modalities . As the focal point of this work , we put forward the greedy learner hypothesis . The greedy learner hypothesis states that a multi-modal DNN learns to rely on one of the input modalities , based on which it could learn faster , and does not continue to learn to use the other modalities . This greediness leads to an imbalance in conditional utilization rates between modalities . In other words , it prevents a multi-modal DNN from learning to exploit all available modalities and often results in worse generalization . It explains the challenge in training multi-modal DNNs and motivates us to design a better multi-modal learning algorithm . According to the greedy learner hypothesis , it is the diverged speed at which a multi-modal DNN learns from different modalities that leads to an imbalance in conditional utilization rate . If we intervene in the training process to adjust these speeds , we may be able to prevent the hurtful imbalance across input modalities . We analyze the learning dynamics of model components and propose a metric named by conditional learning speed using the gradient norm and weight norm of models ’ parameters . It measures the relative learning speed at which the model learns from one modality against the other modality . We empirically show that it is a reasonable proxy for conditional utilization rate . We introduce a training algorithm , balanced multi-modal learning which guides the model to learn intentionally from one of the modalities according to the conditional learning speeds . We show that models trained with this algorithm learn to use all modalities appropriately and achieve stronger generalization on three multi-modal datasets : Colored MNIST dataset ( Kim et al. , 2019 ) , ModelNet40 dataset of 3D objects ( Su et al. , 2015 ) and NVGesture Dataset ( Molchanov et al. , 2015 ) . 2 PROBLEM SETUP . We consider two input modalities , referred to as m0 and m1 , without loss of generality towards tasks with more than two modalities . We denote a multi-modal dataset by D = { ( xi , yi ) } N i=1 , where x = ( xm0 , xm1 ) . We partition the dataset D into training , validation and test sets , denoted by Dtrain , Dval , and Dtest , respectively . The goal is then to use this data set to train a multi-modal DNN such that it accurately predicts y from x. Multi-modal DNNs We use a multi-modal DNN that has two uni-modal branches , each taking one modality as input . The two uni-modal branches are interconnected by layer-wise fusion modules . These fusion modules enable information flow from one branch to another . According to categorization of multi-modal fusion strategies in the deep learning literature , this is intermediate fusion ( Ngiam et al. , 2011 ; Atrey et al. , 2010 ; Baltrušaitis et al. , 2018 ) . It has demonstrated competitive performance against multi-modal DNNs with late fusion in many tasks ( Perez et al. , 2018 ; Joze et al. , 2020 ; Anderson et al. , 2018 ; Wang et al. , 2020b ) . We implement every fusion module by a multi-modal transfer module ( MMTM ) ( Joze et al. , 2020 ) . Each MMTM connects two layers from the two uni-modal branches . There is first the global average pooling applied over spatial dimensions to transform feature maps into a vector . We concatenate the two vectors and apply linear transformation . We refer to its output as context representation . Next , for each uni-modal branch , we implement a fully connected layer on the context representation and get a vector with a dimension of the number of feature maps . Feature maps are re-scaled by this vector before passing to the next layer of the uni-modal branch . We train a multi-modal DNN f on Dtrain . Let ŷ be the prediction of f for an input x : ŷ = f ( xm0 , xm1 ) . As shown in Figure 1 , ŷ = 12 ( ŷ0+ ŷ1 ) , where ŷ0 and ŷ1 are the outputs of the two uni-modal branches . During training , the parameters of the multi-modal DNN are updated by stochastic gradient descent ( SGD ) to minimize the loss : L = CE ( y , ŷ0 ) + CE ( y , ŷ1 ) , where CE stands for cross-entropy . We refer to each of the cross-entropy losses as a modality-specific loss . We train the model until ŷ = y for all samples in Dtrain and take the checkpoint of it when ŷ reaches the highest accuracy on Dval . During training , each uni-modal branch largely focuses on its associate input modality . The fusion modules generate context representation using all modalities and feed such information to the unimodal branches . Both ŷ0 and ŷ1 depend on information from both modalities . We end up with two functions , f0 and f1 , corresponding to the two uni-modal branches : ŷ0 = f0 ( xm0 , xm1 ) , ŷ1 = f1 ( xm0 , xm1 ) . To evaluate the multi-modal DNN ’ s ability in using a single modality , we disable information sharing between branches ad derive f ′0 and f ′ 1 , each taking only one of the input modalities as input . We describe how we derive them in detail in Appendix A.1 . We use ŷ′0 and ŷ ′ 1 to refer to their outputs : ŷ′0 = f ′ 0 ( xm0 ) , ŷ ′ 1 = f ′ 1 ( xm1 ) . In summary , we derive four models from f and we define the ability of each model as its accuracy A on Dtest . We group the four accuracies into two pairs : ( A ( f0 ) , A ( f ′0 ) ) , and ( A ( f1 ) , A ( f ′1 ) ) and define the conditional utilization rates as bellow : Definition 2.1 ( Conditional utilization rate ) For a multi-modal DNN , f , taking two modalities m0 and m1 as inputs , its conditional utilization rates for m1 and m0 are defined as u ( m1|m0 ) = A ( f0 ) −A ( f ′0 ) A ( f0 ) and u ( m0|m1 ) = A ( f1 ) −A ( f ′ 1 ) A ( f1 ) , respectively . The conditional utilization rate is the relative change in accuracy between the two models within each pair . The conditional utilization rate of m0 , u ( m0|m1 ) , measures how important it is for f to use m0 together with m1 . In other words , it estimates the marginal contribution that m0 has in increasing the accuracy of ŷ1 depending on m1 . A multi-modal learning process Given a DNN ’ s architecture and a dataset , every time we run a multi-modal learning process , we perform the following steps : ( 1 ) sample a value of learning rate randomly from a given range , ( 2 ) initialize the DNN ’ s parameters randomly , ( 3 ) update them to fit Dtrain with SGD , and ( 4 ) select the model checkpoint with the highest accuracy on Dval . 3 THE GREEDY LEARNER HYPOTHESIS . In this section , we introduce the greedy learner hypothesis to explain challenges observed in training multi-modal DNNs . Before describing our hypothesis , we discuss some assumptions on the multimodal data and preliminary observations made in the multi-modal learning literature . 3.1 ASSUMPTIONS IN MULTI-MODAL LEARNING . First , in multi-modal learning literature ( Blum & Mitchell , 1998 ; Sridharan & Kakade , 2008 ) , both modalities are assumed to be predictive of the target , i.e. , I ( Y , Xm0 ) > 0 and I ( Y , Xm1 ) > 0 , where I denotes the mutual information . In order to minimize one of the modality-specific losses , e.g . CE ( y , ŷ0 ) , one can either update the parameters of the fusion layers that pass information from m1 to ŷ0 , or the parameters of the uni-modal branch taking m0 as input , or both . Second , modalities are predictive of the target at different degrees . For example , it has been observed that when training DNNs on each modality separately , they usually do not reach the same performance ( Wang et al. , 2020b ; Joze et al. , 2020 ) . In addition , multi-modal DNNs exhibit varying performance when being trained on different subsets of modalities present for the task ( Weng et al. , 2021 ; Pérez-Rúa et al. , 2019 ; Liu et al. , 2018 ) . Third , modalities are learned by the model at different speeds . This has been observed in both uni-modal DNNs and Multi-modal DNNs ( Wang et al. , 2020a ; Wu et al. , 2020 ) .

Summary: This paper focuses on the multi-modal interaction problem, that multi-modal models tend to rely on just one modality while under-utilizing the other modalities. Since conditional utilization rate cannot be computed efficiently during training, they introduce an efficient proxy based on the pace at which a DNN learns from each modality, which we refer to as conditional learning speed. So they propose a training algorithm, balanced multi-modal learning, and demonstrate that it indeed addresses the issue of greedy learning. The proposed algorithm is found to improve the model’s generalization.