What Makes Multi-Modal Learning Better than Single (Provably)

Can multi-modal learning provably perform better than uni-modal ? In this paper , we answer this question under a most popular multi-modal fusion framework , which firstly encodes features from different modalities into a common latent space and seamlessly maps the latent representations into the task space . We prove that learning with multiple modalities achieves a smaller population risk than only using its subset of modalities . The main intuition is that the former has a more accurate estimate of the latent space representation . To the best of our knowledge , this is the first theoretical treatment to capture important qualitative phenomena observed in real multi-modal applications from the generalization perspective . Combining with experiment results , we show that multi-modal learning does possess an appealing formal guarantee . 1 Introduction . Our perception of the world is based on different modalities , e.g . sight , sound , movement , touch , and even smell [ 36 ] . Inspired from the success of deep learning [ 26 , 21 ] , deep multi-modal research is also activated , which covers fields like audio-visual learning [ 8 , 43 ] , RGB-D semantic segmentation [ 35 , 20 ] and Visual Question Answering [ 17 , 2 ] . While deep multi-modal learning shows excellent power in practice , theoretical understanding of deep multi-modal learning is limited . Some recent works have been done towards this direction [ 39 , 48 ] . However , these works made strict assumptions on the probability distributions across different modalities , which may not hold in real-life applications [ 30 ] . Notably , they do not take generalization performance of multi-modal learning into consideration . Toward this end , the following fundamental problem remains largely open : ∗equal contribution †Correspondence to : longbohuang @ tsinghua.edu.cn 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Can multi-modal learning provably performs better than uni-modal ? In this paper , we provably answer this question from two perspectives : • ( When ) Under what conditions multi-modal performs better than uni-modal ? • ( Why ) What results in the performance gains ? The framework we study is abstracted from the multi-modal fusion approaches , which is one of the most researched topics of multi-modal learning [ 3 ] . Specifically , we first encode the complex data from heterogeneous sources into a common latent space Z . The true latent representation is g⋆ in a function class G , and the task mapping h⋆ is contained in a function classH defined on the latent space . Our model corresponds to the recent progress of deep multi-modal learning on various applications , such as scene classification [ 9 ] and action recognition [ 24 , 43 ] . Under this composite framework , we provide the first theoretical analysis to shed light on what makes multi-modal outperform uni-modal from the generalization perspective . We identify the relationship between the population risk and the distance between a learned latent representation ĝ and the g⋆ , under the metric we will define later . Informally , closer to the true representation leads to less population loss , which indicates that a better latent representation guarantees the end-to-end multimodal learning performance . Instead of simply considering the comparison of multi vs uni modalities , we consider a general case , M vs N modalities , which are distinct subsets of all modalities . We focus on the condition that the latter is a subset of the former . Our second result is a bound for the closeness between ĝ and the g⋆ , from which we provably show that the latent representation ĝM learning from theM modalities is closer to the true g⋆ than ĝN learning from N modalities . As shown in Figure 1 , ĝM has a more sufficient latent space exploration than ĝN . Moreover , in a specific linear regression model , we directly verify that using multiple modalities rather than its subset learns a better latent representation . The main contributions of this paper are summarized as follows : • We formalize the multi-modal learning problem into a theoretical framework . Firstly , we show that the performance of multi-modal learning in terms of population risk can be bounded by the latent representation quality , a novel metric we propose to measure the distance from a learned latent representation to the true representation , which reveals that the ability of learning the whole task coincides with the ability of learning the latent representation when we have sufficient training samples . • We derive an upper bound for the latent representation quality of training over a subset of modalities . This directly implies a principle to guide us in modality selection , i.e. , when the number of sample size is large and multiple modalities can efficiently optimize the empirical risk , using multi-modal to build a recognition or detection system can have a better performance . • Restricted to linear latent and task mapping , we provide rigorous theoretical analysis that latent representation quality degrades when the subset of multiple modalities is applied . Experiments are also carried out to empirically validate the theoretical observation that ĝN is inferior to ĝM . The rest of the paper is organized as follows . In the next section , we review the related literature . The formulation of multi-modal learning problem is described in Section 3 . Main results are presented in Section 4 . In Section 5 , we show simulation results to support our theoretical claims . Additional discussions about the inefficiency of multi-modal learning are presented in Section 6 . Finally , conclusions are drawn in Section 7 . 2 Related Work . Multi-modal Learning Applications Deep learning makes fusing different signals easier , which enables us to develop many multi-modal frameworks . For example , [ 35 , 23 , 22 , 31 , 20 ] combine RGB and depth images to improve semantic segmentation ; [ 8 , 16 ] fuse audio with video to do scene understanding ; researchers also explore audio-visual source separation and localization [ 49 , 11 ] . Theory of Multi-modal Learning On the theory side , in semi-supervised setting , [ 39 ] proposed a novel method , Total Correlation Gain Maximization ( TCGM ) , and theoretically proved that TCGM can find the groundtruth Bayesian classifier given each modality . Moreover , CPM-Nets [ 48 ] showed multi-view representation can recover the same performance as only using the single-view observation by constructing the versatility . However , they made strict assumptions on the relationship across different modalities , while our analysis does not require such additional assumptions . Besides , previous multi-view analysis [ 37 , 1 , 45 , 13 ] typically assumes that each view alone is sufficient to predict the target accurately , which may not hold in our multi-modal setting . For instance , it is difficult to build a classifier just using a weak modality with limited labeled data , e.g. , depth modality in RGB-D images for object detection task [ 18 ] . Transfer Learning A line of work closely related to our composite learning framework is transfer learning via representation learning , which firstly learns a shared representation on various tasks and then transfers the learned representation to a new task . [ 40 , 7 , 32 , 10 ] have provided the sample complexity bounds in the special case of the linear feature and linear task mapping . [ 41 ] introduces a new notion of task diversity and provides a generalization bound with general tasks , features , and losses . Unfortunately , the function class which contains feature mappings is the same across all different tasks while our focus is that the function classes generated by different subsets modalities are usually inconsistent . Notation : Throughout the paper , we use ∥ · ∥ to denote the ℓ2 norm . We also denote the set of positive integer numbers less or equal than n by [ n ] , i.e . [ n ] ≜ { 1 , 2 , · · · , n } . 3 The Multi-modal Learning Formulation . In this section , we present the Multi-modal Learning problem formulation . Specifically , we assume that a given data x : = ( x ( 1 ) , · · · , x ( K ) ) consists of K modalities , where x ( k ) ∈ X ( k ) the domain set of the k-th modality . Denote X = X ( 1 ) × · · · × X ( K ) . We use Y to denote the target domain and use Z to denote a latent space . Then , we denote g⋆ : X 7→ Z the true mapping from the input space ( using all of K modalities ) to the latent space , and h⋆ : Z 7→ Y is the true task mapping . For instance , in aggregation-based multi-modal fusion , g⋆ is an aggregation function compounding on K seperate sub-networks and h⋆ is a multi-layer neural network [ 44 ] . In the learning task , a data pair ( x , y ) ∈ X × Y is generated from an unknown distribution D , such that PD ( x , y ) ≜ Py|x ( y | h⋆ ◦ g⋆ ( x ) ) Px ( x ) ( 1 ) Here h⋆ ◦ g⋆ ( x ) = h⋆ ( g⋆ ( x ) ) represents the composite function of h⋆ and g⋆ . In real-world settings , we often face incomplete multi-modal data , i.e. , some modalities are not observed . To take into account this situation , we let M be a subset of [ K ] , and without loss of generality , focus on the learning problem only using the modalities in M. Specifically , define X ′ : = ( X ( 1 ) ∪ { ⊥ } ) × . . .× ( X ( K ) ∪ { ⊥ } ) as the extension of X , where x′ ∈ X ′ , x′k =⊥ means that the k-th modality is not used ( collected ) . Then we define a mapping pM from X to X ′ induced byM : pM ( x ) ( k ) = { x ( k ) if k ∈M ⊥ else Also define p′M : X ′ 7→ X ′ as the extension of pM . Let G′ denote a function class , which contains the mapping from X ′ to the latent space Z , and define a function class GM as follows : GM ≜ { gM : X 7→ Z | gM ( x ) : = g′ ( pM ( x ) ) , g′ ∈ G′ } ( 2 ) Given a data set S = ( ( xi , yi ) ) mi=1 , where ( xi , yi ) is drawn i.i.d . from D , the learning objective is , following the Empirical Risk Minimization ( ERM ) principle [ 27 ] , to find h ∈ H and gM ∈ GM to jointly minimize the empirical risk , i.e. , min r̂ ( h ◦ gM ) ≜ 1 m m∑ i=1 ℓ ( h ◦ gM ( xi ) , yi ) ( 3 ) s.t . h ∈ H , gM ∈ GM . ( 4 ) where ℓ ( · , · ) is the loss function . Given r̂ ( h ◦ gM ) , we similarly define its corresponding population risk as r ( h ◦ gM ) = E ( xi , yi ) ∼D [ r̂ ( h ◦ gM ) ] ( 5 ) Similar to [ 1 , 41 ] , we use the population risk to measure the performance of learning . Example . As a concrete example of our model , consider the video classification problem under the late-fusion model in [ 43 ] . In this case , each modality k , e.g . RGB frames , audio or optical flows , is encoded by a deep network φk , and their features are fused and passed to a classifier C. If we train on the first M modalities , we can letM = [ M ] . Then gM has the form : φ1 ⊕φ2 ⊕ · · · ⊕φM , where ⊕ denotes a fusion operation , e.g . self-attention ( Z is the output of gM ) , and h is the classifier C. More examples are provided in Appendix B . Why Composite Framework ? Note that the composite multi-modal framework is often observed in applications . In fact , in recent years , a large number of papers , e.g. , [ 3 , 12 , 14 , 44 , 43 , 24 ] , appear to have utilized this framework in one way or another , even though the contributors did not clearly summarize the relationship between their methods and this common underlying structure . However , despite the popularity of the framework , existing works lack a very formal definition in theory . What is Special about Multi-modal Learning ? For the multi-modal representation x : = ( x ( 1 ) , · · · , x ( K ) ) consists of K modalities , we allow the dimension of the domain set of each modality X ( k ) to be different , which well models the source of heterogeneity of each modality . The relationships across different modalities are usually of varying levels due to the heterogeneous sources . Therefore , compared to previous works [ 45 , 48 ] , we make no assumptions on the relationship across every single modality in our analysis , which makes it general to allow different correlations . Moreover , the main assumption behind previous analysis [ 37 , 39 , 42 ] is that each view/modality contains sufficient information for target tasks , which does not shed light on our analysis . It may not hold in multi-modal applications [ 47 ] , e.g. , in object detecting task , it is known that depth images are with lower accuracy than RGB images [ 18 ] .

Summary: This work introduces a generalization bound for multimodal learning, which shows that, for a sufficiently large sample size, the model risk decreases with an increasing number of considered modalities. The authors also discuss a novel principle to dictate whether new modalities can be beneficial as a function of the dataset size and considered function classes. An empirical evaluation on both toy and real-world data validates the theory, underlining the benefits of learning from different modalities and laying the foundations for future research.