What Makes Multi-Modal Learning Better than Single (Provably)

Can multi-modal learning provably perform better than uni-modal ? In this paper , we answer this question under a most popular multi-modal fusion framework , which firstly encodes features from different modalities into a common latent space and seamlessly maps the latent representations into the task space . We prove that learning with multiple modalities achieves a smaller population risk than only using its subset of modalities . The main intuition is that the former has a more accurate estimate of the latent space representation . To the best of our knowledge , this is the first theoretical treatment to capture important qualitative phenomena observed in real multi-modal applications from the generalization perspective . Combining with experiment results , we show that multi-modal learning does possess an appealing formal guarantee . 1 Introduction . Our perception of the world is based on different modalities , e.g . sight , sound , movement , touch , and even smell [ 36 ] . Inspired from the success of deep learning [ 26 , 21 ] , deep multi-modal research is also activated , which covers fields like audio-visual learning [ 8 , 43 ] , RGB-D semantic segmentation [ 35 , 20 ] and Visual Question Answering [ 17 , 2 ] . While deep multi-modal learning shows excellent power in practice , theoretical understanding of deep multi-modal learning is limited . Some recent works have been done towards this direction [ 39 , 48 ] . However , these works made strict assumptions on the probability distributions across different modalities , which may not hold in real-life applications [ 30 ] . Notably , they do not take generalization performance of multi-modal learning into consideration . Toward this end , the following fundamental problem remains largely open : ∗equal contribution †Correspondence to : longbohuang @ tsinghua.edu.cn 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Can multi-modal learning provably performs better than uni-modal ? In this paper , we provably answer this question from two perspectives : • ( When ) Under what conditions multi-modal performs better than uni-modal ? • ( Why ) What results in the performance gains ? The framework we study is abstracted from the multi-modal fusion approaches , which is one of the most researched topics of multi-modal learning [ 3 ] . Specifically , we first encode the complex data from heterogeneous sources into a common latent space Z . The true latent representation is g⋆ in a function class G , and the task mapping h⋆ is contained in a function classH defined on the latent space . Our model corresponds to the recent progress of deep multi-modal learning on various applications , such as scene classification [ 9 ] and action recognition [ 24 , 43 ] . Under this composite framework , we provide the first theoretical analysis to shed light on what makes multi-modal outperform uni-modal from the generalization perspective . We identify the relationship between the population risk and the distance between a learned latent representation ĝ and the g⋆ , under the metric we will define later . Informally , closer to the true representation leads to less population loss , which indicates that a better latent representation guarantees the end-to-end multimodal learning performance . Instead of simply considering the comparison of multi vs uni modalities , we consider a general case , M vs N modalities , which are distinct subsets of all modalities . We focus on the condition that the latter is a subset of the former . Our second result is a bound for the closeness between ĝ and the g⋆ , from which we provably show that the latent representation ĝM learning from theM modalities is closer to the true g⋆ than ĝN learning from N modalities . As shown in Figure 1 , ĝM has a more sufficient latent space exploration than ĝN . Moreover , in a specific linear regression model , we directly verify that using multiple modalities rather than its subset learns a better latent representation . The main contributions of this paper are summarized as follows : • We formalize the multi-modal learning problem into a theoretical framework . Firstly , we show that the performance of multi-modal learning in terms of population risk can be bounded by the latent representation quality , a novel metric we propose to measure the distance from a learned latent representation to the true representation , which reveals that the ability of learning the whole task coincides with the ability of learning the latent representation when we have sufficient training samples . • We derive an upper bound for the latent representation quality of training over a subset of modalities . This directly implies a principle to guide us in modality selection , i.e. , when the number of sample size is large and multiple modalities can efficiently optimize the empirical risk , using multi-modal to build a recognition or detection system can have a better performance . • Restricted to linear latent and task mapping , we provide rigorous theoretical analysis that latent representation quality degrades when the subset of multiple modalities is applied . Experiments are also carried out to empirically validate the theoretical observation that ĝN is inferior to ĝM . The rest of the paper is organized as follows . In the next section , we review the related literature . The formulation of multi-modal learning problem is described in Section 3 . Main results are presented in Section 4 . In Section 5 , we show simulation results to support our theoretical claims . Additional discussions about the inefficiency of multi-modal learning are presented in Section 6 . Finally , conclusions are drawn in Section 7 . 2 Related Work . Multi-modal Learning Applications Deep learning makes fusing different signals easier , which enables us to develop many multi-modal frameworks . For example , [ 35 , 23 , 22 , 31 , 20 ] combine RGB and depth images to improve semantic segmentation ; [ 8 , 16 ] fuse audio with video to do scene understanding ; researchers also explore audio-visual source separation and localization [ 49 , 11 ] . Theory of Multi-modal Learning On the theory side , in semi-supervised setting , [ 39 ] proposed a novel method , Total Correlation Gain Maximization ( TCGM ) , and theoretically proved that TCGM can find the groundtruth Bayesian classifier given each modality . Moreover , CPM-Nets [ 48 ] showed multi-view representation can recover the same performance as only using the single-view observation by constructing the versatility . However , they made strict assumptions on the relationship across different modalities , while our analysis does not require such additional assumptions . Besides , previous multi-view analysis [ 37 , 1 , 45 , 13 ] typically assumes that each view alone is sufficient to predict the target accurately , which may not hold in our multi-modal setting . For instance , it is difficult to build a classifier just using a weak modality with limited labeled data , e.g. , depth modality in RGB-D images for object detection task [ 18 ] . Transfer Learning A line of work closely related to our composite learning framework is transfer learning via representation learning , which firstly learns a shared representation on various tasks and then transfers the learned representation to a new task . [ 40 , 7 , 32 , 10 ] have provided the sample complexity bounds in the special case of the linear feature and linear task mapping . [ 41 ] introduces a new notion of task diversity and provides a generalization bound with general tasks , features , and losses . Unfortunately , the function class which contains feature mappings is the same across all different tasks while our focus is that the function classes generated by different subsets modalities are usually inconsistent . Notation : Throughout the paper , we use ∥ · ∥ to denote the ℓ2 norm . We also denote the set of positive integer numbers less or equal than n by [ n ] , i.e . [ n ] ≜ { 1 , 2 , · · · , n } . 3 The Multi-modal Learning Formulation . In this section , we present the Multi-modal Learning problem formulation . Specifically , we assume that a given data x : = ( x ( 1 ) , · · · , x ( K ) ) consists of K modalities , where x ( k ) ∈ X ( k ) the domain set of the k-th modality . Denote X = X ( 1 ) × · · · × X ( K ) . We use Y to denote the target domain and use Z to denote a latent space . Then , we denote g⋆ : X 7→ Z the true mapping from the input space ( using all of K modalities ) to the latent space , and h⋆ : Z 7→ Y is the true task mapping . For instance , in aggregation-based multi-modal fusion , g⋆ is an aggregation function compounding on K seperate sub-networks and h⋆ is a multi-layer neural network [ 44 ] . In the learning task , a data pair ( x , y ) ∈ X × Y is generated from an unknown distribution D , such that PD ( x , y ) ≜ Py|x ( y | h⋆ ◦ g⋆ ( x ) ) Px ( x ) ( 1 ) Here h⋆ ◦ g⋆ ( x ) = h⋆ ( g⋆ ( x ) ) represents the composite function of h⋆ and g⋆ . In real-world settings , we often face incomplete multi-modal data , i.e. , some modalities are not observed . To take into account this situation , we let M be a subset of [ K ] , and without loss of generality , focus on the learning problem only using the modalities in M. Specifically , define X ′ : = ( X ( 1 ) ∪ { ⊥ } ) × . . .× ( X ( K ) ∪ { ⊥ } ) as the extension of X , where x′ ∈ X ′ , x′k =⊥ means that the k-th modality is not used ( collected ) . Then we define a mapping pM from X to X ′ induced byM : pM ( x ) ( k ) = { x ( k ) if k ∈M ⊥ else Also define p′M : X ′ 7→ X ′ as the extension of pM . Let G′ denote a function class , which contains the mapping from X ′ to the latent space Z , and define a function class GM as follows : GM ≜ { gM : X 7→ Z | gM ( x ) : = g′ ( pM ( x ) ) , g′ ∈ G′ } ( 2 ) Given a data set S = ( ( xi , yi ) ) mi=1 , where ( xi , yi ) is drawn i.i.d . from D , the learning objective is , following the Empirical Risk Minimization ( ERM ) principle [ 27 ] , to find h ∈ H and gM ∈ GM to jointly minimize the empirical risk , i.e. , min r̂ ( h ◦ gM ) ≜ 1 m m∑ i=1 ℓ ( h ◦ gM ( xi ) , yi ) ( 3 ) s.t . h ∈ H , gM ∈ GM . ( 4 ) where ℓ ( · , · ) is the loss function . Given r̂ ( h ◦ gM ) , we similarly define its corresponding population risk as r ( h ◦ gM ) = E ( xi , yi ) ∼D [ r̂ ( h ◦ gM ) ] ( 5 ) Similar to [ 1 , 41 ] , we use the population risk to measure the performance of learning . Example . As a concrete example of our model , consider the video classification problem under the late-fusion model in [ 43 ] . In this case , each modality k , e.g . RGB frames , audio or optical flows , is encoded by a deep network φk , and their features are fused and passed to a classifier C. If we train on the first M modalities , we can letM = [ M ] . Then gM has the form : φ1 ⊕φ2 ⊕ · · · ⊕φM , where ⊕ denotes a fusion operation , e.g . self-attention ( Z is the output of gM ) , and h is the classifier C. More examples are provided in Appendix B . Why Composite Framework ? Note that the composite multi-modal framework is often observed in applications . In fact , in recent years , a large number of papers , e.g. , [ 3 , 12 , 14 , 44 , 43 , 24 ] , appear to have utilized this framework in one way or another , even though the contributors did not clearly summarize the relationship between their methods and this common underlying structure . However , despite the popularity of the framework , existing works lack a very formal definition in theory . What is Special about Multi-modal Learning ? For the multi-modal representation x : = ( x ( 1 ) , · · · , x ( K ) ) consists of K modalities , we allow the dimension of the domain set of each modality X ( k ) to be different , which well models the source of heterogeneity of each modality . The relationships across different modalities are usually of varying levels due to the heterogeneous sources . Therefore , compared to previous works [ 45 , 48 ] , we make no assumptions on the relationship across every single modality in our analysis , which makes it general to allow different correlations . Moreover , the main assumption behind previous analysis [ 37 , 39 , 42 ] is that each view/modality contains sufficient information for target tasks , which does not shed light on our analysis . It may not hold in multi-modal applications [ 47 ] , e.g. , in object detecting task , it is known that depth images are with lower accuracy than RGB images [ 18 ] .

Summary: This paper aims to provide a theoretical explanation on whether multimodal learning can perform better than unimodal learning. They study a popular multimodal learning framework which firstly encodes features from different modalities into a common latent space before mapping the latent representations into a label. They prove that under certain conditions multimodal learning enables better estimation of the latent space which then enables better task performance. They also show some experiment results to back up their theory.