What Makes Multi-modal Learning Better than Single (Provably)
Year: 2021

Abstract:
The world provides us with data of multiple modalities. Intuitively, models fusing data from different modalities outperform their uni-modal counterparts, since more information is aggregated. Recently, joining the success of deep learning, there is an influential line of work on deep multi-modal learning, which has remarkable empirical results on various applications. However, theoretical justifications in this field are notably lacking. Can multi-modal learning provably perform better than uni-modal? In this paper, we answer this question under a most popular multi-modal fusion framework, which firstly encodes features from different modalities into a common latent space and seamlessly maps the latent representations into the task space. We prove that learning with multiple modalities achieves a smaller population risk than only using its subset of modalities. The main intuition is that the former has a more accurate estimate of the latent space representation. To the best of our knowledge, this is the first theoretical treatment to capture important qualitative phenomena observed in real multi-modal applications from the generalization perspective. Combining with experiment results, we show that multi-modal learning does possess an appealing formal guarantee.

text: Can multi-modal learning provably perform better than uni-modal?
In this paper, we answer this question under a most popular multi-modal fusion framework, which firstly encodes features from different modalities into a common latent space and seamlessly maps the latent representations into the task space. We prove that learning with multiple modalities achieves a smaller population risk than only using its subset of modalities. The main intuition is that the former has a more accurate estimate of the latent space representation. To the best of our knowledge, this is the first theoretical treatment to capture important qualitative phenomena observed in real multi-modal applications from the generalization perspective. Combining with experiment results, we show that multi-modal learning does possess an appealing formal guarantee.

heading: 1 Introduction
text: Our perception of the world is based on different modalities, e.g. sight, sound, movement, touch, and even smell [36]. Inspired from the success of deep learning [26, 21], deep multi-modal research is also activated, which covers fields like audio-visual learning [8, 43], RGB-D semantic segmentation [35, 20] and Visual Question Answering [17, 2].
While deep multi-modal learning shows excellent power in practice, theoretical understanding of deep multi-modal learning is limited. Some recent works have been done towards this direction [39, 48]. However, these works made strict assumptions on the probability distributions across different modalities, which may not hold in real-life applications [30]. Notably, they do not take generalization performance of multi-modal learning into consideration. Toward this end, the following fundamental problem remains largely open:
∗equal contribution †Correspondence to: longbohuang@tsinghua.edu.cn
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Can multi-modal learning provably performs better than uni-modal?
In this paper, we provably answer this question from two perspectives:
• (When) Under what conditions multi-modal performs better than uni-modal?
• (Why) What results in the performance gains ?
The framework we study is abstracted from the multi-modal fusion approaches, which is one of the most researched topics of multi-modal learning [3]. Specifically, we first encode the complex data from heterogeneous sources into a common latent space Z . The true latent representation is g⋆ in a function class G, and the task mapping h⋆ is contained in a function classH defined on the latent space. Our model corresponds to the recent progress of deep multi-modal learning on various applications, such as scene classification [9] and action recognition [24, 43].
Under this composite framework, we provide the first theoretical analysis to shed light on what makes multi-modal outperform uni-modal from the generalization perspective. We identify the relationship between the population risk and the distance between a learned latent representation ĝ and the g⋆, under the metric we will define later. Informally, closer to the true representation leads to less population loss, which indicates that a better latent representation guarantees the end-to-end multimodal learning performance. Instead of simply considering the comparison of multi vs uni modalities, we consider a general case,M vs N modalities, which are distinct subsets of all modalities. We focus on the condition that the latter is a subset of the former. Our second result is a bound for the closeness between ĝ and the g⋆, from which we provably show that the latent representation ĝM learning from theM modalities is closer to the true g⋆ than ĝN learning from N modalities. As shown in Figure 1, ĝM has a more sufficient latent space exploration than ĝN . Moreover, in a specific linear regression model, we directly verify that using multiple modalities rather than its subset learns a better latent representation.
The main contributions of this paper are summarized as follows:
• We formalize the multi-modal learning problem into a theoretical framework. Firstly, we show that the performance of multi-modal learning in terms of population risk can be bounded by the latent representation quality, a novel metric we propose to measure the distance from a learned latent representation to the true representation, which reveals that the ability of learning the whole task coincides with the ability of learning the latent representation when we have sufficient training samples.
• We derive an upper bound for the latent representation quality of training over a subset of modalities. This directly implies a principle to guide us in modality selection, i.e., when the number of sample size is large and multiple modalities can efficiently optimize the empirical risk, using multi-modal to build a recognition or detection system can have a better performance.
• Restricted to linear latent and task mapping, we provide rigorous theoretical analysis that latent representation quality degrades when the subset of multiple modalities is applied.
Experiments are also carried out to empirically validate the theoretical observation that ĝN is inferior to ĝM.
The rest of the paper is organized as follows. In the next section, we review the related literature. The formulation of multi-modal learning problem is described in Section 3. Main results are presented in Section 4. In Section 5, we show simulation results to support our theoretical claims. Additional discussions about the inefficiency of multi-modal learning are presented in Section 6. Finally, conclusions are drawn in Section 7.

heading: 2 Related Work
text: Multi-modal Learning Applications Deep learning makes fusing different signals easier, which enables us to develop many multi-modal frameworks. For example, [35, 23, 22, 31, 20] combine RGB and depth images to improve semantic segmentation; [8, 16] fuse audio with video to do scene understanding; researchers also explore audio-visual source separation and localization [49, 11].
Theory of Multi-modal Learning On the theory side, in semi-supervised setting, [39] proposed a novel method, Total Correlation Gain Maximization (TCGM), and theoretically proved that TCGM can find the groundtruth Bayesian classifier given each modality. Moreover, CPM-Nets [48] showed multi-view representation can recover the same performance as only using the single-view observation by constructing the versatility. However, they made strict assumptions on the relationship across different modalities, while our analysis does not require such additional assumptions. Besides, previous multi-view analysis [37, 1, 45, 13] typically assumes that each view alone is sufficient to predict the target accurately, which may not hold in our multi-modal setting. For instance, it is difficult to build a classifier just using a weak modality with limited labeled data, e.g., depth modality in RGB-D images for object detection task [18].
Transfer Learning A line of work closely related to our composite learning framework is transfer learning via representation learning, which firstly learns a shared representation on various tasks and then transfers the learned representation to a new task. [40, 7, 32, 10] have provided the sample complexity bounds in the special case of the linear feature and linear task mapping. [41] introduces a new notion of task diversity and provides a generalization bound with general tasks, features, and losses. Unfortunately, the function class which contains feature mappings is the same across all different tasks while our focus is that the function classes generated by different subsets modalities are usually inconsistent.
Notation: Throughout the paper, we use ∥ · ∥ to denote the ℓ2 norm. We also denote the set of positive integer numbers less or equal than n by [n], i.e. [n] ≜ {1, 2, · · · , n}.

heading: 3 The Multi-modal Learning Formulation
text: In this section, we present the Multi-modal Learning problem formulation. Specifically, we assume that a given data x := ( x(1), · · · , x(K) ) consists of K modalities, where x(k) ∈ X (k) the domain
set of the k-th modality. Denote X = X (1) × · · · × X (K). We use Y to denote the target domain and use Z to denote a latent space. Then, we denote g⋆ : X 7→ Z the true mapping from the input space (using all of K modalities) to the latent space, and h⋆ : Z 7→ Y is the true task mapping. For instance, in aggregation-based multi-modal fusion, g⋆ is an aggregation function compounding on K seperate sub-networks and h⋆ is a multi-layer neural network [44].
In the learning task, a data pair (x, y) ∈ X × Y is generated from an unknown distribution D, such that
PD(x, y) ≜ Py|x (y | h⋆ ◦ g⋆(x))Px(x) (1)
Here h⋆ ◦ g⋆(x) = h⋆(g⋆(x)) represents the composite function of h⋆ and g⋆. In real-world settings, we often face incomplete multi-modal data, i.e., some modalities are not observed. To take into account this situation, we let M be a subset of [K], and without loss of generality, focus on the learning problem only using the modalities in M. Specifically, define
X ′ := ( X (1) ∪ {⊥} ) × . . .× ( X (K) ∪ {⊥} ) as the extension of X , where x′ ∈ X ′, x′k =⊥ means that the k-th modality is not used (collected). Then we define a mapping pM from X to X ′ induced byM:
pM(x) (k) = { x(k) if k ∈M ⊥ else
Also define p′M : X ′ 7→ X ′ as the extension of pM. Let G′ denote a function class, which contains the mapping from X ′ to the latent space Z , and define a function class GM as follows:
GM ≜ {gM : X 7→ Z | gM(x) := g′(pM(x)), g′ ∈ G′} (2)
Given a data set S = ((xi, yi))mi=1, where (xi, yi) is drawn i.i.d. from D, the learning objective is, following the Empirical Risk Minimization (ERM) principle [27], to find h ∈ H and gM ∈ GM to jointly minimize the empirical risk, i.e.,
min r̂ (h ◦ gM) ≜ 1
m m∑ i=1 ℓ (h ◦ gM(xi), yi) (3)
s.t. h ∈ H, gM ∈ GM. (4) where ℓ(·, ·) is the loss function. Given r̂ (h ◦ gM), we similarly define its corresponding population risk as
r (h ◦ gM) = E(xi,yi)∼D [r̂ (h ◦ gM)] (5) Similar to [1, 41], we use the population risk to measure the performance of learning.
Example. As a concrete example of our model, consider the video classification problem under the late-fusion model in [43]. In this case, each modality k, e.g. RGB frames, audio or optical flows, is encoded by a deep network φk, and their features are fused and passed to a classifier C. If we train on the first M modalities, we can letM = [M ]. Then gM has the form: φ1 ⊕φ2 ⊕ · · · ⊕φM , where ⊕ denotes a fusion operation, e.g. self-attention (Z is the output of gM), and h is the classifier C. More examples are provided in Appendix B.
Why Composite Framework ? Note that the composite multi-modal framework is often observed in applications. In fact, in recent years, a large number of papers, e.g., [3, 12, 14, 44, 43, 24], appear to have utilized this framework in one way or another, even though the contributors did not clearly summarize the relationship between their methods and this common underlying structure. However, despite the popularity of the framework, existing works lack a very formal definition in theory.
What is Special about Multi-modal Learning? For the multi-modal representation x :=( x(1), · · · , x(K) ) consists of K modalities, we allow the dimension of the domain set of each
modality X (k) to be different, which well models the source of heterogeneity of each modality. The relationships across different modalities are usually of varying levels due to the heterogeneous sources. Therefore, compared to previous works [45, 48], we make no assumptions on the relationship across every single modality in our analysis, which makes it general to allow different correlations. Moreover, the main assumption behind previous analysis [37, 39, 42] is that each view/modality contains sufficient information for target tasks, which does not shed light on our analysis. It may not hold in multi-modal applications [47], e.g., in object detecting task, it is known that depth images are with lower accuracy than RGB images [18].

heading: 4 Main Results
text: In this section, we provide main theoretical results to rigorously establish various aspects of the folklore claim that multi-modal is better than single. We first detail several assumptions throughout this section. Assumption 1. The loss function ℓ(·, ·) is L-smooth with respect to the first coordinate, and is bounded by a constant C. Assumption 2. The true latent representation g⋆ is contained in G, and the task mapping h⋆ is contained inH.
Assumption 1 is a classical regularity condition for loss function in theoretical analysis [27, 41, 40]. Assumption 2 is also be known as realizability condition in representation learning [41, 10, 40], which ensures that the function class that we optimize over contains the true latent representation and the task mapping. Assumption 3. For any g′ ∈ G′ andM⊂ [K], g′ ◦ p′M ∈ G′.
To understand Assumption 3, note that for any N ⊂ M ⊂ [K], by definition, for any gN ∈ GN , there exists g′ ∈ G′, s.t.
gN (x) = g ′(pN (x)) = g ′(p′N (pM(x)))
Therefore, Assumption 3 directly implies gN ∈ GM. Moreover, we have GN ⊂ GM ⊂ G, which means that the inclusion relationship of modality subsets remains unchanged on the latent function class induced by them. As an example, if G′ is linear, represented as matrix G ∈ RQ×K . Also p′M can be represented as a diagonal matrix P ∈ RK×K with the i-th diagonal entry being 1 for i ∈M and 0 otherwise. In this case, Assumption 3 holds, i.e. G×P ∈ G′. Moreover, G×P is a matrix with i-th column all be zero for i /∈M, which is commonly used in the underfitting analysis in linear regression [34].

heading: 4.1 Connection to Latent Representation Quality
text: Latent space is employed to better exploit the correlation among different modalities. Therefore, we will naturally conjecture that the performance of training with different modalities is related to its ability to learn latent space representation. In this section, we will formally characterize this relationship.
In order to measure the goodness of a learned latent representation g, we introduce the following definition of latent representation quality.
Definition 1. Given a data distribution with the form in (1), for any learned latent representation mapping g ∈ G, the latent representation quality is defined as
η(g) = inf h∈H
[r (h ◦ g)− r(h∗ ◦ g∗)] (6)
Here infh∈H r (h ◦ g) is the best achievable population risk with the fixed latent representation g. Thus, to a certain extent, η(g) measures the loss incurred by the distance between g and g⋆.
Next, we recap the Rademacher complexity measure for model complexity. It will be used in quantifying the population risk performance based on different modalities. Specifically, let F be a class of vector-valued function Rd 7→ Rn. Let Z1, . . . , Zm be i.i.d. random variables on Rd following some distribution P. Denote the sample S = (Z1, . . . , Zm). The empirical Rademacher complexity of F with respect to the sample S is given by [5]
R̂S(F) := Eσ [ sup f∈F 1 m m∑ i=1 σif (Zi) ] where σ = (σ1, . . . , σn) ⊤ with σi iid∼ unif {−1, 1}. The Rademacher complexity of F is
Rm(F) = ES [ R̂S(F) ] Now we present our first main result regarding multi-modal learning. Theorem 1. Let S = ((xi, yi))mi=1 be a dataset of m examples drawn i.i.d. according to D. Let M,N be two distinct subsets of [K]. Assuming we have produced the empirical risk minimizers (ĥM, ĝM) and (ĥN , ĝN ), training with theM andN modalities separately. Then, for all 1 > δ > 0, with probability at least 1− δ2 :
r ( ĥM ◦ ĝM ) − r ( ĥN ◦ ĝN ) ≤ γS(M,N ) + 8LRm(H ◦ GM) +
4C√ m + 2C
√ 2 ln(2/δ)
m (7)
where γS(M,N ) ≜ η(ĝM)− η(ĝN ) □ (8)
Remark. A few remarks are in place. First of all, γS(M,N ) defined in (8) compares the quality between latent representations learning fromM andN modalities with respect to the given dataset S . Theorem 1 bounds the difference of population risk training with two different subsets of modalities by γS(M,N ), which validates our conjecture that including more modalities is advantageous in learning. Second, for the commonly used function classes in the field of machine learning, Radamacher complexity for a sample of size m, Rm(F) is usually bounded by √ C(F)/m, where C(F) represents the intrinsic property of function class F . Third, (7) can be written as γS(M,N ) + O( √ 1 m ) in order terms. This shows that as the number of sample size grows, the performance of using different modalities mainly depends on its latent representation quality.

heading: 4.2 Upper Bound for Latent Space Exploration
text: Having establish the connection between the population risk difference with latent representation quality, our next goal is to estimate how close the learned latent representation ĝM is to the true latent representation g⋆. The following theorem shows how the latent representation quality can be controlled in the training process.
Theorem 2. Let S = {(xi, yi)}mi=1 be a dataset of m examples drawn i.i.d. according to D. LetM be a subset of [K]. Assuming we have produced the empirical risk minimizers (ĥM, ĝM) training with theM modalities. Then, for all 1 > δ > 0, with probability at least 1− δ:
η(ĝM) ≤ 4LRm(H ◦ GM) + 4LRm(H ◦ G) + 6C √ 2 ln(2/δ)
m + L̂(ĥM ◦ ĝM,S) (9)
where L̂(ĥM ◦ ĝM,S) ≜ r̂ ( ĥM ◦ ĝM ) − r̂ (h⋆ ◦ g⋆) is the centered empirical loss. □
Remark. Consider sets N ⊂M ⊂ [K]. Under Assumption 3, GN ⊂ GM ⊂ G, optimizing over a larger function class results in a smaller empirical risk. Therefore
L̂(ĥM ◦ ĝM,S) ≤ L̂(ĥN ◦ ĝN ,S) (10)
Similar to the analysis in Theorem 1, the first term on the Right-hand Side (RHS), Rm(H ◦ GM) ∼√ C(H ◦ GM)/m and Rm(H ◦ GN ) ∼ √ C(H ◦ GN )/m. Following the basic structural property of Radamacher complexity [5], we have C(H ◦ GN ) ≤ C(H ◦ GM). Therefore, Theorem 2 offers the following principle for choosing modalities to improve the latent representation quality.
Principle: choose to learn with more modalities if: L̂(ĥN ◦ ĝN ,S)− L̂(ĥM ◦ ĝM,S) ≥ √
C(H ◦ GM) m
− √
C(H ◦ GN ) m
What this principle implies are twofold. (i) When the number of sample size m is large, the impact of intrinsic complexity of function classes will be reduced. (ii) Using more modalities can efficiently optimize the empirical risk, hence improve the latent representation quality.
Through the trade-off illustrated in the above principle, we provide theoretical evidence that when N ⊂ M and training samples are sufficient, η(ĝM) may be less than η(ĝN ), i.e.γS(M,N ) ≤ 0. Moreover, combining with the conclusion from Theorem 1, if the sample size m is large enough, γS(M,N ) ≤ 0 guarantees r ( ĥM ◦ ĝM ) ≤ r ( ĥN ◦ ĝN ) , which indicates learning with theM
modalities outperforms only using its subset N modalities.
Role of the intrinsic property C(·) Hypothesis function classes are typically overparametrized in deep learning, and will be extremely large for some typical measures, e.g., VC dimension, absolute dimension. Some recent efforts aim to offer an explanation about why neural networks generalize better with over-parametrization [28, 29, 4]. [28] suggest a novel complexity measure based on unit-wise capacities, which implies if both in overparametrized settings, the complexity will not change much or even decrease when we have more modalities (using more parameters). Thus, the inequality in the principle is trivially satisfied, and we will choose to learn with more modalities.

heading: 4.3 Non-Positivity Guarantee
text: In this section, we focus on a composite linear data generating model to theoretically verify that the γS(M,N ) is indeed non-positive in this special case.3Specifically, we consider the case where the mapping to the latent space and the task mapping are both linear. Formally, let the function class G andH be:
G = { g | g(x) = A⊤x,A ∈ Rd×n,A } H = { h | h(z) = β⊤z,β ∈ Rn, ∥β∥ ≤ Cb
} (11) where x = ( x(1), · · · ,x(K) ) is a d-dimensional vector, x(k) ∈ Rdk denotes the feature vector for
the k-th modality and ∑K
k=1 dk = d. Here, the distribution Px(·) satisfies that its covariance matrix is positive definite. The data is generated by:
y = (β⋆) ⊤ A⋆⊤x+ ϵ (12)
where r.v. ϵ is independent of x and has zero-mean and bounded second moment. Note that in practical multi-modal learning, usually only one layer is linear and the other is a neural network. For instance, [50] employs the linear matrix to project the feature matrix from different modalities into a common latent space for early dementia diagnosis, i.e., G is linear. Another example is in pedestrian detection [46], where a linear task mapping is adopted, i.e.,H is linear. Thus, our composite linear model can be viewed as an approximation to such popular models, and our results can offer insights into the performance of these models.
We consider a special case that M = [K] and N = [K − 1]. Thus GM = G and we have the following result.
GN = { g | g(x) = [ A1: ∑K−1 k=1 dk
0
]⊤ x,A ∈ Rd×n with orthonormal columns } (13)
Proposition 1. Consider the dataset S = {(xi, yi)}mi=1 generating from the linear model defined in (12) with ℓ2 loss. LetM = [K] and N = [K − 1]. Let ÂM, ÂN denote the projection matrix estimated by M, N modalities. Assume that ÂM, A⋆ has orthonormal columns. If n = d, for sufficiently large constant Cb, we have:
γS(M,N ) ≤ 0 (14)
In this special case, Proposition 1 directly guarantees that training with incomplete modalities weakens the ability to learn a optimal latent representation. As a result, it also degrades the learning performance.

heading: 5 Experiment
text: We conduct experiments to validate our theoretical results. The source of the data we consider is two-fold, multi-modal real-world dataset and well-designed generated dataset.

heading: 5.1 Real-world dataset
text: Dataset. The natural dataset we use is the Interactive Emotional Dyadic Motion Capture (IEMOCAP) database, which is an acted multi-modal and multi-speaker database [6]. It contains three modalities, Text, Video and Audio. We follow the data preprocessing method of [33] and obtain 100 dimensions data for audio, 100 dimensions for text, and 500 dimensions for video. There are six labels here, namely, happy, sad, neutral, angry, excited and frustrated. We use 13200 data for training and 3410 for testing.
Training Setting. For all experiments on IEMOCAP, we use one linear neural network layer to extract the latent feature, and we set the hidden dimension to be 128. In multi-modal network, different modalities do not share encoders and we concatenate the features first, and then map the
3Proving that γS(M,N ) ≤ 0 holds in general is open and will be an interesting future work.
feature to the task space. We use Adam [25] as the optimizer and set the learning rate to be 0.01, with other hyper-parameters default. The batch size is 2048 for the data. For this classification task, the top-1 accuracy is used for performance measurement. We use naively multi-modal late-fusion training as our framework [43, 9]. In Appendix C, we provide more discussions on stable multi-modal training.
Connection to the Latent Representation Quality. The classification accuracy on IEMOCAP, using different combinations of modalities are summarized in Table 1. All learning strategies using multiple modalities outperform the single modal baseline. To validate Theorem 1, we calculate the test accuracy difference between different subsets of modalities using the result in Table 1 and show them in the third column of Table 3.
Moreover, we empirically evaluate the η(ĝM) in the following way: freeze the encoder ĝM obtained through pretraining and then finetune to obtain a better classifier h. Having η(ĝM) and η(ĝN ), the values of γS(M,N ) between different subsets of modalities are also presented in Table 3. Previous discussions on Theorem 1 implies that the population risk difference betweenM and N modalities has the same sign as γS(M,N ) when the sample size is large enough, and negativity implies performance gains. Since we use accuracy as the measure, on the contrary, positivity indicates a better performance in our settings. As shown in Table 3, when more modalities are added for learning, the test accuracy difference and γS(M,N ) are both positive, which confirms the important role of latent representation quality characterized in Theorem 1.
Upper Bound for Latent Space Exploration. Table 3 also confirms our theoretical analysis in Theorem 2. In all cases, the N modalities is a subset of M modalities, and correspondingly, a positive γS(M,N ) is observed. This indicates thatM modalities has a more sufficient latent space exploration than its subset N modalities.
We also attempt to understand the use of sample size for exploring the latent space. Table 2 presents the latent representation quality η obtained by using different numbers of sample size, which is measured by the test accuracy of the pretrain+finetuned modal. Here, the ratio of sample size is set to the total number of training samples. The corresponding curve is also ploted in Figure 2(a). As the number of sample size grows, the increase in performance of η is observed, which is in keeping with the O( √ 1/m) term in our upper bound for η. The phenomenon that the combination of Text, Video and Audio (TVA) modalities underperforms the uni-modal when the number of sample size is relatively small, can also be interpreted by the trade-off we discussed in Theorem 2. When there are insufficient training examples, the intrinsic complexity of the function class induced by multiple modalities dominates, thus weakening its latent representation quality.

heading: 5.2 Synthetic Data
text: In this subsection, we investigate the effect of modality correlation on latent representation quality. Typically, there are three situations for the correlation across each modality [39]. (i) Each modality does not share information at all, that is, each modality only contains modal-specific information. (ii) The other is that, all modalities only maintain the share information without unique information on their own. (iii) The last is a moderate condition, i.e., each modal not only shares information, but also owns modal-specific information. The reason to utilize simulated data in this section is due to the fact that it is hard in practice to have natural datasets that possess the required degree of modality correlation.
Data Generation. For the synthetic data, we have four modalities, denoted by m1,m2,m3,m4, respectively, and the generation process is summarized as follows:
Step 1: Generate mi ∼ N (0, I), i = 1, 2, 3, 4, where mi is i.i.d. 100-dimensional random vector; Step 2: Generate mi ← (1− w) ·mi + w ·m1for i = 2, 3, 4 Step 3: Generate the labels as follows. First, add the four modality vectors and calculate the
sum of coordinates. Then, obatin the 1-dimensional label, i.e. y = (m1 + m2 + m3 + m4).sum(dim = 1)
We can see that information from m1 is shared across different modalities, and w controls how much is shared, which one can use to measure the degree of overlaps. We tune it in {0.0, 0.2, 0.5, 0.8, 1.0}. A high weight w (close to 1) indicates a high degree of overlaps, while w = 0 means each modality is totally independent and is non-overlapping.
Training Setting. We use the multi-layer perceptron neural network as our model. To be more specific, we first use a linear layer to encode the input to a 10-dimension latent space and then we map the latent feature to the output space. The first layer’s input dimension depends on the number of modality. For example, if we use two modalities, the input dimension is 200. We use SGD as our
optimizer, and set a learning rate 0.01, momentum 0.9, batch size 10000, training for 10000 steps. The Mean Square Error (MSE) loss is considered for evaluation in this regression problem.
η vs Modality Correlation Our aim is to discover the influence of modality correlation on the latent representation quality η. To this end, Table 4 shows the η with the varying number of modalities under different correlation conditions, which is measured by the MSE loss of the pretrain+finetuned modal. The trend that the loss decreases as the number of modalities increases is described in Figure 2(b), which also validates our analysis of Theorem 2. Moreover, Figure 2(b) shows that higher correlation among modalities achieves a lower loss for η, which means a better latent representation. This emphasizes the role of latent space to exploit the intrinsic correlations among different modalities.

heading: 6 Discussion
text: It has been discovered that the use of multi-modal data in practice will degrade the performance of the model in some cases[43, 19, 38, 15]. These works identify the causes of performance drops of multimodal as interactions between modalities in the training stage and try to improve the performance by proposing new optimization strategies. Therefore, to theoretically understand them, we need to analyze the training process from the optimization perspective. Our results, on the other hand, mainly focus on the generalization side, which is separated from optimization and assumes that we get the best performance possible in training. Moreover, our theory is general and does not require additional assumptions on the relationship across every single modality, which may be crucial for theoretically analyzing the observations in a multi-modal performance drop. Understanding why multi-modal fails in practice is a very interesting direction and worth further investigation in future research.

heading: 7 Conclusion
text: In this work, we formulate a multi-modal learning framework that has been extensively studied in the empirical literature towards rigorously understanding why multi-modality outperforms single since the former has access to a better latent space representation. The results answer the two questions: when and why multi-modal outperforms uni-modal jointly. To the best of our knowledge, this is the first theoretical treatment to explain the superiority of multi-modal from the generalization standpoint.
The key takeaway message is that the success of multi-modal learning relies essentially on the better quality of latent space representation, which points to an exciting direction that is worth further investigation: to find which encoder is the bottleneck and focus on improving it. More importantly, our work provides new insights for multi-modal theoretical research, and we hope this can encourage more related research.
Acknowledgments and Disclosure of Funding
The work of Yu Huang and Longbo Huang is supported in part by the Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant 2020AAA0108400 and 2020AAA0108403.

Review summaries:
This paper aims to provide a theoretical explanation on whether multimodal learning can perform better than unimodal learning. They study a popular multimodal learning framework which firstly encodes features from different modalities into a common latent space before mapping the latent representations into a label. They prove that under certain conditions multimodal learning enables better estimation of the latent space which then enables better task performance. They also show some experiment results to back up their theory.

This work introduces a generalization bound for multimodal learning, which shows that, for a sufficiently large sample size, the model risk decreases with an increasing number of considered modalities. The authors also discuss a novel principle to dictate whether new modalities can be beneficial as a function of the dataset size and considered function classes. An empirical evaluation on both toy and real-world data validates the theory, underlining the benefits of learning from different modalities and laying the foundations for future research.