What Makes Multi-Modal Learning Better than Single (Provably)
Authors: Yu Huang
Chenzhuang Du
Zihui Xue
Xuanyao Chen
Hang Zhao
Longbo Huang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Multi-modal learning theory

Abstract: The world provides us with data of multiple modalities. Intuitively, models fusing data from different modalities outperform their uni-modal counterparts, since more information is aggregated. Recently, joining the success of deep learning, there is an influential line of work on deep multi-modal learning, which has remarkable empirical results on various applications. However, theoretical justifications in this field are notably lacking.
                        Can multi-modal learning provably perform better than uni-modal?
In this paper, we answer this question under a most popular multi-modal fusion framework, which firstly encodes features from different modalities into a common latent space and seamlessly maps the latent representations into the task space. We prove that learning with multiple modalities achieves a  smaller population risk than only using its subset of modalities. The main intuition is that the former has a more accurate estimate of the latent space representation. To the best of our knowledge, this is the first theoretical treatment to capture important qualitative phenomena observed in real multi-modal applications from the generalization perspective. Combining with experiment results, we show that multi-modal learning does possess an appealing formal guarantee.

What Makes Multi-modal Learning Better than Single (Provably)
Yu Huang!., Chenzhuang Dul.*, Zihui Xue?, Xuanyao Chen 3.4
Hang Zhao 1,4 Longbo Huang 1.
Institute Interdisciplinary Information Sciences. Tsinghua University 2 TTe University of Texas at Austin 3 Fudan University
Shanghai Qi Zhi Institute
Abstract
The world provides us with data of multiple modalities. Intuitively, models fusing data from different modalities outperform their uni-modal counterparts, since more information is aggregated. Recently, joining the success of deep learning, there is influential line of work on deep multi-modal learning, which has remarkable empirical results on various applications. However, theoretical justifications in this field are notably lacking.
Can multi-modal learning provably perform better than uni-modal?
In this paper, we answer this question under most popular multi-modal fusion framework, which firstly encodes features from different modalities into a common latent space and seamlessly maps the latent representations into the task space. We prove that learning with multiple modalities achieves a smaller population risk than only using its subset of modalities. The main intuition is that the former has accurate estimate of the latent space representation. To the best of our knowledge, this is the first theoretical treatment capture important qualitative phenomena observed in real multi-modal applications from the generalization perspective. Combining with experiment results, show that multi-modal learning does possess an appealing formal guarantee.
Introduction
Our perception of the world is based different modalities, e.g. sight, sound, movement, touch, and even smell [36]. Inspired from the success of deep learning [26, 21], deep multi-modal research is also activated, which covers fields like audio-visun learning [8. 43], RGB-D semantic segmentation [35, 20] and Visual Question Answering [17, 2].
While deep multi-modal learning shows excellent power in practice, theoretical understanding of deep multi-modal learning is limited. Some recent works have been done towards this direction [39, 48]. However, these works made strict assumptions the probability distributions across different modalities, which may not hold in real-life applications [30]. Notably, they do not take generalization performance of multi-modal learning into consideration. Toward this end, the following fundamental problem remains largely open:
"qqull contribution "Correspondence to: (mggxhhungg@siighuu..cch
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Latent Space
MM
Z
ON
MM
M1
M4a
N
M
Figure 1: .M vs N modalities latent space representation, where the latter is subset of the former. 2M, 2N and x* are images on the latent space corresponding to the representation mappings 9.M IN and g*, M, denotes modality i.
Can multi-modal learning provably performs better than uni uni-modal?
In this paper, we provably answer this question from two perspectives:
(When) Under what conditions multi-modal performs better than uni-modal?
(Why) What results in the performance gains
The framework study is abstracted from the multi-modal fusion approaches, which is one of the most researched topics of multi-modal learning [3]. Specifically, we first encode the complex data from heterogeneous sources into common latent space Z. The true latent representation is g" function class G, and the task mapping h* contained in a function class H defined on the latent space. Our model corresponds the recent progress of deep multi-modal learning various applications, such as scene classification [9] and action recognition [24, 43].
Under this composite framework, provide the first theoretical analysis shed light on what makes multi-modal outperform uni-modal from the generalization perspective. We identify the relationship between the population risk and the distance between learned latent representation Ò and the g*, under the metric we will define later. Informally, closer the true representation leads to less population loss, which indicates that better latent representation guarantees the end-to-end multi- modal learning performance. Instead of simply considering the comparison multi vs uni modalities, consider general case, M vs N modalities, which distinct subsets of all modalities. We focus on the condition that the latter a subset of the former. Our second result a bound for the closeness between and the g", from which we provably show that the latent representation ÿM learning from the M modalities is closer to the true g than ÔN learning from N modalities. As shown Figure 1, I.M has more sufficient latent space exploration than Åv. Moreover, specific linear regression model, we directly verify that using multiple modalities rather than its subset learns better latent representation.
The main contributions this paper are summarized as follows:
We formalize the multi-modal learning problem into theoretical framework. Firstly, we show that the performance of multi-modal learning in terms of population risk can bounded by the latent representation quality, a novel metric we propose to measure the distance from learned latent representation to the true representation, which reveals that the ability of learning the whole task coincides with the ability of learning the latent representation when we have sufficient training samples.
We derive upper bound for the latent representation quality of training over subset of modalities. This directly implies principle to guide us in modality selection, i.e., when the number of sample size large and multiple modalities can efficiently optimize the empirical risk, using multi-modal to build a recognition or detection system can have a better performance.
Restricted to linear latent and task mapping, we provide rigorous theoretical analysis that latent representation quality degrades when the subset of multiple modalities is applied.

---

Experiments are also carried out to empirically validate the theoretical observation that ON is inferior 9.
The rest of the paper is organized as follows. In the next section, we review the related literature. The formulation of multi-modal learning problem described in Section 3. Main results are presented in Section 4. In Section 5, we show simulation results to support our theoretical claims. Additional discussions about the inefficiency of multi-modal learning are presented in Section Finally, conclusions are drawn in Section 7.
2 Related Work
Multi-modal Learning Applications Deep learning makes fusing different signals easier, which enables us to develop many multi-modal frameworks. For example, [35, 23, 22, 31, 20] combine RGB and depth images to improve semantic segmentation; [8, 16] fuse audio with video to do scene understanding: researchers also explore audio-visual source separation and localization [49, 11].
Theory of Multi-modal Learning On the theory side, in semi-supervised setting, [39] proposed a novel method, Total Correlation Gain Maximization (TCGM), and theoretically proved that TCGM can find the groundtruth Bayesian classifier given each modality. Moreover, CPM-Nets [48] showed multi-view representation can recover the same performance as only using the single-view observation by constructing the versatility. However, they made strict assumptions on the relationship across different modalities, while our analysis does not require such additional assumptions. Besides, previous multi-view analysis [37, 1, 45, 13] typically assumes that each view alone is sufficient to predict the target accurately, which may not hold in our multi- modal setting. For instance, is difficult to build classifier just using weak modality with limited labeled data, e.g., depth modality in RGB-D images for object detection task [18].
Transfer Learning A line of work closely related to our composite learning framework is transfer learning via representation learning, which firstly learns a shared representation on various tasks and then transfers the learned representation complexity bounds in the special case of the linear feature and linear task mapping. [41] introduces a new notion of task diversity and provides generalization bound with general tasks, features, and losses. Unfortunately, the function class which contains feature mappings is the same across all different tasks while our focus is that the function classes generated by different subsets modalities are usually inconsistent.
new task. [40, 7, 32. 10] have provided the sample
Notation:
Throughout the paper, use || - II to denote the lg norm. We also denote the set of positive integer numbers less or equal than by [m)), i.e. [m]] A (1,2, n).
3 The Multi-modal Learning Formulation
In this section, we present the Multi-modal Learning problem formulation. Specifically, assume that a given data x = (zr(),.... x: (K) consists of K modalities, where ,((k) E x4) the domain set of the k-th modality. Denote X' (1)
(KK) We use y denote the target domain and use Z to denote a latent space. Then, we denote g* X' + Z the true mapping from the input space (using all of modalities) to the latent space, and h* 2 + y is the true task mapping. For instance, in aggregation- based multi-mode fusion, g" is an aggregation function compounding on K seperate sub-networks and h* a multi -ulti- neural network [44].
In the learning task, a data pair (x, y) E X' x y is generated from an unknown distribution D, such that
Px y) I (y h" o gx)
(1)
Here h² gx)) h² (g" (x)) represents the composite function h* and g"
In real-world settings, we often face incomplete multi-modal data, i.e., some modalities are not observed. To take into account this situation, we let M be a subset of [K], and without loss of
generality, focus the learning problem only using the modalities in M. Specifically, define
3

---

x= (x0)) .... x (x U {1)} as the extension of X', where x' E X", X =1 means that the k-th modality is not used (collected). Then we define a mapping PM from X' to X" induced by M:
[x(() i else
PM(())
Also define P'M X -> X" as the extension of PM- Let g' denote function elass, which contains the mapping from X' to the latent space 2, and define function class 9M as follows:
GM _ {am : X' - 2 (x(()) =- I' (PM(()) E
Given data set S ((x;, y4 V=)) where (xi, yi is drawn i.i.d. from D, the learning objective is, following the Empirical Risk Minimization (ERM) principle [27], to find E H and 9M E 9M to jointly minimize the empirical risk, i.e.,
m
min f(ogg)) - Σ gM))))))
(3)
m
s.t. he H1 E -M-
(4)
where ((.,) is the loss function. Given (h gM), similarly define its corresponding population risk as
(h gM) = B... (f (h gM)]
(5)
Similar to [1, 41], we use the population risk to measure the performance of learning.
Example. As concrete example of our model, consider the video classification problem under the late-fusion model in [43]. In this case, each modality k, e.g. RGB frames, audio or optical flows, is encoded by a deep network (Pk, and their features are fused and passed to a classifier C. we train on the first M modalities, denotes a fusion operation, e.g. self-attention (2 is the output of gM). and is the classifier C. More examples provided in Appendix B.
let M [M]. Then J.M has the form: y01 y22
@ pPM, where ()
Why Composite Framework? Note that the composite multi-modal framework is often observed in applications. In fact, in recent years, large number of papers, e.g., [3, 12, 14, 44, 43, 24], appear to have utilized this framework in one way or another, even though the contributors did not clearly summarize the relationship between their methods and this common underlying structure. However, despite the popularity of the framework, existing works lack very formal definition in theory.
What is Special about Multi-modal Learning? For the multi-modal representation X (z:(1) ..... , ,z(K)) consists of K modalities, we allow the dimension of the domain set of each modality (k) to be different, which well models the source of heterogeneity of each modality. The relationships across different modalities are usually of varying levels due to the heterogeneous sources. Therefore, compared to previous works [45, 48], we make assumptions on the relationship across every single modality in our analysis, which makes it general to allow different correlations. Moreover, the main assumption behind previous analysis [37, 39, 42] that each view/modality contains sufficient information for target tasks, which does not shed light on our analysis. It may not hold in multi-m moltll applications [47], c.g., in object detecting task, is known that depth images are with lower accuracy than RGB images [18].
4 Main Results
In this section, we provide main theoretical results to rigorously establish various aspects of the folklore claim that multi-modal is better than single. We first detail several assumptions throughout this section.
Assumption The loss function E(.y)) is L-smooth with respect to the first coordinate, and is bounded by a constant C.
Assumption The true latent representation g" contained G, and the task mapping ht is contained in H.

---

Assumption is a classical regularity condition for loss function in theoretical analysis [27, 41, 40]. Assumption 2 is also be known as realizability condition representation learning [41, 10, 40]. which ensures that the function class that we optimize over contains the true latent representation and the task mapping.
Assumption 3. For any o' E G' and M C [K] g' o Pp E
To understand Assumption 3, note that for any N C M C [K], by definition, for any IN E GN. there exists g' E G s.t.
((x))='((xx(et - -ele-s/(((xxmxxii
Therefore, Assumption 3 directly implies 9N E SMM- Moreover, we have GN 9M 9, which means that the inclusion relationship of modality subsets remains unchanged on the latent function class induced by them. As an example, if G' is linear, represented matrix G E RQ*K Also p'M diagonal matrix P I Kx* with the i-th diagonal entry being for i M
can be represented and 0 otherwise. In this case, Assumption holds, i.e. x P E g'. Moreover, G is a matrix with i-th column all be zero for & M, which is commonly used in the underfitting analysis in linear regression [34].
4.1 Connection Latent Representation Quality
Latent space is employed to better exploit the correlation among different modalities. Therefore, we will naturally conjecture that the performance of training with different modalities is related to its ability to learn latent space representation. In this section, we will formally characterize this relationship.
In order to measure the goodness of a learned latent representation we introduce the following definition of latent representation quality.
Definition 1. Given a data distribution with the form in (1), for any learned latent representation mapping E G, the latent representation quality is defined as
n(g) inf [r (h o g) r(h" o g")l
(6)
KCH
Here infReи (h g) is the best achievable population risk with the fixed latent representation g. Thus, certain extent, n(g) measures the loss incurred by the distance between g and g"
Next, we recap the Rademacher complexity measure for model complexity. It will be used in quantifying the population risk performance based on different modalities. Specifically, let F be a class of vector-valued function Rd + IR... Let 7,, 7,,..., 7m be i.i.d. random variables on RR following some distribution P. Denote the sample S = (ZI.... Zm). The empirical Rademacher complexity of JF with respect to the sample S is given by [5]
RisFF) == Eg
Σ orf (Z,) FEF
sup
where a = (o1)...,o,)) with ~ unif 1, The Rademacher complexity of F is
(F) = Ess [Fs(F)]
Now we present our first main result regarding multi modil learning.
Theorem 1. Let S = ((x,, 31)),. =n= be dataset of m examples drawn i.i.d. according to D. Let M, be two distinct subsets of [K]. Assuming we have produced the empirical risk minimizers (hy, 9.) and (hw.gx) (.).. training with the M and N modalities separately. Then, for all 1 > 5 > 0, with probability at least 5:
(hm (amm@gm) 9m) - (hr 2N))
4C
2ln(2/8)
$ <ss(M,N) + 8IH, (H (M))M) Vm + 20
where
7s(M,N) n((m)) -((N)
(8)
5

---

Remark. A few remarks are in place. First of all, (s((..) defined in (8) compares the quality between latent representations learning from M and A modalities with respect to the given dataset S.
Theorem bounds the difference of population risk training with two different subsets of modalities by (s,,) which validates our conjecture that including more modalities is advantageous in learning. Second, for the commonly used function classes in the field of machine learning, Radamacher complexity for a sample of size m,, Rm (FF is usually bounded by v /CFF//m where C(F) represents the intrinsic property of function class F. Third, (7) can be written as 7:5 (M, +
O(V in order terms. This shows that as the number of sample size grows, the performance of using different modalities mainly depends on its latent representation quality.
4.2 Upper Bound for Latent Space Exploration
Having establish the connection between the population risk difference with latent representation quality, next goal is to estimate how close the learned latent representation dM is to the true latent representation 9". The following theorem shows how the latent representation quality can be controlled in the training process.
Theorem 2. Let S {(x, yo) be dataset of m examples drawn i..dd according to D. Let M be a subset of [K]. Assuming we have produced the empirical risk minimizers (hM, training with the M modalities. Then, for all 1 > 8 >> 0, with probability at least 1 - 8:
2l(22/8 + ihh 9M.
1(āM) S 4LR (H o GM) + mL (H g) + GC
(9)
m
where i(hm o S) 2 (hm o 2m) - 8 (h* o g²) is the centered empirical loss.
Remark. Consider sets A' M [K]. Under Assumption 3, GN C GM G, optimizing over larger function class results in a smaller empirical risk. Therefore
L(hm= o Im, S) s L(hx S)
(10)
Similar to the analysis in Theorem 1, the first term the Right-hand Side (RHS), R., (H o GM) / /C(H GM)/m and Rm (H o GN) VC(H o Gy)/m. Following the basic structural property of Radamacher complexity [5], we have C(H - 9x) S C(H o GM). Therefore, Theorem 2 offers the following principle for choosing modalities to improve the latent representation quality.
Principle: choose to learn with more modalities if:
i(hw o S) - i(hm SM, / ((HO m
|CC GN)
What this principle implies are twofold. (i) When the number of sample size m is large, the impact of intrinsic complexity of function classes will be reduced. (ii) Using more modalities can efficiently optimize the empirical risk, hence improve the latent representation quality.
Through the trade-off illustrated in the above principle, we provide theoretical evidence that when N C M and training samples are sufficient, v((m) may be less than n((xx), i.e. (M,)) < O. Moreover, combining with the conclusion from Theorem 1, if the sample size is large enough,
(s) < 0 guarantees (hM 9M) S r (hw âN). which indicates learning with the M modalities outperforms only using its subset. N modalities.
Role of the intrinsic property C(-) Hypothesis function classes are typically overparametrized in deep learning, and will be extremely large for some typical measures, e.g., VC dimension, absolute dimension. Some recent efforts aim to offer an explanation about why neural networks generalize better with over-parametrization [28, 29, 4]. [28] suggest a novel complexity measure based on unit-wise capacities, which implies if both in overparametrized settings, the complexity will not change much or even decrease when we have more modalities (using more parameters). Thus, the incquality in the principle is trivially satisfied, and we will choose to learn with more modalities.

---

4.3 Non-Positivity Guarantee
In this section, we focus on a composite linear data generating model theoretically verify that the Ys ()) is indeed non-positive in this special case. 'Specifically, we consider the case where the mapping to the latent space and the task mapping are both linear, Formally, let the function class G and H be:
g { g(x) = A x. Rdx= A}
(11)
H h h(2) = 3 2. R" |||| S Cb.
where x = (x(1),.. .... x is a d-dimensional vector, x (*) E Rd denotes the feature vector for the k-th modality and Σ dk d. Here, the distribution (xx()) satisfies that its covariance matrix is positive definite. The data is generated by:
y - (3*) T A' A
(12)
where is independent of and has zero-mean and bounded second moment. Note that in practical multi-modal learning, usually only one layer is linear and the other is neural network. For instance, [50] employs the linear matrix project the feature matrix from different modalities into common latent space for early dementia diagnosis, i.e., 9 linear. Another example is in pedestrian detection [46], where linear task mapping is adopted, i.e., H is linear. Thus, our composite linear model can be viewed as an approximation to such popular models, and our results can offer insights into the performance of these models.
- -1]. Thus GM = G and we have the We consider a special case that M = [K] and N = [K--1] following result.
GN = f0
g(x)= (0010000)=1
A du [
A, E R with orthonormal columns
(13)
0
Proposition 1. Consider the dataset S {(x,, Ua)) generating from the linear model defined in (12) with l loss. Let M M= [K] and N [K 1]. Let AM. AN denote the projection matrix estimated by M, N modalities. Assume that A A has orthonormal columns. If n d, for sufficiently large constant Cr, we have:
(s(M,N) 0
(14)
In this special case, Proposition directly guarantees that training with incomplete modalities weakens the ability to learn a optimal latent representation. As result, it also degrades the learning performance.
Experiment
We conduct experiments to validate our theoretical results. The source of the data we consider is two-fold, multi-modes real-world dataset and well-designed generated dataset.
5.1 Real-world dataset
The natural dataset we use is the Interactive Emotional Dyadic Motion Capture (IEMO- CAP) database, which is an acted multi-modal and multi-speaker database [6]. It contains three modalities, Text, Video and Audio. We follow the data preprocessing method of [33] and obtain 100 dimensions data for audio, 100 dimensions for text, and 500 dimensions for video. There are six labels here, namely, happy, sad, neutral, angry, excited and frustrated. We use 13200 data for training and 3410 for testing.
Dataset.
Training Setting. For all experiments on IEMOCAP, we use one linear neural network layer to extract the latent feature, and we set the hidden dimension to be 128. In multi-modal network. different modalities do not share encoders and we concatenate the features first, and then map the
Proovin that ys (M,,) < holds in general is open and will be interesting future work.

---

feature to the task space. We use Adam [25] as the optimizer and set the learning rate to be 0.01, with other hyper-parameterr default. The batch size is 2048 for the data. For this classification task, the top-1 accuracy is used for performance measurement. We use naively multi-modal late-fusion training as our framework [43, 9]. In Appendix C, we provide more discussions on stable multi-modal training.
Connection to the Latent Representation Quality. The classification accuracy on IEMOCAP, using different combinations of modalities are summarized in Table 1. All learning strategies using multiple modalities outperform the single modal baseline. To validate Theorem calculate the test accuracy difference between different subsets of modalities using the result in Table and show them in the third column of Table 3.
Moreover, we empirically evaluate the n(âm) in the following way: freeze the encoder Òas obtained through pretraining and then finetune obtain better classifier h. Having ((āM) and ((xv), the values of is( fssss... (M.N) between different subsets of modalities are also presented in Table 3. Previous discussions Theorem implies that the population risk difference between M and N modalities has the same sign as (sM when the sample size is large enough, and negativity implies performance gains. Since we accuracy the measure, on the contrary, positivity indicates better performance in our settings. As shown in Table 3, when more modalities are added for learning, the test accuracy difference and is (MM.)) are both positive, which confirms the important role of latent representation quality characterized in Theorem I.
Table Test classification accuracy on IEMOCAP, using different combinations of modalities, only Text, Text + Video, Text Audio and Text Video Audio.
Modalities
Test Acc
Text(T)
49.9 51.08=0.66
Text Video(TV)
Text + Audio(TA)
53.03
Text Video + Audio(TVA)
53.89 + 0.47
Table Latent representation quality vs. The number of the sample size on IEMOCAP. Noting that in this table, show the results from naively end-to-end late-fusion training and in Appendix C, discuss on more stable multi-modal training methods.
Modalities
Test Acc (Ratio of Sample Size)
10-4
10-3 3
10-2
10-1
23.66::1.28
29.0843.34
45.63=0.29
48.30+1.31
49.93-0.57
TA
1.05
34.28:4..54
47.28=1.224
50.46+0.61
51.08=0.66
TV
20.71
38.3743.12
46.54=1.62
49.50+1104
53.03=0.21
TVA
24.71
32.24+1.117
46.39=3.82
50.75=1145
53.89=0.47
Table 3: Comparison of test accuracy and latent representation quality among different combinations of modalities.
M Modalities
A Modalities
Test Acc Difference
(s(,,N
TA
T
1.15
1.36
TV TVA TVA
T
3.10
3.57
TV TA
0.86
0.19
2.81
2.4
Upper Bound for Latent Space Exploration. Table also confirms our theoretical analysis in Theorem 2. In all cases, the N modalities is subset of M modalities, and correspondingly. positive (M, observed. This indicates that M modalities has a more sufficient latent space exploration than its subset. N modalities.

---

We also attempt to understand the use of sample size for exploring the latent space. Table 2 presents the latent representation quality 1) obtained by using different numbers of sample size, which is measured by the test accuracy of the pretrain+finetuedd modal. Here, the ratio of sample size is set to the total number of training samples. The corresponding curve is also ploted in Figure 2(a). As the number of sample size grows, the increase in performance of is observed, which is in keeping with the O( (VT[/)) term in our upper bound for 1j. The phenomenon that the combination of Text, Video and Audio (TVA) modalities underperforms the uni-modal when the number of sample size is relatively small, can also be interpreted by the trade-off we discussed in Theorem 2. When there are insufficient training examples, the intrinsic complexity of the function class induced by multiple modalities dominates, thus weakening its latent representation quality.
EMOCAP
Simulated
T
overlap: 1.0
-x- overlap: 0.8
TVA
everlap: 0.2
overiap: 0.0
Acuracy
50
10
10
number sample size
The number of modelittes
(a) Latent representation quality vs. The ratio of the sample size IEMOCAP
(b) Latent representation quality vs. The number of modalities simulated data
Figure 2: Evaluation of the latent representation quality on different data sets
5.2
Synthetic Data
In this subsection, we investigate the effect of modality correlation on latent representation quality. Typically, there are three situations for the correlation across each modality [39]. (i) Each modality does not share information at all, that is, each modality only contains modal-specific information. (ii) The other is that, all modalities only maintain the share information without unique information on their own. (iii) The last is a moderate condition, i.c., each modal not only shares information, but also owns modal-specific information. The reason to utilize simulated data in this section is due to the fact that it is hard in practice to have natural datasets that possess the required degree of modality correlation.
Data Generation. For the synthetic data, we have four modalities, denoted by m1, m2, m3, ma, respectively, and the generation process is summarized as follows:
Step 1: Generate my N,,, i- 1,2,3, 4, where m; is i.i.d. 100-dimensional random vector; Step 2: Generate mn +- (1 w) m. + w- my for i = 2,3,4
Step 3: Generate the labels as follows. First, add the four modality vectors and calculate the sum of coordinates. Then, obatin the 1-dimensional label, i.c. y (m1 + m2 + ma m4).sum (dim 1)
We can see that information from my shared across different modalities, and w controls how much is shared, which one can use to measure the degree of overlaps. We tune it in (0.0, 0.2, 0.5, 0.8, 1.0). high weight w (close to 1) indicates a high degree of overlaps, while w = means each modality is totally independent and is non-overlapping.
Training Setting. We use the multi-layer perceptron neural network as our model. To be more specific, we first use a linear layer to encode the input to a 10-dimension latent space and then we map the latent feature to the output space. The first layer's 's input dimension depends on the number of modality. For example, if we use two modalities, the input dimension is 200. We use SGD as

---

optimizer, and set learning rate 0.01, momentum 0.9, batch size 10000, training for 10000 steps. The Mean Square Error (MSE) loss is considered for evaluation this regression problem.
Modality Correlation Our aim is to discover the influence of modality correlation on the latent representation quality 7. To this end, Table shows the y with the varying number modalities under different correlation conditions, which is measured by the MSE loss of the pretrain+finetunel modal. The trend that the loss decreases as the number of modalities increases is described in Figure 2(b), which also validates our analysis of Theorem 2. Moreover, Figure 2(b) shows that higher correlation among modalities achieves a lower loss for 7,, which means better latent representation. This emphasizes the role of latent space exploit the intrinsic correlations among different modalities.
Table 4: Latent representation quality among different correlation situations synthetic data
Modalities
MSE Loss (Degree of Overlap) )
0.8
0.5
0.2
0.0
0
0..9
75.89+1.25
193.28+1.00
301.92
m1, m2
0
8. 1.177
51.25+1.06
129.81+4.36
207.45=4.68
m, M2, my
0
0..05
26.06:10.69
1..17
103.23=0.61
mi; 172 Ma, m7
0
6 Discussion
has been discovered that the use of multi-modal data in practice will degrade the performance of the model in some cases[43, 19, 38, 15]. These works identify the causes of performance drops of multi- modal as interactions between modalities the training stage and try to improve the performance by proposing new optimization strategies. Therefore, to theoretically understand them, we need to analyze the training process from the optimization perspective. Our results, on the other hand, mainly focus on the generalization side, which is separated from optimization and assumes that we get the best performance possible in training. Moreover, our theory is general and does not require additional assumptions the relationship across every single modality, which may be crucial for theoretically analyzing the observations in a multi-modal performance drop. Understanding why multi-modal fails in practice is very interesting direction and worth further investigation in future research.
7 Conclusion
In this work, we formulate multi-modal learning framework that has been extensively studied in the empirical literature towards rigorously understanding why multi-modality outperforms single since the former has access to better latent space representation. The results answer the two questions: when and why multi-modal outperforms uni-modal jointly. To the best of our knowledge. this the first theoretical treatment to explain the superiority of multi-modal from the generalization standpoint.
The key takeaway message is that the success of multi-modal learning relies essentially on the better quality of latent space representation, which points to an exciting direction that is worth further investigation: to find which encoder is the bottleneck and focus on improving it. More importantly, our work provides new insights for multi-modal theoretical research, and we hope this can encourage more related research.
Acknowledgments and Disclosure of Funding
The work of Yu Huang and Longbo Huang is supported in part by the Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant A00000000 and 2020AAA00084005
10

---

References
[1] Massih-Reza Amini, Nicolas Usunier, Cyril Goutte, et al. Learning from multiple partially observed views-an application to multilingual text categorization. In NIPS, volume 22, pages 28-36, 2009.
[2] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pages 3674-3683, 2018.
aanoo-and-language
[3] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learn- ing: survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423-44 443-4433343, 2018.
[4] Peter Bartlett, Dylan J Foster, and Matus Telgarsky. npermazlly-normalzzeed margin bounds for neural networks. arXiv preprint arXiv:1706. 08498, 2017.
[5] Peter I. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov)44634482 2002.
[6] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang. Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335 2008,
[7] Giovanni Cavallanti, Nicolo CessaBBianchi, and Claudio Gentile. Linear algorithms for online multitask classification. The Journal of Machine Learning Research, 11:2901-2933 2010.
[8] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721-725. IEEE, 2020.
[9] Chenzhuang Du, Tingle Li, Yichen Liu, Zixin Wen, Tianyu Hua, Yue Wang, and Hang Zhao Improving multi-modal learning with uni-modal teachers. arXiv preprint arXiv:2 1059, 2021.
[10] Simon S Du, Wei Hu, Sham M Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.0994: 2020.
[11] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen the cocktail party: speaker- independent audio-visual model for speech separation. arXiv preprint arXiv: 03.6,0669, 2018.
[12) Haytham M Fayek and Anurag Kumar. Large scale audiovisual learning of sounds with weakly labeled data. arXiv preprint arXiv:2006.0 01595, 2020.
[13] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07017 2020.
[14] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1933-1941, 2016.
[15] Itai Gat, Idan Schwartz, Alexander Schwing, and Tamir Hazan. Removing bias in multi- modal classifiers: Regularization by maximizing functional entropies. arXiv preprint arXiv:2010. 100000 2020.
[16] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and Human-labelen dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776-780. IEEE, 2017.
[17] Yash Goyal, Tejas Khot, Douglas Summers-Shee Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904-6913, 1144-6913,220 2017.
[18] Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2827-2836, 2016.
11

---

[19] Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification. arXiv preprint arXiv:2102.02057, 2021.
[20] Caner Hazirbas, Lingni Ma, Csaba Domokos, and Daniel Cremers. Fusenet:
Incorporating depth into semantic segmentation via fusion-based based enn architecture. In Asian conference on computer vision, pages 213-228. Springer, 2016.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[22] Xinxin Hu, Kailun Yang, Lei Fei, and Kaiwei Wang. Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation. In 2019 IEEE International Conference on Image Processing (ICIP), pages 1440-1444. IEEE, 2019.
[23] Jindong Jiang. Lunan Zheng, Fei Luo, and Zhijun Zhang. Rednet: Residual encoder-decoder network for indoor rgb-d semantic segmentation. arXiv preprint arXiv: 01054, 2018.
[24] M Esat Kalfaoglu, Sinan Kalkan, and A Aydin Alatan. Late temporal modeling in 3d cnn architectures with bert for action recognition. In European Conference on Computer Vision, pages 731-747. Springer, 2020.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXfvv1412.6980, 2014.
[26] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097- 1105, 2012.
[27] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
[28] Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of wverpaaramttiizatoon in generalization of neural networks arXiv preprint arXiv: 1805. /2076, 2018
[29]
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. based capacity control in neural networks. In Conference on Learning Theory, pages 1376-1401. PMLR, 2015.
[30] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In ICML. 2011.
[31] Seong-Jin Park, Ki-Sang Hong. and Seungyong Lee. Rdfnet: Rgb-d multi-level residual feature fusion for indoor semantic segmentation. In Proceedings of the IEEE international conference on computer vision, pages 4980-4989, 2017.
[32] Massimiliano Pontil and Andreas Maurer. Excess risk bounds for multitask learning with trace norm regularization. In Conference on Learning Theory, pages 55-76. PMLR, 2013.
[33] Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis- Morency. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th annual meeting of the association for computational linguistics (volume
1: Long papers), pages 873-883, 2017.
[34] George AF Seber and Alan J Lee. Linear regression analysis, volume 329. John Wiley & Sons, 2012.
[35] Daniel Seichter, Mona Köhler, Benjamin Lewandowski, Tim Wengefeld, and Horst-Michael Gross. Efficient rgb-d semantie segmentation for indoor scene analysis. arXiv preprint arXiv:2011.06961, 2020.
[36] Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial life. 11(1-2):13-29 2005.
[37] Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi-view learning. 2008
[38] Mahesh Subedar, Ranganath Krishnan, Paulo Lopez Meyer, Omesh Tickoo, and Jonathan Huang. Uncertainty-awan audiovisual activity recognition using deep bayesian variational inference. Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 6301-6310, 2019.
12

---

[39] Xinwei Sun, Yilun Xu, Peng Cao. Yuqing Kong, Lingjing Hu, Shanghang Zhang, and Yizhou Wang. Tegm: An -hhhoaation-thhoreiii framework for semi-supervised multi- multi-modality learning. In European Conference Computer Vision, pages 171-188. Springer, 2020.
[40] Nilesh Tripuraneni, Chi Jin, and Michael I Jordan. Provable meta-learning of linear representa-
tions. arXiv preprint arXiv:2002. /684, 2020.
[41] Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. Advances Neural Information Processing Systems, 33, 2020.
[42] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self- supervised learning from a multi-view perspective. arXiv preprint arXiv:2006.055776 2020.
[43] Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 12692-12702. IEEE, 2020.
[44] Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, and Junzhou Huang. Deep multimodal fusion by channel exchanging. Advances in Neural Information Processing Systems, 33, 2020.
[45] Chang Xu, Dacheng Tao, and Chao Xu. A survey on multi-view learning. arXiv preprint arXiv: 1304.5634, 2013.
[46] Dan Xu, Wanli Ouyang, Elisa Ricci, Xiaogang Wang, and Nicu Sebe. Learning cross-modal deep representations for robust pedestrian detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5363-5371, 2017.
[47] Yang Yang, Han-Jia Ye, De-Chuan Zhan, and Yuan Jiang. Auxiliary information regularized machine for multiple modality feature learning. In Twenty-Fouthh International Joint Conference on Artificial Intelligence 2015.
[48] Changqing Zhang. Huazhu Fu, Jocy Tianyi Zhou, Qinghua Hu, et al. Cpm-nets: Cross partial multi-view networks. 2019.
[49] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pages 570-586, 2018.
[50] Tao Zhou, Kim-Han Thung, Mingxia Liu, Feng Shi, Changqing Zhang, and Dinggang Shen. Multi-modal latent space inducing ensemble svm classifier for early dementia diagnosis with neuroimaging data. Medical image analysis, 60: 10360, 2020.
13


Reviews:
[{"review_id": "l2DQyW19Sda", "reviewer": "Reviewer_3VyC", "summary": "This work introduces a generalization bound for multimodal learning, which shows that, for a sufficiently large sample size, the model risk decreases with an increasing number of considered modalities.\nThe authors also discuss a novel principle to dictate whether new modalities can be beneficial as a function of the dataset size and considered function classes.\nAn empirical evaluation on both toy and real-world data validates the theory, underlining the benefits of learning from different modalities and laying the foundations for future research.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces a formal definition of the multimodal learning problem, clearly stating the initial assumptions and introducing the concept of \"latent representation quality\" to show under which condition learning can benefit from multiple modalities.\nThe paper is clear and well-written, and all the main results are clearly stated and proved.\nTo the best of the reviewer's knowledge, the analysis reported in this paper is novel and relevant to the field of multimodal learning.\n\nMain comments:\n\n1) Can the authors further comment on the role of the intrinsic properties mentioned in the principle in lines 189-190 with respect to the modality selection procedure? What role does the change in the size of the possible representation space $\\mathcal{G}_{\\mathcal{N}}$ play?\n\n2) The data generation process described in lines 277-281 is unclear. How is the 1-dimensional label defined? What procedure used to correlate the modalities? Further details would improve clarity and reproducibility.\n\n3) Reducing the number of available modalities can be considered as a restriction of the size of the hypothesis class. How does the proposed analysis relate to classic PAC bounds in learning theory [1]?\n\nOther Comments and general questions:\n\n4) According to the definition, for a sufficiently large $\\mathcal{H}$ the representation quality seems to be closely related to the amount of predictive information loss: as long as the representation mapping $g$ does not discard predictive information, we can find a task mapping $h$ so that $r(h \\circ g) = r(h^* \\circ g^*)$. In this sense, the difference of representation quality for $\\mathcal{N}\\subset\\mathcal{M}$ could be interpreted as the amount of task-relevant information loss caused by removing informative modalities: $I({\\bf y};p_{\\mathcal{M}}({\\bf x})) \\ge I({\\bf y};p_{\\mathcal{N}}({\\bf x}))$ for $\\mathcal{N}\\subseteq\\mathcal{M}$. On the other hand, the addition of new modalities increases the rademacher complexity and the total amount of information $I({\\bf x};p_{\\mathcal{M}}({\\bf x})) \\ge I({\\bf x};p_{\\mathcal{N}}({\\bf x}))$.\nCan the authors underline any connection with the Information bottleneck principle? Can the addition of redundant modalities $I({\\bf y};p_{\\mathcal{M}}({\\bf x})) = I({\\bf y};p_{\\mathcal{N}}({\\bf x}))$ ever be favorable according to the proposed view-selection principle.\n\n5) Line 159-160: The expression for the Rademacher complexity of $\\mathcal{F}$ should contain $f(Z_i)$ instead of $g(Z_i)$.\n\nReferences:\n[1] Shalev-Shwartz, Shai, and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper has a strong theoretical focus and does not mention any limitation or societal impact of the proposed analysis.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5 h"}, "extra_text": {}}, {"review_id": "UjUdH_U3c_Y", "reviewer": "Reviewer_eSGN", "summary": "This paper aims to provide a theoretical explanation on whether multimodal learning can perform better than unimodal learning. They study a popular multimodal learning framework which firstly encodes features from different modalities into a common latent space before mapping the latent representations into a label. They prove that under certain conditions multimodal learning enables better estimation of the latent space which then enables better task performance. They also show some experiment results to back up their theory.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The problem tackled is an important one and an important shortcoming in current work.\n2. There are some empirical results supporting their results.\n\nWeaknesses:\n1. There are some severe issues with the setup, results, and writing which makes the paper hard to follow and understand:\n\na. In line 38 you say you analyze 'a widely used composite multimodal framework [35]' but [35] was only proposed in 2020 so I'm not sure how it can be deemed as a 'widely used' framework.\n\nb. Also [35] uses a SVM on each modality's (total M modalities) features to get them into a common latent space from which a single classifier is used from that space to the label, and each modality is used for predicting the label to get M labels, from which you take an ensemble to predict the final label. But in your setup you don't do this, and the example you give in line 124-128 talks about self-attention to combine the features. I'm not sure why there is inconsistency here.\n\nc. In line 115 I'm not sure why noisy modalities are being introduced. If you want to analyze multimodal learning why don't you start by analyzing the case where all modalities are present. I'm also not sure what p'_M means as an extension of p_M - are you adding noisy modalities on top of noisy modalities? I'm not sure what is going on here.\n\n2. There are many multimodal datasets that can be used for research, but this paper only uses 1 dataset and a synthetic dataset. You can refer to https://arxiv.org/abs/2107.07502 for a set of comprehensive benchmarks to test on.\n\n3. There are severe clarity issues with the writing such that I have a hard time understanding the experimental results.\n\na. The experiments are on concatenating features which seems to be different from your analysis and also different from [35].\n\nb. line 245: 'freeze the encoder gˆM obtained 246 through pretraining and then finetune a better classifier h.' I'm not sure what this means.\n\nc. Many of the figure captions are not informative (e.g. figure 2) which makes it hard to understand.\n\nd. Many details are missing in the synthetic data, including what exactly 'overlap' means and how the labels are generated.\n\n4. It would be good to refer to 'What Makes Training Multi-Modal Classification Networks Hard? https://arxiv.org/abs/1905.12681' as an example of where simple multimodal learning (i.e. concatenating features) fails. I believe the theory proposed here is unable to explain this phenomenon. I'm also not sure if the assumptions made in this paper are too strict or too loose to reconcile these empirical findings.\n\nOverall this paper would benefit from a significant revision. I would suggest carefully reading the multimodal literature (see https://arxiv.org/abs/2107.07502 and https://arxiv.org/abs/1705.09406) for comprehensive models, benchmarks, and surveys. It would also be good to see under what settings do multimodal models fail (like in https://arxiv.org/abs/1905.12681 or https://aclanthology.org/2020.emnlp-main.62.pdf) because just by looking at methods that work will make it hard to come up with an explanation that is consistent with both sides of the story. The writing and clarity of this paper can also be significantly improved. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors can say a bit more about the societal impact of multimodal models, including their applications in robotics, healthcare, education etc.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns as far as I can tell.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "J0jmxWQbkBs", "reviewer": "Reviewer_zJa7", "summary": "The paper tries to prove that learning with multiple modalities can achieve a smaller population risk than only using its subset of modalities. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am glad to see the work which aims to answer the question: Can multimodal learning provably performs better than unimodal? Under a specific setting, the authors provide theoretical results.\n\nAlthough under a simple setting, the theoretical results are good. There is another work [1] that has provided some analysis about why multiple modalities are better. I think it is interesting to discuss the relation between the proposed one and [1]. \n\nSome minors:\n“which points to an interesting direction that are worth further…” should be “which points to an interesting direction that is worth further…”\n\nThe authors claim “we provide the first theoretical analysis to shed light on what makes multimodal outperform unimodal…”, “this is the first theoretical treatment to capture important qualitative phenomena observed in real multimodal applications”. Since the CPM-Nets has also provided related theoretical results (from other perspective), it not proper to claim the work as the first one. \n\n[1] CPM-Nets: Cross Partial Multi-View Networks, NeurIPS 2019\n\nThe authors claim that “two common principles, consensus and complement, which are crucial in the theoretical analysis of multi-view learning, are not applicable in multimodal learning”. Since in machine learning, multimodal and Multiview learning are quite similar, so the claim is not reasonable. Furthermore, the description for the difference between multimodal learning and multiview learning is arguable and even misleading. \n\nWhy choose the composite multimodal framework, since there are lots of multimodal learning methods and why the chosen one is widely used?\n\nIn the CPM-Net, the versatility also implies the advantages of using multi-view information, and the paper also provides analysis in terms of empirical error.  It will be better if the authors could provide a detailed discussion to clarify the relationship between them and advantages of the proposed one. Providing theoretical results is a good direction but the current results seem a little straightforward. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 "}, "extra_text": {}}, {"review_id": "7nDeGrHjsCB", "reviewer": "Reviewer_VZFZ", "summary": "This paper focuses on proving multimodal learning performs better than unimodal under the most popular multimodal learning framework. Specifically, they proved that in the linear case, learning with multiple modalities achieves a smaller population risk than only using its subset.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality*: As far as I know, this is the first theoretical paper to try to prove multimodal learning performs better than unimodal. This is an important direction and lack of analysis before.\n\n*Quality*: Authors conduct sufficient theoretical analysis and proof. I don’t find any technical flaws. Meanwhile, the author conducted experiments to validate the conjectures and theoretical results. \n\n*Clarity*: The paper is easy to follow. I only find one typo, e.g., “When the number of *simple* size” on line 190.\n\n*Significance*: This paper could provide new insight for multimodal theoretical research.\n\n*Comments*:  Although this paper theoretically proves that in the linear case multimodal learning better than single, many potential risks may reduce the performance of the multimodal model in practice. Therefore, I think the paper should discuss why the use of multimodal data in practice will reduce the performance of the model in some cases. For example, [1] gives an analysis from the perspective of different model convergence speed. [2][3] explores multi-modal learning from the perspective of uncertainty. [4] removes the bias in multimodal classifiers to avoid some modalities contribute more to the classification results than others.\n\n[1] What makes training multi-modal classification networks hard? In CVPR 2020\n\n[2] Uncertainty-Aware Audiovisual Activity Recognition Using Deep Bayesian Variational Inference? In ICCV 2019\n\n[3] Trusted multi-view classification. In ICLR 2021\n\n[4] Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies. In NeurIPS 2020\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative social impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a formal theoretical analysis of the utility of observing multimodal information over unimodal information from the perspective of empirical risk minimisation. The analysis is followed up with some empirical evidence seeking to confirm theoretical findings.

The reviewers all agreed that the theoretical contributions were both sound and principled.

There were a few concerns about related work (eg. CPM-Nets) and making things clear and precise that the authors should address.  For example the characterisation of [35] as 'widely-used' is odd to say the least.
  
The main concern with this paper is with the experiments---specifically the design and the observed results.

It appears to be the case, as the authors themselves point out (l265-267), that the complexity of the function class has a pretty strong effect on the _significance_ of the results (cf. Table 3). The 'bold' values are only significant when the ratio of sample size is 1.0; in all other cases, they're within the reported standard deviations, and cannot be claimed to particularly significant.

Moreover, as reviewer VZFZ points out with relevant citations, in practice, multimodal learning tends to perform worse. In the discussion, this is somewhat hand-waved away as having to do with optimisation, but it would seem to be a pretty serious issue affecting the theory if it is something typically not observed in practice!
A proper discussion of what the cause for this apparent contradiction is and how it might relate to the complexity of the function class/sample size ratio in such cases would strengthen the paper.

I would also urge the authors to incorporate the clarifications and edits in discussion with all the reviewers, especially reviewer eSGN into the updated manuscript.

If there such a thing as a conditional accept, this paper would be a prime candidate as it does have relevant contributions, but also leaves a few things to be desired before it can be classed a complete piece of publishable work.
It would probably be reasonable to accept the paper on the merits of its theoretical contributions, but *strongly* encourage the authors to address the question of applicability, relation to contradictory results in practice, how this relates to the function-class complexity vs empirical observations trade-off they discuss (esp Sec 5.1).

Decision: Accept (Poster)