A Theory of Multimodal Learning
Authors: Zhou Lu
Venue: NeurIPS 2023 poster (2023)
Keywords: Multimodal Learning

Abstract: Human perception of the empirical world involves recognizing the diverse appearances, or 'modalities', of underlying objects. Despite the longstanding consideration of this perspective in philosophy and cognitive science, the study of multimodality remains relatively under-explored within the field of machine learning. Nevertheless, current studies of multimodal machine learning are limited to empirical practices, lacking theoretical foundations beyond heuristic arguments. 
An intriguing finding from the practice of multimodal learning is that a model trained on multiple modalities can outperform a finely-tuned unimodal model, even on unimodal tasks. This paper provides a theoretical framework that explains this phenomenon, by studying generalization properties of multimodal learning algorithms. We demonstrate that multimodal learning allows for a superior generalization bound compared to unimodal learning, up to a factor of $O(\sqrt{n})$, where $n$ represents the sample size. Such advantage occurs when both connection and heterogeneity exist between the modalities.

A Theory of Multimodal Learning
Zhou Lu Princeton University zhoul edu
Abstract
Human perception of the empirical world involves recognizing the diverse ap- pearances, or 'modalitie'', of underlying objects. Despite the longstanding con- sideration of this perspective in philosophy and cognitive science, the study of multimodality remains relatively under-explored within the field of machine learn- ing. Nevertheless, current studies of multimodal machine learning are limited to empirical practices. lacking theoretical foundations beyond heuristic arguments. An intriguing finding from the practice of multimodal learning is that model trained on multiple modalities can outperform finely-tuned unimodal model, even on unimodal tasks. This paper provides theoretical framework that explains this phenomenon, by studying generalization properties of multimodal learning algorithms. We demonstrate that multimodal learning allows for superior gener- alization bound compared to unimodal learning. up to a factor of O (V)), where represents the sample size. Such advantage occurs when both connection and heterogeneity exist between the modalities.
Introduction
Even before the common era, the concept of viewing an object as the collection of its appearances, has already sprouted in early philosophy. The Buddha, in the 'Diamond Sutra', separated the essence of universe from various modalities such as sight, sound, smell, taste and touch. Two centuries ago, Immanuel Kant made further step, positing that humans perceive only the representations of "noumena" from the empirical world. He wrote:
"And we indeed, rightly considering objects of sense as mere appearances, confess thereby that they are based upon thing in itself, though we know not this thing in internal constitution, but only know its appearances, viz., the way in which our senses are affected by this unknown something. Prooegomena"
From this perspective, human cognition of the world, may therefore be considered effectively equivalent to the multiple modalities of the underlying objects. The importance of multimodality extends beyond metaphysics to everyday life: children learning languages often rely on illustrations, and even mathematicians benefit from visual aids.
However, machine learning, which could be seen as the cognition of computer systems, has not fully harnessed the power of multimodality. Multimodal machine learning, which processes and learns from data with multiple modalities, remained relatively under-explored until recently. Despite the impressive success of multimodal learning in empirical applications, such Gato 31 and GPT-4 26). the corresponding theoretical understanding is largely absent, often limited to heuristics.
A fascinating observation from empirical multimodal learning is that model trained with multiple modalities can outperform a finely-tuned unimodal model, even on population data of the same unimodal task. It's 's not immediately clear why multimodality offers such an advantage, considering that the trained model's 's focus is spread across different modalities.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

While it seems challenging to outperform unimodal learning asymptotically when sufficient data available. multimodal learning can still provide an edge under a fixed data budget. Different modalities might focus on different aspects of an object, and for specific classification problem, one modality may require a smaller sample complexity. This phenomenon often occurs with large models handling many tasks and a vast amount of training data, suggesting that:
Training across tasks learns a common connection between modalities efficiently, allowing the model to adapt to the modality with the smallest sample complexity.
An intuitive example of how multiple modalities help is learning parametric sine functions. The samples come in the form of
x E (0, 1...1y = == =inz=//in(///) (in()/)),
where v are the two modalities and is the label. Given data from both modalities the learning problem is trivial, even with a single training data point, while learning solely on a is hard albeit there bijective mapping between 2; y. From perspective of VC-dimension, there is gap between the class of linear functions (0x) and the class of parametric sine functions (sin(1 /(x)). in that the former one has VC-dimension 1 while the latter has infinite VC-dimension. More details will be provided later.
The theory problem we study in this paper, is thus how to formalize the above heuristic with provable guarantees. To this end, we examine generalization bounds of a simple multimodal ERM algorithm, which involves two parallel stages: learning a predictor f E F based on multimodal training data, and learning a connection 9 that maps one modality another with potentially unlabeled data. During inference, the composition o ğ is used to perform prediction on unimodal population data.
In this setting, we prove that the learnt unimodal predictor f o can achieve vanishing generalization error against the best multimodal predictor /" as if given multiple modalities, whenever ς expressive enough realize the training data. In addition, such generalization bound depends the complexities of both hypothesis classes F, F separately, better than unimodal approaches which typically involve the complexity of F G or a worst-case complexity of F, up to an O( factor where denotes the size of training data. On the other hand, we show a separation between multimodal and unimodal learning, by constructing a hard instance learnable by multimodal learning, in which no matter what hypothesis class is chosen for the unimodal learning problem, it's either under- expressieee or ever-expressive and thus incurs constant error. Putting the two pieces together, our theory suggests that with both connection and heterogeneity, multimodal learning is provably better than unimodal learning.
The paper is organized as follows. In section we formalize the setting of multimodal learning and provide motivating example. Section 3 proves generalization upper bound of the two-stage multimodal ERM algorithm on semi-supervised multitask learning problems. The lower bound on the separation between multimodal and unimodal learning is given in section 4 then we discuss the limitations of this paper and future directions in section 5
1.1
Related Works
Theoretical Multimodal Learning: while empirical multimodal learning has shown significant progress, theoretical studies are relatively sparse, lacking a firm foundation. Prior theoretical in- vestigations often focus on specific settings or incorporate additional assumptions. Some of these perspective, proposing algorithms based on total correlation or utilizing partial information decomposition to quantify relationships between modalities [35] I7 Other studies approach the problem from a multi-view setting, typically assuming that each view alone suffices for prediction 40
studies adopt an information-theoree
An important work in theoretical multimodal learning is [6], which also considered the advantage of multimodal learning in generalization and is the first and probably the only general theoretical result this field so far. In particular, they considered the population risk of a representation learning based approach where learning under different subsets of modalities is performed on the same ERM objective with shared hypothesis classes. They proved that the gap between the population risks of different subsets of modalities is lower bounded by the difference between what they called the latent

---

representation quality, which is the best achievable population risk with the learnt representation on the chosen subset of modalities.
There are two limitations in this result: 1, there is no quantitative characterization on how large the gap between latent representation qualities can be; 2, the comparison is not-only instance-dependent, but also carried over the same hypothesis classes and doesn't I exclude the possibility that the smaller subset of modalities could potentially use different class bypass the gap, making the lower bound somewhat restricted. We strengthen this result by showing the gap can be as large as (2(1) (Theorem 7. even we allow the smaller subset of modalities to use any hypothesis class. They also showed an upper bound on the excess population risk via a standard representation learning analysis, which involves the complexity of a composition of hypothesis classes F o G. while our analysis decouples the complexities of hypothesis classes, leading to an improved upper bound up to a factor of (VV)).
A recent work of [32] made more fine-grained study on multimodal learning, analyzing the benefit of contrastive loss in training dynamics. They considered the aspect of optimization instead of generalization for particular problem, focusing on the setting of linear data- gatta-enerating model. They proved that the use of contrastive loss is both sufficient and necessary for the training algorithm to learn aligned and balanced representations.
the inception of multimodal learning applications dates back to the last century, initially devised to enhance speech recognition using both vision and audio [39] 25
Empirical Multimodal Learning:
Multimedia is another aspect in which multimodal learning inspired new methods for indexing and searching [T] 19]. With the development of deep learning and its success in computer vision [14]
and natural language processing [38], people start studying deep multimodal learning in related tasks For more comprehensive introduction
such generating one modality from the other g 15 30
of multimodal learning, we refer the readers to the excellent survey paper 4 see also [
[3
Recently, the power of multimodal learning was carried out in large-scale generalist models. In 31 the training data includes a wide variety of modalities such as image, text, robotics and so on. The resulting model is reported to be able to beat fine-tuned unimodal models in some tasks. In a more recent ground- breakiin result, the super-large language model GPT-4 [26] makes use of not only text data available on internet, but also data from other modalities such as audio, demonstrating excellent capabilities in integrating knowledge from multiple domains.
Representation Learning: this field, closely related to our work, focuses on learning common underlying representation across multiple tasks. The typical framework of representation learning involves solving ERM problem with a composition of hypotheses ft g. where fe is task-specific while g is the common representation. Generalization bounds of representation learning usually involve the complexity of F G or a worst-case complexity of F.
Starting from Baxter's study [5] which gave theoretical error bounds via covering numbers on the inductive bias learning approach [36.. long line of work has followed, each improving upon and 27
generalizing the previous results 6 2
For more recent works we detail several representative ones here. The work of 23 studied both representation learning and transfer learning in the setting of multitask learning, achieving dimension independent generalization bounds with a chain rule on Gaussian averages (22). For the problem of transfer learning, (37) improved the leading term of (23 by a O(vn) factor under task diversity assumption, while [9] obtained a similar bound under low-dimension and linear function assumptions.
B analyzed generalized setting called contrastive learning inspired by the success of empirical language models.
2 Setting
In this paper, we consider straightforward yet non-trivial case of two modalities to ensure clarity in our presentation. Formally, we denote the set of possible observations to be (X, y/ R). Here, each element s e S constitutes a pairing of inputs from both modalities æ e X C R", y E y C R and their associated label E RR thus forming a tuple (Ir, y, z). Assuming without loss of generality that both X' and y are contained within their respective Euclidean unit balls. Given probability measure ja on S and a loss function f, the performance of a learning algorithm A is measured by the population loss if we interpret A as a function, namely
(( 2),
3

---

To leverage the hidden correlation between different modalities, we aim to learn both connection function g bridging the modalities and prediction function that accepts inputs from both modalities. Although we focus on learning a connection from X to y for simplicity, symmetrical approach can handle the reverse direction.
In particular, we will consider learning algorithms as a composition of functions A(z, y) = (()), where f E F and g E represent the hypothesis classes for both functions. This form is most general and common practical forms such as fusion A(g(T), h(y)) can be subsumed by the general form.
The goal is to identify f.9 using multi-modal training data, to minimize the excess population risk
E (rr...))-n (())),)(()))))))))))))()) 2) min E (x..v.))) y). JEF
(1)
In this context, we compare with the optimal predictor f as if given complete observations of both modalities, because our objective is to achieve a performance comparable to the best predictor given both modalities. The reason is that such a predictor could have significant advantage over any unimodal predictor that does not learn these connections (either explicitly or implicitly).
We seek statistical guarantees for f and ÿ. To provide generalization bounds on the excess population risk, we require a complexity measure of hypothesis classes, defined follows.
Definition (Gaussian average). Let Y be non-empty subset of IR", the Gaussian average of Y is defined as
G(Y) = EEg
sup NEY
where oj are iid standard normal variables. Similarly, we can define the function version of Gaussian average. Let & be function class from the domain X' R$ and X {.1,.... Inf be the set of input sample points. We define the Gaussian average of the class G on sample X as:
G(G(X)) = Eyg
sup gEG
==
where ozy are iid standard normal variables.
We make the following Lipschitz assumption the class F and the loss function which is standard in literature.
Assumption 1. We assume that any function f Ba R the class F is L-Lipschitz, for some constant L O. The loss function E takes value in [o, 1). and is 1-Lipschitz in the first argument for every value of z.
2.1 A Motivating Example
To introduce our theoretical findings, we present a straightforward but insightful example, illustrating the circumstances and mechanisms through which multimodal learning can outperform unimodal learning. Despite its simplicity and informal nature, this example captures the core concept of why multimodal learning requires both connection and heterogeneity, and we believe it is as vital as the more formal statements that follow.
Consider the problem where X y (0, 1]. Any potential data point (xx, y, 2) from S governed by parameter 8* (0, 1], such that
y = 0 sin (1/v).
The loss function choice is flexible in this case, and any frequently used loss function like the & loss will suffice.
Suppose we have prior knowledge about the structure of the problem, and we select g(x) = 0z E (0, 1] and F = (1x)) as our hypothesis classes. If we have sampled data from both modalities, we can easily learn the correct hypothesis via Empirical Risk Minimization (ERM): simply take any (x, y, 2) sample and compute 0 = y/z.

---

However, if we only have sampled data with the y modality concealed, there could be multiple e values that minimize the empirical loss, making the learning process with F G significantly more challenging. To formalize this, we can calculate the Gaussian averages for both scenarios. In the multimodal case, the Gaussian average of is zero since it's singleton. G()(X)) can be upper bounded by
12
G(G(X)) IE.
sup (E(U,1)
Σ o
Eg
Σ
O(vn).
i=1
In contrast, the Gaussian average of F 9 is larger by factor of vn
n
n
G(Fog(X)) Es
sup (0.1)
o
sin
E.
07
2(n),
Ox;
for some sample X (we leave the proof in the appendix), see also
[24].
This separation in Gaussian averages implies that unimodal learning can be statistically harder than multimodal learning, even if there exists a simple bijective mapping between x, y. We summarize the
intrinsic properties of multimodal learning leading such separation as follows:
Heterogeneity: multimodal data easier to learn than unimodal data.
Connection: mapping between multiple modalities is learnable.
Thereby, the superiority of multi-modality can naturally decomposed into two parts: model trained with multi-modal data performs comparably on uni-modal population data as if multi-modal data is provided (connection), model trained and tested with multi-modal data outperforms any model given only uni-modal data (heterogeneity).
We note that both connection and heterogeneity are crucial achieving such advantage: connection allows efficiently learning of )' from X, while heterogeneity guarantees that learning with X' is harder than learning with both X. Lacking either one can lead ill-conditioned cases: when x = y the connection is perfect while there is no heterogeneity and thus no need to learn anything about y at all. When r a random noise the heterogeneity is large while there can't be any connection between X, V. and it's impossible to come up with a non-trivial learner solely on X.
Remark 2. The example above can be converted to be hard for each single modality. Any potential data point (I, y, =) is now generated by three parameters E (0, 1), 01 E (1, 2), O2 E (-2,-1), under the constraint that 01 + 82 + 0, and (z, 2) is of form (cf1, cB2. c(01 + 02)). The hypothesis classes are now G {g(z)
0z, 0 E (-1,0) (0, 1)] and F sin(1/x). For any uni-modal data (0,1-1/8j) This range is a subset of 1-10 2 1/4. As result, G(F G(X) in this case is at least 1/4 of that the simpler example, thus the term remains Q(n).. On the other hand, we have that max([1 2/0,1, 1/0,1) < G(G(X)) O(yn) holds still. The same argument holds for y similarly.
I = e01, the range of ratio (I + 7)/z is (1 (1- 2/0,
(-1,0) (0, 1) and
have that max([1
The Case of Semi-supervised Multitask Learning
The efficacy of practical multimodal learning often hinges on large models and extensive datasets, with the majority potentially being unlabeled. This is especially true when the training data encompasses broad spectrum of tasks. Given these conditions, within the realm of semi-supervised multitask learning, where the model leverages substantial amounts of unlabeled data from various tasks.
interested in the power of multimodality
Consider the following setup for semi-supervised multitask multimodal learning. The training data taken from number of tasks coming in the form of multi- partitioned into parts S, St. The labeled sample S takes form = (S..... Sr) where S, = (St1,..,, stn) ~ μن in which μt represents the probability measures of the T different tasks from which we draw the independent data points $;; (Its, Vts, 2ts) from. The unlabeled sample S' takes a similar form, that S' = (Si,..,, St) where St ((1a1, W/)), ..... (Iem, )tm)) ~ P(((z.))> drawn independently of S, here by Me.(n.v) denote the marginal distribution of pa. We assume S' has larger size m >>> n than the labeled sample S.
5

---

Using S", we aim to learn connection function, and with S, we learn predictor that leverages both modalities. A common approach to this learning problem is empirical risk minimization (ERM). In particular, we solve the following optimization problem, where the predictors ft on both modalities are learnt via ERM on the labeled sample S.
n
fi..., argmin STE EF nT
Σ
([elTrei Uti), ( zti)-
Meanwhile, the connection 9 is learned by minimizing the distance to the true input, using the unlabeled sample S' instead:
T
m
g = argmin gE9 mf
Σ
(gg(((h))) - sull-
:=1
Our focus is not on solving the above optimization problems (as modern deep learning techniques readily address ERM) but rather the statistical guarantees of the solutions these ERM problems.
To measure the performance of the solution 9. f., fr the modality X, define the task- averaged excess risk as follows:
L(g.fi,...
fT)
1 + T
E E Ln (f(((,,
min
Σ B(c. FEF
In order bound the excess risk, it's crucial to require the class to contain, least, approximate of a "ground truth" connection function, which maps x to for any empirical observation. Later we will show that such requirement is inevitable, which be seen fundamental limit of our theoretical model.
Definition (Approximate realizability). We define the approximate realizability a function class on of input data 5 {(111 V1), (Iins Bn))
RGG
minn .EG
Ie9(x)) ynll-
When the set S is labeled, we abuse the notation RG, S) to denote R(G, (X,Y)) for simplicity.
We have the following theorem that bounds the generalization error of our ERM algorithm in terms of Gaussian averages and the approximate realizability.
Theorem 4, For any > 0, with probability at least 5 in the drawing of the samples S, S", we have that
2 T. Σ
2 (ICI
log(8/0) 2n.T'
L(g, f...... Fr)
C(F(X,
8))
mTT
where Y)) denotes the set of (Iti ((xa)) I,....
Remark 5. It's important to note that the Gaussian average typically on the order of O(vNdl when N is the sample size and d is the intrinsic complexity of the hypothesis class, such as the VC dimension. If we treat as a constant, for most hypothesis classes in machine learning applications, the term G(G(X') typically scales as O (VmT) and each term G(F(X, Y2)) scales as O (Vvn). In practice, learning the connection is often more challenging than learning predictor fe, so it's encouraging to see the leading term (Xx())))mm decay both m, n.T.
Theorem asserts that the ERM model trained with und-modal test data to the optimal model as if multi-mode modall test data is provided, when connection is learnable. This result can naturally extended to accommodate multiple modalities a similar way. In this case, the ERM algorithm would learn mapping from subset of modalities to all modalities, which involves only one hierarchy as in the two-modality case, thus our analysis naturally carries over to this new setting.
data, achieves low excess risk on

---

3.1 Necessity of A Good Connection
Recall that in the upper bound of Theorem 4 all the terms vanish as n, tend to infinity, except for IR(G, S*). It's therefore important to determine whether the term is defect of our analysis or a fundamental limit of our theoretical model. Here we present simple example showing that the dependence on approximate realizability is indeed inevitable.
Let X' = = {0. } and n. T' 2 2. Each probability measure ae is determined by Boolean function be : {0, + {0, 1}, and for each observation st, the label žn = (y()) is purely determined by yr- In particular, the four possible observations
(0, (0, 0, (0)), (1, (1))))
happen with the same probability for any t.
For the hypothesis classes, 9 includes all Boolean functions g {0,
1} - {0, 13, while includes
all 1-Lipschitz functions R - IR. It's straightforward verify that
LRIG S)
( 1 n7
co
C1 nT
-i=1 oi
20o
201
where og are iid Rademacher random variables, and co, ei denotes the number observations with
0 and respectively. The loss function E is set as [f(x, z/-
The simplest version of Bernstein inequality states that for any > 0 and e N+
Σ
m
0
2e
20+
m
m
therefore with probability least 3. we have that 1c0 100-c1 cal s nT since lco - c1 itself can be written in the form of E ozl-
Condition on cco cil < 3vTT, we have that co, < nI 2 + 2 3nT 1. Using the Bernstein
inequality again, with probability least g. it holds Ei= 0il S 8v and similarly for eı Putting
together, with probability least 5, the term LR(G, S) can be lower bounded
1 4v3 LR(G,S)
2 VnT
On the other hand, the population loss of any (x)) composition is clearly at least 1 because the label z independent of x, while the population loss of (f:(I, y)} with the choice of fa y) = by (y) is zero. As a result the excess risk on population is at least which doesn't 't scale with T. When n, T are large enough, the term LR(G, 5) matches the optimal achievable excess risk.
The Role of Heterogeneity
So far we have demonstrated that as long as a good connection indeed exists, learning with multimodal training data using simple ERM algorithm yields a unimodal model which is guaranteed to perform as well as the best model / (ar, y) with both modalities. The story is not yet complete. In order to explain the empirical phenomenon we' re investigating, we still need to determine in what circumstance learning with multimodal data is strictly casier than unimodal learning. A good connection itself isn't 't sufficient: in the case of y = which admits perfect connection function, bringing y into consideration apparently gives no advantage.
To address this question, we move our eyes on heterogeneity, another fundamental property of multimodal learning that describes how modalities diverge and complement each other. Intuitively, heterogeneity can potentially lead to a separation between learnability in the following way: learning from a single modality is much harder than learning from both modalities, in the sense that requires much more complicated hypothesis class. As a result, either the sample complexity is huge due to complicated hypothesis class, or the hypothesis class so simple that even the best hypothesis performs poorly population.
7

---

Consequently, we compare not only the best achievable population risks, but also the generalization errors. For unimodal learning denote as the hypothesis class, we consider the ERM solution
1
arg min
(()(x),
gEG
The generalization error of ğ can be bounded via Gaussian average of the hypothesis class G in the following way if we denote g" = argmin gee peg
[G((())
E w (T(x), 2) ((o.
((
(z),
which is tight in general without additional assumptions. For multimodal learning, F and f* can be defined in the same way.
We are going to show that, either the intrinsic gap of risk between unimodality and multimodality [(xy.))) (æ)) 2) - ((x.g.=)~x.(ff (x,
(3)
substantial, or the Gaussian average gap is large. This implies a separation between multimodal learning and unimodal learning, when heterogeneity is present. Consequently, we define the hetero- geneity gap as follow.
Definition 6 (Heterogeneity gap). the heterogeneity gap between w.r.t some distribution jr and hypothesis class G is defined as
Given fixed hypothesis class F and number of sample 2
H( = G()(X)) + ((
G(F(X,Y)) + E( ( (I. v),
(gg (x)) 2)
n
The above definition measures the population risks between learning with single modality X' or with both modalities X, y. When H (p, 9) is large, unimodal learning is harder since ERM is arguably the optimal algorithm in general. As long as Theorem 4 admits a low risk, the heterogeneity gap itself directly implies the superiority of multi-modality by definition. Therefore, the only question left whether such desired instance (large heterogeneity gap + perfect connection) actually exists.
To this end, the following theorem provides the existence of such instance, proving that our theory is indeed effective. Let's sightly abuse the notation L(ğ) denote the excess population risk of ğ which is the output of the unimodal ERM. We have the following lower bound w.r.t heterogeneity gap. Theorem 7. There exist X, X,Y, F and class U of distributions (X, y, TR), such that 4G, Vn N+, Ap E U.S.I. Hu, 2(11, and Ex (T(()] 2(1).
Meanwhile, let f.9 denote the outputs of the multimodal ERM algorithm in section we have that
39, s.t. Ap U,An N+ L(g. f)
Remark 8. We compare with the work of [16. They showed similar separation in terms of the intrinsic gap (the difference between optimal hypotheses), under a representation learning framework. We make a step further by taking the Gaussian average (how hard learn the optimal hypothesis) into consideration, which is crucial to the (1) gap in the more general setting.
Theorem 7 shows the existence of hard instances, where not only the heterogeneity gap is large, but the difference between actual risks is also substantial. It implies that under certain circumstances, multimodal learning is statistically easy, while unimodal learning incurs constant error matter what hypothesis classes are used.
To sum up. our theory demonstrates that the superiority of multi-modaliit can be explained as the combined impact of connection and heterogeneity: when connection (Theorem 4 and heterogeneity (Theorem 7 both exist, multimodal learning has an edge over unimodal learning even if tested on unimodal data, providing an explanation to the empirical findings. Nevertheless, our theory also suggests a simple principle potentially useful for guiding empirical multimodal learning:
1. Collect numerous unlabeled multimodal data. Learn a connection via generative models.
2. Learn predictor based modest amount of labeled multimodal data.
Such framework can be easily carried out with modern deep learning algorithms, for example we can leam the connection by generative models 22 33).
8

---

4.1 Comparison with Representation Learning
It's possible to Iearn f.g and minimize based observations of single modality X, via representation learning. In particular, representation learning solves the following unimodal ERM problem by treating unknown representation of x, labeled sample where y is hidden
9. fr...,fr argmin fee.
((fe(zei ze).
nl
Unfortunately, although this representation learning method uses the same set of hypotheses, it leads to a worse sample complexity bound. Such method fails to exploit two essential parts of semi-super multimodal learning, namely the rich observation of unlabeled data S", and separate learning of g- Failing to utilize S' will lead to a worse factor G(G(X)) which scales as 1/vnT, while in Theorem the factor G(G(X"))/ml scales as 4/vmf which is much smaller than 1/vnT.
Failing exploit the "explicit representations" y from the training data requires learning composition fo o g from scratch, which typically leads to worst case Gaussian average term, for example in [37] they have maxzeg (FF((g))) instead where S(g) (( g(xx)))) is random set induced by As comparison, our approach decouples the learning of f, 9. and the Gaussian average term G(F Y.) is only measured over the "instance" Ś which can be smaller. In fact, G(F(X, Y))) can be smaller than max G(F(S(g)) up a factor of (Vyn). see the example in appendix.
5 Limitations and Future Directions
In this paper we propose theoretical framework explaining the empirical success of multimodal learning, serving as stepping stone towards more in-depth understanding and development of the theory. Nevertheless, as preliminary study on the relatively unexplored field of theoretical multimodal learning, our result comes with limitations, and potentially opportunities for future research well. We elaborate on these points below.
More natural assumptions: the current set of assumptions, while generally consistent with practical scenarios, does not fully align with them. Although most assumptions are satisfied in practice, the assumption on containing only Lipschitz functions, restrictive: the predictor class F typically comprises deep neural networks. Future work could seek to overcome the Lipschitz assumption, which is not only fundamental to our results but also a cornerstone in representation learning theory.
sppothesii-indeeendeer definitions: our study characterizes multimodal learning through two central properties: connection and heterogeneity. However, their current definitions, RG S) and H (μ, G), depend on both the data and the hypothesis class G. It's interesting to see if we can develop theories with bypothesis-independer definitions, such as mutual information or correlation. A potential difficulty is that such statistical notions are not totally aligned with the perspective of machine learning, for example it could happen that the two modalities are independent with zero mutual information, while there exists a trivial connection mapping.
Fine-grained analysis: our theory, which focuses the statistical guarantees of ERM solutions, is fairly abstract and does not make specific assumptions about the learning problem. To study other more concrete multimodal learning algorithms, for example the subspace learning method, need more fine-grained analysis which takes the particular algorithm into account.
More realistic examples: while the example in section 2. provides theoretical justification, it remains somewhat artificial and diverges from typical multimodal learning problem structures. Ideally, would like to see examples that closely mirror real-world scenarios. For instance, can we progress beyond the current example to provide one that captures the inherent structures of NLP and CV data?
Benefit in optimization: our work demonstrates the advantages of multimodal learning primarily in terms of generalization properties. However, optimization, another crucial aspect of machine learning, could also benefit from multimodality. One potential direction to explore the possibility that multimodal data is more separable and thus casier to optimize. We provide simple example

---

in the appendix that demonstrates the existence of linearly separable multimodal data, where the decision boundary for any single modality is arbitrary.
Conclusion
In this paper we study multimodal learning from perspective of generalization. By adopting classic tools in statistic learning theory on this new problem, we prove upper and lower bounds on the population risk of simple multimodal ERM algorithm. Compared with previous works, our framework improves the upper bound up to an O(yn) factor by decoupling the learning of hypotheses, and gives quantitative example in the separation between multimodal and unimodal learning. Our results relate the heuristic concepts connection and heterogeneity to provable statistical guarantee, providing explanation to important phenomenon in empirical multimodal learning, that a multimodal model can beat a ane-tuned unimodal one. We hope our result, though being a preliminary step into deeper understanding of multimodal learning, can shed some light the directions of future theoretical studies.
References
[1] Massih Amini, Nicolas Usunier, and Cyril Goutte. Learning from multiple partially observed views-an application to multilingual text categorization. Advances in neural information processing systems, 22, 2009.
[2] Rie Kubota Ando, Tong Zhang, and Peter Bartlett. framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6(11), 2005.
[3] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv: /ttp9990..999... 2019.
[4] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learn- ing: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423 443,, 2018.
[5] Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12:149-198, 2000.
[6] Shai Ben-David and Reba Schuller. Exploiting task relatedness for multiple task learning. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003, Proceedings, pages 567-580. Springer, 2003.
[7] Giovanni Cavallanti, Nicolo Cesa-Bianchi, and Claudio Gentile. Linear algorithms for online multitask classification. The Journal of Machine Learning Research, 11:2901-2934, 2010.
[8] Angel Chang, Will Monroe, Manolis Savva, Christopher Potts, and Christopher D Manning. Text to scene generation with rich lexical grounding. arXiv preprint arXiv:/505.06220 2015.
[9] Simon S Du, Wei Hu, Sham Kakade, Jason Lee, and Qi Lei. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002. 09434, 2020.
[10] Georgios Evangelopoulos, Athanasia Zlatintsi, Alexandros Potamianos, Petros Maragos, Kon- stantinos Rapantzikos, Georgios Skoumas, and Yannis Avrithis. Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention. IEEE Transactions Multimedia, 15(7): 2013.
[11] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata, Learning robust representations via view information bottleneck. arXiv preprint arXiv:2002.07017 2020.
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63441413991144.0 2020.
10

---

[13] Wenzhong Guo, Jianwen Wang, and Shiping Wang. Deep multimodal representation learning: survey, IEEE Access, 7:63373-63394, 2019.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[15] Micah Hodosh, Peter Young, and Julia Hockenmaier, Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853- 899, 2013.
[16] Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 34: 34:10944-10956, 2021.
[17] Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Faisal Mahmood, Ruslan Salakhutdinov, and Louis-Philippe Morency. Quantifying & modeling feature interactions: An information decomposition framework. arXiv preprint arXiv:2302. 12247, 2023.
[18) Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint wrXiv:2209.03330. 2022.
[19] Rainer W Lienhart. Comparison of automatic shot boundary detection algorithms. In Storage and retrieval for image and video databases VII, volume 3656, pages 290-301. SPIE, 1998.
[20] Karim Lounici, Massimiliano Pontil, Sara Van De Geer, and Alexandre B Tsybakov. Oracle inequalities and optimal inference under group sparsity. 2011.
[21] Andreas Maurer. Bounds for linear multi-task learning. The Journal of Machine Learning Research, 7:117-139 2006.
[22] Andreas Maurer. A chain rule for the expected suprema of gaussian processes. Theoretical Computer Science, 650: 109-122, 2016.
[23] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. Journal of Machine Learning Research, 17(81):1 2016.
[24] Mehryar Mohri, Afshin Rostamizadeh, and Amect Talwalkar. Foundations of machine learning. MIT press, 2018.
[25] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 689-696, 2011.
[26] OpenAl. Gpt-4 technical report. arXiv, 2023.
[27] Anastasia Pentina and Christoph Lampert. A pac-bayesian bound for lifelong learning. In International Conference on Machine Learning, pages 991-999, PMLR, 2014.
[28] Massimiliano Pontil and Andreas Maurer. Excess risk bounds for multitask learning with trace norm regularization. In Conference on Learning Theory, pages 55-76. PMLR, 2013.
[29] Dhanesh Ramachandram and Graham W Taylor. Deep multimodal learning: A survey recent advances and trends. IEEE signal processing magazine, 34(6);96-108, 2017.
[30] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text image synthesis. In International conference on machine learning, pages 1060-1069. PMLR, 2016.
[31] Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenherg, et al. A generalist agent, arXiv preprint arXiv:2205.06175. 2022.

---

[32] Yunwei Ren and Yuanzhi Li. On the importance of contrastive loss in multimodal learning. arXiv preprint arXiv:2304.03717, 2023.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[34] Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi-view learning. 2008
[35] Xinwei Sun, Yilun Xu, Peng Cao. Yuqing Kong, Lingjing Hu, Shanghang Zhang, and Yizhou Wang. Tegm: An information-theorctii framework for semi-supervised multi-modality learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28,
2020, Proceedings, Part III 16, pages 171-188. Springer, 2020.
[36] Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. Learning to learn, pages 3-17, 1998.
[37] Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory transfer learning: The importance of task diversity. Advances in neural information processing systems, 33:7852-7862. 2020.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[39] Ben P Yuhas. Moise H Goldstein, and Terrence J Sejnowski. Integration of acoustic and visual speech signals using neural networks. JEEE Communications Magazine, 27(11)::6--7,, 1989.
[40] Changqing Zhang, Zongbo Han, Huazhu Fu, Joey Tianyi Zhou, Qinghua Hu, et al. Cpm-nets: Cross partial multi-vien networks. Advances in Neural Information Processing Systems, 32, 2019.
12


Reviews:
[{"review_id": "i95O7j3Stw", "reviewer": "Reviewer_PsHQ", "summary": "The authors presents an interesting theoretical framework that allow them to estimate a generalisation bound for multimodal learning.\nThe main result consist in proving the the bound for the multimodal case is superior to the unimodal one up to a factor $O(\\sqrt{n})$ that depends on the sample size of the dataset considered.", "questions": "Some questions that I would like to have feedback from the others on are:\n- How would you proceed to relax the condition of having a predictor class containing not only Lipschitz functions?\n- How would you generalise the results presented to the case of considering more than two modalities? Namely having $m$ modalities with multiple mapping functions linking single and combination of modalities. Do you see any specific limitation in the current framework?", "limitations": "First of all, the authors did an amazing job in highlighting the limitations of their work in their manuscript.\nHere I simply highlight the major ones:\n- Lipschitz assumption on $\\mathcal{F}$ (predictor class) are restrictive as it prevents a direct extension to DNN\n- Lack of a concrete analysis for specific multimodal learning algorithms", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper is pleasant to read and the work is refreshingly original. Moreover, even if at an abstract level, it builds a theoretical framework for multimodal learning that was long missing in the community, especially given the importance and the relevance that multimodal models  are acquiring in recent times.", "weaknesses": "As also the author points out the results presented are rather abstract and more fine-grained analysis of specific multi-modal learning algorithms would be of broad interest.\nAnother note is the notation usage, some key terms and notations should be briefly explained (it is sufficient in the appendix) to allow less experienced readers to follow the paper more straightforwardly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful feedback and constructive suggestions!\n\n**Fine-grained analysis:** the scope of this current study is primarily to establish a general theoretical framework for multi-modal learning. We will certainly delve into specific algorithms based on our framework in future works.\n\n**Notation usage:** we will clarify the key terms and notations to ensure they are clearly explained.\n\n**Lipschitz assumption:** in general, we believe it's necessary to jump out of the current hypothesis-composition mindset to relax the Lipschitz assumption, since this assumption is also known to be a fundamental limit of representation learning generalization bounds.\n\nWithin our existing framework, assuming zero training loss of $\\mathcal{G}$ on $X'$ ($\\mathcal{R}(\\mathcal{G},X')=0$ which is natural in practice) can partially mitigate this problem. The $L$-dependent term is then $O(\\frac{L}{\\sqrt{m}})$ where $m$ the size of unlabeled data is huge enough to lighten the effect of $L$ (see line 444). Moreover, if we consider the supervised setting, assuming zero training loss will immediately reduce the $L$-dependent term to zero. Future explorations could focus on modifying the analysis for this setting. \n\n**Multiple modalities:** this work is focused on the case of two modalities for the sake of clarity, but our theoretical framework can be extended to accommodate multiple modalities in a similar way. In this case, the ERM algorithm would learn a mapping from a subset of modalities to all modalities, which involves only one hierarchy as in the two-modality case. Therefore our analysis naturally carries over to this new setting. In particular, the hypothesis class $\\mathcal{G}$ will include mappings from the subset to all modalities.\n\nIf a more complicated relationship graph between multiple modalities with more hierarchies is considered and the data don't contain full modalities, we believe new tools beyond the current representation learning framework are required, potentially from the field of graph theory."}, {"review_id": "YwTLF0RdK9", "reviewer": "Reviewer_iaEv", "summary": "The paper establishes theoretical bounds for generalization in multimodal learning, where functions mapping between two modalities and to the label are learned. The authors demonstrate that multimodal learning achieves better generalization by decoupling the learning of hypotheses and provide insights regarding the connection and heterogeneity between modalities.", "questions": "In addition to the questions raised in Weaknesses, I am also curious if the framework presented in the paper can offer insights or potentially be extended to provide analysis for other methods, such as CLIP, where the objective is to learn shared representations for both modalities instead of learning a map from one modality to another.", "limitations": "The authors have adequately discussed limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe paper aims at addressing an important topic and offers theoretical insights into the superiority of multimodal learning over unimodal learning, particularly regarding the decoupling of hypothesis learning.\n2.\tThe use of simple examples at the beginning enhances the readability of the paper and helps illustrate certain aspects of the main ideas.", "weaknesses": "1.\tThe motivating example appears to be restrictive as it relies on F containing only the ground truth mapping from modality Y to label Z, and by the construction the map from Y to Z is exatly the only `hard part' of the learning. It might be slightly better to use the harder example in Remark 2. How would the comparison between multimodal and unimodal learning be in this case?\n2.\tTheorem 7 only shows the existence of cases where multimodal learning surpasses unimodal learning, without delving into a deeper discussion on the required conditions for such cases. As a result, it remains unclear whether these instances solely rely on trivial constructions, such as letting F be a class containing a single function, which is exactly the true map between Y and Z, with learning class G (map between X and Y) constructed as a trivial task. More importantly, from proof of theorem 7, this is exactly how the existence is proved, i.e., by taking the motivating example in section 2.1. Such constructions imply that (1) learning the mapping between modalities is exceedingly simple, and (2) one can directly determine the relationship between a modality and the labels without the need for learning, both of which are impossible in practice. This greatly limits the paper's ability to capture the underlying mechanisms behind the success of multimodal learning. On direction for improvement could be to seek for more realistic instances.\n3.\tThe authors claim that heterogeneity (e.g., lines 167, 255) is one of the two factors leading to the superiority of multimodal learning. However, heterogeneity appears to be more of a consequence rather than the underlying reason. The statement in line 167, explaining heterogeneity as 'multimodal data being easier to learn than unimodal'. Furthermore, in definition 6, heterogeneity is directly defined as the difference between population risks of learning with single and multimodal approaches. This raises my concerns about circular reasoning within the argument. As a result, section 4 and the arguments regarding heterogeneity currently lack clarity and informativeness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive feedback and the thoughtful questions posed. We'll address each of your concerns below.\n\n**The choice of the example:** we discussed the simpler example purely for the sake of clarity, and the harder example in Remark 2 is strictly stronger because the separations between Gaussian averages hold for **both** modalities simultaneously. A slight modification to the construction of Remark 2 can fully recover such separations for both modalities as in the simpler example. Details are given below.\n\nAny potential data point $(x,y,z)$ is now generated by three parameters $c\\in (0,1), \\theta_1 \\in (1,2),\\theta_2 \\in (-2,-1)$, under the constraint that $\\theta_1+\\theta_2\\ne 0$, and $(x,y,z)$ is of form $(c\\theta_1,c\\theta_2,c(\\theta_1+\\theta_2))$. For the learning problem, parameters $\\theta_1,\\theta_2$ are pre-fixed and unknown, data is then generated from some distribution of $c$. The hypothesis classes are now $\\mathcal{G}=\\{g(x)=\\theta x, \\theta \\in (-1,0)\\cup(0,1)\\}$ and $\\mathcal{F}=\\sin (1/x)$. \n\nFor any uni-modal data $x=c\\theta_1$, the range of ratio $(x+y)/x$ is $(1-2/\\theta_1,0)\\cup(0,1-1/\\theta_1)$. This range is a subset of $(-1,0)\\cup(0,1)$ and we have that $\\max(|1-2/\\theta_1|,|1-1/\\theta_1|)\\ge (2/\\theta_1-1+1-1/\\theta_1)/2\\ge 1/4$. As a result, $G(\\mathcal{F} \\circ \\mathcal{G}(X))$ in this case is at least $1/4$ of that in the simpler example because the flip of sign doesn't affect Gaussian averages, thus the term remains $\\Omega(n)$. On the other hand, we have that $\\max(|1-2/\\theta_1|,|1-1/\\theta_1|)\\le 1$, so $G(\\mathcal{G}(X))=O(\\sqrt{n})$ holds still. The same argument holds for $\\mathcal{Y}$ as well since it mirrors $\\mathcal{X}$.\n\n**Lower bound conditions:** in our understanding, your question is can we go beyond worst case constructions and derive a general condition for such separation? While our current construction seems simple, it effectively serves its purpose in illustrating a lower bound. Deriving general instance-dependent separation bounds seems challenging, since this task is not much easier than deriving closed-form estimations of the generalization error. Nevertheless, we agree that more realistic examples can be more insightful. A potential method is to trade some flexibility from $\\mathcal{G}$ to $\\mathcal{F}$. We can also modify the example in Appendix D which is used for a similar purpose, but this example is less intuitive than the one in the main-text and it has other restrictions.\n\n**The role of heterogeneity:** we acknowledge your concern about the role of heterogeneity, and whether it forms circular reasoning. Let's briefly clarify the logic behind. The superiority of multi-modal model over uni-modal model on uni-modal tasks consists of two parts: multi-modal model is comparable to uni-modal model as if multi-modal data is provided (connection), and multi-modal data is easier to learn than uni-modal by **any** uni-modal ERM algorithm (heterogeneity). Therefore, the superiority is the consequence of the two factors connection and heterogeneity.\n\nWhy heterogeneity isn't the consequence of such superiority? Because heterogeneity has stronger requirements: from Definition 6, heterogeneity is defined to be the gap for **any** uni-modal ERM algorithm, while for a particular algorithm with such superiority, the hypothesis class in use doesn't exclude the possibility that there exists better hypothesis classes with a smaller gap. To sum up, connection + heterogeneity is a sufficient condition for superiority, while superiority alone doesn't necessarily imply heterogeneity, and Theorem 7 shows the existence of a scenario with connection + heterogeneity. We will clarify and emphasize this point to avoid potential confusion.\n\n**Extension to CLIP:** we appreciate your suggestion to explore our theory's applicability to CLIP. There are similarities between our work and the approach CLIP employs, although a more nuanced analysis is needed due to CLIP's use of cosine similarity as a specific measure. We'll delve deeper into this in our future work."}, {"review_id": "2jUSPKmIzO", "reviewer": "Reviewer_TbhH", "summary": "This study proposes a new theoretical foundation for multimodal learning. In particular, regarding the phenomenon that models trained in multiple modalities perform better on single-modality tasks than fine-tuned single-modality models, this paper proposes that multimodal learning is a composition of connection and label predictor, and shows that generalization performance is better than single-modality learning when both connection and heterogeneity are present.", "questions": "I would appreciate it if the authors could respond to the above-mentioned weaknesses.", "limitations": "The authors explain the limitations of this theory very clearly.", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper is well-organized and very easy to understand.\n- Although I am not an expert in learning theory, I have checked throughout and found the theory to be valid. In particular, it is interesting and novel to compare the generalization bound with the usual unimodal learning case by setting multimodal learning as a composition of connection and label predictor.", "weaknesses": "- In this study, multimodal learning is considered to be prediction from unimodal data, consisting of prediction by connections from one modality to another and label prediction from them. However, this setting may appear somewhat unnatural when viewed from the perspective of normal multimodal learning. It would be valuable to have a more in-depth discussion on this aspect.\n- In multimodal learning, it is common to consider more than two modalities. It would be even more insightful to explore the potential implications and extensions of this theory when applied to settings involving multiple modalities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback and inquiries!\n\n**Unnatural setting:** we agree that the mapping + predictor learning process we described, may not perfectly align with practical multi-modal learning scenarios. However, the goal of our work is to propose a general theoretical framework, therefore clarity comes with a degree of departure from real-world applicability. The perspective we offer on multi-modal learning is one of many, and we believe that it provides an intuitive way to understand the mechanisms underlying the success of multi-modal learning.\n \nIn addition, beyond the theorem statement, our theory may have implications to more general scenarios: the process of learning connections between modalities could be implicitly occurring in empirical multi-modal learning. This poses an intriguing research question: can we establish similar generalization bounds for a more practical \"one-pass\" algorithm, which does not require learning connections and predictors separately?\n\n**Multiple modalities:** this work is focused on the case of two modalities for the sake of clarity, but our theoretical framework can be extended to accommodate multiple modalities in a similar way. In this case, the ERM algorithm would learn a mapping from a subset of modalities to all modalities, which involves only one hierarchy as in the two-modality case. Therefore our analysis naturally carries over to this new setting. In particular, the hypothesis class $\\mathcal{G}$ will include mappings from the subset to all modalities.\n\nIf a more complicated relationship graph between multiple modalities with more hierarchies is considered and the data don't contain full modalities, we believe new tools beyond the current representation learning framework are required, potentially from the field of graph theory."}, {"review_id": "VZI0Ccmrcx", "reviewer": "Reviewer_wXhy", "summary": "This paper introduces a theoretical framework aimed at elucidating the phenomenon wherein a multimodal neural network, trained on multimodal data, can exhibit strong performance on unimodal data. The framework incorporates Gaussian averaging and operates within a semi-supervised context. The findings demonstrate that under the conditions of heterogeneity and connectivity, it is possible to identify a bridging function $\\hat{g}$ and a projection function $\\hat{f}$ that enable comparable generalization error on unimodal data, as if the network were provided with multimodal data.", "questions": "Overall, I believe the paper is of good quality and deserves acceptance. The questions I'm about to raise are intended for discussion purposes and should be considered by the authors. Lastly, I have some suggestions for improving the writing, particularly in areas that could benefit from greater clarity, especially for readers encountering the content for the first time.\n\n1) I recommend that the authors rephrase the explanations regarding Equation (1). Based on Section 3 with T=1, my understanding is that we solve for $\\hat{f_1}$ and $\\hat{g}$ using ERM, substituting them back into $L(\\hat{g},\\hat{f}_1)$ as shown between lines 192 and 193, which is identical to Equation (1). In other words, we learn $\\hat{f}_1$ and $\\hat{g}$ on multimodal data. However, the current wording in line 137 states that we are attempting to minimize Equation (1) with respect to $\\hat{f}$ and $\\hat{g}$, which may confuse readers. When I first encountered Equation (1), I assumed that the second term had no impact because it does not depend on $\\hat{f}$ and $\\hat{g}$, leading me to believe that Equation (1) simplifies to solving for $\\hat{f}$ and $\\hat{g}$ based solely on $x$.\n\n2) If my understanding is correct, I suggest explicitly writing out $s_{t1}=(x_{t1},y_{t1},z_{t1})$ in line 186.\n\n3) The learning framework presented in this section assumes learning the bridging function $\\hat{g}$ on a set of unlabeled data and the projection functions $\\hat{f_t}$ on the labeled dataset. While this is certainly a valid learning framework, it seems to me that the most common one might be a fully supervised setting where paired data $(x_i, y_i, z_i)$ are given.\n\n4) In line 8, the authors state that a multimodal network trained on a multimodal dataset can outperform a unimodal network even on unimodal tasks, and this paper aims to demonstrate that. However, in my understanding, this paper actually shows that a multimodal network can perform comparably on a unimodal dataset **as if** multimodal data is provided, as indicated by Equation (1). These two statements differ.\n\n5) The model considered by the authors takes the form of $f(x,y\\approx g(x))$. It seems to deviate from what is typically used in the literature. A common form found in the literature is $f(g(x),h(y))$, where $g$ and $h$ are two encoders, and $f$ could represent early/middle/late fusion.\n\n6) I would like to bring an important topic to the attention of the authors. There have been several explorations on multimodal knowledge distillation. Under this topic, it would be highly interesting to theoretically analyze the performance of multimodal (or unimodal) teachers (or students). For further reference, please see the following paper: https://arxiv.org/abs/2206.06487. It would be fascinating to extend this work to knowledge distillation settings as a future area of research.\n\n7) I would suggest the authors to include literature on multi-view learning as Ref [16] did. Because that is closely related to multimodal learning, or some might argue their theory are essentially the same. \n\n8) It will be good to include numerical experiments to support the theorems (or at least the claim in line 258-260). ", "limitations": "The authors clearly stated the limitation of the work in the last section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear presentation, nice illustrative sinusoidal function, solid technical contribution.", "weaknesses": "Some clarifications are needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your thorough feedback and constructive suggestions! We will address each of your questions individually and please feel free to let us know if you have further concerns.\n\n**Q1:** thank you for highlighting this. Your understanding is accurate, and we will revise our language to make this clearer and prevent potential confusion.\n\n**Q2:** we agree with your suggestion and will incorporate this change in our revised manuscript.\n\n**Q3:** we agree that a fully supervised setting is more common, however, we decided to explore the more general semi-supervised setting for a few reasons: first, using unlabeled data can provide sharper bounds when available and utilized; second, in practice, labeled multi-modal data is rarer than unlabeled data, and the latter is crucial to the empirical success of recent large-scale multi-modal models; lastly, the supervised setting can be subsumed by the semi-supervised setting by partitioning the data.\n\n**Q4:** you have precisely interpreted what Theorem 4 states. However, in line 8, we were also implicitly taking the separation between multi-modal and uni-modal (Theorem 7) into account. We will make the statement more precise.\n\n**Q5:** we focus on a general theory, thus we only consider the simplest case $f(x,y)$. Notice that the encoder form can be subsumed by the general form, by setting the hypothesis class to include $f(g,h)$. This opens up an interesting research question, that under what assumption (for example, when data is low-rank) the encoder form can lead to better generalization bounds.\n\n**Q6:** we greatly appreciate your suggestion. We will delve into the referenced materials and investigate the potential application of our theory in knowledge distillation.\n\n**Q7:** we will expand our discussion on multi-view learning, and make clear distinctions where necessary as you suggested.\n\n**Q8:** we feel re-validating the theorems with toy experiments somewhat marginal, as this work provides a theoretical explanation   rather than a better algorithm. On the other hand, large-scale experiment is beyond the scope of the current work. We agree experiments might be more useful for the principle part (line 258-260), but for practical purposes modification of the algorithm is needed, and we leave it to future study."}, {"review_id": "t530lmxaye", "reviewer": "Reviewer_XfDh", "summary": "This paper aims to provide a theoretical framework for multimodal learning, that aims to explain why multimodal learning is sometimes better than unimodal learning even when the model is only applied on unimodal tasks. To do this, the authors measure the generalization bounds of multimodal learning on the excess population with Gaussian average, and showed multimodal has a superior generalization bound. They also try to formulate the problem with connection and heterogeneity, and show when multimodal is better than unimodal under their evaluation metrics.", "questions": "NA", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Motivation is strong. Theoretical understanding of multimodal learning is an important yet less investigated problem.\n2. The theoretical investigation is interesting and relatively novel to me. It is particularly interesting to characterize/formulate the multimodal problem into connection and heterogeneity.", "weaknesses": "1. The authors nicely concluded their weakness in their limitations section, I think the below points are the major weakness that are non-neglectable (although pointed out in the final section): The work needs more natural assumptions, it currently requires too many hypothesis, and it needs more realistic examples.\n2. Related to the above point -- I think the results would be much stronger if it can be backed up by some toy experiments with synthetic datasets to demonstrate how the proof are useful in real-world cases. The current theoretical results are nice, but that content is not sufficient enough to be published on its own for NeurIPS.\n3. The presentation/writing of the paper could be further improved (by a lot). e.g. Line 17 ’Diamond Sutra’; line 224 \"The story is not yet complete\"; line 245 \"Let’s\". \n\nOverall, I think this is an interesting work that needs further iterations/improvements to be able to be published at NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive feedback and the thoughtful questions posed. We'll address each of your concerns below.\n\n**More natural assumptions:** our work is devoted to providing a general theory of multi-modal learning, which inevitably comes at the cost of some loss in practicality. We feel the Lipschitz assumption is the major one, which other reviewers also agree upon. In general, we believe it's necessary to jump out of the current hypothesis-composition mindset to relax the Lipschitz assumption, since this assumption is also known to be a fundamental limit of representation learning generalization bounds.\n\nWithin our existing framework, assuming zero training loss of $\\mathcal{G}$ on $X'$ ($\\mathcal{R}(\\mathcal{G},X')=0$ which is natural in practice) can partially mitigate this problem. The $L$-dependent term is then $O(\\frac{L}{\\sqrt{m}})$ where $m$ the size of unlabeled data is huge enough to lighten the effect of $L$ (see line 444). Moreover, if we consider the supervised setting, assuming zero training loss will immediately reduce the $L$-dependent term to zero. Future explorations could focus on modifying the analysis for this setting. \n\n**Toy experiments:** in our opinion, the benefit of toy experiments with synthetic data somewhat marginal. Our theory is on explaining an existing phenomenon instead of proposing a better algorithm, thus re-validating the theory seems less significant to us: toy experiments can only serve as a sanity check to what our theory has rigorously proven. On the other hand, large-scale experiment is beyond the scope of the current work, and such phenomenon is already widely observed on large-scale practices of multi-modal learning. For these reasons, we believe experiment is more of a complement rather than a necessity to our theory.\n\nIs there a specific experimental setup that the reviewer would like to see?\n\n**Theory is not sufficient:** we believe the theoretical results are significant enough on their own. Here are a few points raised also by you and other reviewers: the problem studied is important yet less investigated, which became even more urgent due to recent empirical success; the proposed theory is intuitive and original; the technical contributions are solid. \n\n**Writing:** thank you for catching some typos we missed. We will correct them and make a pass of the paper to improve writing."}]

Author rebuttal:

Meta-review: The paper provides a theoretical framework for multimodal learning and shows that when functions mapping between two modalities and to the label are learned, multimodal learning achieves a genearlization performance that is superior to the unimodal one up to a factor that depends on the square root of the sample size. I believe the paper provides valuable insights for the community and can be accepted as is. However, I strongly encourage the authors to incorporate the clarifications on the theory suggested by iaEv, discussion on and comparison to multi-view learning suggested by wXhy, and add an experiment on the example used in Section 2.1 suggested by XfDh. These will greatly improve the clarify of the paper.
Decision: Accept (poster)