SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhou Lu
zhoul@princeton.edu
Human perception of the empirical world involves recognizing the diverse appearances, or 'modalities', of underlying objects. Despite the longstanding consideration of this perspective in philosophy and cognitive science, the study of multimodality remains relatively under-explored within the field of machine learning. Nevertheless, current studies of multimodal machine learning are limited to empirical practices, lacking theoretical foundations beyond heuristic arguments. An intriguing finding from the practice of multimodal learning is that a model trained on multiple modalities can outperform a finely-tuned unimodal model, even on unimodal tasks. This paper provides a theoretical framework that explains this phenomenon, by studying generalization properties of multimodal learning algorithms. We demonstrate that multimodal learning allows for a superior generalization bound compared to unimodal learning, up to a factor of  $O(\sqrt{n})$ , where  $n$  represents the sample size. Such advantage occurs when both connection and heterogeneity exist between the modalities.
Even before the common era, the concept of viewing an object as the collection of its appearances, has already sprouted in early philosophy. The Buddha, in the 'Diamond Sutra', separated the essence of universe from various modalities such as sight, sound, smell, taste and touch. Two centuries ago, Immanuel Kant made a further step, positing that humans perceive only the representations of 'noumena' from the empirical world. He wrote:
"And we indeed, rightly considering objects of sense as mere appearances, confess thereby that they are based upon a thing in itself, though we know not this thing in its internal constitution, but only know its appearances, viz., the way in which our senses are affected by this unknown something. - Prolegomena"
From this perspective, human cognition of the world, may therefore be considered effectively equivalent to the multiple modalities of the underlying objects. The importance of multimodality extends beyond metaphysics to everyday life: children learning languages often rely on illustrations, and even mathematicians benefit from visual aids.
However, machine learning, which could be seen as the cognition of computer systems, has not fully harnessed the power of multimodality. Multimodal machine learning, which processes and learns from data with multiple modalities, remained relatively under-explored until recently. Despite the impressive success of multimodal learning in empirical applications, such as Gato [31] and GPT-4 [26], the corresponding theoretical understanding is largely absent, often limited to heuristics.
A fascinating observation from empirical multimodal learning is that a model trained with multiple modalities can outperform a finely-tuned unimodal model, even on population data of the same unimodal task. It's not immediately clear why multimodality offers such an advantage, considering that the trained model's focus is spread across different modalities.
While it seems challenging to outperform unimodal learning asymptotically when sufficient data is available, multimodal learning can still provide an edge under a fixed data budget. Different modalities might focus on different aspects of an object, and for a specific classification problem, one modality may require a smaller sample complexity. This phenomenon often occurs with large models handling many tasks and a vast amount of training data, suggesting that:
Training across tasks learns a common connection between modalities efficiently, allowing the model to adapt to the modality with the smallest sample complexity.
An intuitive example of how multiple modalities help is learning parametric sine functions. The samples come in the form of
$$
x \in (0, 1 ], y = \theta x, z = \sin (1 / y),
$$
where  $x, y$  are the two modalities and  $z$  is the label. Given data from both modalities the learning problem is trivial, even with a single training data point, while learning solely on  $x$  is hard albeit there is a bijective mapping between  $x, y$ . From a perspective of VC-dimension, there is a gap between the class of linear functions  $\{\theta x\}$  and the class of parametric sine functions  $\{\sin(1/\theta x)\}$ , in that the former one has VC-dimension 1 while the latter one has infinite VC-dimension. More details will be provided later.
The theory problem we study in this paper, is thus how to formalize the above heuristic with provable guarantees. To this end, we examine generalization bounds of a simple multimodal ERM algorithm, which involves two parallel stages: learning a predictor  $\hat{f} \in \mathcal{F}$  based on multimodal training data, and learning a connection  $\hat{g} \in \mathcal{G}$  that maps one modality to another with potentially unlabeled data. During inference, the composition  $\hat{f} \circ \hat{g}$  is used to perform prediction on unimodal population data.
In this setting, we prove that the learnt unimodal predictor  $\hat{f} \circ \hat{g}$  can achieve vanishing generalization error against the best multimodal predictor  $f^{*}$  as if given multiple modalities, whenever  $\mathcal{G}$  is expressive enough to realize the training data. In addition, such generalization bound depends on the complexities of both hypothesis classes  $\mathcal{F}, \mathcal{G}$  separately, better than unimodal approaches which typically involve the complexity of  $\mathcal{F} \circ \mathcal{G}$  or a worst-case complexity of  $\mathcal{F}$ , up to an  $O(\sqrt{n})$  factor where  $n$  denotes the size of training data. On the other hand, we show a separation between multimodal and unimodal learning, by constructing a hard instance learnable by multimodal learning, in which no matter what hypothesis class is chosen for the unimodal learning problem, it's either under-expressive or over-expressive and thus incurs constant error. Putting the two pieces together, our theory suggests that with both connection and heterogeneity, multimodal learning is provably better than unimodal learning.
The paper is organized as follows. In section 2 we formalize the setting of multimodal learning and provide a motivating example. Section 3 proves a generalization upper bound of the two-stage multimodal ERM algorithm on semi-supervised multitask learning problems. The lower bound on the separation between multimodal and unimodal learning is given in section 4, then we discuss the limitations of this paper and future directions in section 5.
Theoretical Multimodal Learning: while empirical multimodal learning has shown significant progress, theoretical studies are relatively sparse, lacking a firm foundation. Prior theoretical investigations often focus on specific settings or incorporate additional assumptions. Some of these studies adopt an information-theoretic perspective, proposing algorithms based on total correlation or utilizing partial information decomposition to quantify relationships between modalities [35, 17]. Other studies approach the problem from a multi-view setting, typically assuming that each view alone suffices for prediction [40, 1, 11, 34].
An important work in theoretical multimodal learning is [16], which also considered the advantage of multimodal learning in generalization and is the first and probably the only general theoretical result in this field so far. In particular, they considered the population risk of a representation learning based approach where learning under different subsets of modalities is performed on the same ERM objective with shared hypothesis classes. They proved that the gap between the population risks of different subsets of modalities is lower bounded by the difference between what they called the latent
representation quality, which is the best achievable population risk with the learnt representation on the chosen subset of modalities.
There are two limitations in this result: 1, there is no quantitative characterization on how large the gap between latent representation qualities can be; 2, the comparison is not-only instance-dependent, but also carried over the same hypothesis classes and doesn't exclude the possibility that the smaller subset of modalities could potentially use a different class to bypass the gap, making the lower bound somewhat restricted. We strengthen this result by showing the gap can be as large as  $\Omega(1)$  (Theorem 7), even if we allow the smaller subset of modalities to use any hypothesis class. They also showed an upper bound on the excess population risk via a standard representation learning analysis, which involves the complexity of a composition of hypothesis classes  $\mathcal{F} \circ \mathcal{G}$ , while our analysis decouples the complexities of hypothesis classes, leading to an improved upper bound up to a factor of  $O(\sqrt{n})$ .
A recent work of [32] made a more fine-grained study on multimodal learning, analyzing the benefit of contrastive loss in training dynamics. They considered the aspect of optimization instead of generalization for a particular problem, focusing on the setting of a linear data-generating model. They proved that the use of contrastive loss is both sufficient and necessary for the training algorithm to learn aligned and balanced representations.
Empirical Multimodal Learning: the inception of multimodal learning applications dates back to the last century, initially devised to enhance speech recognition using both vision and audio [39, 25]. Multimedia is another aspect in which multimodal learning inspired new methods for indexing and searching [10, 19]. With the development of deep learning and its success in computer vision [14] and natural language processing [38], people start studying deep multimodal learning in related tasks such as generating one modality from the other [8, 15, 30]. For a more comprehensive introduction of multimodal learning, we refer the readers to the excellent survey paper [4], see also [29, 13, 18].
Recently, the power of multimodal learning was carried out in large-scale generalist models. In [31], the training data includes a wide variety of modalities such as image, text, robotics and so on. The resulting model is reported to be able to beat fine-tuned unimodal models in some tasks. In a more recent ground-breaking result, the super-large language model GPT-4 [26] makes use of not only text data available on internet, but also data from other modalities such as audio, demonstrating excellent capabilities in integrating knowledge from multiple domains.
Representation Learning: this field, closely related to our work, focuses on learning a common underlying representation across multiple tasks. The typical framework of representation learning involves solving an ERM problem with a composition of hypotheses  $f_{t} \circ g$ , where  $f_{t}$  is task-specific while  $g$  is the common representation. Generalization bounds of representation learning usually involve the complexity of  $\mathcal{F} \circ \mathcal{G}$  or a worst-case complexity of  $\mathcal{F}$ .
Starting from Baxter's study [5] which gave theoretical error bounds via covering numbers on the inductive bias learning approach [36], a long line of work has followed, each improving upon and generalizing the previous results [6, 2, 21, 7, 20, 28, 27].
For more recent works we detail several representative ones here. The work of [23] studied both representation learning and transfer learning in the setting of multitask learning, achieving dimension independent generalization bounds with a chain rule on Gaussian averages [22]. For the problem of transfer learning, [37] improved the leading term of [23] by a  $O(\sqrt{n})$  factor under a task diversity assumption, while [9] obtained a similar bound under low-dimension and linear function assumptions. [3] analyzed a generalized setting called contrastive learning inspired by the success of empirical language models.
In this paper, we consider a straightforward yet non-trivial case of two modalities to ensure clarity in our presentation. Formally, we denote the set of possible observations  $S$  to be  $(\mathcal{X},\mathcal{Y},\mathbb{R})$ . Here, each element  $s\in S$  constitutes a pairing of inputs from both modalities  $x\in \mathcal{X}\subset \mathbb{R}^q,y\in \mathcal{Y}\subset \mathbb{R}^k$  and their associated label  $z\in \mathbb{R}$ , thus forming a tuple  $(x,y,z)$ . Assuming without loss of generality that both  $\mathcal{X}$  and  $\mathcal{Y}$  are contained within their respective Euclidean unit balls. Given a probability measure  $\mu$  on  $S$  and a loss function  $\ell$ , the performance of a learning algorithm  $\mathcal{A}$  is measured by the population loss if we interpret  $\mathcal{A}$  as a function, namely
$$
\mathbb {E} _ {(x, y, z) \sim \mu} \ell (\mathcal {A} (x, y), z).
$$
To leverage the hidden correlation between different modalities, we aim to learn both a connection function  $g$  bridging the modalities and a prediction function  $f$  that accepts inputs from both modalities. Although we focus on learning a connection from  $\mathcal{X}$  to  $\mathcal{Y}$  for simplicity, a symmetrical approach can handle the reverse direction.
In particular, we will consider learning algorithms as a composition of functions  $\mathcal{A}(x,y) = f(x,g(x))$ , where  $f\in \mathcal{F}$  and  $g\in \mathcal{G}$  represent the hypothesis classes for both functions. This form is most general and common practical forms such as fusion  $\mathcal{A}(g(x),h(y))$  can be subsumed by the general form.
The goal is to identify  $\hat{f},\hat{g}$  using multi-modal training data, to minimize the excess population risk
$$
\mathbb {E} _ {(x, y, z) \sim \mu} \ell (\hat {f} (x, \hat {g} (x)), z) - \min  _ {f \in \mathcal {F}} \mathbb {E} _ {(x, y, z) \sim \mu} \ell (f (x, y), z). \tag {1}
$$
In this context, we compare with the optimal predictor  $f^{*}$  as if given complete observations of both modalities, because our objective is to achieve a performance comparable to the best predictor given both modalities. The reason is that such a predictor could have a significant advantage over any unimodal predictor that does not learn these connections (either explicitly or implicitly).
We seek statistical guarantees for  $\hat{f}$  and  $\hat{g}$ . To provide generalization bounds on the excess population risk, we require a complexity measure of hypothesis classes, defined as follows.
Definition 1 (Gaussian average). Let  $Y$  be a non-empty subset of  $\mathbb{R}^n$ , the Gaussian average of  $Y$  is defined as
$$
G (Y) = \mathbb {E} _ {\sigma} \left[ \sup  _ {y \in Y} \sum_ {i = 1} ^ {n} \sigma_ {i} y _ {i} \right]
$$
where  $\sigma_{i}$  are iid standard normal variables. Similarly, we can define the function version of Gaussian average. Let  $\mathcal{G}$  be a function class from the domain  $\mathcal{X}$  to  $\mathbb{R}^k$ , and  $X = \{x_{1},\ldots ,x_{n}\}$  be the set of input sample points. We define the Gaussian average of the class  $\mathcal{G}$  on sample  $X$  as:
$$
G (\mathcal {G} (X)) = \mathbb {E} _ {\sigma} \left[ \sup  _ {g \in \mathcal {G}} \sum_ {i = 1} ^ {k} \sum_ {j = 1} ^ {n} \sigma_ {i, j} g _ {i} \left(x _ {j}\right) \right],
$$
where  $\sigma_{i,j}$  are iid standard normal variables.
We make the following Lipschitz assumption on the class  $\mathcal{F}$  and the loss function  $\ell$ , which is standard in literature.
Assumption 1. We assume that any function  $f: \mathbb{R}^{q + k} \to \mathbb{R}$  in the class  $\mathcal{F}$  is  $L$ -Lipschitz, for some constant  $L > 0$ . The loss function  $\ell$  takes value in  $[0,1]$ , and is 1-Lipschitz in the first argument for every value of  $z$ .
2.1 A Motivating Example
To introduce our theoretical findings, we present a straightforward but insightful example, illustrating the circumstances and mechanisms through which multimodal learning can outperform unimodal learning. Despite its simplicity and informal nature, this example captures the core concept of why multimodal learning requires both connection and heterogeneity, and we believe it is as vital as the more formal statements that follow.
Consider the problem where  $\mathcal{X} = \mathcal{Y} = (0,1]$ . Any potential data point  $(x,y,z)$  from  $\mathcal{S}$  is governed by a parameter  $\theta^{*}\in (0,1]$ , such that
$$
y = \theta^ {*} x, z = \sin (1 / y).
$$
The loss function choice is flexible in this case, and any frequently used loss function like the  $\ell_1$  loss will suffice.
Suppose we have prior knowledge about the structure of the problem, and we select  $\mathcal{G} = g(x) = \theta x|\theta \in (0,1]$  and  $\mathcal{F} = \sin(1/x)$  as our hypothesis classes. If we have sampled data from both modalities, we can easily learn the correct hypothesis via Empirical Risk Minimization (ERM): simply take any  $(x,y,z)$  sample and compute  $\theta = y/x$ .
However, if we only have sampled data with the  $\mathcal{V}$  modality concealed, there could be multiple  $\theta$  values that minimize the empirical loss, making the learning process with  $\mathcal{F} \circ \mathcal{G}$  significantly more challenging. To formalize this, we can calculate the Gaussian averages for both scenarios. In the multimodal case, the Gaussian average of  $\mathcal{F}$  is zero since it's a singleton.  $G(\mathcal{G}(X))$  can be upper bounded by
$$
G (\mathcal {G} (X)) = \mathbb {E} _ {\sigma} \left[ \sup  _ {\theta \in (0, 1 ]} \theta \sum_ {i = 1} ^ {n} \sigma_ {i} x _ {i} \right] \leq \mathbb {E} _ {\sigma} \left[ | \sum_ {i = 1} ^ {n} \sigma_ {i} x _ {i} | \right] = O (\sqrt {n}).
$$
In contrast, the Gaussian average of  $\mathcal{F} \circ \mathcal{G}$  is larger by a factor of  $\sqrt{n}$
$$
G (\mathcal {F} \circ \mathcal {G} (X)) = \mathbb {E} _ {\sigma} \left[ \sup  _ {\theta \in (0, 1 ]} \sum_ {i = 1} ^ {n} \sigma_ {i} \sin \left(\frac {1}{\theta x _ {i}}\right) \right] \geq \mathbb {E} _ {\sigma} \left[ \sum_ {i = 1} ^ {n} \frac {1}{2} | \sigma_ {i} | \right] = \Omega (n), \tag {2}
$$
for some sample  $X$  (we leave the proof in the appendix), see also [24].
This separation in Gaussian averages implies that unimodal learning can be statistically harder than multimodal learning, even if there exists a simple bijective mapping between  $x, y$ . We summarize the intrinsic properties of multimodal learning leading to such separation as follows:
Heterogeneity: multimodal data is easier to learn than unimodal data.
Connection: a mapping between multiple modalities is learnable.
Thereby, the superiority of multi-modality can be naturally decomposed into two parts: a model trained with multi-modal data performs comparably on uni-modal population data as if multi-modal data is provided (connection), a model trained and tested with multi-modal data outperforms any model given only uni-modal data (heterogeneity).
We note that both connection and heterogeneity are crucial to achieving such advantage: connection allows efficiently learning of  $\mathcal{V}$  from  $\mathcal{X}$ , while heterogeneity guarantees that learning with  $\mathcal{X}$  is harder than learning with both  $\mathcal{X},\mathcal{Y}$ . Lacking either one can lead to ill-conditioned cases: when  $x\equiv y$  the connection is perfect while there is no heterogeneity and thus no need to learn anything about  $\mathcal{V}$  at all. When  $x$  is a random noise the heterogeneity is large while there can't be any connection between  $\mathcal{X},\mathcal{Y}$ , and it's impossible to come up with a non-trivial learner solely on  $\mathcal{X}$ .
Remark 2. The example above can be converted to be hard for each single modality. Any potential data point  $(x,y,z)$  is now generated by three parameters  $c\in (0,1),\theta_1\in (1,2),\theta_2\in (-2, - 1)$  under the constraint that  $\theta_{1} + \theta_{2}\neq 0$  , and  $(x,y,z)$  is of form  $(c\theta_{1},c\theta_{2},c(\theta_{1} + \theta_{2}))$  . The hypothesis classes are now  $\mathcal{G} = \{g(x) = \theta x,\theta \in (-1,0)\cup (0,1)\}$  and  $\mathcal{F} = \sin (1 / x)$  . For any uni-modal data  $x = c\theta_{1}$  , the range of ratio  $(x + y) / x$  is  $(1 - 2 / \theta_{1},0)\cup (0,1 - 1 / \theta_{1})$  . This range is a subset of  $(-1,0)\cup (0,1)$  and we have that  $\max (|1 - 2 / \theta_1|,|1 - 1 / \theta_1|)\geq 1 / 4$  . As a result,  $G(\mathcal{F}\circ \mathcal{G}(X))$  in this case is at least  $1 / 4$  of that in the simpler example, thus the term remains  $\Omega (n)$  . On the other hand, we have that  $\max (|1 - 2 / \theta_1|,|1 - 1 / \theta_1|)\leq 1$  , so  $G(\mathcal{G}(X)) = O(\sqrt{n})$  holds still. The same argument holds for  $\mathcal{V}$  similarly.
3 The Case of Semi-supervised Multitask Learning
The efficacy of practical multimodal learning often hinges on large models and extensive datasets, with the majority potentially being unlabeled. This is especially true when the training data encompasses a broad spectrum of tasks. Given these conditions, we are interested in the power of multimodality within the realm of semi-supervised multitask learning, where the model leverages substantial amounts of unlabeled data from various tasks.
Consider the following setup for semi-supervised multitask multimodal learning. The training data is taken from a number of tasks coming in the form of a multi-sample, partitioned into two parts  $S$ ,  $S'$ . The labeled sample  $S$  takes form  $S = (S_{1},\dots,S_{T})$  where  $S_{t} = (s_{t1},\dots,s_{tn}) \sim \mu_{t}^{n}$ , in which  $\mu_{t}$  represents the probability measures of the  $T$  different tasks from which we draw the independent data points  $s_{ti} = (x_{ti},y_{ti},z_{ti})$  from. The unlabeled sample  $S'$  takes a similar form, that  $S' = (S_1',\dots,S_T')$  where  $S_t' = ((x_{t1},y_{t1}),\dots,(x_{tm},y_{tm})) \sim \mu_{t,(x,y)}^m$ , drawn independently of  $S$ , here by  $\mu_{t,(x,y)}$  we denote the marginal distribution of  $\mu_t$ . We assume  $S'$  has a larger size  $m \gg n$  than the labeled sample  $S$ .
Using  $S'$ , we aim to learn a connection function, and with  $S$ , we learn a predictor that leverages both modalities. A common approach to this learning problem is empirical risk minimization (ERM). In particular, we solve the following optimization problem, where the predictors  $\hat{f}_t$  on both modalities are learnt via an ERM on the labeled sample  $S$ ,
$$
\hat {f} _ {1},.., \hat {f} _ {T} = \mathbf {a r g m i n} _ {f _ {1}, \dots , f _ {T} \in \mathcal {F}} \frac {1}{n T} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {n} \ell (f _ {t} (x _ {t i}, y _ {t i}), z _ {t i}).
$$
Meanwhile, the connection  $\hat{g}$  is learned by minimizing the distance to the true input, using the unlabeled sample  $S^{\prime}$  instead:
$$
\hat {g} = \mathbf {a r g m i n} _ {g \in \mathcal {G}} \frac {1}{m T} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {m} \| g (x _ {t i} ^ {\prime}) - y _ {t i} ^ {\prime} \|.
$$
Our focus is not on solving the above optimization problems (as modern deep learning techniques readily address ERM) but rather on the statistical guarantees of the solutions to these ERM problems.
To measure the performance of the solution  $\hat{g},\hat{f}_1,\dots ,\hat{f}_T$  on the modality  $\mathcal{X}$ , we define the task-averaged excess risk as follows:
$$
L (\hat {g}, \hat {f} _ {1}, \dots , \hat {f} _ {T}) = \frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} _ {(x, y, z) \sim \mu_ {t}} \ell (\hat {f} _ {t} (x, \hat {g} (x)), z) - \min _ {f \in \mathcal {F}} \frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} _ {(x, y, z) \sim \mu_ {t}} \ell (f _ {t} (x, y), z).
$$
In order to bound the excess risk, it's crucial to require the class  $\mathcal{G}$  to contain, at least, an approximate of a "ground truth" connection function, which maps  $x$  to  $y$  for any empirical observation. Later we will show that such requirement is inevitable, which can be seen a fundamental limit of our theoretical model.
Definition 3 (Approximate realizability). We define the approximate realizability of a function class  $\mathcal{G}$  on a set of input data  $S = \{(x_{1},y_{1}),\ldots ,(x_{n},y_{n})\}$  as
$$
\mathcal {R} (\mathcal {G}, S) = \min  _ {g \in \mathcal {G}} \frac {1}{n} \sum_ {i = 1} ^ {n} \| g (x _ {n}) - y _ {n} \|.
$$