SlowGuess/NeurIPS_2023_OCR huggingface dataset:
When the set  $S$  is labeled, we abuse the notation  $\mathcal{R}(\mathcal{G}, S)$  to denote  $\mathcal{R}(\mathcal{G}, (X, Y))$  for simplicity.
We have the following theorem that bounds the generalization error of our ERM algorithm in terms of Gaussian averages and the approximate realizability.
Theorem 4. For any  $\delta >0$ , with probability at least  $1 - \delta$  in the drawing of the samples  $S,S^{\prime}$  we have that
$$
L (\hat {g}, \hat {f} _ {1}, \dots , \hat {f} _ {T}) \leq \frac {\sqrt {2 \pi}}{n T} \sum_ {t = 1} ^ {T} G (\mathcal {F} (\hat {X} _ {t}, \hat {Y} _ {t})) + \frac {2 \sqrt {2 \pi} L}{m T} G (\mathcal {G} (X ^ {\prime})) + L \mathcal {R} (\mathcal {G}, S ^ {\prime}) + (8 L + 4) \sqrt {\frac {\log (8 / \delta)}{2 n T}},
$$
where  $(\hat{X}_t,\hat{Y}_t)$  denotes the set of  $\{x_{ti},\hat{g} (x_{ti})|i = 1,\dots,n\}$
Remark 5. It's important to note that the Gaussian average is typically on the order of  $O(\sqrt{Nd})$  when  $N$  is the sample size and  $d$  is the intrinsic complexity of the hypothesis class, such as the VC dimension. If we treat  $d$  as a constant, for most hypothesis classes in machine learning applications, the term  $G(\mathcal{G}(X'))$  typically scales as  $O(\sqrt{mT})$  and each term  $G(\mathcal{F}(\hat{X}_t, \hat{Y}_t))$  scales as  $O(\sqrt{n})$ . In practice, learning the connection  $g$  is often more challenging than learning a predictor  $f_t$ , so it's encouraging to see the leading term  $G(\mathcal{G}(X')) / mT$  decay in both  $m, T$ .
Theorem 4 asserts that the ERM model trained with multi-modal data, achieves low excess risk on uni-modal test data to the optimal model as if multi-modal test data is provided, when connection is learnable. This result can be naturally extended to accommodate multiple modalities in a similar way. In this case, the ERM algorithm would learn a mapping from a subset of modalities to all modalities, which involves only one hierarchy as in the two-modality case, thus our analysis naturally carries over to this new setting.
3.1 Necessity of A Good Connection
Recall that in the upper bound of Theorem 4, all the terms vanish as  $n,T$  tend to infinity, except for  $LR(\mathcal{G},S^{\prime})$ . It's therefore important to determine whether the term is a defect of our analysis or a fundamental limit of our theoretical model. Here we present a simple example showing that the dependence on approximate realizability is indeed inevitable.
Let  $\mathcal{X} = \mathcal{Y} = \{0,1\}$  and  $n,T\geq 2$ . Each probability measure  $\mu_t$  is determined by a Boolean function  $b_{t}:\{0,1\} \to \{0,1\}$ , and for each observation  $s_t$ , the label  $z_{t} = b_{t}(y_{t})$  is purely determined by  $y_{t}$ . In particular, the four possible observations
$$
(0, 0, b _ {t} (0)), (1, 0, b _ {t} (0)), (0, 1, b _ {t} (1)), (1, 1, b _ {t} (1))
$$
happen with the same probability for any  $t$ .
For the hypothesis classes,  $\mathcal{G}$  includes all Boolean functions  $g:\{0,1\} \to \{0,1\}$ , while  $\mathcal{F}$  includes all 1-Lipschitz functions  $\mathbb{R}^2\rightarrow \mathbb{R}$ . It's straightforward to verify that
$$
L \mathcal {R} (\mathcal {G}, S) = \frac {c _ {0}}{n T} \left(\frac {1}{2} - \frac {| \sum_ {i = 1} ^ {c _ {0}} \sigma_ {i} |}{2 c _ {0}}\right) + \frac {c _ {1}}{n T} \left(\frac {1}{2} - \frac {| \sum_ {i = 1} ^ {c _ {1}} \sigma_ {i} |}{2 c _ {1}}\right),
$$
where  $\sigma_{i}$  are iid Rademacher random variables, and  $c_{0}, c_{1}$  denotes the number of observations with  $x = 0$  and  $x = 1$  respectively. The loss function  $\ell$  is set as  $|f(x,y) - z|$ .
The simplest version of Bernstein inequality states that for any  $\epsilon > 0$  and  $m \in \mathbb{N}^{+}$
$$
\mathbb {P} \left(\frac {1}{m} | \sum_ {i = 1} ^ {m} \sigma_ {i} | > \epsilon\right) \leq 2 e ^ {- \frac {m \epsilon^ {2}}{2 (1 + \frac {6}{3})}},
$$
therefore with probability at least  $\frac{3}{4}$ , we have that  $|c_0 - c_1| \leq 3\sqrt{nT}$  since  $|c_0 - c_1|$  itself can be written in the form of  $|\sum_{i=1}^{nT} \sigma_i|$ .
Condition on  $|c_0 - c_1| \leq 3\sqrt{nT}$ , we have that  $c_0, c_1 \leq \frac{nT}{2} + 2\sqrt{nT} \leq \frac{3nT}{4}$ . Using the Bernstein inequality again, with probability at least  $\frac{7}{8}$ , it holds  $|\sum_{i=1}^{c_0} \sigma_i| \leq 8\sqrt{c_0}$  and similarly for  $c_1$ . Putting it together, with probability at least  $\frac{1}{2}$ , the term  $L\mathcal{R}(\mathcal{G}, S)$  can be lower bounded as
$$
L \mathcal {R} (\mathcal {G}, S) \geq \frac {1}{2} - \frac {4 \sqrt {3}}{\sqrt {n T}}.
$$
On the other hand, the population loss of any  $f(x,g(x))$  composition is clearly at least  $\frac{1}{2}$  because the label  $z$  is independent of  $x$ , while the population loss of  $\{f_t(x,y)\}$  with the choice of  $f_{t}(x,y) = b_{t}(y)$  is zero. As a result the excess risk on population is at least  $\frac{1}{2}$  which doesn't scale with  $n,T$ . When  $n,T$  are large enough, the term  $L\mathcal{R}(\mathcal{G},S)$  matches the optimal achievable excess risk.
4 The Role of Heterogeneity
So far we have demonstrated that as long as a good connection indeed exists, learning with multimodal training data using a simple ERM algorithm yields a unimodal model which is guaranteed to perform as well as the best model  $f_{t}^{*}(x,y)$  with both modalities. The story is not yet complete. In order to explain the empirical phenomenon we're investigating, we still need to determine in what circumstance learning with multimodal data is strictly easier than unimodal learning. A good connection itself isn't sufficient: in the case of  $y \equiv x$  which admits a perfect connection function, bringing  $\mathcal{V}$  into consideration apparently gives no advantage.
To address this question, we move our eyes on heterogeneity, another fundamental property of multimodal learning that describes how modalities diverge and complement each other. Intuitively, heterogeneity can potentially lead to a separation between learnability in the following way: learning from a single modality is much harder than learning from both modalities, in the sense that it requires a much more complicated hypothesis class. As a result, either the sample complexity is huge due to a complicated hypothesis class, or the hypothesis class is so simple that even the best hypothesis performs poorly on population.
Consequently, we compare not only the best achievable population risks, but also the generalization errors. For unimodal learning denote  $\mathcal{G}$  as the hypothesis class, we consider the ERM solution
$$
\tilde {g} = \underset {g \in G} {\arg \min } \frac {1}{n} \sum_ {i = 1} ^ {n} \ell (g (x), z).
$$
The generalization error of  $\tilde{g}$  can be bounded via Gaussian average of the hypothesis class  $\mathcal{G}$  in the following way if we denote  $g^{*} = \operatorname*{argmin}_{g\in \mathcal{G}}\mathbb{E}_{(x,y,z)\sim \mu}\ell (g(x),z)$ :
$$
\mathbb {E} _ {(x, y, z) \sim \mu} \ell (\tilde {g} (x), z) - \mathbb {E} _ {(x, y, z) \sim \mu} \ell (g ^ {*} (x), z) \leq \tilde {O} \left(\frac {G (\mathcal {G} (X))}{n}\right),
$$
which is tight in general without additional assumptions. For multimodal learning,  $\mathcal{F}$  and  $f^{*}$  can be defined in the same way.
We are going to show that, either the intrinsic gap of risk between unimodality and multimodality
$$
\mathbb {E} _ {(x, y, z) \sim \mu} \ell \left(g ^ {*} (x), z\right) - \mathbb {E} _ {(x, y, z) \sim \mu} \ell \left(f ^ {*} (x, y), z\right) \tag {3}
$$
is substantial, or the Gaussian average gap is large. This implies a separation between multimodal learning and unimodal learning, when heterogeneity is present. Consequently, we define the heterogeneity gap as follow.
Definition 6 (Heterogeneity gap). Given a fixed hypothesis class  $\mathcal{F}$  and number of sample  $n\geq 2$  the heterogeneity gap between w.r.t some distribution  $\mu$  and hypothesis class  $\mathcal{G}$  is defined as
$$
H (\mu , \mathcal {G}) = \left[ \mathbb {E} \frac {G (\mathcal {G} (X))}{n} + \mathbb {E} _ {(x, y, z) \sim \mu} \ell (g ^ {*} (x), z) \right] - \left[ \mathbb {E} \frac {G (\mathcal {F} (X , Y))}{n} + \mathbb {E} _ {(x, y, z) \sim \mu} \ell (f ^ {*} (x, y), z) \right].
$$
The above definition measures the population risks between learning with a single modality  $\mathcal{X}$  or with both modalities  $\mathcal{X},\mathcal{Y}$ . When  $H(\mu ,\mathcal{G})$  is large, unimodal learning is harder since ERM is arguably the optimal algorithm in general. As long as Theorem 4 admits a low risk, the heterogeneity gap itself directly implies the superiority of multi-modality by definition. Therefore, the only question left is whether such desired instance (large heterogeneity gap + perfect connection) actually exists.
To this end, the following theorem provides the existence of such instance, proving that our theory is indeed effective. Let's slightly abuse the notation  $L(\tilde{g})$  to denote the excess population risk of  $\tilde{g}$  which is the output of the unimodal ERM. We have the following lower bound w.r.t heterogeneity gap.
Theorem 7. There exist  $\mathcal{X},\mathcal{Y},\mathcal{F}$  and a class  $U$  of distributions on  $(\mathcal{X},\mathcal{Y},\mathbb{R})$ , such that
$$
\forall \mathcal {G}, \forall n \in \mathbb {N} ^ {+}, \exists \mu \in U, s. t. H (\mu , \mathcal {G}) = \Omega (1), a n d \mathbb {E} _ {X} [ L (\tilde {g}) ] = \Omega (1).
$$
Meanwhile, let  $\hat{f},\hat{g}$  denote the outputs of the multimodal ERM algorithm in section 3, we have that
$$
\exists \mathcal {G}, s. t. \forall \mu \in U, \forall n \in \mathbb {N} ^ {+}, L (\hat {g}, \hat {f}) = 0.
$$
Remark 8. We compare with the work of [16]. They showed a similar separation in terms of the intrinsic gap (the difference between optimal hypotheses), under a representation learning framework. We make a step further by taking the Gaussian average (how hard to learn the optimal hypothesis) into consideration, which is crucial to the  $\Omega(1)$  gap in the more general setting.
Theorem 7 shows the existence of hard instances, where not only the heterogeneity gap is large, but the difference between actual risks is also substantial. It implies that under certain circumstances, multimodal learning is statistically easy, while unimodal learning incurs constant error no matter what hypothesis classes are used.
To sum up, our theory demonstrates that the superiority of multi-modality can be explained as the combined impact of connection and heterogeneity: when connection (Theorem 4) and heterogeneity (Theorem 7) both exist, multimodal learning has an edge over unimodal learning even if tested on unimodal data, providing an explanation to the empirical findings. Nevertheless, our theory also suggests a simple principle potentially useful for guiding empirical multimodal learning:
1. Collect numerous unlabeled multimodal data. Learn a connection via generative models.
2. Learn a predictor based on a modest amount of labeled multimodal data.
Such framework can be easily carried out with modern deep learning algorithms, for example we can learn the connection by generative models [12, 33].
4.1 Comparison with Representation Learning
It's possible to learn  $\hat{f},\hat{g}$  and minimize 1, based on observations of a single modality  $\mathcal{X}$ , via representation learning. In particular, representation learning solves the following unimodal ERM problem by treating  $y$  as an unknown representation of  $x$ , on a labeled sample  $S$  where  $y$  is hidden
$$
\hat {g}, \hat {f} _ {1},.., \hat {f} _ {T} = \operatorname {a r g m i n} _ {g \in \mathcal {G}, f _ {1}, \dots , f _ {T} \in \mathcal {F}} \frac {1}{n T} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {n} \ell \left(f _ {t} \left(x _ {t i}, g \left(x _ {t i}\right)\right), z _ {t i}\right).
$$
Unfortunately, although this representation learning method uses the same set of hypotheses, it leads to a worse sample complexity bound. Such method fails to exploit two essential parts of semi-supervised multimodal learning, namely the rich observation of unlabeled data  $S'$ , and separate learning of  $f, g$ . Failing to utilize  $S'$  will lead to a worse factor  $G(\mathcal{G}(X)) / nT$  which scales as  $1 / \sqrt{nT}$ , while in Theorem 4 the factor  $G(\mathcal{G}(X')) / mT$  scales as  $1 / \sqrt{mT}$  which is much smaller than  $1 / \sqrt{nT}$ .
Failing to exploit the "explicit representations"  $y$  from the training data requires learning a composition  $f \circ g$  from scratch, which typically leads to a worst case Gaussian average term, for example in [37] they have  $\max_{g \in \mathcal{G}} G(\mathcal{F}(S(g)))$  instead where  $S(g) = \{(x_{ti}, g(x_{ti}))\}$  is a random set induced by  $g$ . As a comparison, our approach decouples the learning of  $f, g$ , and the Gaussian average term  $G(\mathcal{F}(\hat{X}_t, \hat{Y}_t))$  is only measured over the "instance"  $\hat{S}$  which can be smaller. In fact,  $G(\mathcal{F}(\hat{X}_t, \hat{Y}_t))$  can be smaller than  $\max_{g \in \mathcal{G}} G(\mathcal{F}(S(g)))$  up to a factor of  $O(\sqrt{n})$ , see the example in appendix.
5 Limitations and Future Directions
In this paper we propose a theoretical framework on explaining the empirical success of multimodal learning, serving as a stepping stone towards more in-depth understanding and development of the theory. Nevertheless, as a preliminary study on the relatively unexplored field of theoretical multimodal learning, our result comes with limitations, and potentially opportunities for future research as well. We elaborate on these points below.
More natural assumptions: the current set of assumptions, while generally consistent with practical scenarios, does not fully align with them. Although most assumptions are satisfied in practice, the assumption on  $\mathcal{F}$  containing only Lipschitz functions, is restrictive: the predictor class  $\mathcal{F}$  typically comprises deep neural networks. Future work could seek to overcome the Lipschitz assumption, which is not only fundamental to our results but also a cornerstone in representation learning theory.
Hypothesis-independent definitions: our study characterizes multimodal learning through two central properties: connection and heterogeneity. However, their current definitions,  $\mathcal{R}(\mathcal{G}, S)$  and  $H(\mu, \mathcal{G})$ , depend on both the data and the hypothesis class  $\mathcal{G}$ . It's interesting to see if we can develop theories with hypothesis-independent definitions, such as mutual information or correlation. A potential difficulty is that such statistical notions are not totally aligned with the perspective of machine learning, for example it could happen that the two modalities are independent with zero mutual information, while there exists a trivial connection mapping.
Fine-grained analysis: our theory, which focuses on the statistical guarantees of ERM solutions, is fairly abstract and does not make specific assumptions about the learning problem. To study other more concrete multimodal learning algorithms, for example the subspace learning method, we need more fine-grained analysis which takes the particular algorithm into account.
More realistic examples: while the example in section 2.1 provides theoretical justification, it remains somewhat artificial and diverges from typical multimodal learning problem structures. Ideally, we would like to see examples that closely mirror real-world scenarios. For instance, can we progress beyond the current example to provide one that captures the inherent structures of NLP and CV data?
Benefit in optimization: our work demonstrates the advantages of multimodal learning primarily in terms of generalization properties. However, optimization, another crucial aspect of machine learning, could also benefit from multimodality. One potential direction to explore is the possibility that multimodal data is more separable and thus easier to optimize. We provide a simple example
in the appendix that demonstrates the existence of linearly separable multimodal data, where the decision boundary for any single modality is arbitrary.
In this paper we study multimodal learning from a perspective of generalization. By adopting classic tools in statistic learning theory on this new problem, we prove upper and lower bounds on the population risk of a simple multimodal ERM algorithm. Compared with previous works, our framework improves the upper bound up to an  $O(\sqrt{n})$  factor by decoupling the learning of hypotheses, and gives a quantitative example in the separation between multimodal and unimodal learning. Our results relate the heuristic concepts connection and heterogeneity to a provable statistical guarantee, providing an explanation to an important phenomenon in empirical multimodal learning, that a multimodal model can beat a fine-tuned unimodal one. We hope our result, though being a preliminary step into a deeper understanding of multimodal learning, can shed some light on the directions of future theoretical studies.
[1] Massih R Amini, Nicolas Usunier, and Cyril Goutte. Learning from multiple partially observed views—an application to multilingual text categorization. Advances in neural information processing systems, 22, 2009.
[2] Rie Kubota Ando, Tong Zhang, and Peter Bartlett. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6(11), 2005.
[3] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229, 2019.
[4] Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423-443, 2018.
[5] Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12:149-198, 2000.
[6] Shai Ben-David and Reba Schuller. Exploiting task relatedness for multiple task learning. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings, pages 567-580. Springer, 2003.
[7] Giovanni Cavallanti, Nicolo Cesa-Bianchi, and Claudio Gentile. Linear algorithms for online multitask classification. The Journal of Machine Learning Research, 11:2901-2934, 2010.
[8] Angel Chang, Will Monroe, Manolis Savva, Christopher Potts, and Christopher D Manning. Text to 3d scene generation with rich lexical grounding. arXiv preprint arXiv:1505.06289, 2015.
[9] Simon S Du, Wei Hu, Sham M Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.09434, 2020.
[10] Georgios Evangelopoulos, Athanasia Zlatintsi, Alexandros Potamianos, Petros Maragos, Konstantinos Rapantzikos, Georgios Skoumas, and Yannis Avrithis. Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention. IEEE Transactions on Multimedia, 15(7):1553-1568, 2013.
[11] Marco Federici, Anjan Dutta, Patrick Forre, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07017, 2020.
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
[32] Yunwei Ren and Yuanzhi Li. On the importance of contrastive loss in multimodal learning. arXiv preprint arXiv:2304.03717, 2023.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[34] Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi-view learning. 2008.
[35] Xinwei Sun, Yilun Xu, Peng Cao, Yuqing Kong, Lingjing Hu, Shanghang Zhang, and Yizhou Wang. Tcgm: An information-theoretic framework for semi-supervised multi-modality learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 171-188. Springer, 2020.
[36] Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. Learning to learn, pages 3-17, 1998.
[37] Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. Advances in neural information processing systems, 33:7852-7862, 2020.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[39] Ben P Yuhas, Moise H Goldstein, and Terrence J Sejnowski. Integration of acoustic and visual speech signals using neural networks. IEEE Communications Magazine, 27(11):65-71, 1989.
[40] Changqing Zhang, Zongbo Han, Huazhu Fu, Joey Tianyi Zhou, Qinghua Hu, et al. Cpm-nets: Cross partial multi-view networks. Advances in Neural Information Processing Systems, 32, 2019.