ICLR 2025

Published as a conference paper at ICLR 2025
TOWARDS A LEARNING THEORY OF REPRESENTATION
ALIGNMENT
Francesco Insulla
Institute of Computational and Mathematical Engineering
Stanford University
Stanford, CA 94305, USA
franinsu@stanford.edu
Shuo Huang
Istituto Italiano di Tecnologia
Genoa, GE 16163, Italy
shuo.huang@iit.it
Lorenzo Rosasco
MaLGa Center, DIBRIS, Universit`a di Genova, Genoa, GE 16146, Italy
CBMM, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
Istituto Italiano di Tecnologia, Genoa, GE 16163, Italy
lrosasco@mit.edu
ABSTRACT
It has recently been argued that AI models’ representations are becoming aligned
as their scale and performance increase. Empirical analyses have been designed to
support this idea and conjecture the possible alignment of different representations
toward a shared statistical model of reality. In this paper, we propose a learning-
theoretic perspective to representation alignment. First, we review and connect
different notions of alignment based on metric, probabilistic, and spectral ideas.
Then, we focus on stitching, a particular approach to understanding the interplay
between different representations in the context of a task. Our main contribution
here is to relate the properties of stitching to the kernel alignment of the underlying
representation. Our results can be seen as a first step toward casting representation
alignment as a learning-theoretic problem.
1
INTRODUCTION
In recent years, as AI systems have grown in scale and performance, attention has moved towards
universal models that share architecture across modalities. Examples of such systems include CLIP
(Radford et al., 2021), VinVL (Zhang et al., 2021), FLAVA (Singh et al., 2022), OpenAI’s GPT-4
(OpenAI, 2023), and Google’s Gemini (Google, 2023). These models are trained on diverse datasets
containing both images and text and yield embeddings that can be used for downstream tasks in
either modality or for tasks that require both modalities. The emergen...