Towards a learning theory of representation alignment
Authors: Francesco Insulla
Shuo Huang
Lorenzo Rosasco
Venue: ICLR 2025 Poster (2025)
Keywords: learning theory
representation learning
model stitching
representation alignment
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: properties of stitching and connection to kernel alignment to better understand multimodal representation learning.

Abstract: It has recently been argued that AI models' representations are becoming aligned as their scale and performance increase. Empirical analyses have been designed to support this idea and conjecture the possible alignment of different representations toward a shared statistical model of reality. In this paper, we propose a learning-theoretic perspective to representation alignment. First, we review and connect different notions of alignment based on metric, probabilistic, and spectral ideas. Then, we focus on stitching, a particular approach to understanding the interplay between different representations in the context of a task. Our main contribution here is to relate the properties of stitching to the kernel alignment of the underlying representation. Our results can be seen as a first step toward casting representation alignment as a learning-theoretic problem.

Published as a conference paper at ICLR 2025
TOWARDS A LEARNING THEORY OF REPRESENTATION ALIGNMENT
Francesco Insulla
Shuo Huang Istituto Italiano di Tecnologia Genoa, GE 16163, Italy shuo www.geainnii...... it
Institute of Computational and Mathematical Engineering Stanford University Stanford, CA 94305, USA
frantnau@stanfoord.ee
Lorenzo Rosasco
MaLGa Center, DIBRIS, Università di Genova, Genoa, GE 16146. Italy CBMM, Massachusetts Institute of Technology, Cambridge, MA 02139, USA Istituto Italiano di Tecnologia, Genoa, GE 16163, Italy Trosasco@mit.cc
ABSTRACT
It has recently been argued that AI models' representations are becoming aligned as their scale and performance increase. Empirical analyses have been designed support this idea and conjecture the possible alignment of different representations toward shared statistical model of reality. this paper, we propose learning- theoretic perspective to representation alignment. First, we review and connect different notions of alignment based on metric, probabilistic, and spectral ideas. Then, we focus stitching, particular approach to understanding the interplay between different representations in the context of a task. Our main contribution here is to relate the properties of stitching the kernel alignment of the underlying representation. Our results can be seen as a first step toward casting representation alignment as leaming-theoretic problem.
INTRODUCTION
In recent years, Al systems have grown in scale and performance, attention has moved towards universal models that share architecture across modalities. Examples of such systems include CLIP (Radford et al. 202I), VinVL (Zhang et al.] (021). FLAVA (Singh et al.) [022), OpenAl's GPT-4 OpenAI 2023), and Google's Gemini [ooglee[0223 These models are trained on diverse datasets containing both images and text and yield embeddings that be used for downstream tasks in either modality or for tasks that require both modalities. The emergence of this new class of multi- modal models poses interesting questions regarding alignment and the trade-off's between unimodal and multimodal modeling. While multimodal models may provide access to greater scale through dataset size and computational efficiency, how well do features learned from different modalities correspond to each other? How do we mathematically quantify and evaluate this alignment and feature learning across modalities?
Regarding alignment, Huh et al. (2024) observed that the scale and performance of deep networks increase, the models' representations tend to align. They further conjectured that the limiting repre- sentations accurately describe reality known Platonic representation hypothesis. Their analysis also suggests that alignment correlates with performance, implying that improving the alignment of learned features across different modalities could enhance model's generalization ability. However, alignment across modalities has yet to be evaluated in a more interpretable manner, and theoretical guarantees of alignment under realistic assumptions still lacking.
One way to quantify alignment is by kernel alignment, introduced by Cristianini et al. (2001). which evaluates the correlation of two kernel matrices K Ki,n; K2,, through Frobenius norms

---

Published as a conference paper at ICLR 2025
(K1. In K2n2F
A(K1,n K2,n) ((Kin K1 k1,a) ( F2,n)
Following this direction, methods such as Centered Kernel Alignment (CKA) (Korbblth et al. 2019) and Singular Vector Canonical Correlation Analysis (SVCCA) (Raghu et al. 2017) were developed to compare the learned representations. Another class of metrics is derived from independence testing, including the Hilbert-Schmid Independence Criterion (HSIC) (Gretton al., (005a) and Mutual Information (Mf) (Hjelm et al.] (019). However, further research is needed to clarify the relationships among these methods and other frameworks for assessing alignment.
Reality (E, {)
m
2122
Objects/data (X1, p1)
Objects/data (22, F2)
fi
81,2
Representation (1)
Representation (Zz, X2)
h1
he
g1
82,1
Output (V1, v1)
Output (22 122)
Figure 1: Diagram illustrating the process of multi-modal learning. It contains spaces and measures of reality, objects/data, representation, and outputs as well as the functions connecting them. A detailed explanation of these symbols is in Section 2
To quantify the aligmment of representation conditioned on task, one approach is to use the stitching method (Lenc& & Vedaldi 2015). Bansal al. (202T) revisited this technique and used it high- light that good models trained different objectives (supervised vs self-supervised) have similar representations. By evaluating how well one representation integrates into another model, stitching provides more interpretable framework for assessing alignment. To describe the setup, h1.2 == G2 51,2 fi to represent the function after stitching from model to model (Figure gives detailed illustration of the whole process). Here Og and fy are parts of model Hy : X. Yg with = 1,2 2, and $1,2 is the stitcher. We consider the generalization error after stitching between two models:
R(g2 $1,2 fi) (((),,((z)) y)]
We can use the risk of the stitched model in excess of the risk of model
min R(h122 - R(h2)
to quantify the impact of using different representations, fixing 92-
In this paper, we aim formalize and refine some of these questions, and our contributions are summarized as follows:
(a) We compile different definitions of alignment from various communities, demonstrate their connections, and give spectral interpretations.
Starting from the empirical Kernel Aligmment (KA), we reformulate empirical KA and pop- ulation version of KA using feature/representation maps, operators in Reproducing Kerell Hilbert Space (RKHS), and spectral interpretation. addition, we discuss the statistical properties of KA.
2

---

Published as a conference paper at ICLR 2025
We integrate various notions of alignment, ranging from kernel alignment in independence testing and learning theory to measure and metric alignment, and demonstrate their relation- ships and correlations. This comprchensive exploration provides deeper understanding for practical applications.
(b) We provide generalization error bound of linear stitching with the kernel alignment of the underlying representation.
A linear 9y results the stitching error being equivalent to the risk from the model Hgg This occurs, for example, when Hg represents RKHSs or neural networks, then By is linear combination of features in RKHSs or the output linear layers of neural networks.
The excess stitching risk can be bounded by kernel alignment when Bg are nonlinear functions with the Lipschitz property. A typical scenario is stitching across the intermediate layers of neural networks.
For models involving several compositions, such deep networks, if we stitch from layer further from the output to a layer closer to the output (stitching forward) and 9g is Lipschitz, the difference in risk can be bounded by stitching.
Structure of the paper In the following of this paper, we introduce the problem settings and some notation in Section[] Different definitions for representation alignment from different communities and the relationship among them will be derived in Section Section 4 demonstrates that the stitching error could be bounded by the kernel alignment metric. And the conclusion given in Section 5
PRELIMINARIES
Empirical results demonstrate that well-aligned features significantly enhance task performance. However, there is pressing need for more rigorous mathematical tools to formalize and quantify these concepts in uni/multi-monn modal settings. In this section, we provide mathematical formalization of uni/ multi-modal learning, introducing key notation facilitate deeper understanding of the underlying processes.
Setup Without loss of generality, we focus on the case of two modalities, as illustrated in Figure I which outlines the corresponding process. For 2, let (Xg, M4) and (Zy, Ag) be probability spaces, and let Fg be spaces of functions fq Xg Z, = Rd, We regard X, as the space of objects (or data), Fg as the space of representation (or embedding) maps, and 2g as the space of representations. We relate Hl and Ag by assuming > = (ff) #M We also assume that μl and H2 are the marginals of joint probability space (K with X X1 x X2, M4g (Tg)#14 where Tg X - X, is the projection map. Moreover, let (N,, My) be the task-based output spaces and define Gg (gg Zg - yg} with Vg (g.) Ag. Each overall model is generated by H, == (hy Xr + Vu hg = 9o faJ.
Reality Consider a space abstract objects, called the reality space and denoted by E, which generates the observed data in various modalities through maps m, Ny. These maps may be bijective, lossy, or stochastic. Reality can be modeled as a probability space (E, Alternatively, one may define reality as the joint distribution over modalities by setting m, = mq.
We may want consider the case of single modality, where only data space exists, versus multiple modalities, where several such spaces are present. Two modalities are deemed equal if ạ n22
Uni/Multi-modal
Representation alignment A representation mapping is function f X - Ed that assigns latent feature vector in Rdd to each input in the data domain X' Alignment provides a metric to eval- uate how well the latent feature spaces obtained from different representation mappings, whether
(fo) is the pushforward measure of defined as (fa))Ho(A) = (e()=((()) In terms of random variables X, and Zg with measures (4g and Xg. respectively, this is equivalent to fo (X₂) and 7g being equal in law.

---

Published as a conference paper at ICLR 2025
from uni-modal multi-modal data, aligned or similar. Commonly used metrics for quantify- ing alignment include those derived from kernel alignment, contrastive learning, mutual informa- tion, canonical correlation analysis, and cross-modal mechanisms, among others. However, they are introduced in very fragmented manner, without an integrated or unified concept. A detailed introduction and analysis of these methods will be provided in Section
3 FRAMEWORKS FOR REPRESENTATION ALIGNMENT
In this section, we describe various definitions of representation alignment from different commu- nities and demonstrate the relationship among them. We begin with detailed presentation of em- pirical and population Kernel Alignment and its statistical properties. We then cover other notions of alignment coming from metrics, independence testing, and probability measures, as well as their spectral interpretations. We draw connections to kernel alignment which emerges as central object.
3.1 KERNEL ALIGNMENT (KA)
Based on the work of Cristianini et (2001), who introduced the definition of kernel alignment using empirical kernel matrices, we propose different perspectives to understand kernel alignment in both empirical and population settings and derive its statistical properties accordingly.
A reproducing positive definite kernel K X' x X' + R captures the notion of similarity between objects by inducing an inner product the associated reproducing kernel Hilbert space (RKHS) H. Specifically, K (xxx,')) = (J),, (ar') for any representation (feature) map f E H, and x' E X. For the multi-modal case, we define (g x²) (T),, TT(x') Kg ( where K2 is the reproducing kernel associated with Hg. In other words, Kg acts on (æ1, 22) by first applying the projection (z) = Zg. In the following, the subscript Ty denotes the q-th modality, and the superscript x' indicates the i-th sample.
3.1.1 EMPIRICAL KA
From Cristianini et al. (201I). we adopt the following formulation for kernel alignment for kernel matrix Kg,n E ux with samples {ar ] drawing according to the probability measure μ
(Ki, K2,r)
A(Ki,n (1,,n, = V(Ki,g (Ki,, Ki,a) F (K2 K2,)) F
(20 where (K,,),K2))) Em.==1 K1, x3) K2) (z*, 25). One modification is to first demean the kernel by applying matrix H = In 1 1 on the left and right of each Kg,n. with ga being the identity matrix and I, being the ones vectors. This results in Centered Kernel Alignment (CKA).
n
Representation interpretation of KA Denote the empirical cross matrix between the representation maps fi and fz as Σ1, En [f(())f(()']] = (('')afa e Rdixd, xds Then the empirical KA will become
(KKim,K2.
En.1 |l |2222l1
RKHS operator interpretation of KA Inspired by the equation I we construct a consistent def- inition of Kernel Aligmment using tools of RKHS, where it suffices to consider output in one di- mension?] Consider RKHS H, containing functions hy X₁ IR with kernel Kg. Given eval- uation (sampling) operators S, : Hg - RR" defined by (S,h,)
(xa) = (har K Knaz). It is not hard to check that the adjoint operator S F." + Hy can be written as S7 (w)), ...., wn)= =
We can generalize the definition to vector-valued functions by recasting hy X2 + Rtv ho Xa [t.] IR i.c. with kernels of the form (3 24, for integers i, S tg.
4

---

Published as a conference paper at ICLR 2025
w Kg (2) -) and the empirical kernels be written Kgn//
SyS0
De Vito et al.
2005
Smale & Zhou 2007). Then the empirical KA may be written as
(S,Si,3551 A(Ki, K2n) ||
1 S |F S
where S1 RKHS.
n k1, @ Kg,nz and it coincides with the literature about learning theory with
3.1.2 POPULATION VERSION OF KA
For the population setting (infinite data limit of the evaluation operator) in L the restriction operator
Sq : Hg + (Xx μ1) is defined by (zh = (hy: (x, k, and its adjoint S L 2 11) +
Hg given by 9 = Jx (() (1, .dd. Then the integral operator Lx, = S ; I (x H) +
(,2 1u) is given by Lx,g(z) = Jx (2 (x,x²) ()(((')d((()) and the operator Σg = S Hy 7
Hg can be written as Eg Ix (2, (2, (ed)((((z) (De Vito et al. 2005
Rosasco et al. 2010
Similarly, the population KA between two kernels K1, K2 can be defined by
Tr (LK, LK.)
A(K1, (2) = Ir (Lx,,, Tr (L)
where the summation in nK K2,ar becomes the integration as
/ dp(zi, x2),, x2) 2 (#), x 22)
Tr (LK, LK.) =
If (x,,,²') = (z)()), ). fg (x')), then S is a projection onto the span of coordinates of fq- The population version of CKA is KA with Sy replaced with H.
Spectral Interpretation of KA The understanding of kernel alignment (KA) can be deepened via the spectral decomposition of the associated integral operator. The mercer kernel K can be decomposed as K = >; т): de & di, where are the eigenvalues and 6, are the eigenfunctions of the integral operator Lx (Cucker & Smale, 2002 Schölkopf (002). Defining the features as fo VTe; and expressing the target function as h
E, W.tif. we obtain
Σ, 1 w?
A(K,hoh)=
VE,hh mn??
Similarly, given two kernels Kg E, 19.0 @ 09. with fg.s = Vng we have
A(K, K2) (fii Eis M1,572.5 (01, d22.)² Σ: nř. E, n2,
V E.
Letting [Ci.2las = (6h,is 22.5) and defining îj = m//llm/ll we can equivalently write
A(Ki, K2) Tr Tr|Cr.2 diag(í2) CT22 diag(mn) )] (m. (C1.2 2 @ C1.2) îz)) (fn C C1,2)
with @ as the Hadamard product. This formulation provides insight into kernel alignment by relating it to the similarity between the eigenfunctions of the two integral operators. In particular, if m and 2/2 are constant, then A(K,, K2) x ||Ci,all2: and Ci = then A(K1, K2) = (f, 12))
3.1.3 STATISTICAL PROPERTIES OF K.A.
Having introduced both the empirical and population versions of KA we explore its statistical properties. Cristianinni et al. (2006) shows that empirical KA concentrates to its expectation by McDiarmid's inequality and gives an (LI/vn) bound. For completeness, we state the following
lemma summarizing this statistical property and the proof is provided in Appendix 66.
Lemma 1. Let K1, K2 be two kernels for different representations and K Ki,, R2,n E B be kernel matrices generated by n samples, then with probability at least 1 - 5, we have
Å(K1,a5 K2 A(Ki, K2) & //22nn) log(2/05
5

---

Published as a conference paper at ICLR 2025
3.2 ALIGNMENT FROM DISTANCE ALIGNMENT
Distance alignment (DA) of two spaces by
Given distances dg 2g X2x + R. then we can compare the difference
D(d d2)
(d((0,x')) - -d((,,) -
Equivalence between KA and DA Suppose dy = = Kg) Igel et 2007 and K (T (gg) which emerges naturally from assuming Kz(2, (fg()), (x))))) fz 1, and (agg syg)) (zg) [g((2)) (i.e., Kg represents mapping onto ball). Also assume [| = C. Then, D(d1, d2) 8C((1 A(K1, K2)), hence the two paradigms equivalent.
3.3 ALIGNMENT FROM INDEPENDENCE TESTING
Independence testing is statistical method used to assess the degree dependence between vari- ables. It often involves examining the covariance and correlations between random variables and can also be applied quantify kernel-based independence. this section, we outline several approaches from independence testing within the alignment framework and investigate their connections to the kernel alignment method discussed earlier.
Hilbert-Schmidt Independence Criterion (HSIC) The cross-covariance operator for two func- tions (Baker 1973) is given by C1.2 Ez, (22 (22)
E., as [hn (x1)
Ez, (22)))22222))
for h1 E H1, h2 E H2. From Gretton et al. 2005a)
HSIC(,, H1, H2) = ||C1,all4s:
where μ is the joint distribution of Xr and X2. We can also note that HSIC(₁, H1, H2) ||TE (Ka, @ Kny || |
Hence HSIC is effectively and unnormalized version of CKA, or, explicitly,
HSIC(H, H2) v /HSIC(H,, H, HSIC(H2, H2)
CKA(K1, K2)
Statistical property of HSIC Gretton al. (2005)) shows that, excluding the 0( diago- nal bias, centered empirical HSIC concentrates to population and [Song et al. (2012) provides an unbiased estimator of HSIC and shows its concentration, both by U-statistic arguments.
Remark 1 (Other notions from independence testing). There are other concepts of d'independence test- ing for alignment such us Constrained Covariance (COCO) (Gretton et al., 2005a),, Kernel Canonical Correlation (KCC), Kernel Mutual Information (KMI) (Bach & Jordan 2002). They are also related kernel alignment and more detailed explanations can be found in Appendix 6.2
3.4 ALIGNMENT FROM MEASURE ALIGNMENT
There are several methods for comparing measures on the same space. One can then quantify inde- pendence by comparing a joint measure with the product of its marginals. This principle allows us to interpret HSIC as test for independence given two function classes
MMD to HSIC Following Gretton et al. (0022). we start by introducing the so-called Maximum Mean Discrepancy (MMD). Let H be class of functions h X and let By be different measures on X. Then, letting Vig Aq.
MMD(41, P2; H) = sup [[b(z)) - h(x2))
heR
Let H be RKHS and restrict ball of radius 1, then MMD(H1, pa; H)2 || (Kz, K2z 11 [K(1),4)) + K(x2, 2K 2K(x1

---

Published as a conference paper at ICLR 2025
Now construct measure of independence by applying MMD μ versus P1 H2 where H. is replaced with H1 x H2 and get HSIC
MMD(H, A1 @ P2: H1 @ H2)² HSIC(I, H1. H2) |1,,2|| Σ Ps
where { is the spectrum 11,2
We can also use tests of independence that don't explicitly depend on a function class, such as mutual information, by letting μ be a Gaussian Process measure on two functions in their respective RKHS with covariance defined by their kernels.
KL Divergence to Mutual Information
Given KL divergence
DRL (rllu)) / /d((z) log du du
(x))
we can define mutual information as
/ du((z1 (22 log
(((2)/11) f22 (122)
[p)) DRr (pll))))))))))
du(xi x2) log
M41 (31))22(32) (3:2)
For multivariate Gaussian μa, with marginals M4g = (O, En),
1
[
MI(P)=
log
=
log
2
2
EE Σ2,1 2q
Σ1,
For the simplest case of Mg = 1, then this simplifies to
MI(V) log((T E = Σ log(1 - p2))
Wasserstein distance For the Wasserstein distance
W2(u,v) v) = inf (E(,g(m)) [|| - y|l3 71 = 72 = v}.
applying i and μn @ M2 measure independence, we have
W2(u, in @ u2) inf ( (a)) [||e] x1112 + || 2/21 :n M,72 1n 22)
For mean zero Gaussians
W2(11, Tr|E, + Eg 2(111/2 2
and as a measure of independence with Σg
W2(u, pn @ u2) 2Tr// - -x (1---1)))
In summary, we've introduced several popular metrics for alignment between two representations and related them via spectral decompositions to a central notion of kernel alignment generalized for RKHS. Moreover, similar notions can be used to quantify alignment between model and task to estimate generalization error, and more details provided in the Appendix 6.1
4 STITCHING: TASK AWARE REPRESENTATION ALIGNMENT
Building on our understanding of kernel alignment a fundamental metric for evaluating the align- ment of representations detailed in the previous section-we now explore stitching, a task-aware concept of alignment. Stitching involves combining layers components from various models to create a new model which can be used to understand of how different parts contribute to overall per- formance to compare the learned features for task. In this section, mathematically formulate this process and provide some intuition by demonstrating that the generalization error after stitching can be bounded by kernel alignment using spectral arguments.
7

---

Published as a conference paper at ICLR 2025
4.1 STITCHING ERROR BETWEEN MODELS
In the following, we focus on stitching between two modalities. Figure provides a detailed illus- tration of the functions, spaces, and compositions in question. Denote the function space for task learning
Hg = [hg : Rg + Vylhng = 9q fq: 94 E Gg. fg E Fg} with 4 2. Here Fg : Xg + 2y and Gg Zg + y... Denote S1.2 == {$1.2 21 + 22] as the stitching map from Z to 22 and 2, == {52,1 22 + 21} reversely. Define the risk concerning the least squares loss as
(ha) = E (1)) - y/1²] = //- xY,
[1n((x) - upr v), hy E Hq-
Here, (z, y) the joint distribution of X and V/ and use the notation || represent || 1ly, associated with space Vi for simplicity, i.c. absolute value for 3/ R, la norm for 3, Rt and L2 norm for J'g being the function space. For hg E Hgg denote any minimizer of R(hg) among Hy as hy. that is,
(H, == (b)) min (h), q = 1,2.
hEH,
Moreover, denote the function spaces generated after stitching from 21 to 22
H1,2 (h1,2 92 81 81,2 E S1,2)
and conversely as H2.1.
Lenc & Vedaldi (2015) proposed describe the similarity between representations by quanti-
fying how usable representation fi is when stitching with O2 through a function $1,2 21 - Z₂ or oppositely through $2 E S2,1- To quantify the similarity, we provide detailed definition of the stitching error.
Stitching error
Define the stitching error as
R1 (81.2 = R2(02 o 81,2 8 fi) = R2(h11)
and the minimum as
Ratitch (S1,2)
min R2(h122 = R2(2112). $1,2ES1,2
To quantify the difference in the use of stitching, we define the excess stitching risk
Ratitch (S122 - R2(h2).
Note that R (S1.2) (R2(h2) quantifies a difference in use of representation (fix 92, compare 81,2 o f1 f2), while if 3/ 3/2 and R2 R2 then Ri (S R1 (hi) quantifies difference between 82 81,2 and 91 (fix f1).
The functions in S1.2 typically simple maps such linear layers convolutions of size one, to avoid introducing any learning, as emphasized in Bansal et al. (2021). The aim is to measure the compatibility of given representations without fitting representation another. One perspec- tive inspired by Lenc & Vedaldi (2015) is that we should not penalize certain symmetries, such as rotations, scaling, or translations, which do not alter the information content of the representations. Furthermore, the amount of unwanted learning may be quantified by stitching from a randomly initialized network.
4.2 STITCHING ERROR BOUNDS WITH KERNEL. ALIGNMENT
In this section, focus simplified setting where $1,2 Z1 + Zg is linear stitching, that is, $1,) S1 with Si,2 &d 2g Rdd Additionally, assume V R R In this section, we quantify the stitching error and excess stitching risk using kernel alignment and provide lower bound for the stitching error when stitching forward.
The following lemma shows that when G, are linear, stitching error only measures the difference in risk of Hi versus H2.
8

---

Published as a conference paper at ICLR 2025
Lemma 2. Suppose dim()/) dim()/2 = d and R1 R2. Let 9q Gg be linear with 9g (g(2y) Wazg and Wa E Rdxdd, Let 81.2 21 Zg be linear with (1,2(21) S1 and S1 E Rdz Then 1,2 R1,2ch(S1,2) = Ri(h) (h1).
Remark 2. The lemma applies when H, represents a neural network with Gg as the output linear layer, as well when H, an RKHS with a Mercer kernel and Gg is the linear map of representa- tions 3
The next theorem shows the case when 9g are nonlinear with the ri-Lipsciizz property, l19(2) g(>'Ill & K||2 - 2/11. One intermediate example is the stitching between the middle layers of neural networks.
Theorem 1. Suppose 92 is Kg-LLisshitz. Again let $1,2 be linear, identified with matrix S1.2- With the spectral interpretations E1,2 = =R [f = diag(m) 2ddaggni) and d
||Cr, al as Paragraph 3.1.2 we have
R1 (S1,2) Ra(ha) x3.A2 2x² (A2R22(22))
(2)
Proof. Breaking Ri (81,2) into two parts and using Cauchy- Schwarz we get
E |11o(S) =|1
-E [1 - (x) (v- (2((2)
[Ra(h2) + E (1.2((1))))()=) - (=(22 + 2(E [119 - (2(f2) (2
As 92 N2- kg-shihsh we can bound with the error from linearly regressing f2 on fi
E (12((S12f1)(0) o=(f2)(x))1 Ro ( fa(=)]]²
(1 ?, + +|/7||
- xxT
2(5
with || M = (M, (n)) Taking derivatives, we note that the minimizer of the RHS is S1,2 = Ef diag(m)) Plugging in, the RHS reduces to Ag. Thus
R1 (S1.2) S R (E,,2) Rz(he) R2A2 2x² RA
Remark 3. In arguing that kernel alignment bounds stitching error for Theorem I we made sev- eral simplifying assumptions, which now assess. Firstly, we restricted the stitching S to linear maps, following the transformations commonly used in practice (Bansal et al. [022]] Csiszärik et al., we relax this assumption, observe that similar result holds, with Ag inf, (f (x)) - [z) Interest- ingly, for $1.2 to use only information about the covariance of fI, f2. similarly to kernel alignment, $1,2 must be linear. Furthermore, we note that for stitching classes that include all linear maps, the linear result remains valid.
2021). and preserve the significance of the original representations.
Remark 4. Note that the notion of alignment that appears here, namely ||4112 As
||Ci, 11 ||Ci,,adiag())) (12))1 is similar to, yet distinct from, kernel alignment given by || Idiag( (m) 1//CC,2diag(1)) 1/21(2 In particular, the spectrum 7/, is irrelevant for the bound. However, this does not hold if regularization added to S1,2 by analogy linear regression.
Remark 5. If two representations are similar in the alignment sense, they are also similar in the stitching sense; however, the converse does not necessarily hold. By loose analogy to topology, this suggests that kernel alignment a stronger notion of similarity.
More explicitly, if the RKHS kernel Kg is sum of separable kernels, then by Mercer's theorem we can decompose it as K,, Dne 2y. & dg.o where ne.r 2 0 are the eigenvalues, and Φe.p Po + g are the orthonormal eigenfunctions of the integral operator associated with the kernel Kg. Then any hg E H, can be decomposed hg 9q far where fa E Fg is the feature map fa ( and 9q 9, is linear go (2g) 004 2g

---

Published as a conference paper at ICLR 2025
Excess stitching risk also serve intermediate result bound the difference in risk. Let J1 = 3/2 and R1 = R2- To obtain a lower bound for 1,,2 (S1,2) in a practical setting, we can assume that 2 o G2 S 9. For models involving several compositions, such deep networks, this condition can hold when stitching from a layer further from the output to a layer closer to the output (i.e., stitching forward), provided that the networks are similar and the layer indices aligned the end.
Lemma 3. Let = 2 = and R₁ = R₂ = R. If o 92 S 91 then RT,
(S1.2) 2 R1 (H1).
The following theorem derives directly from equation 2 and Lemma
Theorem 2. Let 31- V/2 and R1 R2 R. Assume S1,2 G2 C 91, 92 is ke-liipschiz, and
R(h2) = R(H2). Then
(RHH) R(H2) S Rithhh R(H2) S n2A22 + 2nz (A²R
Remark 6. If we consider deep models and keep the Hi, H2 the same but iterate over layers stitching forward, then
((H)) - R(H2) & min ( + 2nd
2n (AL R(H2)))))
Alternatively, by making similar assumptions and swapping the index 1 2, which requires G1 T2 up to a linear layer (due to the S1,2 o G2 S 91 condition), we get
(H(H) R(H2))
max (E
K + 2%, 2 (AR(())
The above result can be stated informally as "alignment similar depth (measured backward from the output) bounds differences in risk".
The results presented have several practical implications. First, we build on the experiments from Huh et al. (2024). which provide evidence for the alignment of deep networks large scale using measures similar to kernel alignment. By establishing connection between kernel alignment and stitching, our work supports building universal models that share architectures across modalities scale increases. Second, can elucidate the experiments from Bansal et al. (2021), which suggest that typical SGD minima have low stitching costs (stitching connectivity). This aligns with works that argue feature learning under SGD can be understood through the lens of adaptive kernels RRadhakrishaan et al. 2022 Atanasov et al. 2022).
5 CONCLUSION
In this paper, we review and unify several representation alignment metrics, including kernel align- ment, distance alignment, and independence testing, demonstrating their equivalence and interrela- tionships. Additionally, we formalize the concept of stitching, a technique used in uni/multi-moda settings to quantify alignment in relation to given task. Furthermore, we establish bounds on stitch- ing eтror across different modalities and derive stitching error bounds based on misalignment, along with their generalizations and implications.
ACKNOWLEDGMENT
L.R. is thankful the CBMM-Hebrew University workshop organizers and Philipp Isola for the talk that inspired this work. L. R. acknowledges the financial support of the European Research Council (grant SLING 819789), the European Commission (Horizon Europe grant ELIAS 101 101230,,, the US Air Force Office of Scientific Research UTA8655-22-1-7033 versity and Research (FARE grant ML4IP R20ST712KP: grant BAC FAIR PE00000013 funded by the EU NGEU). This work represents only the view of the authors. The European Commission and the other organizations are not responsible for any use that may be made of the information contains.
the Ministry of Education, Uni-
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kemell learners: The silent alignment effect. International Conference on Learning Representations, 2022.
Francis R Bach and Michael Jordan. Kernel independent component analysis. Journal of Machine Learning Research, 3(Jul):1-48, 2002.
Charles Baker. Joint measures and cross-covariance operators. Transactions of the American Mathematical Society, 186:273-283 22992, 1973.
Yamini Bansal, Prectum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances in Neural Information Processing Systems, 34:225-236. 2021.
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature Commu- nications, 12(1):2914) 14, 2021.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. Journal of Machine Learning Research, 13:795-828, 2012.
Peter Craven and Grace Wahba. Smoothing noisy data with spline functions: estimating the correct degree smoothing by the method of generalized cross-validation. Numerische mathematik, (4):377-403, 1978.
Nello Cristianini, John Shawe- Taylor, Andre Elisseeff, and Jaz Kandola. On kernel-target alignment. Advances in Neural Information Processing Systems, 14, 2001.
Nello Cristianini, Jaz Kandola, Andre Elisseeff, and John Shawe-Taylor. On Kernel Target Align- ment, pp. 205-256. Springer Berlin Heidelberg, 2006.
Adrián Csiszárik, Péter Kórösi-Szabó, Akos Matszangosz, Gergely Papp, and Dániel Varga. Similar- ity and matching of neural network representations. Advances in Neural Information Processing Systems, 34:565655668, 2021.
Felipe Cucker and Steve Smale. On the mathematical foundations of learning. Bulletin of the American mathematical society, 39(1):1-49, 2002.
Ernesto De Vito, Lorenzo Rosasco, Andrea Caponnetto, Umberto De Giovannini, Francesca Odone, and Peter Bartlett. Learning from examples as inverse problem. Journal of Machine Learning Research, 6(5), 2005.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850-5861,
Gene H Golub, Michael Heath, and Grace Wahba. Generalized volsdvalliotion as a method for choosing good ridge parameter. Technometrics, 21(2):115-223, 1979.
Google. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1180. 2023.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical de- pendence with hilbert-schmidt norms. International Conference on Algorithmic Learning The- ory, pp. 63-77. Springer. 2005a.
Arthur Gretton, Ralf Herbrich, Alexander Smola, Olivier Bousquet, Bernhard Schölkopf, and Aapo Hyvärinen. Kernel methods for measuring independence. Journal of Machine Learning Research, 6(12), 2005b.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(1):723-773, 2012.
11

---

Published as a conference paper at ICLR 2025
R Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
Karan Grewal, Phil Bachman, Adam
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic represen- tation hypothesis. In Forty-first International Conference on Machine Learning, 2024.
Christian lgel, Tobias Glasmachers, Britta Mersch, Nico Pfeifer, and Peter Meinicke. Gradient- based optimization of kernel-target alignment for sequence kernels applied to bacterial gene start detection. JEEE/ACM Transactions on Computational Biology and Bioinformatics, 4(2):216-226, 2007.
Arthur Jacot, Berfin Simsek, Francesco Spadaro, Clément Hongler, and Franck Gabriel. Kernel alignment risk estimator: Risk prediction from training data. Advances in Neural Information Processing Systems, 33:15568-15578 2020.
Dmitry Kopitkov and Vadim Indelman. Neural spectrum alignment: Empirical study. In Artificial Neural Networks and Machine Learning-ICAN: 2020: 29th International Conference Artifi- cial Neural Networks, Bratislava, Slovakia, September 15-18, 2020, Proceedings, Part II 29, pp- 168-179. Springer, 2020.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivari- ance and equivalence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 991-999, 2015.
OpenAl. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Jonas Paccolat, Leonardo Petrini, Mario Geiger, Kevin Tyloo, and Matthieu Wyart. Geometric compression of invariant manifolds in neural networks, Journal of Statistical Mechanics: Theory and Experiment, 2021 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- $748-8763 PMLR, 2021.
Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Mecha- nism of feature learning deep fully connected networks and kernel machines that recursively learn features. arXiv preprint arXiv:2212. 13881, 2022.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances Neural Information Processing Systems, 30, 2017.
Lorenzo Rosasco, Emesto De Vito, and Alessandro Verri. Spectral methods for regularization in learning theory. DISI, Universita degli Studi di Genova, Italy, Technical Report DIS1-7R-05-18, 2005.
Lorenzo Rosasco, Mikhail Belkin, and Ernesto De Vito. On learning with integral operators. Journal of Machine Learning Research, 11(2). 2010.
B Schölkopf. Learning with kernels: support vector machines, regularization, optimization, and beyond. The MIT Press, 2002.
Haozhe Shan and Blake Bordelon. A theory of neural tangent kernel alignment and its influence on training. arXiv preprint arXiv:2105. 14301, 2021.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Mar- cus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15638-15650, 2022.
12

---

Published as a conference paper at ICLR 2025
Steve Smale and Ding-Xuan Zhou. Learning theory estimates via integral operators and their ap- proximations. Constructive approximation, 26(2):153-172 2007.
Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature selection via dependence maximization. Journal of Machine Learning Research, 13(5), 2012.
Alexander Wei, Wei Hu, and Jacob Steinhardt. More than toy: Random matrix models predict how real-world neural representations generalize. In International Conference on Machine Learning. pp. 23549-25555 23588. PMLR, 2022.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations vision- lsionnlanguage models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3579-5588, 2021.
APPENDIX
6.1 ALIGNMENT TO TASK
Here we mention ideas of alignment between a representation and task used to estimate generaliza- tion and characterize spectral contributions to sample complexity.
Kernel alignment risk estimator (KARE) for KARE which is estimator for risk.
In Jacot et al. (2020) we have the following definition
((Kn/ A yn
p(), Un. Kn)
Tr(( A1) 11)
This also obtained in Golub et al. (979). Wei et al.
(022))
Craven & Wahba (1978).
Spectral task-model alignment From Canatar et al. 2021), we have a definition for the cumula- tive power distribution which quantifies task alignment.
C(n) Eign n
Einer
E; nadi & Φ;. (0:, = 8z.j. and target h2 = E, WRR C(n) can be interpreted fraction of variance of h explained by first features. The faster C(n) goes the higher the alignment.
Here K ==
From Rosasco al. (005) have bounds generalization of kemel ridge assuming some regularity of hus called source condition
Source Condition
hy E 02 = {h E (x, p) Liu ||l|| S R}
Assuming hp E, wi Mho then the statement can be rewritten
m
Remark 7. KTA appears in several theoretical applications. Cristianini et al. (2001) bounds gener- alization error of Parzen window classifier ! Cristianini al. (2006); Cortes al. (2012) show that there exist predictors for which kernel target alignment (KTA) A(K, wy bounds risk.
Fex .' [K (3r, xt Do'l R(h) I 2(1 vV))
h(x))
Ex,, (x, 2²
Furthermore, several authors including Atanasov et al. (0022) y Paccolat et al. (002)): Kopitkov & Indelman (2020); Fort et al. (2020); (Shan & Bordelon (2021 use KTA to study feature learning and Neural Tangent Kernel evolution.
Cortes et max, F.,. (K" F
al.
2012
notes
eπrr in proof since implicitly assumes 1 making kernel constant.
However proof can be saved
with additional assumption.
13

---

Published as a conference paper at ICLR 2025
6.2 OTHER NOTIONS FOR ALIGNMENT FROM INDEPENDENCE TESTING
Constrained Covariance (COCO) Then Gretton et al. (2005a) proposed the concept of con- strained covariance the largest singular value of the cross-covariame operator,
COCO( H2) = (x)) (2(2)) : h1 E Hi, h2 E H22
Kernel Canonical Correlation (KCC)
From Bach & Jordan 2002)
cov[hi h2 (32)
KCC(μ, Hi, H2, k) = sup
h E Hi, E H2
(var(h) (x1)) + x x/|/n| ) (1ar((vva22222)))) + x|| l 11/2
The next two are bounds on mutual information from correlation and covariance respectively
Kernel Mutual Information (KMI) From Bach & Jordan (2002
KM, H2) - log(1 - (n 1, (K
where kernels are centered and Kg.n min; E, (5 Fays but empirically 1/n suffices.
6.3 ADDITIONAL PROOFS
In this section, we provide the detailed proofs of Lemmas presented in and Section 3 and Section We begin with the proof of Lemma [ For completeness, we first restate the lemma below.
Lemma 4. Assume [K d'g)l S Cer Let A1,((X) = A x2), ..., (of, xz))
1 nZ (K1, K2)F. Let A1, = EA1 = ÅL. VAI,A and A = A1.2
VAL1A222 Then with probability
at least 1 - 8, and c (332nn) log (2/)), we have IÁ A C(X),, where C(X) is non-trivial function.
Proof. Let (xf' 2²b') = (of, xb)) for all 1,... n except Then
Dij K Ki(x) of K2) - Ki "xd')("((())))) (aa wxl") and note Dijl S 42,C2. Then
( = - AA -ha(x)-Ane(X'))
Σ
Dayl 1Dal
< C2
2n 1
8C
n?
n
Applying McDiarmid, we get
-
P( A1,2l 20 < 2exp
320
which can also be read as, with probability least 1 5, LÅj.2 A1,2 S (32/n)) /3= log(2/6)
Finally, we show that |dc, - Ar,l S e for i.j E {1,2) gives IÁ - Al $ C(X)e.
A - A = (1, 1122 -,, 1/2
=|Ar,2 - A ( A,, + A1,2 (11,1 -1/2 - -A1,1
=|A1 A12
+ A1. (2,21
(2,1
(2,, -1/2 (2,)
14

---

Published as a conference paper at ICLR 2025
Lastly, use
1/2 (x-1/2-y-11 1-1/2 y1/2 x/12
y (rz) (zy) 1/2 + x 1/2)
Now we are in the position to prove Lemma 2 For completeness, we first restate the lemma below. Lemma 5. Suppose dim()i) = dim()/2) = d and R1 = R2. Let E 9g be linear with Og (2n) = Wz and Wg E Rdxd, Let 81,2 21 - Zz be linear with (1,) = S1,221 and S1 E R xds. Then Ri (1 = (2H))
Proof. For the linear there exists vector Wg E Rdxd, such that gg(2g) Wg 2g R We can write the error of stitching as
RT.2 ($1.2) = E |||25112/1 -
E [I[(WS11,2 - Wi) + E [IWifi(=)) v113]
[W2S1,2- Will. (h1)..
where we used that for W1 to be optimal, we require dw; R1 = E [(Wif v)fl = 0. Minimizing with respect to S1,2 yields
RI12 (S1.2) = 11T2 Wi + (HG),
where IT / - (W2 diag(m) WW) diag(m) denote the residual of the generalized ---projection onto (columm) span of W2. We note that in general, as long as d < d2. we have Retitch (S1.2) R2 (H4).
15


Reviews:
[{"review_id": "NLA0x5uH2m", "reviewer": "Reviewer_embS", "summary": "In this manuscript the authors first give a review of various measures of representation alignment, with a particular focus on showing how they relate. They show that many different approaches can be straightforwardly related to kernel alignment.\n\nThe authors then formalise the stitching approach to measuring task-relevant representation alignment. They demonstrate that bounds on stitching error can be derived and related to the alignment metrics that they reviewed in the first section.", "questions": "Just a few nits:\n\nFigure 1: it would nice to note for the reader that these symbols will be defined in section 2.\n\nL130: I’m not familiar with the notation using the #. Perhaps add a note or reference, or a an explanatory word or two in the sentence where it is first used?\n\nL188: I didn’t understand why H_q had codomain \\mathbb{R}. This seems to be assuming something about Y_q that up until this point hasn’t been assumed. Could you clarify?\n\nL241 & L670: McDiarmin -> McDiarmid!\n\nL351: modals -> models?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The manuscript is clear and well-written.\n\nAlthough perhaps a little dense, the first section does a good job of introducing various notions of representation alignment and elucidating their relationships. I found this precise and succinct presentation to be useful and digestible.\n\nThe formalisation of stitching methods is also clear and straightforward. While the assumption of linear stichers is quite restrictive, it does lead to some nice proofs.", "weaknesses": "I don’t think the manuscript has any significant weaknesses.\n\nI do think that at the end of section 3 it might be helpful to give a few sentence summary of the relationships between the various measures of alignment. While the preceding material is complete, it would help readers who perhaps have not digested all of that material on a first pass to more easily comprehend the manuscript.\n\nAs mentioned above, the restriction to linear stitching functions is potentially quite limiting. It might be nice to comment more on the impact of this assumption on the analysis.\n\nIt should be noted that I am not very familiar with the field of representation alignment, so I am not in a position to comment on this work relative to the existing literature. I will have to leave that to the other reviewers. My score reflects this uncertainty, but it should be understood that this is not a comment on the manuscript, but rather my uncertainty about its impact, novelty, and relevance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jdjbibPmzh", "reviewer": "Reviewer_smSY", "summary": "The authors compiled broad notions of representational alignment metrics and provide connected interpretation. Next, they mathematically formulate stitching method, which is used to evaluate similarity of representations between different models given a task by plugging representation from one model to another. They provide the generalization error bound of linearly stitched model based on kernel alignment.", "questions": "- In theorem 3, why does swapping the index 1,2 require $\\mathcal{G}_1 = \\mathcal{G}_2$ up to linear?\n\n- On remark 3, could you elaborate little more on the regularization of $S_{1,2}$?\n\n- It looks like the stitching setting is quite relevant to transfer learning. Could the current result tell anything about kernel alighment and its relevance on transfer learning perspective?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "While I am not familiar with all the references,the first section of the paper where the authors make a overview of different formulation of representation alignment and provide connected interpretation from different community seems to be a nice contribution.\n\nThe theoretical formulation of stitching method that is used frequently in practice is relevant point and the results can give valuable insights to use of stitching method in practice\n\nIn general the paper is well-written and well structured.", "weaknesses": "- I appreciate the first section of the paper, but it would be nice to have the last paragraph where authors could summarize and make a brief overview of everything in one place. \n\n- While it being a solid theoretical work, I think it is great that the settings in questions are still relevant for practice. Having a couple of sentences summarizing practical implication of the results in conclusion might be helpful for broader audience.\n\n- It is little confusing about significance of 'task defined representation' and 'modality specific representation' in stitching setting. \n\nMinor comments on the manuscript\n- little confused by term 'representation $\\mathcal{H}_q$' online 4 as I thought it's entire input-output mapping\n- I think there are some indexing, sqrt typos on \"Spectral interpreation of KA\" section; what is $\\rho$?\n- Typo on line 494", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Wz3xgzkts", "reviewer": "Reviewer_YJw7", "summary": "The paper proposes a theoretical perspective on representation alignment of AI models. \nThe authors state  that while there is empirical work  that supports the idea of representation alignment for bigger models, there is little theoretical perspective on the idea of representational alignment. The authors focus on the practice of (model) stitching as a device to study representational alignment. More specifically, \nthey first present different ways to measure representation alignment and then focus on stitching as task to contextualise a theory for representational alignment. \nThe contribution of this paper is showing that a stitching error and stitching error bound can be derived based on the kernel alignment of the underlying representations.\n\nOverall, I think this is a good paper attempting to provide a theoretical framework to representational alignment which is usually an empirical field of research. The derived stitching error and stitching error bound seem important in order to quantify generalisation error of different models. To improve the submission I would suggest making Section 4 more coherent with Section 3, since at the moment they seem a bit disconnected to me notation wise. I would also suggest either adding a small experiment showing the usefulness of the derived theory or contextualising it within current empirical methods.", "questions": "- Would it be possible to design a small experiment with, e.g., MNIST where the stitching error (bound) is used to quantify the possible generalisation error?\n- Alternatively you could add a small discussion on how your theory fits in with current stitching methods or how it could be applied to learn more better representations.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The authors do a good job of introducing the different concepts of kernel alignment and independence testing needed for their theory. \n- Using Kernel Alignment to provide a generalisation error bound for stitching seems novel and useful given the rise of representation learning.\n- I think the paper is generally well written making it accessible for readers that are not that familiar with a learning theoretic perspective on this problem.", "weaknesses": "- It is unclear to me from Section 4 how the stitching error and stitching error bound can be used. Since the paper makes reference to existing empirical work, I would like to see a more concrete example or a reference to where the derived theory fits in existing empirical work.\n- For instance how does the proposed theory quantify the stitching error in current approaches (e.g., can different layers be compared equivalently, does normalisation, regularisation play a role?).\n- I feel there is a slight disconnect between sections 3 and 4. While in section 3 a comprehensive listing of ways to measure representation alignment is presented, the introduced notation is not really used in Section 4. This makes it a bit difficult to follow what parts of Section 3 inspired Section 4. Perhaps you could write a small paragraph at the beginning of Section 4 stating how Section 3 serves as building blocks for deriving the stitching error and stitching error bound in Section 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LpudCcdj4z", "reviewer": "Reviewer_EzC8", "summary": "This work proposes a theoretical foundation for understanding representations alignment, especially the alignments for multimodal models.\nThe main contribution is in the idea of connecting different representations through a so-called \"stitching\" mechanism.", "questions": "1. The reviewer understands that the authors try to avoid complexity by only arguing about linear stitching. But how is the generality of the linear stitching? And what is its limitation? Do different types of representations require different stitching?  Please add this argument in the discussion part of the paper.\n\n2. Please provide at least one experiment to demonstrate the correctness of the proposed theory, for example, through transfer learning.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is generally well-written. The authors gave an excellent review of measures of representation alignment, which is a nice entry point for understanding the paper. \nThis paper is also supported by a thorough mathematical argument about the stitching error between models.", "weaknesses": "1. The paper provides no empirical experiment demonstrating the proposed theory's correctness.\n2. Only one type of stitching, Linear Stitching, is argued. It would be helpful if the authors could briefly explain other stitching types, as this will help to familiarize the readers to this idea.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)