Contextures: Representations from Contexts
Authors: Runtian Zhai
Kai Yang
Burak Varıcı
Che-Ping Tsai
J Zico Kolter
Pradeep Kumar Ravikumar
Venue: ICML 2025 poster (2025)
Keywords: representation learning
pretraining
learning theory
foundation models
scaling law
Primary area: general_machine_learning->representation_learning
TL;DR: We develop the contexture theory that clarifies that representations are learned from the association between the input and a context variable.

Abstract: Despite the empirical success of foundation models, we do not have a systematic characterization of the representations that these models learn. 
In this paper, we establish the contexture theory.
It shows that a large class of representation learning methods can be characterized as learning from the association between the input and a context variable. Specifically, we show that many popular methods aim to approximate the top-d singular functions of the expectation operator induced by the context, in which case we say that the representation learns the contexture.
We demonstrate the generality of the contexture theory by proving that representation learning within various learning paradigms -- supervised, self-supervised, and manifold learning -- can all be studied from such a perspective.
We prove that representations that learn the contexture are optimal on those tasks that are compatible with the context.
One important implication of our theory is that once the model is large enough to approximate the top singular functions, scaling up the model size yields diminishing returns, so further improvement requires better contexts.
To this end, we study how to evaluate a context without knowing the downstream tasks. We propose a metric and show by experiments that it correlates well with the actual performance of the encoder on many real datasets.

Contextures: Representations from Contexts
Runtian Zhai"
Kai Yang? Burak Varier' Che-Ping Tsai' Zico Kolter! ! Pradeep Ravikumar'
Abstract
useful, or perhaps even what representations are these mod- els learning? Unlike classical statistical learning theory, where there is no mystery regarding the statistical estimand, the very target itself is unclear in representation learning. For example, what are the representations that BERT (De- vlin et al., trriedt to do cloze tests- -ss learning, and why are they useful in understanding the sentiment of user reviews on Netflix? What representations do deep neural networks learn, and why they useful if they cause neural collapse (Papyan et al., 2020), where deep representations could collapse to few clusters? Do the many different self-supervised learning methods (Balestriero et al., 2023) all learn similar or disparate representations?
Despite the empirical success of foundation mod- cls. we do not have a systematic characterization of the representations that these models learn. In this paper, we establish the contexture theory. It shows that a large class of representation learning methods can be characterized as learning from the association between the input and context variable. Specifically, we show that many popular methods aim to approximate the top-d singular functions of the expectation operator induced by the context, in which case we say that the repre- sentation learns the contexture. We demonstrate the generality of the contexture theory by proving that representation learning within various learn- ing Suattiign--uuprrissss
The responses to these questions are often muddled. Many analyses are conflated with the mystery of deep learning generalization the ability of large neural networks to learn function approximations that generalize to unseen points. However, this is a different problem from the mechanism of representation learning. Our focus is on what representa- tion learning (or "pretraining" in the context of foundation models) aims to capture, and why it can be applied to tasks completely different from the objectives used train the representations. Another way this question is muddled is by recourse to scaling. A popular viewpoint argues that even if the encoder performs poorly on one task, increasing the model size while keeping everything else the same could allow better performance to "emerge" (Wei et al., 2022). However, substantial evidence suggests that certain abilities cannot emerge solely from scaling, Additional training sig- nals, such alignment (Ouyang et al., 2022), necessary.
self-supervised, and manifold learning can all be studied from such a perspective. We prove that representations that learn the contexture are optimal on those tasks that are compatible with the context. One im- portant implication of our theory is that once the model is large enough to approximate the top sin- gular functions, scaling up the model size yields diminishing returns, so further improvement re- quires better contexts. To this end, we study how to evaluate a context without knowing the down- stream tasks. We propose a metric and show by experiments that correlates well with the actual performance of the encoder on many real datasets.
The above questions are naturally interesting to learning theorists, but why should the broader machine learning community care about understanding the mechanism of representation lcarning, empirical success seems to be always achievable with existing approaches by scaling up the model size, an empirical observation known as scaling laws (Kaplan et al., 2020)? This is because sustainable suc- cess progress is not always guaranteed. Ilya Sutskever recently remarked that "pretraining as we know it will end" (Sutskever, 2024), largely because the current pretraining paradigm is producing diminishing returns. Understanding what representations are learned by foundation models crucial for designing future generations of pretraining meth- ods, and this is how this field can make scientific progress.
1. Introduction
Representation learning underpins the modern deep learning revolution, leading up the remarkable recent successes of foundation models (Bommasani et al., 2021). But critical question that has remained unanswered to a satis- factory extent is: why are these models learning anything
'Camegie Mellon University, Pittsburgh, PA, USA "Peking University, Beijing, China. Correspondence to: Runtian Zhai <rzhai@ alwmmi.cmu.cdu.......
Proceedings of the 4rnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Contextures: Representations from Contexts
In this work, we establish the contexture theory, which provides unified lens for inspecting a large class of rep- resentation learning methods. The central argument of this theory is that representations are learned from the associ- ation between the input X and a context variable A. This framework is general enough to encompass a wide variety of learning paradigms, as we demonstrate in Section 3.
typically the one before the last linear layer, as the represen- tation. However, it has never been fully explained why the penultimate layer works so well across disparate tasks.
superviid learning (SSL) is currently the most com- mon way of learning representations. There are two main types of SSL: multi-view learning and masked autoencoders. Multi-vier vieww learning includes contrastive learning (Oord et al., 2018; Chen al., 2020) and non-contrastive learning (Grill et al., 2020; Zbontar et al., 2021; Bardes et al., 2022). Masked autoencoders have wide applications, including lan- guage (Devlin et al., 2019; Radford et al., 2019), vision (He et al., 2022), videos (Gupta et al., 2023), and more.
Now, suppose we are given a context variable A along with X, how should we learn the representation? In Section 4 we prove that the optimal method is to approximate the top singular functions of the expectation operator induced by the context, in which case we say that the encoder "learns the contexture". Such an encoder is optimal as long as the task is compatible with the context, and in Section we define a quantitative measurement of such compatibility.
Manifold learning is a classical method that aims to capture the geometry of the data. Examples such as locally linear embedding (LLE) (Roweis & Saul, 2000) and Laplacian eigenmaps (Belkin & Niyogi, 2003) formulate manifold learning as node representation learning on a graph, where connected nodes should have similar embeddings.
Our theory implies that the main consequence of enlarging the model is that the learned representation space will be brought closer to the span of the top-d singular functions of the expectation operator, which we empirically verify in Section 4.2. Once the two spaces are close enough, further scaling will yield diminishing returns. We envision that future breakthroughs in pretraining require context scaling, where better contexts are learned from data, not heuristics.
2. Definitions and Examples
Let X' be the input space. Pretraining aims learn a feature encoder +b X Rd. We call (xx) the embedding of z, and df the embedding dimension. Let Px be the data distribution. In this work, we assume Px to be fixed.
we study how to evaluate contexts. This is a prerequisite for context scaling because if we cannot even determine which contexts are good, then we cannot create better contexts. The key takeaway is that for a context to be useful (meaning that it can lead to good representations), the association between X and A should be sedeerate-neihhen too strong nor too weak. For example, if A X, then their association is the strongest; if A is independent of X, then their ussociation is the weakest. However, neither context is useful because A does not provide additional information about X. In Section 5.2, we propose a quantitative measure- ment of context usefulness that can be efficiently estimated and does not require knowledge of the downstream task. We also empirically verify that the metric correlates with the performance of the encoder on many real datasets.
In Section 5
The central argument of the contexture theory is that rep- resentations are learned from the association between two random variables: the input X X' and a context variable A E A. A is called the context space. Let P+ (Jr, a) be the joint distribution of X and A, with marginal distributions Px and PA. Let (²x(Px) be the L function space w.r.t. Px. with inner product (f1, f2) fa) Px = Ex-Px (xx()) and norm I| = (f.f) Px Define (²(PA), (.:) PA I for PA similarly.
2.1. Examples of Contexts
1. Labels are a common type of context. They can take different forms, such as discrete categories in classification, continuous values in regression, or text captions of images. Labels may be obtained from human annotators or in pseudo-forms, such as clusters teacher models. Typically, labels are provided compatible pairs sampled from the joint distribution p+ (1x a).
In one sentence, our key contribution is clarity on the target of a large class of representation learning methods the singular functions of the expectation operator. We do not discuss the numerical aspect of approximating these func- tions, which requires an expressive model architecture and good optimizer, and such analyses are left to future work.
2. Random transformations generate different views of the same data point. Common transformations include adding random noise to inputs, as seen in diffusion models and denoising autoencoders, or randomly corrupting/masking inputs, as in SimCLR and masked autoencoder. These transformations are typically defined by domain experts and are specified through a predefined conditional distribution p+ (a) x)).
1.1. Examples of Representation Learning Methods
Supervised learning is the simplest way to learn represen- tations. For example, neural networks pretrained on Ima- geNet (Russakovsky et al., 2015) were very popular in the early days of the deep learning boom (Huh et al., 2016). Specifically, one uses the output of an intermediate layer,
2

---

Contextures: Representations from Contexts
3. Graphs provide locality information about the inputs. The edge weights represent the similarity between two inputs. In this case, we have A = X' with the conditional distribution P (a] 2) proportional to the edge values between x and a. See Section 3.3.
Here A S 1 because of Jensen's inequality, and v0 1 is always an eigenfunction of Tk; with Ao 1. Similarly, we can order the eigenvalues of * by = Ko 2 Ki Z
... 2 0, and the eigenfunctions 14: M1,'.. form an ONB of [Px))), where fuu = 1. We also have the following result.
4. Features are predefined or pretrained mappings from X' to a vector space, which we denote by a = 9(x)). This encompasses teacher models that provide stochas- tic pretrained feature encoders. In contrast to previous instances, here p+ (a 2²) is directly described via a repa- rameterization structural equation for in terms of x.
Lemma 2.3 (Duality property. Zhai et al. (2024), Proposi- tion 1). For all i, we have X = KG E [0. 1]. And if X > O, then we have p4 = X, TT Us, and vs = X TTp p5-
We call 8; = = a singular value of Tp- associated with left singular function μr E L²(Px) and right singular function v; E (PA)).. Since μo = 1 and vo = 1, all other Ps (v)) must have zero mean as they are orthogonal to pu (Vo). Now, we can spectrally decompose P+ as follows.
2.2. Induced Kernels and the Expectation Operator
The joint distribution p fully determines the association between X and A. It induces the positive-pair kernel (John- son et al., 2023) and the dual kernel (Zhai et al., 2024).
Lemma 2.4. The spectral decomposition of P+ is P+ p+(,,)) a) = E.sinte)uullPPP(((PPAA
Definition 2.1. The positive-pair kernel A and its dual kernel xix are defined as
Proof. Vi, < pt (t,a)
(Plla)aaaa) u1) PA
= Pp x))s(a)da==
(t(a,d) d') pt(a,d)
(Tp+):)(t)= = (>² p4: (xr) = (x))((z). Since (v) is an (t.aa
PA(a)
Pa(a)Pa(a') P (PPaa)
ONB, we have Px( Ei (z))s)))
kz Pt (t(xz')
Px(x)Px(x))
(x(x))
The first key result of the contexture theory is that the opti- mal d-dimensional representation should recover the linear space spanned by the top-d singular functions -1..... Mdd- We say that such a representation learns the contexture. Note that the constant function μ40 = 1 is excluded, as it does not need to be learned there is no benefit in allocating a dimension to encode something already universally present.
The kernels are density ratios between joints and marginals for A and X, respectively. They capture how more likely (a, or (x, x') appear together than independently given Pt. p+ also induces the following expectation operator. Intuitively, given function g E (Pxx,,, the operator com- putes the expectation of ((4) conditioned on any x.
Definition 2.5. A d-dimensional encoder & = = Ad] learns the contexture of p+, if there exists a set of top-d singular functions {u1,-.. ,ad} of Tp+, such that span p {ut = span ,paa). We also say that & extracts the top-d eigenspace of Tx
Definition 2.2. The expectation operator Tp+ (P(( (Px( is defined as for all E (PA))
(Tp+))(x) = /
a))a = E[g(A) | x].
Its adjoint operator Th+ (P(Px) + (P(PA).
If the multiplicity of sa more than 1, then recovering the span of any top-d singular functions suffices. The in- tuition why learning the contexture is ideal is that such representation keeps the most variance of the context, anal- ogous to principal component analysis (PCA) in the finite- dimensional case. Suppose X' and A are both finite sets. Let N = | and M = [A]. Then, a function f E 22((x) is a vector in RN, 9 E ('(PA) is a vector in R and Ty- is essentially a matrix T E R NXM The goal is learning a d-dimensional embedding E E R for the N samples in X. PCA states that we should use the top-d left singular vec- tors of T as E, or the top-d eigenvectors of TT because they maximize the explained variance. Similarly, functional spaces are essentially dimesiodiieesiooall vector spaces, so
which satisfies U.Tp+g) = (Tpi (VS E
(P(Px),g g E (P()))). is given by (a)+ =
f() pt (1)))))))))) dx = E[(( a). PA(A)
Now we discuss the spectral properties of these opera- tors. Define the kernel integral operator Taj : (PA) + (PA)) as (Tx; (a) = gg(a") (a, a²) (a'). Define the other operator Fst (Pxx)) + L2 similarly. It easy to see that TØ= = T, Tp+, and T = Tp+Tp+ T
We call A E IR an eigenvalue of T Tx* with eigenfunction v E (PA)) if Tx = Xv. Suppose Tx= is Hlllbert-Schmide integral operator. Then, we can order its eigenvalues by 1 = Au 2 A 2 ... 2 0, and the corresponding eigenfunctions Vo V1, form an orthonormal basis (ONB) of (.P(P)).
the d-dimensional encoder that preserves the most variance of Tp+ consists of the top-d left singular functions of T Tp+, or the top-d eigenfunctions of T Ty+ T
3

---

Contextures: Representations from Contexts
3. Learning the Contexture
Interestingly, this result can partially explain neural col- lapse. Papyan al. (2020) empirically showed that when there are d classes of equal sizes and the label A is determin- istic, sufficiently trained deep representation will collapse to an equiangular tight frame (ETF) Φ1, ... 2d.. which are defined as (1()) = c(I[z belongs to class i] - d for all E [d] and some constant c. The span of 01,,... 22 is the same as the top-d eigenspace of Tp ATPi However, cannot explain why Φ1,--. &d converge to the exact func- tions as above-it only proves that they will span the same space. To prove this, one needs to analyze the training dy- namics of the specific optimizer, such as gradient- -based methods, while our results are independent of the optimizer.
In this section, we show that every example method in Sec- tion 1.1 does one of the following:
(i) Extracts the top-d eigenspace of T = Tp+ p+ (learns the contexture of Pt), that is, recovering the span of M1," Ma (excluding 1o0 =1);
(ii) Extracts the top-d eigenspace of TT+ ATp, where A is the integral operator of a kernel & (a, a'), that is (Ag)(a) = f g(a')) k (a, a') ((a"). hA is called the loss kernel, which is defined by the loss function used in the objective. Since the constant function is not necessarily the top eigenfunction of ATp in this case, we do not exclude any eigenfunction.
When the classes have different sizes, the dual kernel of Tr ATP, is (x,,, x') = 1[r and x' have the same label]. This is equivalent to the simplex-encoded labels interpola- tion (SELI) defined by Thrampoulidis et al. (2022, Defini- tion 2), which generalizes neural collapse.
Notation: For f E L2(Px). denote its mean by f [px(f((), and its centered version by f- The same notation is used for multi-dim functions and random variables, when the distribution is clear from context. The covariance matrix of & : X + Rdd denoted by Covpx is a d x d matrix C' where Cli.j] = ( Φ3) Px
For a regression task where A is an arbitrary Euclidean vector, using the same objective Equ. (1), a similar result can be proved. See Appendix A.3.
3.2. Self-supervised Learning: Transformation Context
3.1. Supervised Learning: Label Context
Two major types of SSL are multi-view learning and de- noising autoencoders. Multi-view learning independently samples two views A, A+ from p+ (-|z)) for every x. A+ is called a positive sample of A. Then, one trains V :AA - Rd such that VD (AA) x ((+)). This V is an encoder on A in- stead of X, so at downstream we need to convert T (a) to ((x),, which is typically done via the average encoder:
Let A be the label of X. The mean squared error (MSE) is
R( = min
E. W.b (X,A)~PI
W + b- - AIl] (1)
where (XX) is the output of the layer before the last linear layer in a neural network, and denotes the bias. If can be an arbitrary vector, then the linear layer is biased; if b = 0 is fixed, then the linear layer is unbiased. For classification where A is one-hot, we have the following result.
((x) (x) = (Tpi () = (aao))((((ddptt ((a)dP+(a x).
Theorem 3.1 (Proof in Appendix A.1). Let A be a one- hot random vector. Suppose the linear layer is unbiased, that is b- O. Then, 4² minimizes R(Φ)
By Lemma 2.3, we have the following corollary.
Corollary 3.3. Let > O. The averuge encoder &o span- ning the span of the left top-d singular functions Tf is equivalent to W spanning the span of the right top-d singular functions of TT
) if and only if it extracts the top-d eigenspace of Tp+ ATx p where ka(a, a') = Ila a'), or (Ag)(a) = g(a) classes have the same size, then the top-d eigenfunctions of Tpp AT7, and Tp, are the same.
If all
Enforcing (A) = (A+) alone leads to the degenerate solution where v gives the same embedding to all a. This is called the feature collapse problem. There are two solutions: contrastive learning and non-contrastive learning. Prior work by HaoChen et al. (2021); Johnson et al. (2023); Zhai et al. (2024) showed that the spectral contrastive loss Lo and non-contrastive loss Lx can learn the contexture of p+, Let A+ + be positive sample of A, and A be a negative sample drawn from PA independently. Define
This theorem works for randomized labels, where each x can belong to multiple classes with certain probabilities. Kernel ka stems from class imbalance; it puts more weights larger classes. Indeed, in practice, smaller classes are harder to Iearn. To get rid of A, we can use the class-balance risk (also known as importance weighting (Shimodaira, 2000)):
[W*()) + 6- b-1
Roal ( min E W.b (X+A)~P+
(PA(A)
(w((A) ((AA))(())) + 1 { (ū(A)))))))))))) ((A 27
Lc E
Theorem 3.2 (Proof in Appendix A.2). Under the setting of Thm. 3.1, let the linear layer be biased. Then, x minimizes Reail ()) if and only if learns the contexture of P+.
CN E [IU((A) *(A+))1 s.t. Cov [ I,
4

---

Contextures: Representations from Contexts
where the (i,j)-th entry of C is (Vi, thy) px i E [d]. Minimizing LN is constrained optimization problem. The constraint CovPA [ = I is called the orthonormality constraint. It makes sure that y must be rank-d, so that it cannot be a constant function on A.
We say that context p+ and task f" are compatible, if P+ can help learn good predictor for f". Formally, consider the scenario where we have corrupted training set {(a, 3:)}. where a; ~ (- xz)) and y/- = (zz). That is, we cannot see the original samples Ty but can only see the corrupted samples 0g To learn a predictor on this training set, we can train a predictor 9 : A + Y and then use f = ((A æ]] At test time, given input 2, we can draw ~ P+ (. x)) and then output the average of ga()). For this procedure to work, two conditions are necessary:
Theorem 3.4 (Proof in Appendix A.4). Vp* minimizes Le or LN if and only if o* = Tt T learns the contexture.
For denoising autoencoders, similar result can be proved. See Appendix A.5.
3.3. Node Representation Learning: Graph Context
There exists g" E LLP(PA) S.t. f = Elg" a]]
Let G = EV,E) be an undirected graph, where edge (u, v) has a weight w(2, v) such that w(u, v) = w(v, u) 2 0. Let the degree of node u be d(u) = VVED w(u, e), and
This g" has low Varlg" (Ar) z]] average over æ.
If Varlg" [t] is high, then 9 (a) will be far away from y = (z), so fitting 9 on (a, y) will not work. Based on these insights, we define compatibility as follows.
dmam= E. d(). Let (x(u)=
d(n) be a distribution on
dumm
V, and Pv (u, v) w((na)) be a distribution on E. Define dum
Definition 4.1. The compatibility with Pt of any non-zero f (Px) is defined as
pt (+lu) = d(aa The following problem with a similar orthonormality constraint learns the contexture of Pt.
(f.
minimize 0:X-<20
2 (u [10(w)) [u))
(2)
P(S,
max 9EL-(PA).
Px E 10,
11911x
S.L.
Covrx (5) I.
Px
Theorem 3.5 (Proof in Appendix A.6). Let dh be any solu- tion to Eqn. (2) (so that for any constant c, xh + e is also a solution). Then, 8x learns the contexture of Pt.
For further insight, let S = Σ, Uaits and g = E, uay. Then, p(f, Pt) max E,2
Exz ofu? Ess uf
VE
by Cauchy-Schward incquality (the optimal vi satisfy uo = O and v; x 8,u; for i 2 1). For any c > 0, we define the class of (1 - -ocopaiiiee tasks
4. Optimality of the Contexture
So far, we have shown that commonly used learning objec- tives can learn the contexture. Now, we address the question of why and when learning the contexture is optimal. Ar- guably no representations can be good for all downstream tasks, but we show that a feature encoder that learns the con- texture is optimal for the class of tasks that are compatible with the context. This provides quantitative characteri- zation of when a task respects the human prior knowledge the context incorporates. Interestingly, as we detail in the sequel, this has intriguing implications for scaling laws.
F.(PP) = {f e L2(x)) Elfl = 0, p() 21c
This class of tasks satisfies the two conditions, i.e. we can find g" with low variance Varig" x]:
Theorem 4.2 (Proof in Appendix B.1). For any f F. (Pt), there exists g" E LP((A)) such that f*( Elg"(A) æ), and g" satisfies
( (A)) Acllg"|1?2
E XxPx P
E
4.1. Compatibility
The ultimate evaluation of an encoder is its performance on relevant downstream tasks. Most downstream tasks, such as prediction, clustering, and segmentation, can be associated with a target function f" (Px).. For example, multi- class classification can be associated with multiple one-vs- all labeling functions. Moreover, if we are fitting a linear predictor on top of d then the mean and variance of f do not matter because we can change the weight and bias of the predictor accordingly. Thus, we can assume that f is normalized, that is, it has zero mean and unit variance.
Now that we have a class of tasks compatible with P+, we evaluate &b by its worst-case approximation error on F. F.()). The most common way to evaluate is to fit a linear predictor on top, also called a linear probe, which is the focus of our attention (other methods for using t include fitting a small neural network on top, using kernel method, or using KNN). Specifically, the worst-case approximation error of O on F C (Px) ) is the maximum error of the optimal linear probe in estimating any function in F. In this work, we focus on the L22 error.
5

---

Contextures: Representations from Contexts
Definition 4.3. Let F C (2xx be function class where F = of E F for all a E R. The worst-case approxi-
0.9
Depth
Alignment
0.8
mation error of 4 : X + Rd on F is defined as
2
0.7
max err(D::)= = fe P
err(
3
0.6
Width
64
128
256
512
1024
2048
4000
8000
12000
16000
20000
250000
30000
where e
min WERE BER
I To + b 10+b--1lpp Slipxx Px
0.9
Width
Alignment
0.8
512
The following key result shows that the & that minimizes err(#: (; (P+)) over all d-dimensional encoders must re- cover the linear space spanned by the M1, ... 14g- Here 1o0 is excluded since the bias b in the linear predictor implicitly contains μo-
1024
0.7
2048
0.6
Depth
100
150
200
Theorem 4.4 (Proof in Appendix B.2). Suppose 1 - 81 S
Figure I. Alignment between the learned representation and the top-d eigenfunctions of Tat on the aba 1 one dataset. Solid curves: CCA. Dashed curves: mutual KNN. Depth here means the number of hidden layers.
ES-- / For any d, among all o- = [01,... ,aal where di [2x)))) & minimizes enr(#;; (((t)) if and only if it learns the contexture of p+. The error given by
= $ (1-6) -)
to the span of the top-d singular functions of Ty-, explain- ing why big models learn aligned representations. The key difference that contexts are designed by humans and thus are more subjective than the so-called "underlying reality".
min X-X-RR, ((LLA(()))
err( (e( (((tt) s s8+1
Conversely, for any d-dimensional encoder & and any > O, there exists f E (Px) such that p(f, = 1- c, and
We substantiate this extrapolation with experiment the abal I one dataset from OpenML. We use KNN (K = 30) as context, where A = X. and p+ maps I to one of its K nearest neighbors equiprobably. We compare two d- dimensional representations with = 128 learned in the following two ways: (i) Kernel PCA to obtain the exact top- d eigenfunctions of TeTy (ii) non-contrastive learning (CN in Theorem 3.4) implemented with VICReg (Bardes et al., 2022). For (ii), we use a fully-connected neural network with Tanh activation, skip connections, and Adam opti- mizer (Kingma & Ba, 2015; Loshchilov & Hutter, 2017). We use the same number of training epochs for every model. For each width and depth, we run the experiments 15 times with different random initializations, and report the average alignment. See Appendix for more details.
-1-66)
err( f) 2 s 33
This result has two parts. First, we show that if f is com- patible (f* E (Pt)), the optimal encoder achieves low error on f*. Second, we ask what if f* is incompatible. We cannot claim that no t works for if one knows f pri- ori, then one can set Φ1 f to achieve zero error. Instead, we show that for any 4, there exists an f with the same compatibility as f for which performs poorly. Therefore, compatibility reflects whether context is suitable for task.
Evaluating an arbitrary encoder. The above result bounds the approximation error of the encoder that learns the contexture. We can also bound the approximation error of an arbitrary encoder. See Appendix C.
We measure the alignment between the two representa- tions using the canonical-correlation analysis (CCA) metric RoCA and the mutual KNN metric with 10 neighbors like Huh et al. (2024). We center and whiten the representations (making the covariance identity) when using mutual KNN. CCA is invariant to all invertible linear transformations on T which is ideal because such transformations do not affect the performance of the downstream línear probe, since one can adjust W and b of the línear probe accordingly. We do not use linear CKA proposed by Kornblith et al. (2019) because it is only invariant to orthogonal transformations.
4.2. Implications for Neural Scaling Laws
Scaling laws (Kaplan et al., 2020) state that the performance of large neural networks grows with their size. Moreover, models of different architectures learn highly aligned repre- sentations when scaled up. Huh et al. (2024) thus proposed the platonic representation hypothesis that scaling makes representations more aligned with an underlying reality, though they did not formally define this reality.
The contexture theory provides a new perspective on the role of scaling. The function class from which the feature encoders & are trained is a subset of (Px)), and as the model gets larger, the class approaches L (Px). This sug- gests that scaling brings the learned representation closer
Figure 1 plots the alignment between the exact top-d eigen- functions and the learned deep representation while varying the depth and width of the neural network. When they are chosen optimally, the CCA can be as high as 0.9, and the
6

---

Contextures: Representations from Contexts
mutual KNN can be higher than 0.8. Note that these align- ment metric values are very high. For example, in Huh et al.
between the X and A of a context is too weak, then the signals will be insufficient. If the association is too strong, then capturing the signals will be too hard. The association level affects the spectrum of the context-the stronger the association, the slower the decay of the singular values.
(2024), the mutual KNN metric value is usually below 0.2. Hence, the representation learned by the neural network is highly aligned with the top-d eigenfunctions.
The top plot studies neural networks with increasing widths. We observe that when the neural network is relatively nar- row, increasing the width improves alignment. However, once the neural network is sufficiently wide, further increas- ing the width may have a negative effect. For example, when the depth is 3, the aligmment is the highest when the width is 512, and the alignment becomes lower when the network is wider than 512. Since increasing the width can only make the function class of & larger, this phenomenon is not due to the expressivity of the neural network. We hypothesize that it arises from optimization difficulty, that is larger models are harder to train effectively. Consequently, with the same number of pretraining steps, a larger model will be farther away from the minima, leading to reduced alignment.
Weak association. Consider the extreme case where is independent of X. This context clearly useless because it provides no information. In this case, only the trivial singular function μi all the other singular values are O. When X and A are nearly independent, kx (I, x²) is very close to 1, which causes the singular values to decay too fast. Formally, we have:
Case 1:
has positive singular value;
Lemma 5.1 (Proof in Appendix G.I). When ( (a, x') 1 < e for all x E X, we have Σi>o o7 << 6.
In Appendix F, we empirically verify that low association leads to small |k* (xx, x²) 1| for all x, In such scenario, Fe (P+) is very small set, so very few tasks are compatible with and can benefit from the context.
The bottom plot studies neural networks with increasing depths, and a similar trend is observed. When the network is shallow, increasing the depth improves the alignment. However, once the network is sufficiently deep, further in- creasing the depth may have negative effect.
Case 2: Strong association. The context A = X is useless Contexts whose singular values decay too slow are bad because (i) for pretraining, there are non-smooth singular functions with large singular values, which are hard to learn; (ii) for downstream, larger d is needed, as more singular functions have non-trivial contributions, and it leads to a higher sample complexity. In Appendix F, we empirically verify that kernel ki has high Lipschitz constant when the association is strong, meaning that the kernel is non-smooth and thus the singular functions are non-smooth.
In summary, we draw two conclusions from this experiment: (i) the representation Iearned by a large neural network is highly aligned with the top-d eigenfunctions; (ii) once the neural network is large enough, further increasing its size does not increase the alignment. Hence, we argue that once the model is large enough such that & is already highly aligned with the top-d eigenfunctions, further increasing the model size inevitably yields diminishing returns.
5.2. aaass-agnostic Evaluation of Contexts
A good measurement of context usefulness should be task- agnostic, because we would like the pretrained encoder be transferable to a variety of tasks, which we might not know at pretrain time. Note that for any task-agnostic metric, one can adversarially create task for which the metric fails, so there is no universal task-agnostic metric. However, a metric can still be very useful if it provides guidance for most real tasks. To this end, we propose the following metric:
5. Context Evaluation
How to create better contexts a challenging problem. In this section, take a first step by studying when context is useful and how to efficiently evaluate its usefulness. The key result is that the usefulness of a context is largely de- termined by the association level between X and A, and a useful context should have a moderate association. The association level affects the decay rate of the singular values. We propose a usefulness metric that only depends on the singular values. Then, we empirically verify that this metric has strong correlation with the actual performance of the encoder on many real datasets. As such, the proposed metric can help practitioners to select among various pretraining methods or hyperparameter settings efficiently.
Ta = 1 - 1
3
T =
-=mming Tdi
(3)
Σ
where 3 > 0 is a parameter, and dg is the maximum embed- ding dimension we consider. Typically dg ranges from 512 to 8192. We choose 3 1 and do 512 in our experiments. Ta is proxy of the prediction error when the embedding dimension is d. Thus, the d that minimizes Ta can be viewed as the predicted optimal embedding dimension, and 7 evalu- ates the context when d is chosen optimally.
5.1. The Effect of Context Association
A useful context should provide sufficient training signals that are easy for the model to capture. If the association
Metric derivation. Let the target function be f = fo + fi, where (fo. fi) = 0. f1 is compatible with the
7

---

Contextures: Representations from Contexts
1
1
1
Cosine similarity
0.5 x
0.5 x
0.4
0.4
0.3
0.3
0.5
0.5
0.5
0.2
0.2
0.1
0.1
0
0
0
0
100 200
0
100 200
0
100
200
0
50
100 150
0
50
100
150
0.8=
0.8 1
0.8 x
4
(a) abalone
(b) MNIST
0.7
0.7
0.7
Figure 4. Comparison of the dowastream task between abalone and MNTST. The y-axis is the cosine similarity hetween the down- stream task and the i-th eigenfunction.
0.6
0.6
0.6
0.5
0.5
0.5
0.4
0.4
0.4
0 100 200
0 100 200
0
100 200
(a) Weak association
(b) Moderate
(c) Strong
the context, similar to Section 4.2. We adjust the association
Figure 2. Metric illustration on abal 1 one. Top row: context spec- tra. Bottom row: black solid curves are 7a divided by 6; red dashed curves are the actual downstream prediction error. We divide Ta by 6 to fit it in the same plot.
level by changing K: K = 150 (weak), K = 30 (moderate) and K = 5 (strong). We obtain the exact eigenvalues and eigenfunctions of Tut using kemnel PCA. In Figure 2, we plot the spectra of the three contexts in the top row. Then, in the bottom row, we compare Ta against the prediction error of the linear probe under different d. We can see that when the association is weak or moderate, Ta first decreases and then increases, which tracks the actual error. However, when the association is too strong, monotonically decreases with d, and it cannot track the actual error.
1 ?
1
0.5
0.5
0.5
0
0
0
100
200
0
100 200
0
100
200
0.8
0.8
0.8
0.6
0.6
0.6
Second, we use the MNIST dataset. The context is ran- dom cropping with crop ratio a. We adjust the association level by changing at a 0.5 (weak), a 0.2 (moderate) and a = 0.05 (strong). Since kernel PCA is not scalable to datasets large as MNIST, we instead train LeNet (LeCun et al., 1998) using the non-contrastive learning ob- jective (CN in Theorem 3.4) and the A optimizer. Then, we estimate the top eigenvalues using the method in Appendix D. The downstream task is binary classification task whether the digit is greater than 4. After pretraining, linear probe is fit on top of & using ridge regression. The result is plotted in Figure 3.
0.4
0.4
0.4
0.2
0.2
0.2
0 100 200
0
100
200
0
100 200
(a) Weak association
(b) Moderate
(c) Strong
Figure 3. Metric illustration MNIST, similar to Figure 2.
context, and fo is not compatible with the context. The prediction error can then be decomposed into (i) the approx- imation error of fi; (ii) the approximation error of fo; (iii) the estimation error. Theorem 4.4 bounds component (i) by ei-(1-0)" In practice, 81 is usually very close to 1, and we sf-s2-1 simplify this bound to the first term of Eqn. (3), up to con- stant factor. For component (ii), stronger associations imply that more tasks are compatible with the context, reducing this approximation error. Thus, this component should be negatively correlated with Dee sR. Component (iii), the estimation error, increases with stronger associations, since higher association typically requires larger d, and thus greater sample complexity. Based on the results in Zhai et al. (2024), this component can be essentially understood as positively correlated with Σd= s?. The second term in Eqn. (3) combines the contributions from components (ii) and (iii), and is designed to be bounded by 1. This met- ric can be efficiently estimated. It only requires the top-dg eigenfunctions of Tag. which can be estimated in O(m²) time using a random subset of m = ((do log do) samples. See Appendix D for details.
Unlike abalone, on MNIST the downstream error mono- tonically decreases with d. This disparity stems from the difference between the two downstream tasks. To demon- strate this, in Figure 4 we plot the cosine similarity between the target function f and the estimated i-th eigenfunction the two datasets. We can see that the variance of f" on abaloone is mostly concentrated on the top-5 eigenfunc- tions, with the first cosine similarity being almost 0.5. In contrast, the variance of f on MNIST is more scattered, and the cosine similarity is still close to 0.1 for the 150-th eigenfunction. Consequently, having large d on abal 1 one will have a little impact on the approximation error but will increase the estimation error significantly. On the other hand, having larger d on MNIST will decrease the approx- imation error more than it increases the estimation error, which is why the total error monotonically decreases with d. The takeaway from this experiment is that, while a context with moderate association is generally good, its effective- ness ultimately depends on the specific downstream task.
We now conduct an experiment that examines Ta on two datasets. First, we use the abal lone dataset and KNN as
8

---

Contextures:
Representations from Contexts
The implication is that evaluation metric would univer- sally work for all contexts and downstream tasks. However, metric would still be useful if it correlates well with the actual error in most scenarios, and thus can provide insights into choosing the right context and the right hyperparame- ters, such as the mask or crop ratio.
Table I. Correlation between and enr,- on all 4 types of contexts (clipped), Full results reported in Table 3 in Appendix G.
Dataset
Size (t)
WFeat.
Type
Pearson
Dist.
diabetes
768
Cls
0.737
0.740
Moneyball
1232
14
Reg
0.680
0.650
yeast
1269
Cls
0.221
0.256
splice
3190
60
Cls
0.831
0,801
5.3. Empirical Verification
abalone
4177
Reg
0.028
0.470
mushroom
8124
22
Cls
0.185
0,340
Now we examine if our metric correlates with the encoder's performance on real datasets. In practice, the performance is influenced by many factors. To create a setting where all factors but the context are controlled, we let the encoder be the exact top-d singular functions obtained by kernel PCA.
pumadyn32nh
8192
32
Reg
0.938
0.961
SpeedDating
8378
120
Cls
0.590
0.656
grid stability
10000
12
Reg
0.925
0.911
brazilian_houser
10692
Reg
-0,290
0,563
fifa
19178
28
Reg
-0.349
0.663
kings county
21613
21
Reg
0.842
0.882
Each dataset is randomly split into a pretrain set, down- stream labeled set, and test set. The downstream linear predictor is fit via ridge regression. Hyperparameter grid search is conducted at both encoder learning and down- stream stages. The evaluation metric is the mean squared error. Let erra be the actual prediction error when is d- dimensional. We test d up to dg 512. Let d' be the one that minimizes erra. We use four types of contexts:
cps88wages
28155
6
Reg
0.250
0.479
Mean on 28 datasets
0.431
0.611
Median on 28 datasets
0.587
0.659
does not work on all datasets. For example, the Pearson correlation is very negative brazi iataailianni hoossessa and houses fifa. To understand the failure modes, in Appendix G we do more analysis on the datasets where our metric fails.
RBF kernels: k(z,, a) = exp(->/|= -al3) We define p+ as pt (a z) x k(x,, a) for each x.
6. Conclusion
We advance the science of representation learning by articu- lating the target of representation learning-the top singu- lar functions of particular operator induced by a context, which we term contextures. We further prove that such a representation is optimal because it minimizes the worst- case approximation on the class of tasks compatible with the context. We show that most representation learning approaches could be cast as estimating contextures, and em- pirically verified that large neural networks can learn the top singular functions well. We further analyze when context can be useful, relating that to its spectrum, and proposed a task agnostic usefulness metric that correlates well with the encoder's performance on real datasets.
KNN: P+ (a x) = K-1 if a is KNN of xr else O.
mask: First, randomly mask 20% of the features, and then apply RBF kernels to the other features. Specifically, we randomly draw 50 masks, and use the average of pt over all masks as the context.
KNN 20% random masking and then apply KNN.
For each of these contexts, A- = X. For each type, we use 35 contexts by adjusting y for RBF kernels and K for KNN. By doing so, we adjust the association level between X and A. We make sure that contexts in every type range from very weak to very strong association. We do not use masking alone because its dual kernel is hard to estimate.
Our analysis has three limitations, which lead to three open problems. First, our analysis focused on the minimizers of the objectives. However, Cohen et al. (2021) showed that deep models trained by popular gradient methods do not find the minimizers, but instead oscillate around the edge of stability. The open problem is how this phenomenon affects our results. Second, we did not discuss the impact of the inductive bias of the model architecture, such as the trans- lation invariance of convolutional neural networks. Such inductive biases can affect the context and, therefore, the encoder. We pose how to integrate the effect of these bi- ases into our theory an open problem. Third, our theory assumes that Px is fixed. In practice, however, there is al- ways a data distribution shift from upstream to downstream. Refining our theory to handle such distribution shifts is an exciting direction for future work.
In Table 1 report the correlation between 7 and err over all 140 contexts from the 4 types 28 classification (Cls) and regression (Reg) datasets from OpenML (Vanschoren et al., 2013) widely used in machine learning research. The most common metric is the Pearson correlation, but it can only detect linear correlations, while the correlation between T and erra- is not necessarily linear. Thus, we also report the distance correlation (Székely et al., 2007) that can detect non-linear correlations, but it cannot tell if the correlation is positive or negative because it is always non-negative.
The median reported in the table shows that on more than half of the datasets, there is a Pearson correlation of over 0.5, which is in general considered a strong correlation. The distance correlation is even higher. As expected, the metric

---

Contextures: Representations from Contexts
Acknowledgements
Cohen, J., Kaur, S., Li, Y., Kolter, J. Z., and Talwalkar, A. Gradient descent on neural networks typically oc- curs at the edge of stability. In International Confer- ence on Learning Representations, 2021. https: / //openrevi eww. net/ forum? d= heiiiddd-h--rteeeeee rTtvkGeM.
We thank Rattana Pukdee, Hugo Contant, Chenhao Zhang and Zihao Ye for their feedback on this paper. Kai Yang did this work at Carnegie Mellon University, under the PKU- CMU undergraduate research program. We acknowledge the support of NSF via IIS-2211907, ONR via N00014- 23-1-2368, AFRL via FA8750-23-2-1015, and DARPA via HR00112020006
Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, NAACL-HLT, Volume / (Long and Short Papers), Minneapolis, MN, June 2019. Association for Computational Linguistics.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z Gheshlaghi Azar, M., Piot, B., Kavukcuoglu, K., Munos, R., and Valko, M. Bootstrap your own latent - a new approach to self-supe learning. In Proc. Advances Neural Information Processing Systems, December 2020.
Code and Data Availability
The code for this paper can be found at https: //colaba www.ereeeercccc..e r/c.laab.eeeercch.cogo 1GdJ00n-PPKAB www.le.c..ooooooooooo.ooo. com/ drive/ = sharing. The data can be downloaded from OpenML.
Gupta, A., Wu, J., Deng, J., and Li, F.-.. Siamese masked autoencoders. In Advances in Neural Information Pro- cessing Systems, New Orleans, LA, December 2023.
References
Balestriero, R., Ibrahim, M.. Sobal, V., Morcos, A. S., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., Schwarzschild, A., Wilson, A. G., Geip- ing, J., Garrido, Q.. Fernandez, P., Bar, A., Pirsiavash, H., LeCun, Y., and Goldblum, M. A cookbook of self- supervised learning. arxiv:2304. 12210, 2023.
HaoChen, J. Z., Wei, C., Gaidon, A., and Ma, Provable guarantees for self-supervised deep learning with spectral contrastive loss. In Proc. Advances in Neural Information Processing Systems, December 2021.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, June 2022.
Bardes, A., Ponce, J., and LeCun, Y. VICReg: Variance- Dovariane-covvaiiane regularization for self-supervised learning. In International Conference on Learning Rep- resentations, 2022. URL https: /ttp///ppeeeeeeeeeeeeeeeeeeeeeee net/
Huh, M., Agrawal, P., and Efros, A. A. What makes im- agenet good for transfer learning? wrXiv:1608.08614 2016.
Baudat, G. and Anouar, Generalized discriminant analysis using kemnel approach. Neural computation, 12(10): 2385-2404, 2000.
Huh, M., Cheung, B., Wang, T., and Isola, P. Position: The platonic representation hypothesis. In Proc. International Conference on Machine Learning, Vienna, Austria, July 2024.
Belkin, M. and Niyogi, P. Laplacian eigenmaps for di- mensionality reduction and data representation. Neural computation, 1373):137))136, 2003.
Jing, L., Vincent, P., LeCun, Y.. and Tian, Understanding dimensional collapse in contrastive self-supervised learn ing. In International Conference on Learning Represen- tations, 2022. URL https: www.ooveeeeeeeee ew net// forum? Yed-eevo0SDDN...
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E.,, et al. On the opportunities and risks of foundation models. 108.0:2008.072588 2021.
Johnson, D. D., Hanchi, A. E., and Maddison, C. J. Con- trastive learning can find an optimal basis for approxi- mately view-invariant functions. In International Confer- ence on Learning Representations, 2023. URL https: / //oppereeelve ew net/ forum? d=A
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual repre- sentations. In Proc. International Conference on Machine Learning, July 2020.
10

---

Contextures: Representations from Contexts
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv:2001.0836,, 2020.
Roweis, S. T. and Saul, L. K. Nonlinear dimensionality reduction by locally linear embedding. Science, 290 (5500):2333-3226, 2000.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,, Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition chal- lenge. International Journal of Computer Vision, 1 115: 211-252, 2015.
Kingma, D. P. and Ba, J. Adam: A method for stochas- tic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Represen- tations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: //arxiv. org/ abs/ / 1112. 6980.
Shawe- Taylor, J., Williams, C. K., Cristianini, N., and Kan- dola, J. On the eigenspectrum of the gram matrix and the generalization error of kemnelPCCA. IEEE Transactions on Information Theory, 51(7):2510-222,, 2005.
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Similar- ity of neural network representations revisited. In Proc. International Conference on Machine Learning, Long Beach, CA, June 2019.
Shimodaira, H. Improving predictive inference under covari- ate shift by weighting the log- log-likeldhoo function. Jour- nal of Statistical Planning and Inference, 90(2):227-244, 2000.
LeCun, Y., Bottou, L.,, Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86(1) 1):2278 2:2278-2324, 1998.
Sutskever, I. Test of time award talk: Sequence to sequence learning with neural networks. Advances in Neural Infor- mation Processing Systems, 2024.
Liu, Q. Lu, H., and Ma, S. Improving kernel fisher discrim- inant analysis for face recognition. IEEE transactions on circuits and systems for video technology, 14(1):42 49, 2004.
Székely, G. J., Rizzo, M. L.., and Bakirov, N. K. Mea- suring and testing dependence by correlation of dis- tances. The Annals of Statistics, Statistics,356)))22211 35(6):2769-27994 2007. ISSN 00905364. URL http: //ww. jstor. stable/25464600
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint arXiv: 17100500, 2017.
Mika, S., Ratsch, G., Weston, J., Scholkopf, B., and Mullers, K.-R. Fisher discriminant analysis with kernels. In Neural networks for signal processing IX: Proceedings of the 1999 IEEE signal processing society workshop (cat. no. 98th8468), pp. 41 41-48. leee, 1999.
Thrampoulidis, C., Kini, G. R., Vakilian, V., and Behnia, T. Imbalance trouble: Revisiting neural-collapse geome- try. In Proc. Advances in Neural Information Processing Systems, New Orleans, LA, December 2022.
Vanschoren, J., van Rijn, J. N., Bischl, B., and Torgo, L. Openml: Networked science in machine learn- ing. SIGKDD Explorations, 15(2):49-6) 60, 2013. doi: 10. 10.1145/2641190.26611 org/10 0ei.1..14452244411 0t1111199..6411999 2641198.
Oord, A. d., Li, Y., and Vinyals, O. Representation learn ing with contrastive predictive coding, arXiv:1807.03748, 2018.
URL http:://oi.....
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. In Proc. Advances in Neural Infor- mation Processing Systems, New Orleans, LA, December 2022.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https:/ openreview weiile.nnet.... ht////ormm/// seerum?idyyyssUSzddd. Sur- vey Certification.
Paninski, L. Estimation of entropy and mutual information. Neural computation, 15(6): 191-1253, 2003.
Papyan, V., Han, X., and Donoho, D. L. Prevalence of neural collapse during the terminal phase of deep learn- ing training. Proceedings of the National Academy of Sciences, (44-))):266222266666 2020.
Yarotsky, D. Optimal approximation of continuous func- tions by very deep relu networks. In Conference on Learn- ing Theory, pp. 639-649, Stockholm, Sweden, July 2018
Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In Proc. International Conference on Machine Learning, pp. 12310-12320, July 2021.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.

---

Contextures: Representations from Contexts
Zhai, R., Liu, B., Risteski, A., Kolter, Z., and Ravikumar, Understanding ougmentation-based self-supervised repre- sentation learning via rkhs approximation and regression. In International Conference on Learning Represemtations, 2024. URL https::////. / open ew, n forum? Ax2yhh0rr...
12

---

Contextures: Representations from Contexts
A. Proofs for Section 3
A.1. Proof of Theorem 3.1
Theorem 3.1. Let A be a one-hot random vector. Suppose the linear layer is unbiased, that is b 0. Then, 4- minimizes R( if and only if extracts the top-d eigenspace of AT+ATp+ where (a a') = IIa a²). or (Ag) - Pa))) If all classes have the same size, then the top-d eigenfunctions of Tp+ AT=+ and Tp+ p+ the same.
The following lemma will be very useful in the proof.
Lemma A.I. Tp: ATpp is the integral kernel operator of the following kernel
k(x, x²) = // (a(s) (d'da'dddd'.. )dada'
Proof. By definition, have
(T h))') =
Thus we can get
(ATp:h)(a) (a) = (T
// (d))))(((ddd)a) da'
// (())Pt (2 da'.
This implies that
/ (a)p
(Tpi (Tp,
/] (athaaaaaadd)PPPpPPp"P"11 (z))dddd dx!
/ Mo)))(2,
as desired.
Then, we finish the proof of Theorem 3.1.
Proof. For any fixed 4 define
R(D, W) = Ep+ [|A [
[IA w
XxPx (x
Assuming, without loss of generality, that Ex~rx [ and change the value of W respectively. Thus, it amounts to minimize
diy; otherwise can perform Gram-Schmidt process क;
R(Φ, W)
E
E
|A W
XxPx A (11)
E X~Px
I (X))
122
E (+1x)
(A, W
+ E |AA A~PA
X~Px
|| - 2 E
E
(A, W (x(X)) +
E ||41/2 A~PA
X~Px
Denote W = (ww))sssd. We have
JR
2wij -
E
E
(x)),
Dwij
X (xP+(-))
13

---

Contextures: Representations from Contexts
which implies that for fixed Φ, the optimal W that minimizes this loss should satisfy
Waj
E
(-1x) (X)]
E
X-Px
Combining the minimizer of W with R and notice that BANPa |All is a constant, it suffices to maximize
F(Ø)
Σ
E
E (x1,
XxPx (x))
/
1/ (oo))r))))) (02la2)dz do dx
(x)
= / Σ 2
where
b21, = // (a))
(4)
// I[ai = az) (azl (22 d
Thus d* is minimizer of R( (() if 4- extracts the top-d cigenfunctions of k(11, x2). Combining with Lemma A.1 yields that ka a²) = Ila a'}. Furthermore, we have (a) g(a') (a a' d')dPa(a') = (a(), desired.
all classes have the same size, we have Pa(a) = E (0, where e is a constant. Thus (Ag) = s(a) Pa = ca(a), which implies that Tpi ATP- = cTpi Tp.. This concludes that Tp ATp. and Tpi Tpr share the same top-d eigenfunctions.
A.2. Proof of Theorem 3.2
Theorem 3.2. Under the setting of Theorem 3.1, let the linear layer be biased. Then, d minimizes Rbal ()) if and only learns the contexture of p+.
Proof. For any fixed define
R( W) Ep+
1 |4
(x)x)
E XxPx
B
11- W
(PP(AA)
(x
(P((A)
Assuming, without loss of generality,
E
E (-P-(11x)
D,
=dgj
X~Px
Pa(A)
otherwise we can perform Gram-Schmidt process on i and change the value of W respectively. Thus it suffices to minimize
R(D, R))
E
E
| (x)
X~Px (-P((1X)
/ PA(A)
=
E
X~Px Amp+(1X)
/ PA(A)
-2 E
E X AmP+(-X)
A
w )+
+ IE AnPA
||AI1
(PA(A)
PA (A)
A
||A1/
= ||] - E
B
PP(AA) w
+ E A~PA
XxPx Amp+(1x)
(P((A)
14

---

Contextures: Representations from Contexts
Denote W 11sd.sss)) We have
ØR
A (P((A)
2wij
2 E X-Px (~P*(1XX)
E
1x(X)
Swij
which implies that for a fixed Φ, the minimizer of W satisfies
A (P)(A) 7x(x)
Wij
E X~Px (-X)
E
Combining the minimizer of W with R, it suffices to maximize
R'_ = Σ
A:
-1 (/ dx2;
E
E
XxPx
(-I)
(P((A)
where
x211, |1
(a1,02)
pt
P1(as)
I|a a2)
p+ ((aall
PA((a)) (a((a))
P+ (ala1)Pt P+ (al12) dy.
Pa(a)
Thus, 4- minimizer of R(Φ) if x- extracts the top-d eigenfunctions of (2). Combining with Definitions 2.1 and 2.5 yields the desired results.
A.3. Result for Regression
For regression where A is an arbitrary Euclidean vector, using the same objective as Eqn. (I), we can prove the following result.
Theorem A.2. 4. minimizes Eqn. (1) if and only if 4 extracts the top-d eigenspace TT+AP. Ife linear layer is unbiased (b = 0), then ka(a, a') = (a, a'); if is biased (b can be arbitrary), then KA (a, a') = (ā. a².
Remark A.3. Kernel (a d') (a, a') is called the linear kernel on A, and (a, (ā, a') is called the centered linear kernel w.r.t. PA. Theorem 3.1 is a special case of Theorem A.2.
Proof. For the unbiased linear model, the proof is similar to that of Theorem 3.1. Combining Eqn. (4) and Lemma A. yields the desired result.
Next, we consider a biased linear model. For a variable 2, we denote = =EZ), and ă = -E[2] as its centered version.
For any fixed $ define
R(D, W.
E X-Px
E 1-P-(110)
|| W b1(
E XxPx P-(-11
||4 w b12]
|| w 6 b1?]
E X-Px A~P
E
Ā w क
(x)
+
X~Px A-P (1x)
where
Thus, for any fixed 4 W, the optimal b = - (EXx)x
Assuming, without loss of generality, Ex-Px (x = sty; otherwise we can perform Gram-Schmidt process on o and change the value of W respectively. Thus, it suffices to minimize
15

---

Contextures: Representations from Contexts
R(2, W)
E
E
A (1-w6(X)(1)
X-Px AmPt(1AA (-1x)
(x)
-2 E X~Px AxPi(1))
( (y,
+
A
X-Px
AnPa
E -|I 2 X~Px AmPt(1)x)
(A, (A.
E AmPa
A |//2
Denote W- = (Ss, We have
[x),,((]]
2wzz - E
E
Twiy
XxPx A-P (1x)
which implies that for fixed 4, the minimizer of W satisfies
[A,
Wij = X~Px A (+(())
E
[ is constant, it suffices to maximize
Combining the minimizer of W with R and notice that EA~PA
R'=
Σ
E ,,0,(x)
X A- (+xx)
1
/
Sataaa/kkaaaaaaaaak//////T/aaPaa dxzda2
/ Σ
(1) (clc
where
22) /f (a (2))
Notice that
Pat = ((), (I2, day
= -/ Paa =0,
thus Ø" is a minimizer of R(D) if क extracts the top-d eigenfunctions of k(21, Combining with Lemma A.1 yields the desired results.
A.4. Proof of Theorem 3.4 Theorem 3.4.
T minimizes Le or Lx and only if क. Tpt 5n learns the contexture.
Proof. (i) The spectral contrastive loss is
1 E [ (M(A)) ((A))) ((t)) + 2A-~PA
(AAA),
Lc(V)-
E XaPx P+
E
Suppose v/s = Ejz0 Ci where Vj is the ONB of (P2) in Lemma 2.3. Since vj is the ONB of (Px) and vy = 1, we can get for j 1, Er (a)) = do,j = Thus we can get wh db - E[4/)] Σ5z1 Czpyz-
16

---

Contextures: Representations from Contexts
Denote matrix C (ct))sisddj2 matrix B (byz) CCC. C, and matrix D diag ..,....) where 8, the singular value of T We have
(ύ(A), (
E X PxA,
/// (0
/ ( (())Pt (a ( tr)dat p((x)de
/ (Tp+ Tp- Tp- (T== = |T p
Σ sabst
and
( (K(A)) -1
Σ (
dPA(o)dPa(a))
A,A PPA
//
E
ISiJsd
Σ by
=
i
Thus, we have
Lc(V) = Σ 8iby +
1
Σ 1.j
bi = | D1 D
2
So if suffices to minimize |B - D where rank(B) S d. By Eckart- Young-Mirsky Theorem, we know the minimizer Thus the minimizer of should be C Udiag(s1, 82) where E &Rdxdd an orthonormal matrix. This indicates the minimizer 1" extracts the top-d singular functions of T Tp+. and 4" learns the contexture of p+
B-D
of B B' = .... ...
iii The non-contrastive loss is
Cx(VV-
E XNPx A.A+
E
(ū(()) (A
(V) X~Px A,A (AIx
((()) (A |12]
where COVPA [V] I. Since for any v
17 (AA)
(V( ( 2 E
22d
AnPA
is constant, thus T minimizes (V(VV T minimizes L'x (V).
Suppose 1b: Σ 220 120 CapPy where Pj is the ONB of L²(PA) in Lemma 2.4. Since Ep, [a)]
S0,5. we can get
th vh B[v:] E5Σ1 Coyy.
We consider the minimizer of LN (V) By the calculation in (i), we obtain
(x(V)- = X~PxA.AI ~P (1x)
[ ((+))) ((A)))
11p' 1 = Σ bi
By EPA [d455] = Sjjj we have
Σ bis Σ 2 d.
17

---

Contextures: Representations from Contexts
Since v is ONB of (Pa))), v1,"". Ad are orthogonal, we have
2
Usi
Vi Us
S Sluuile =1
(5)
PA
==1
==1
Thus, we conclude that
d
d
+ CV(V)
sf b(s)
8
bis
Σ s2(11 - -bú)
Σ sb6 = 0, <
1=1
12d
which implies that (V) 2 - Ei= sf. To attain equality, we will have b;: = 1 for i = 1, d, and by: = 0 for 2 d + 1. By Eqn. (5), we can know & extracts the span of 1/,- Va. indicating that Tir extracts the top-d singular functions of T++ and 4* learns the contexture of p+
-
A.5. Result for Denoising Autoencoders
For denoising autoencoders, suppose X' C Rdx, Then, consider minimizing the following objective:
R(V)=
min WER x bER (X,A)-P+
E
(6)
Theorem A.4. Let TT* be any minimizer of Eqn. (6). Then, T extracts the top-d eigenspace Tpp ATp+, where A is the integral operator of ky (x,xx')) x') = (z,²") if b can an arbitrary vector, or kx(x, = 2x, if = O. Consequently, hh = p+ V extructs the top-d eigenspace of p+ Tpt A. p+
Proof. The proof the same as Theorem A.2.
A.6. Proof of Theorem 3.5
Theorem 3.5. Let d' be any solution to Eqn. (2) (so that for any constant c, Φ + c is also a solution). Then, :- learns the contexture of p+
Proof. Without loss of generality, suppose T- 0. We have
Σ f(v) w(u, t) d(u))
(Tp: f.9) - w(u, v) daum
(Te: fl
(f, (f.Tr19)
which implies that Tp- is self-adjoint. Therefore, the eigenfunctions of Tp+ are the same as those Tpp Tpt, with square root eigenvalues.
For the objective of Eqn. (2), have
E P3, [[00(u) - ((t))) = E
[|0(2))12 (0(u),
(u
(10ille2 - -6a,, Tpi (x))))
d Σ (0r, pp+
Note that (u, and (v, u) can be drawn from P., with equal probability. We conclude that T extracts the of Tpt, which are the same the top-d eigenfunctions of T Tpt the top-d singular functions of Tpt. This implies that & learns the contexture of Tpt.
18

---

Contextures: Representations from Contexts
B. Proofs for Section 4
B.I. Proof of Theorem 4.2
Theorem 4.2. For any f E Fett),, there exists ag E (PPa) such that (()) = (lg)(A) z). and g" satisfies
E X~Px A. x) ( < 4el1g"11ia
E
Proof. Let g" = s.u.y. We have already explained that if f E F.(Pt), i.e. BIf and p(f", Pt) 2 c, then it must satisfy the condition w.r.t g":
(f" ("*Tp+ Px 1
|| Px 11g" Il
For Eqn. (7), we have P(A]A a) f (A'IX x)Pt (z|AA a)dz, where (=(aa)
P (x (x) Pa(a) by Bayes
rule. Then, using Definition 2.1 we have | a) = d')a, which implies that
EX-PREANN (A) = -EA-PA BA ( و (A)9"
(AA) / ('dd)P(a((A)dd) = EA ('A) / ((aa))
= -EA ((A)
= <
Since T240" = Tp+9 = = s uiVi Eqn. (7) is equivalent to (s - xt < 2e > s.ug Meanwhile, we have
Ed 2 - ))² Σ u 2 (1 - 26) uu. By Cauchy-Schward Schwarz inequality, we have
( { ut) 2 (Z
(1- 26) (E sFuz),
which proves Eqn. (7).
B.2. Proof of Theorem 4.4
Theorem 4.4. Suppose 1 81 S S Se51-1
f.. 2
For any d, among all = - [ol..... ...
Aal where di E (2(Px) %
minimizes err( (F;FF(((P)) if and only if it learns the contexture of Tp+. The error is given by
err (T( = s c1
min R:X-RR dFEL)(Px)
Conversely, for any d-dimensional encoder & and any e > 0, there exists f E (Px) such that p(J, P+) = 1 - c, and e -1-
Proof. Necessity:
Since span( () is at most rank-d, thus there exists f1 E span ... M241] with || = 1 that is
orthogonal to span($). Thus there exists f1, f2 E span(ui, ... m44+13 } with [| = [|2ll = 1, S1 is orthogonal to span(@) and fz E (T) (thus fi 1 f2). and μ1 E span (f1, fe). Suppose μI aifi + +affa (without loss of
generosity, assuming (n1 02 [0, 1)) and denote fo a2f1 f1.2. Then fl Ixx 1 and (1I, fo) 0. Since
fi, f2 E span( Maa+1). we have fo E span
Ma-1) } and thus E[fo] = O.
Consider f = 81p1 + B2fo where 87 + 52 1, B1, B2 E [0, 1]. Denote f = Σiz1 Hipi we can get E, ue = 1 and
B3 < 3 - (1 c)-
Σ
3 + 3 ot - (s} 5 )32 (1 11- c)² Σ 47
of 241
Obviously, f E F(Pt). We have
== 31141 + B2fu 31aaff + a2f2) + 32 a2 = (a131 + 2232)f1 + (a231 - B2)
By the definition of f1, /2 we can know the approximation error for ʃ is { + 22 We can show F(a)) 0133 + 02 = a + v1 - 1/ (o) E [0, first increases then decreases when 81 E [0. Thus F(a))2 2
min (m()),,), = (i, (3). Take 33 = 81-8011
01-0
< 1. we can get for f. the approximation error is always at
least 3-0-0
8/-82+1
19

---

Contextures: Representations from Contexts
To attain equality, we must have Eis1 oFuţ कM + 1 This implies that f1 Md-1- indicating that span(o1, , +d) span(1u1 [(d).. Thus + learns the contexture of T
Furthermore, denote fo = kz/l2 +... ..... kat 1 and consider f = -B1 + B2f0 where B8 + = 1, 31, B2 E [0, By
the definition of fo and f. we have l| = 1 and f = B1 + 82 = 51 + B2k2t12 +.... Pyka+1 Thus
d+1
d+1
p() = 5t + 3
Σ
k = of
$7
Σ
k
32.
2=2
i=2
Take
8/
s E)
81 (1 e)
5
81
8 +
1 2
sf -
k?
8 82
2 82
we have P*) = 1 -- e. Similarly, the approximation error for is
s7 - (1 - e)²
87 (1 6) e)²
(a1 + x232)) 2 min 33
$7 33-(1t (E s716?
87 8
Sufficiency: For any f E F(P+) with IIfl = and E[f] 0, denote f = Eiz1 UzH4, where Σ1z1 uf 1. Obviously we have (1 (1=c)² s Ei21 su S 1. Notice that when (pan ... (a) = span span(1,, (A1, ... , (aa) since learns the contexture of Tpt, the approximation of f will be Σizd+1 uz : A. By the given conditions, we have
d
(1 (1-6)2 2
u S
2
Σ
u
+
d+1
Σ 12d+1
uf = 89 1 -07 1)=1),, A
121
i=1
and this implies that
$7 (1 c)²
A
min wER*, 4 BER
w TT + b-
fl
Px
o7 8
When uf = . (1-6)
-1-()-)) of -
- u
the equality holds. Thus the approximation error reaches its lower bound
when & learns the contexture of Tp+
C. Evaluating an Arbitrary Encoder
Given a context that is compatible with the task, the encoder that learns the contexture is optimal. Now, what about an arbitrary encoder t? Is it possible to bound its worst-case approximation error on the class of compatible tasks? To derive such a bound, two key objects are necessary: the induced RKHS and the ratio trace. They were originally defined in (Zhai et al., 2024) for sepersupe learning, and here we extend them to broader scope.
Denote the range of Tp+ by R(T}-) (Ti+f f E (Px))
Definition C.1. The induced RKHS of p+, denoted by Hy+, is the Hilbert space R(TP-) with the inner product given by (Tp+ Tp+ f2) Hpt = (f1, f2)
An alternative formula is that for any h1, h2 E Hp+ where h1 = Σ u.uy and h2 = Σ 0uU, there is (hz, 12) Hpt = Σ 4:22
Proposition C.2. The induced RKHS Hpt has the following properties:
(i) ky is the reproducing kernel, such that h(a)= = (h, ka(a, 74, for all Hpt.
(ii Hpp is isometric to span(ps s; > 0}. which is subspace of (Px).
(iii) f" (P+))) is equivalent to h" Tp+5 f satisfying the the following isometry property:
(1 = s $11a511
PA - h
| "4pp"
(8)
Hpp
20

---

Contextures:
Representations from Contexts
Proof. For any h Hp+ where h TTp+ S anddd / = E Welles have
(h, 10, (a. ()) +1+
<> SyUgVi, Σ (kvu(a)ss) Hpt
Σ s,uvi(a) = h(a):
which proves (i). (ii) is obvious. Regarding (iii), recall that f E uilli E F((Pt)
equivalent to Σ121
sfu? z
(1-e)² () Ei>1 ui, and this is
[
PA 2 (1 - -) Hpl
Mcanwhile, it is obvious that [ PA
S
hr
always
Mpp
holds.
Definition C.3. Define covariance matrices C+ Cov [8). and B Co (T D). If Cp is invertible, then the ratio trace of & w.t.. pt is defined as RT( P+) = RT ... 0d: pt) == C BR); otherwise, let &' = [or, .. 6,,] be the maximal linearly independent subset of (61, Dd). and define the ratio trace of & the same as the ratio trace of +'.
The ratio trace of any & essentially measures how well 4 is aligned with the contexture of p+ Multiplying by any invertible matrix does not change its ratio trace. If & leamss the contexture, then its ratio trace is sî +.... + sg. which can be easily shown by setting Φ; p4- In fact, this is the maximum ratio trace of any d-dimensional encoder.
Lemma C.4. Suppose "1. du are orthonormal and all have zero mean. Then, we have
IT 101 1 + +.................. 1 sbal PA S s
Proof. Let ci = EEz 9ij4y for i E [d]. Then, Q (4tz) is a matrix with d orthonormal rows and infinitely many columns. It is easy to see that the left-hand side is equal to TrQQDQT ), where D = diag $7. s22 ..... Let qj be the j-th column of Q. For all j E [d], there is Ei= q. qi S j: and for any j > d, there is E/= q. qi d. Thus, using Abel transformation, we have
x
8
j
d
Tr(QDQT) TT(DQ -T-
Σ
3
Σ
Σ
(sg (5+1 S
9
qi qi
j=1
j=1
as desired.
The ratio trace induces a key quantity in the approximation error bound called the trace gap, which reflects the gap between 4 and the top-d singular functions. The larger the trace gap is, the larger the approximation error will be. A simple definition is of +... 2 RT P+), whose lower bound sy+1 can be achieved by the top-d singular functions, the optimal encoder. However, there is an issue with this definition. For example, consider an encoder with d = 1000. It Iearns the t0p-10 singular functions, but the other 990 dimensions are complete noise that has zero contribution to RT P+). The approximation error of this encoder should be no higher than that of the top-10 singular functions, because adding more dimensions will never make the approximation error higher. However, if becomes larger and RT (b; P+) stays the same, then oì + ..... + 2 2 P+) will become larger, so this quantity does not correlate with the approximation error in this scenario. The following definition fixes this issue.
Definition C.5. For any linearly independent fi,..., fo E LP(xx), denote F = [f1, frl, Cp = Cov and B. = [F]. The trace gap of & w.E.f p+ is defined as
TG( (t); inf inf d'Sd St Jar
{
8 2/ Tr(CF'
Obviously, this definition of trace gap is upper bounded by oî +....++ 2 RT; Pt). ). It solves the issue in the previous example because having completely noisy dimensions does not affect the trace gap. The following result bounds the approximation error. Theorem C.6. Suppose TG( P+) < si and 6 > 1 - 81. Then,
err( Φ; (.(P(())) S of - (1- s P+)
oी TT
Remark C.7. This bound is fairly tight. If o learns the contexture, then by Theorem 4.4 we have enr(O:F. = d]-(1-)) and TG;;; p+) 8d+1- Compared to this exact formula, the above upper bound only has an extra 81 TG(0; P+) ai- term the numerator.
21

---

Contextures: Representations from Contexts
Proof. Let J1, , Sa be the functions that minimize of + +...... + 5 Tr(Cp Without loss of generality, assume that f1.... ; fa have zero mean and are orthonormal. Let F = ff, far J. and H = (Tp fi.... - } For any f E Fe*))t with fl 1, let h = T f HHp.. and let fo be the projection of f onto F. Since is upper bounded by || fx it suffices to show that || - fzllex is upper bounded by the right-hand
side.
Let a² = Il and 132 = 115 1-1 5z where a and 3 are non-negative. Then, a² B² =11111px = =1 = 1111 ,t pt The isometry property says that (1 - =) + 3²) S ||Alle, Let f fr = fo where ll = 1. Let hy T hr +...+
and ho = -o. Then, we have || pl I ffllf = sîa?. Mcanwhile, since fo is orthogonal to fi. .. fd', by Lemma C.4 we have fol= Px + TT- fi Il +.... fa S of +...+ sz which implies that Tr BF ). Let TG(0; pt). Then, we have
(Tpt foll PA I oी +... +zy+1
|MBle = ||اs + 3ho 11 > | + 3 1holll
23 1 ||ollle S 5s9a² + 22
Thus, we have (1 - (a² + (²) 2 a + 32 + 3a, which implies that (sf -6f--99 -2) = [s] - (1 - e)² + +B) +
28 3a S [s] - (1 - c)2 + +1 + 8²) as desired.
Connection to Fisher discriminant analysis. Fisher discriminant analysis (Mika et al., 1999; Baudat & Anouar, 2000; Liu et al., 2004), or more generally linear discriminant analysis (LDA), is a classical method of learning linear classifiers in statistics. Here we show that Fisher discriminant analysis has a strong connection to the contexture theory. Suppose X C Rdx, Fisher discriminant analysis defines the following between-class covariance matrix SB E Rd xdx and within-class covariance matrix Sw E Rdxxxx:
/ʃ { [B[x A= a1] - E|X A = [E A= an - E A= A= a2l)")
Sg =
Sw
Epi (x- E[X A a)) - E[X A al) T
A A= a= dP)(aa)
In the original formulation of Fisher discriminant analysis, A is the label of X. Here we extend it to a general context variable. Consider a linear encoder ((x)) = Wx where W E Rdxxxx Then, one solves the following optimization problem to find W:
maximize J(W Tr ( [WSSWT) (W (WSWWTTT > s.t. WSw W is invertible. W WER
Here, J(W) is called the Fisher discriminant. Define v(a) = Ep+ W X| = a]. Then, we can see that
WS&W = // (v(()) - -(0)) vaz)) T dPa(aa)dPa(os)) WSw WSwWT. T (0(()(((x)) ((G)))0))) -())((())) A = of dl'
Let C+ = =(( 7 and Bg = (A) J. Then, we have
W. = [ () - V 1
2E
1 (A) = =2Bs:
WSW T / |Ep+ [E
(a) | A A= ad]]Pa((a)
=E(O(X)((X] E (A))
= E -E E (AA)((A))
= =C4 - B4.
Therefore, J(W)= [T - Bb)
which is very similar to the ratio trace defined in Definition C.3. Recall that an encoder that learns the contexture maximizes the ratio trace. well-known result that J(W) is maximized when W consists of the top-d eigenvectors of Sw Sg. Hence, Fisher discriminant analysis is almost equivalent to contexture learning under the constraint that the encoder must be linear.
22

---

Contextures: Representations from Contexts
1
Ground truth
= 300
0.5
0.5
0.5
m 1000
m 2000
m = n
0
0
100
200
100
200
0
100
200
(a) abalone (n = 1177)
(b) fifa (n = 19178)
(c) kings.county (n1 = 21613)
Figure 5. Estimating the cigenvalues using a random subset of m samples, from total training samples.
D. Efficient Estimation of the Context Usefulness Metric
To estimate the metric тa defined in Eqn. (3), suffices to estimate the top-dlo eigenvalues of the context. This can be efficiently done with the following procedure:
Train an encoder & whose output dimension is at least dg to learn the contexture with random subset of m samples.
(ii) Estimate the covariance matrix C+ E R.dxd = Cov [+) with Monte Carlo.
(iii) Estimate B& E RRddd, where [i = (bi, Tax 2b
with Monte Carlo.
Px
(iv) Solve the generalized eigenvalue problem B& = \C The eigenvalues \ 2 ... 2 Ado are estimates of si, " , Moreover, Q = [,,,.. v2) where (v;) are the orthonormal eigenvectors corresponding (Ai). Let o- be the normalized version of &Q such that each dimension is scaled to unit variance. Then, oi, O2, are estimates of p1.... Hdg.
If we only need to estimate the eigenvalues, then Schawe-Taylor et al., 2005) showed that for any fixed d. the sum s + $2 be estimated with low error using (c)) i.i.d. samples. By union bound, all si.... 22, be estimated with low error using = (dg log do) i...d. samples. However, if we want estimate the eigenfunctions well, then usually we need to use the entire training set.
Let us demonstrate this method on 3 real datasets from OpenML (Vanschoren et al., 2013); abalone fifa, and kings_county. We use KNN with K = 60 as context, where AA = X, and P (z'[z) = K-1 if x' a K-nearest neighbor of a and O otherwise. For this context, we can exactly compute k and thus we can obtain the exact eigenvalues (ground truth) using kernel PCA. Meanwhile, we pretrain & with one of the variational objectives using a random subset of m samples, and estimate the eigenvalues using the post-hoc approach. Then, we compare the estimation with the ground truth.
We use a 2-layer wide Tanh-activated neural network with embedding dimension d = 512 and hidden dimension 20,000 as Φ. We train the model through non-contrastive learning with the orthonormality constraint implemented by VICReg (Bardes et al., 2022), and AdamW (Kingma & Ba, 2015; Loshchilov & Hutter, 2017) as the optimizer. We vary m and compare the estimated top-do eigenvalues with the ground truth, where d, = 256. The estimated eigenvalues and the ground truth are plotted in Figure 5. From the plots, we observe that the eigenvalues estimated by our estimation method decay faster than the ground truth, even if the full dataset is used. We hypothesize that the main reason is that even though we use very wide neural network, its function class is still subset of (Pxx. Consequently, the inductive bias of the model architecture has an impact on the encoder, and therefore the learned contexture can be viewed as mixture of the inductive bias and the original KNN context. This mixture causes the eigenvalues to decay faster, which explains the observation in Figure 5. Another reason is related to optimization. Since the model is non-convex, gradient methods cannot find the minima of the objective.
The average estimation error of the top-256 eigenvalues is reported in Table 2. The error is defined as de Dde 1 sf1. where A the estimated eigenvalue. The table shows that when m E [600, 1000) [0.5d, log do, 0.7dg log do). the
23

---

Contextures: Representations from Contexts
Dataset
= 100
m = 300
m = 600
m 1000
m. = 2000
Full dataset
abalone fifa
0.157
0.124
0.088
0.104
0.110
0.088
0.218
0.151
0.137
0.134
0.133
0.131
ki kings county
0.278
0.264
0.190
0.183
0.177
0.177
Table 2. Average estimation emor of the top-256 eigenvalues.
performance is comparable to using the full dataset, which verifies the theoretical result of (Shawe-Taylor et al., 2005). The estimation eπrоr is not zero even if the full dataset is used due to the aforementioned reasons. In summary, the post-hoc method can estimate the eigenvalues using a small subset of samples, but the estimated eigenvalues decay faster than the ground truth.
E. Scaling Law Experiment Details
Here we provide more detailed description of the experiment setting in Section 4.2.
Experiment overview. The purpose of this experiment is to examine whether large neural network can learn the contexture well, and whether scaling up the model size makes the learned representation more aligned to the top-d cigenfunctions. We compare two encoders. The first encoder is obtained via kernel PCA on the dual kernel, so it consists of the exact top-d eigenfunctions. The second encoder is obtained via training a large neural network to optimize an objective that can learn the contexture. Then, we compute the representational alignment of these two encoders. The most classical metric the canonical-correlation analysis (CCA) metric Ri which is invariant under invertible linear transformations to the encoders. (Kornblith et al., 2019) proposed a variant called linear CKA, which is only invariant under orthogonal transformations. In setting, since we only care about the span of Φ1. oa. would like the metric to be invariant under all invertible transformations, which is why we use CCA. In addition, we also use the mutual KNN metric with 10 neighbors proposed by (Huh et al., 2024), which measures the intersection over union (IoU) of nearest neighbors between the two representations. This metric is not invariant under invertible linear transformations, so we whiten the two representations such that their covariance matrices are both identities.
Setup. We use the abalone dataset from OpenML, and split the dataset into a pretrain set, a downstream train set and a downstream test set by 70%- 15%-11144 15%. We use K-nearest neighbors (KNN) with K 30 as the context. The embedding dimension is set d 128. For the second encoder, we train a fully-connected neural network with Tanh activation and skip connections for sufficient number of steps with full-batch AdamW, and vary the depth and width of the network so that we can study their effect on the alignment. Here, "depth" refers to the number of hidden layers- for example, a 2-layer neural network has depth 1. For each width and depth, we run the experiments 15 times with different random initializations and report the average alignment.
In our experiments, we observe the dimension collapse problem (Jing et al., 2022)-if we set the output dimension of the neural network be d, then the rank of the learned representation will usually be less than d, meaning that can only extract the top-d' eigenspace for some d' < d. (Jing al., 2022) proved that the training dynamics of self-supervised learning can cause this problem, that is, a large neural network trained with gradient method cannot find the exact minima, but will find low-rank solution instead.
To fix this issue, we set the output dimension of the neural network to be d, = 512 > After we obtain the d didimensionnal encoder, similar to Appendix D we estimate the matrices Cs and Bg, and solve the generalized eigenvalue problem B& = xC Let V [Vi...., va]] E Iddxxe xd be the top-d eigenvectors; then, we use &V as the d-dimensional representation. In other words, we use the 128 principal components of the 112-iimenioonal embedding.
F. More on Context Evaluation
In this section, we offer guidance for practitioners on identifying contexts with weak or strong associations with inputs. We then show that both excessively weak and overly strong associations degrade downstream performance and demonstrate that the proposed quantitative measurements accurately capture association strength in our controlled experiments. Moreover, we provide full experimental results that complements Table 1. Finally, we provide proofs for the lemmas Section 5.1.
24

---

Contextures: Representations from Contexts
F.1. Quantitative measurements for level of association
While mutual information captures mutual dependence between random variables, estimating from samples remains long-standing challenge it requires the joint density function be known (Paninski, 2003). To address this, propose alternative metrics that are computationally more tractable.
Decay rate (all association): As we mentioned, the decay rate of the singular values (s:)120 reflects the strength of association. To estimate the decay rate X, we assume the singular values decay exponentially and fit the regression model s exp (-Xe). When À is large, it indicates fast decay rate and context has low association. Conversely, when À is low, it implies a slow decay and highly associated context.
Expected kernel deviation (weak association): Since the kernel values A (a, a') can be close to for the contexts with low association, we propose using the expected absolute deviation from 1, i.c. Ex,x'~ [ (2, 1I]. as measure to indicate the weak association setting. Given samples "1,"" X, we use Monte Carlo sampling to approximate it, i.e.
Bx (01, 1/.
n²
1=1
Lipschitz constant (strong association): We empirically measure the Lipschitz constant of k for detecting contexts with strong association. Specifically, given samples Fn X, we use the following estimation:
sup (2, (x v)l
1 (#R, x.) - ( 2,,1 || - 25112
max isKssnn
x y11
We note that the first metric requires estimating singular values (s )220 and the last two metrics rely estimating kernel values {x(x,, (z, 2²). For estimating singular values, employ the same technique as the task agnostic metric in Eqn. (3), Estimating kernel values, on the other hand, necessitates training an encode & and approximating & (I. as ((x) ((r))(((1)))) ((x),,
which may require a large training set. Thus, decay rate measurements are preferred due to their simpler estimation process.
Additionally, for the non-smooth kernel kg. we have the following lemma showing that we may have the singular functions could be non-smooth and are difficult to estimate.
Lemma F.. Let the Lipschitz constant for the positive pair kernel k be Luxg = sup P2 (=t( (2siiaa) and the maximum Lipschitz constant of its eigenfunctions be L max421 sup eigenfanctions are bounded by i.e. pa(x) S c.for all i > 0 and x E X. Then we have Lugg S cla Σi=1 87.
||=1 s2al2
[|at -walla Assume that all the
Proof. We have
k (13, x1) (13,
L = sup
[2] 22l12
=1 sup
Eiso s: (x3) (x(1)
Eiso (T
| 222 l
Ees s2 (p: (x())
sup
|11 x2ll2
2 sup
Eisn 57 (23) (1z (x1) - (x2))
(Ho = 1)
| x2l12
Σ Diso (2)(())))))) (x(1))))))))))))) (x(2)
sup
|,] x2 l
2
|M (21) - 1x
sup
(133))
sup #1 R
|| 22112
< sup Σ
Sf (1))
Ex 1
25

---

Contextures: Representations from Contexts
sup clup zyEX 1>0
Σ 8/.
Assume that there exists universal e that bounds eigenfunctions. a highly non-smooth kernel & with high Lipschiz constant Lxx" the lemma implies that we have either (i) smooth singular functions with large Ln and slow decay in singular values with small Ei=1 of (ii) non-smooth singular functions with large T, and fast decay in singular values with small Ei=1 s². For (i), we need a larger d to approximate the kernel well, which leads to a higher downstream sample complexity. For (ii), the function approximation by neural networks becomes more difficult for non-smooth functions (Yarotsky, 2018).
F.1.1. EMPIRICAL VERIFICATION
Setup. We provide empirical evidence showing (I) downstream performance worse for contexts with weak and strong associations, (2) the proposed quantitative measurements are positively correlated with the association level. To control the level of association,
RBF kernels and KNN.
For the estimation kernel value x we use (a,,x)) (ale)) da since P+ (a(2) can be efficiently computed Pa(a)
for these contexts. The decay rate is estimated using a non-linear least squares approach to fit the regression model. For computing the expected kernel deviation, we utilize the entire training set. estimate the Lipschitz constant, we restrict the sample size
1000 for computational efficiency. Other experimental setups are the same as in Section 5.3.
Results. Figure 6 and Figure 7 illustrate the relationship between association level and both the linear probe error erra- and the decay rate for KNN and RBF contexts, respectively. The results show that errą- increases at both extremes, with most blue curves exhibiting U-shape. This suggests that both weak and strong association levels lead to higher errors. Additionally, the red curve indicates that the decay rate À increases as the association level strengthens, highlighting a strong correlation between association level and spectral decay, which is effectively captured by the estimated decay rate.
We report the relationship between association level and both the expected kernel deviation and the Lipschitz constant L in Figure 8 for the KNN context and Figure 9 for the RBF context. The results show that contexts with low association exhibit small expected kernel deviations, while those with high association have large Lipschitz constants. These findings align with our theoretical developments Section 5.1.
G. More Experiment Details and Results
See Table for the full results.
G.1. Proof of Lemma 5.1
Proof. Define k (xxxx') kt (xxx'))-1 -1= Eiso (z)(()))((')) that is the positive pair kernel without the trivial mode (so, (o), where the equality follows the definition of Txy- We also denote the corresponding kernel integral operator as (T (T)) = f g(x)) (x, x')d)x(x'), Then we have
Σ 2 )T(T 20
(ddPx('')=
desired.
26

---

Contextures: Representations from Contexts
redir-approvll
breast-w
Gabetes
solar flare
Monoyball
yeast
Us4
-22
Em
956
Vine
scenee
dha
splice
abalone
spambbae
50
b
Offa
collages
rushroom
Bns4m
p.maeynst2hh
cpu succcctiyyy
Speadibating
grid. stabiitty
140
180
sifhur
braziloor houses
ntt
supercanductiuity
lengr_county
health insurance
<pssEwages
072
354
0.0
KIN
Figure 6. Association level vs emry and the decay rate À for the KNN context on the 28 datasets. A larger K/N indicates a lower association level, while smaller K/N corresponds higher association level.
urditteppoooed
treest-w
diabetes
solar Fare
Moneybell
10
511
022
255
100
200
100
200
100
200
200
Wine
scene
dna
splice
lenvs kp
abalone
spambase
11
10
37-
05-
300
30
20
50
collegee
mustroom
kinthmm
pumadyn322hh
cpU activii
SpeedDDeing
grd grd_siabiity
Decay
-.
sulfur
brazilion houses
ffa
superconductivity
löngs county
health insurance
as88wages
105
100
10C
6M
50
20
V
Figure 7. Association level vs erry- and the decay rate X for the RBF context on the 28 datasets. A larger indicates higher association level, while smaller y corresponds to higher association level.
27

---

Contextures: Representations from Contexts
approval
dilbenes
flore
Honeyball
yeast
up
spambase
activityy
Speadiaating
gnd.
Great
udffir
howr lan hronses
fix
suparrondirrtuug
linge
health jourreee
TeeBBagge
20
Figure 8. Association level vs the kernel deviation Px 14 (2, x²) 1 and the Lipschitz constant L for the KNN context the datasets. larger K/N indicates a lower association level, while smaller K/N corresponds a higher association level. multiply Tu by the input feature dimension d'eat to normalize the L2 distance in the denominator.
rrdit approval
brnost
dabetes
solar flare
Moneyball
yeast
cmD
Wine
scenee
splice
ko-s--p
abalone
Spambae
F0k 0x,21 111
38
100
5.0
2.5
colleges
mustroam
pamaoyes78h
<pu aot_actyyty
Speeditating
grid gyddsttabllny
10
20
20
hrasellanhhuues
superconductiiity
king_county
heathh insurance
cos8Rwages
13-
200
10
50
10
20
10
20
10
20
10
20
y
Figure Association level vs the kernel deviation -x. |& (z, xx) - 1] and Lipschitz constant 4 RBF context on the 28 datasets. A larger indicates a higher association level, while a smaller corresponds to a higher association level. We multiply L by the input feature dimension dos to normalize the Le distance in the denominator.
28

---

Contextures: Representations from Contexts
0.4
0.8
erra
0.8
0.3
0.6
0.7
00
04
A
7
1.6
1.8
2
1.2
1.5
1.8
1.6
1.8
2
(a) diabetes
(b) splice
(c) kings-county
0.4
0.06
OBBF O KNN
etr
A RBF Masking
* KNN * Masking
0.03
0.3
T
1.85
1.9
1.95
1.6
1.8
2
(d) brazilian houses
(e) fifa
Figure 10. Scatter plots of versus etr. Dashed line: Linear fit.
29

---

Contextures: Representations from Contexts
Table 3, Correlation between 7 and the actual enror err,- on all types of contexts.
Dataset
Size (t)
#Feature
Type
Pearson
Distribution
aredit-approval
690
15
Cls
0.583
0.683
breast-w
699
9
Cls
0,072
0.255
diabetes
768
Cls
0.737
0.740
solar_flare
1066
Reg
0.019
0.262
Moneyball
1232
14
Reg
0.680
0.650
yeast
1269
8
Cls
0.221
0.256
cme
1473
9
Cls
0.867
0.860
Wine
1599
11
Reg
-0.084
0.212
scene
2407
299
Cls
0.608
0.685
dna
3186
180
Cls
0.881
0.843
splice
3190
Cls
0.831
0.801
kr-vs-kp
3196
Cls
0.543
0.512
abalone
4177
8
Reg
0.028
0.470
spambase
4601
57
Cls
0.775
0.858
colleges
7603
44
Reg
0.155
0.387
mushroom
8124
22
Cls
0.185
0.340
kin8nm
8192
Reg
0.805
0.760
pumadyn32nh
8192
Reg
0.938
0.961
cpu actiicivv
8192
21
Reg
0.709
0.825
SpeedDating
8378
120
Cls
0.590
0.656
grid-stabiliig
10000
12
Reg 0.925
0.911
sulfur
10081
6
Reg
-0.180
0.487
brazilian
10692
9
Reg
-0,290
0.563
fifa
19178
28
Reg
-0.349
0.663
superconductivity
21263
Reg
0.141
0.367
kings.county county
21613
Reg
0.842
0.882
health
22272
Reg
0.601
0.749
cps88wages
28155
6
Reg
0.250
0.479
Mean
0.431
0.611
Median
0.587
0.659
30


Reviews:
[{"review_id": "8OiT05n117", "reviewer": "Reviewer_asLG", "summary": "The paper introduces a theoretical framework to characterize representations learned by a neural network is: each data sample is represented by a variable and a context variable (e.g. labels in classification settings, augmentations in self supervised learning, or its K nearest neighbors). The authors proposed to consider an operator, denoted as contexture, induced by data and its context, which is defined as the average. They show that certain classes of representation learning algorithms, such as supervised classification of self supervised learning, can be phrased in the presented theory. In particular they claim that most algorithm aim at extracting the largest singular components of the contexture operator. In order to define the performance , they evaluate if the performance of an average representation over the context matched the one of the sample.  They propose a metric, based on computing the first d singular values of the contexture operator, to estimate how downstream performance without having direct access to the downstream task itself. Depending on the relation of the context with the downstream, tasks (i.e. how much one can solve the downstream task from the context) the metric should predict better performance. The validate the theory on  a collection of small scale datasets from the OpenML repository.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We're glad that you find our theory generally good, interesting and valuable. We address your concerns below:\n## 1. Experiments in Section 4.2 (scaling) updated\n**We improved the setting and observed much higher alignments.** Due to the 5000-character limit, please see point 1 in our response to Reviewer [R5n1](https://openreview.net/forum?id=4GZwFPzLgW&noteId=szOgxgY7bm). \n\nAll experiments use 15 different random seeds. The standard deviation plot (of one experiment) at https://i.postimg.cc/2651NYg4/Picture3.png shows low variation.\n## 2. Our results are applicable to larger datasets\n**We added experiments on larger datasets such as MNIST and CIFAR-10.**\nSee point 1 in our response to Reviewer [RMLf](https://openreview.net/forum?id=4GZwFPzLgW&noteId=NtaiQsXUZU).\n## 3. Figure 2\nThis figure illustrates how the association between $X$ and $A$ affects the spectrum’s decay rate, and how $\\tau_d$ tracks the prediction error of a $d$-dimensional encoder.\nWe divide $\\tau$ by 5 for visualization. We’ll enlarge the figures to improve readability.\n## 4. Metrics in [2, 3] do not fit our problem\nThere is a fundamental difference between our metric $\\tau$ and those in [2, 3]. \nThey use the eigenvalues $\\lambda_i$ of $\\Phi \\Phi^{\\top}$: $\\langle \\Phi, \\Phi \\rangle f_i = \\lambda_i f_i$. In contrast, our theory is based on the spectrum of the context: $\\langle \\Phi, T_{P^+}\\Phi\\rangle f_i=s_i^2 \\langle\\Phi, \\Phi\\rangle f_i$.\nThese spectra are fundamentally different. $s_i^2$ is invariant under invertible linear transformations on $\\Phi$, whereas $\\lambda_i$ is not. Since such transformations don’t affect downstream linear probe performance (weights can be adjusted accordingly), metrics in [2, 3] are not suitable for our setting.\n## 5. Elaboration on the two failure cases of our metric\nBoth failure cases stem from the fact that the metric depends only on the spectrum (singular values) of the context.\n- Case 1: The association of the context is too weak/strong. Our metric will indicate that the context is bad, but it is still possible that for a particular task, the context is good.\n- Case 2: The metric might not be able to compare different types of contexts. It may indicate similarity, though one context may perform well on a specific task while the other does not.\n\nHowever, the experiment in Sec.5 shows that such failure cases are rare.\n\n## 6. Why we use kernel PCA and VICReg\nKernel PCA is the standard way to extract the exact top eigenfunctions. It is not scalable, and we compare deep learning based methods with it. \n\nWe only use VICReg in the experiments because, as shown in Section 3, many objectives such as spectral contrastive loss and masked autoencoders, are equivalent to VICReg in that they are all minimized by the top-d singular functions. To demonstrate this, we run the experiment in Sec.4.1 with both VICReg and spectral contrastive loss (with width 512, varied depths). Results at https://i.postimg.cc/Hncjj2Yn/Picture2.png show very similar performance, differing slightly due to optimization. Hence, we use a single method in experiments, given their equivalence.\n## 7. Whether the methods \"solve the task\"\nNo encoder solves all tasks. While we proved that many pretraining objectives learn the contexture, they are not guaranteed to solve arbitrary tasks. In Sec. 4.1, we proved that the encoder’s performance depends on the compatibility between the task and the context. If they are compatible, then an encoder that learns the contexture is guaranteed to succeed.\n## 8. Whether the context variable depends on the encoder\nIn general, the context is defined prior to training the encoder, so it does not depend on the encoder. There are two ways to encode additional context in the encoder:\n- The inductive bias of the architecture (such as translation invariance of CNNs)\n- \"Smoothed\" encoders, such as adding Gaussian noise to the input\n\nThe first method changes the model and the second method changes the input. In both cases, adding additional context weakens the association, hence implicitly changes the level of association. Our theory does not analyze the effect of the inductive bias, and we pose it as an open problem.\n## 9. What is $g$ in supervised learning? \nIn supervised learning, $\\Phi$ extracts the eigenfunctions of $T_{P^+}\\Lambda T_{P^+}^*$, so $g(a) = (T_{P^+}^* \\Phi)(a) = E[\\Phi(X) | \\text{class } a]$. By neural collapse, the representations of samples from the same class $a$ collapse to a cluster, and $g(a)$ is the center of that cluster.\n## 10. Paper updates\n**We have updated the related work section.** See our response to Reviewer [R5n1](https://openreview.net/forum?id=4GZwFPzLgW&noteId=szOgxgY7bm).\n\nWe also added a new limitation paragraph. See our response to Reviewer [rV84](https://openreview.net/forum?id=4GZwFPzLgW&noteId=25ftscT8C3).\n\nWe appreciate your feedback and hope that we've answered all your questions. We are happy to address any remaining concerns.", "claims_and_evidence": "The paper claims the following contributions:\n\n- Theory and formalization looks generally good supporting the following claims (see below), although is not completely clear to me how much this correspond to practical settings and how they related with previous work in detail (see related work section). \n\n    - (i) *representation learning captures the relationship between the inputs and a context variable*  supported by formalization in Section 3 and 4, although it should be related to previous works (e.g. [8,4,5]). \n\n    - (ii) *contexture theory encompasses a large variety of learning paradigms* supported by corresponding proofs in Section 3. \n\n    - (iii) *Optimal representations (representation minimizing the loss) are the one in which the contexture is learned* supported by corresponding proofs in section 4. \n\n    - (iv) *Task and context association dependance*: Supported by section 5.1. How much this is related to the encoder considered? (see related questions in the questions section). \n\nPoints (i) and (iii) could be related to causal representation learning and disentanglement point of view on topicality of representations see e.g. [1,2,3] \n\n\n- Experimental claims are less obvious and convincing. see the experiments section of the review for more details : \n\n    -  *the main role of scaling up the model size is bringing the learned representations close to the top singular values of contexture operator*: the experiments provided are small scale and also the evidence is sometimes weak and motivation for this are not discussed in depth (for example not very high correlation scores in Figure 1 are attributed to non convex optimization, without providing deeper analysis nor explanation of the reason why this could be the case) \n\n    - *Reflecting current status of deep learning models*: while the theory is interesting and valuable, there is not direct evidence of reflecting the current status of deep learning models: Experiments are performed on small scale dataset which are very far even from simple benchmark dataset as MNIST (60k samples).  \n   \n   -  Metric: Is not clear from the experiments for the proposed metric should be compared and contextualized with respect to previous work, e.g. [2,3]", "methods_and_evaluation_criteria": "- The use of the datasets for empirical evaluation is not justified in the paper, in particular given that all datasets are very small scale and don't reflect the current status of deep learning datasets and models.", "theoretical_claims": "- I didn't check the correctness of proofs in detail, but the formalization looks good at a first glance.", "experimental_designs_or_analyses": "## Weaknesses: \n\n- All experiments are performed on small scale datasets (of the order of few hundreds to 20k samples), not reflecting the current status of deep learning models trained on dataset at much larger scales. \n\n- Figure 1: The trend of the correlation plot is not explained in depth: what is the motivation why the correlation decreases at first? What support that the reason of the correlation is always below 0.5: is it hard to disentangle what is affected by the optimization from what the authors want to show here from the theory. It should be verified that training with different seeds attain low variance results. One strategy could be to show for synthetic data that the correlation trend I alway increasing if it expected so. \n\n- Figure 2 plots in the second row are not very readable: a correlation plot like the ones in Figure 3, between errors and \\tau measure would allow to understand better the association level. \n\n-  Results in Figure 3 and Table 1. Although failure cases are highlighted, not a deep analysis or explanation is provided concerning why on certain dataset the metric fails to predict the downstream error.", "relation_to_broader_scientific_literature": "The related work section in the Appendix is quite short and not comprehensive of many works. A more comprehensive discussion with what has been proved in (Zhai 2024) and how has been extended should be included, in order to understand better the contribution of the paper. \n\nSome additional work would need to be discussed. Some examples: [1], in which it is proposed map between operators eigenspaces defined on graphs in latent spaces of arbitrary neural models. [2,3] Discuss representation quality measuring the decay of the eigenspectrum of the covariance operator in representation space. In [6] is introduced relative representation where each data point is represented as function of other points in representations space ( close to the context variable idea) and it's showed that this representation is universal across models, architectures, and training algorithms. Before in [7] was introduced the idea of using second order information to compare representations. It would be interesting to relate the unification perspective proposed by contextures with the one analyze in the causal representation learning setting and disentanglement [4,5,8]. \n\n_[1] Fumero, M., Pegoraro, M., Maiorca, V., Locatello, F., & Rodolà, E. (2024). Latent Functional Maps: a spectral framework for representation alignment, NeurIPS 2024_\n\n_[2] Agrawal, Kumar K., et al. \"$\\alpha $-ReQ: Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay.\" Advances in Neural Information Processing Systems 35 (2022)_ \n\n_[3] Nassar, Josue, et al. \"On 1/n neural representation and robustness.\" Advances in neural information processing systems 33 (2020)_ \n\n_[4] Yao, Dingling, et al. \"Unifying Causal Representation Learning with the Invariance Principle.\" arXiv preprint arXiv:2409.02772 (2024)_\n\n_[5] Zimmermann, Roland S., et al. \"Contrastive learning inverts the data generating process.\" International conference on machine learning. PMLR, 2021._\n\n_[6] Moschella, Luca, et al. \"Relative representations enable zero-shot latent space communication, ICLR 2023_ \n\n_[7] Kriegeskorte, Nikolaus, Marieke Mur, and Peter A. Bandettini. \"Representational similarity analysis-connecting the branches of systems neuroscience.\" Frontiers in systems neuroscience 2 (2008)_\n\n_[8] Achille, Alessandro, and Stefano Soatto. \"Emergence of invariance and disentanglement in deep representations.\" Journal of Machine Learning Research 19.50 (2018)_", "essential_references_not_discussed": "As discussed in the previous paragraph more comprehensive literature review would be needed. Some examples reported below and discussed in the previous section: \n\n_[1] Fumero, M., Pegoraro, M., Maiorca, V., Locatello, F., & Rodolà, E. (2024). Latent Functional Maps: a spectral framework for representation alignment, NeurIPS 2024_\n\n_[2] Agrawal, Kumar K., et al. \"$\\alpha $-ReQ: Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay.\" Advances in Neural Information Processing Systems 35 (2022)_ \n\n_[3] Nassar, Josue, et al. \"On 1/n neural representation and robustness.\" Advances in neural information processing systems 33 (2020)_ \n\n_[4] Yao, Dingling, et al. \"Unifying Causal Representation Learning with the Invariance Principle.\" arXiv preprint arXiv:2409.02772 (2024)_\n\n_[5] Zimmermann, Roland S., et al. \"Contrastive learning inverts the data generating process.\" International conference on machine learning. PMLR, 2021._\n\n_[6] Moschella, Luca, et al. \"Relative representations enable zero-shot latent space communication, ICLR 2023_ \n\n_[7] Kriegeskorte, Nikolaus, Marieke Mur, and Peter A. Bandettini. \"Representational similarity analysis-connecting the branches of systems neuroscience.\" Frontiers in systems neuroscience 2 (2008)_\n\n_[8] Achille, Alessandro, and Stefano Soatto. \"Emergence of invariance and disentanglement in deep representations.\" Journal of Machine Learning Research 19.50 (2018)_", "other_strengths_and_weaknesses": "### Strengths\n\n- **significance**: The paper targets a very important goal of considering different learning algorithms under a unified framework and how to evaluate quality of representation learned by deep learning algorithms. \n\n- **broad theory**  the theory proposed is broad and applies to many setting of deep learning and representations. \n\n- **failure cases** : The demonstration of failure cases for the proposed metric in Figure 3 (d) and (e)  is useful and appreciated.\n\n\n### Weaknesses\n\n- **Clarity**: parts of the text could be improved in clarity: for example, in the section introducing contextures, integral operators and kernels, practical examples accompanying this would be helpful to improve the reader's understanding. \n\n- **Related work**: Related work section should be improved. See specific section. \n\n- **Experimental evidence**: validation in experimental setting of the theory proposed is not very strong and far from models and dataset used in practice (see experimental section).", "other_comments_or_suggestions": "I spotted the following typos: \n\n- line 131: \"The kernels captures\" -> \"The kernels capture\"", "questions_for_authors": "- Why considering just Kernel PCA and VICReg as encoders? How much the contexture operator depends on this choices? \n\n- In the experiments in Figure 2 and Table 1: Do the learned encoders solve the task? I.e. the kernel pac embedding are sufficient to solve the regression and classification  tasks on the test set?   \n\n- How much the context variable depends on which encoder one learns? I.e. for a given data context variable one could optimize for an encoder that controls the level of association despite solving the training task. \n\n- In the setting of supervised classification what is $g$ in practice? Can be just a selection operator or It is an encoder for labels? \n\n- Figure 2: why the \\tau scores are divided by 5?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "smqKxZZBf8", "reviewer": "Reviewer_RMLf", "summary": "The manuscript presents a framework called contextures, showing that many representation learning methods aim to capture the top singular functions of an operator defined by the relationship between inputs and a context variable. It shows that such representations are optimal for tasks that align with the context and that further improvements require better context design rather than just scaling up model size.\n\n## update after rebuttal\nI have increased the score to Weak Accept from Weak Reject, primarily due to additional experiments on MNIST and CIFAR-10.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We address your concerns as follows:\n## 1. Our results are applicable to larger datasets\n**We conduct more experiments on larger datasets such as MNIST and CIFAR-10.**\nWe initially used datasets with $<$ 30K samples (largest: 28,155; see App. F) since most experiments compare the pretrained $\\Phi$ with kernel PCA, which does not scale well to large datasets. However, our theoretical results are independent of dataset size, and our metric is applicable more broadly. Here we show it on MNIST and CIFAR-10.\n\nOn MNIST, we use LeNet as $\\Phi$ and random cropping with various crop ratios as contexts. Spectra of these contexts are estimated using the method in lines 356–368. For instance, see the estimated spectrum for crop ratio of $0.3$: https://i.postimg.cc/bYQP1N46/mnist1.jpg.\n\nWe plot the prediction error of a linear probe on $\\Phi$ (blue) alongside $\\tau_d$ (orange, scaled) at crop ratio 0.3 here: https://i.postimg.cc/3rg4SGkm/mnist3.jpg, which shows that $\\tau_d$ tracks the actual error across different $d$.\nWe further vary the crop ratio from 0.1 to 0.9 and compare $\\tau$ with the prediction error of a linear probe trained on $\\Phi$. The result (https://i.postimg.cc/2yzTMwGX/mnist2.png) shows a strong correlation.\n\nOn CIFAR-10, we use ResNet-18 with SimCLR augmentation as the context. See the estimated spectrum at https://i.postimg.cc/hGTcDLW0/cifar1.jpg. In https://i.postimg.cc/ZqQcWJpL/cifar2.png, we plot the prediction error of a linear probe trained on $\\Phi$ (blue) with $\\tau_d$ (orange, scaled to fit the image). Again, $\\tau_d$ tracks the actual error, supporting its usefulness in choosing dimension $d$.\n\nWe will add these experiments to the paper.\n\n## 2. The benefits of designing better contexts\nWe proved that for a fixed context, making the model larger has diminishing returns, so it follows as a natural logical consequence that better contexts are necessary for further improvement. Thus, we only intended our experiments to provide a simple empirical illustration rather than corroboration. We also note the increasing empirical evidence in the literature: many recent advancements in pretraining are due to better contexts. For example, as the authors of SimCLR noted, its success owed greatly to its aggressive crop ratio and color distortion.\n## 3. The usefulness of representations\nThe experiment in Section 4.2 aims to show that as the model gets larger, $\\Phi$ becomes more aligned with the top-d eigenfunctions. However, a better alignment does not necessarily imply better performance at a downstream task. Whether $\\Phi$ is good on a particular task depends on the compatibility between the context and the task, which we prove in Section 4.1.\n\n\nWe also improved this experiment. See point 1 in our response to Reviewer [R5n1](https://openreview.net/forum?id=4GZwFPzLgW&noteId=szOgxgY7bm). \n## 4. CCA versus CKA\nWe use CCA but not CKA because our setting is different from that in Kornblith et al. In their setting, they want the alignment metric to be invariant under orthogonal transformations, but not under other invertible linear transformations on $\\Phi$.\nThat's why they used CKA. In contrast, we want the metric to be invariant under invertible linear transformations on $\\Phi$ because they do not affect the performance of the downstream linear probe, so we use CCA.\n\nMoreover, CCA is equivalent to linear CKA if both representations are centered and whitened. We also evaluate the CKA when the representation is $[s_1 \\mu_1, \\cdots, s_d \\mu_d]$, that is each singular function is multiplied by the singular value.\nWe run the experiment in Section 4.2 with a fixed width 512 and varied depths. The result is plotted at https://i.postimg.cc/Jzrbc00M/Picture1.png. The plot shows that the trends of CCA and CKA are very similar.\n## 5. Reproducibility\nWe will add details to the appendix. We also kindly point out that we have attached our code in the supplementary material, and all experiments are run with fixed random seeds, so all results can be exactly reproduced.\n## 6. Neural collapse and scaling laws\nOur remarks show that our theory can explain many empirical phenomena and cover different ML paradigms. We believe that including these remarks is important since they show that our theory has real explanatory power. These should be viewed similar to brief remarks in a later section such as conclusions where such discussions are common. We are happy to move these to a later section if the reviewers see fit.\n## 7. Paper updates\n**We have updated the related work section.** See our response to Reviewer [R5n1](https://openreview.net/forum?id=4GZwFPzLgW&noteId=szOgxgY7bm).\n\nWe appreciate your feedback and hope that our response answers all your questions. We are happy to answer any follow-up questions.", "claims_and_evidence": "Some practical claims, particularly the implications for neural scaling laws and the benefits of designing better contexts, are less extensively validated. These claims are based on experiments with a limited set of small datasets.", "methods_and_evaluation_criteria": "The experimental validation relies on relatively small and simple datasets (up to 21k samples).", "theoretical_claims": "Theoretical claims are sound. Additional experimentation on these assumptions and their implications in real-world scenarios can enhance the work.", "experimental_designs_or_analyses": "The experiments rely on relatively simple datasets (up to 21k samples) again.", "relation_to_broader_scientific_literature": "The paper builds on SSL representation learning methods. It frames many self-supervised learning (SSL) approaches as the recovery of the top singular functions of a context-induced operator with the focus on extracting dominant eigenfunctions. This perspective unifies diverse SSL methods (contrastive, non-contrastive, masked autoencoders) in the way of learning low-dimensional representations.", "essential_references_not_discussed": "For example, recent unification frameworks in self-supervised learning provide an important context that is not discussed. \n\nOne work by Balestriero and LeCun (2022) shows that many SSL methods—both contrastive (e.g., SimCLR) and non-contrastive (e.g., Barlow Twins, VICReg)—can be seen as recovering top eigenfunctions of certain operators, akin to classical spectral methods like Laplacian Eigenmaps or PCA. \n\nBalestriero, Randall, and Yann LeCun. \"Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods.\" Advances in Neural Information Processing Systems 35 (2022): 26671-26685.\n\nSimilarly, Munkhoeva et al. (2023) interpret SSL losses as implicitly performing Laplacian-based eigenfunction learning under data augmentations. \n\nMunkhoeva, Marina, and Ivan Oseledets. \"Neural harmonics: Bridging spectral embedding and matrix completion in self-supervised learning.\" Advances in Neural Information Processing Systems 36 (2023): 60712-60723.\n\nThe paper misses several methods, such as GPT, I-JEPA, data2vec 2.0, and DINO v2, that have significantly advanced self-supervised learning in language and vision.", "other_strengths_and_weaknesses": "- While the manuscript attempts to unify many approaches across different domains, it does not perform experiments on real datasets or, analyses, for example, with discussed SSL models. The experiments were performed on very small datasets with a sample size of no more than 21613 samples. There is a lack of application of this theory to more common datasets like ImageNet, or at least CIFAR10 and STL10.\n- In Figure 1, even if we train models with different depths and widths, it is not clear how useful the representation will be with respect to correlations.\n- The classical canonical correlation analysis (CCA) has been shown not to work well with neural network representations (Kornblith et al., 2019).\n- There are only vague details on how the models were trained. It is hard to reproduce.\n\nKornblith, Simon, et al. \"Similarity of neural network representations revisited.\" International conference on machine learning. PMLR, 2019.", "other_comments_or_suggestions": "The manuscript is trying to cover too many topics and ideas, specifically neural collapse and neural scaling laws, which should be discussed in separate publications. Then, you will have space to experiment on SSL and natural image datasets.", "questions_for_authors": "No additional questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yy6qtAqLYr", "reviewer": "Reviewer_R5n1", "summary": "The author proposes a framework that encapsulates common representation learning methods as learning the joint distribution of inputs and contexts. It’s possible to decompose this joint distribution with eigen-decomposition. This decomposition shows that optimal representations are learned via learning the subspace spanned by the top-d singular functions. Thus lots of tasks benefits from learned representation whenever the encoder recovers the span of the top-d singular functions. The authors explain that contexts with either too weak or too strong association with inputs are less effective, and proposes a task-agnostic metric to evaluate context usefulness. Finally, the authors show empirically that the proposed contexture metrics correlate with downstream performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n## 1. Experiments in Section 4.2 (scaling) are updated\nWe changed the embedding dimension from $d=16$ to $d=128$ (which is more common in practice) and reran the experiment. The results are plotted in https://i.postimg.cc/FRncq3Zb/alignment.jpg\n\nThe main observation is that the alignment becomes much higher (CCA generally $>0.8$ and can be $\\approx 0.9$). Also, as the model becomes wider/deeper, the alignment first increases and then decreases. We interpret the results as follows: \n- **Effect of $d$**: The previously used $d=16$ was too small. For instance, the $16^{th}$ eigenvalue is quite large ($>0.3$), and is close to the $25^{th}$ eigenvalue. Thus, it is hard to distinguish which of the top-$25$ singular functions belong to the top-$16$, which is why the original alignment was low. In contrast, the $128^{th}$ eigenvalue is close to zero, so when $d=128$, the network can learn the top singular functions pretty well. Prior work mostly used $d \\ge 128$, such as Kornblith et al., 2019 and Huh et al., 2024, so using $d = 128$ is more reasonable. Moreover, CCA$\\approx 0.9$ is considered very high in prior work.\n- **Alignment trend**: When the model gets larger, the alignment first increases, because the function class of $\\Phi$ becomes closer to the entire $L^2$ space, so the optimizer of the pretraining objective is closer to the top eigenfunctions. However, when the model is already large enough, making it larger makes optimization harder. Hence, with the same number of pretraining steps, a larger model will be farther away from the minima, and the alignment decreases. \n## 2. Updated related work\n**Based on the suggestions of the reviewers, we extend the discussion of the present paper's relation to the prior work as follows, which will be moved to the main body in the final version.**\n\n# Related Work\nUnderstanding the representations learned by an encoder has long been a central topic in machine learning research. Prior work has developed quite different theoretical frameworks for different pretraining methods, but these frameworks are typically not transferable across different settings. This paper provides a unified framework that has a very broad scope, covering learning, SSL based on augmentation, supervised learning, graph representation learning, and more. \n\nIn the pre-deep-learning-boom era, early work on manifold learning revealed the connection between representation learning and extracting the top eigenfunctions of a kernel (Bengio et al., 2004; Coifman & Lafon, 2006). Moreover, using the eigenvectors of the graph Laplacian as node representations on graphs was a classical technique in graph applications (Belkin & Niyogi, 2002; Zhu et al., 2003). \n\nOn the theoretical understanding of deep representations, there are two lines of work that are closely related to this paper. The first line studies **representation alignment** (Kornblith et al., 2019; Canatar et al., 2021; Huh et al., 2024; Fumero et al., 2024). Representation similarity has also been studied in neuroscience (Kriegeskorte et al., 2008). These papers mainly focus on comparing two representations. While we aim to evaluate a single representation and the context on which it is trained, we use the tools developed by these papers in our analysis.\n\nThe second line develops the **spectral theory of self-supervised learning (SSL)**. SSL has achieved remarkable success in recent years (Radford et al., 2019; Chen et al., 2020; Zbontar et al., 2021; Bardes et al., 2022; He et al., 2022; Assran et al., 2023). HaoChen et al. (2021); Johnson et al. (2023) related contrastive learning to the spectrum of the augmentation graph and the positive-pair kernel, and Munkhoeva & Oseledets (2023) related SSL to matrix completion. Later, Zhai et al. (2024) extended the spectral theory to all SSL methods, not just contrastive ones. The present work further extends this theory to representation learning in its most general form, beyond SSL. \n\nBesides, other prior theoretical work on SSL studied its training dynamics (Damian et al., 2022; Jing et al., 2022; Tian, 2022) and built its connection to information theory (Achille & Soatto, 2018; Balestriero & LeCun, 2022; Shwartz-Ziv et al., 2023). \n\nAnother line of work for characterizing the representations aims to learn disentangled or causally associated representations (Higgins et al., 2018; Scholkopf et al., 2021). It is shown that such representations can be provably recovered, provided there is sufficient variability in the environments, for instance by indexing the environments via sufficiently varying auxiliary variables or interventions (Khemakhem et al., 2020; Varıcı et al., 2024; Buchholz et al., 2023; Yao et al., 2024). Some of these results further require stringent parametric assumptions.", "claims_and_evidence": "The observation on optimal representation under learning the contextures is theoretically supported.", "methods_and_evaluation_criteria": "Yes. The authors compare a variety of context types and there are also controlled experiments on the level of association between inputs and contexts.", "theoretical_claims": "I closely verified section 2.", "experimental_designs_or_analyses": "Figure 1 & 2 makes sense.", "relation_to_broader_scientific_literature": "This paper proposes a theoretical framework of contexture which explains why the learned representations are useful for downstream tasks. I see this framework quite generally applicable to a variety of setting as listed in section 2.1", "essential_references_not_discussed": "Canatar, A., Bordelon, B. & Pehlevan, C. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nat Commun 12, 2914 (2021). https://doi.org/10.1038/s41467-021-23103-1\n- This paper discussed similar ideas of learning via eigenfunctions alignment", "other_strengths_and_weaknesses": "Strength\n- The contexture framework provides a perspective to understand why representation learning is useful for downstream tasks. For example, it's not clear why the representation learned via self-supervised learning in vision helps reducing generalization errors on downstream supervised tasks. This top-d singular function learning perspective provides an angle for understanding these phenomenon. \n\nWeakness\n- Figure 1 results are a bit weak. It's still unclear to me why does the correlation decrease as width goes up.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rwTa9rwzrO", "reviewer": "Reviewer_rV84", "summary": "The authors propose a framework for understanding representation learning through the concept of contextures, which are the top singular functions of an operator induced by a context variable. The goal is to characterize learned representations across supervised, self-supervised, and manifold learning paradigms. The main contributions are: a unifying theoretical framework for representation learning, a demonstration that scaling model size yields diminishing returns once optimal contextures are approximated, and a task-agnostic metric for evaluating the usefulness of contexts. The study shows that learned representations align with the top singular functions, and the proposed metric correlates well with downstream performance across multiple datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We are glad that you find our theoretical analysis and the proposed metric valuable. We would like to answer your questions as follows:\n## 1. Efficiency\nFor extracting the top singular functions, the time complexity of kernel PCA is the same as eigendecomposition, which is about $O(m^3)$, where $m$ is the number of pretrain samples. The time complexity of extracting the singular functions using a pretraining objective (the deep learning method) depends on the optimizer, and there is a rich body of work on the convergence rate of popular optimizers such as Adam and SGD. \nMoreover, there exist many deep-learning-based methods that can extract the eigenfunctions, including VICReg, which is used in the experiments, NeuralEF (Deng et al., 2022), and NeuralSVD (Ryu et al., 2024). The complexity of these methods grows linearly with the size of the dataset and does not suffer from the scalability issues of kernel PCA. For instance, for a mini batch of size $B$, NeuralSVD only has complexity $O(B^2 d + d^2 B)$, where $d$ is the dimension of representation.\nIn other words, the deep learning method is scalable and much faster than kernel PCA, though it is not guaranteed to find the exact singular functions because the optimization problem is non-convex.\nWe will add this discussion to the paper.\n\nDeng et al. \"NeuralEF: Deconstructing kernels by deep neural networks.\" ICML 2022.\n\nRyu et al.  \"Operator SVD with neural networks via nested low-rank approximation.\" ICML 2024.\n \n## 2. Evaluation metric \nWe evaluate a context with a task-agnostic metric because we would like the encoder to be transferable to various tasks. Moreover, foundation models are often applied to tasks they are not designed for, so we might not know all the tasks at pretrain time. On the contrary, a probing classifier only evaluates the context on a specific task, so it cannot reflect the transferability of the encoder. There is a strong connection between our theory and alignment metrics, which we discuss in Section 4.2. Alignment metrics compare two encoders (focusing on their relative relationship), while our metric evaluates the contexture of one context (assessing its intrinsic information-capturing properties).\n## 3. Noisy contexts \nThe effect of a *noisy* context depends on how *noisy* is defined. We can think of two possible definitions.\n1)  The noise reduces the compatibility between the context and the task (Section 4.1). In this case, our analysis shows that the learned representation will have a lower quality.\n2)  The noise weakens the association between $X$ and $A$. Section 5 shows that a *good* context should have a moderate association. If the association level of the original context is already moderate, then the noise will make it worse. However, if the association level of the original context is strong, then adding noise might even have a positive effect. For example, it is widely observed in self-supervised learning that strong augmentations make the learned representations better. For example, the authors of SimCLR attribute its success largely to the aggressive crop ratio and color distortion it uses.\n## 4. Paper updates\nWe added a new limitation paragraph in the conclusion (see below), and a new related work section (please see our response to reviewer [R5n1](https://openreview.net/forum?id=4GZwFPzLgW&noteId=szOgxgY7bm)).\n\n\nWe appreciate your feedback and hope that our response answers all your questions. We are happy to address any follow-up questions.\n\n# Updated limitation and open problem paragraph (to be added to conclusion)\nOur analysis has three limitations, which lead to three open problems.\nFirst, our analysis focused on the minimizers of the objectives.\nHowever, Cohen et al. (2021) showed that deep models trained by popular gradient methods do not find the minimizers, but instead oscillate around the *edge of stability*.\nThe open problem is how this phenomenon affects our results.\nSecond, we did not discuss the impact of the inductive bias of the model architecture, such as the translation invariance of convolutional neural networks. Such inductive biases can affect the context and, therefore, the encoder. We pose how to integrate the effect of these biases into our theory as an open problem.\nThird, our theory assumes that $P_X$ is fixed. In practice, however, there is always a data distribution shift from upstream to downstream.\nRefining our theory to handle such distribution shifts is an exciting direction for future work.", "claims_and_evidence": "The authors claim that contextures unify representation learning by framing learned representations as top singular functions of a context-induced operator. They provide theoretical proofs, empirical validation showing neural networks approximate these functions, and a task-agnostic metric that correlates with downstream performance.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited for analyzing representation learning through the contextures framework. The use of singular function decomposition provides a theoretical foundation, and the proposed task-agnostic metric aligns with the goal of evaluating context usefulness. Also, the correlation analysis with the downstream performance is strong.", "theoretical_claims": "No immediate flaws are evident, and the theoretical claims appear sound.", "experimental_designs_or_analyses": "1. The paper evaluates the contextures framework through neural network scaling experiments to test its alignment with theoretical predictions.\n2. The authors compare learned representations with the top singular functions of the context-induced operator in order to validate their claims on scaling limits.\n3. The authors also evaluated their proposed task-agnostic metric by measuring its correlation with downstream task performance across multiple datasets.\n4. The use of canonical correlation analysis and KNN-based metrics provides a structured approach to evaluating representation quality.", "relation_to_broader_scientific_literature": "The framework proposed by the authors utilizes singular functions to characterize learned representations, which aligns with SVD techniques used in Principal Component Analysis (PCA) and Latent Semantic Analysis (LSA).  For the proposed task-agnostic metric for evaluating context quality, it is conceptually similar to CCA.", "essential_references_not_discussed": "The paper's key contributions are grounded in established concepts in representation learning, particularly spectral methods and feature space learning. Most of the essential references were cited in the paper.", "other_strengths_and_weaknesses": "Strengths:\n\n1. Clear description of background knowledge and motivations needed to understand the proposed representation model.\n2. Clear exposition of the proposed method.\n3. The authors introduce the contextures framework that provides a unified mathematical perspective on representation learning across multiple paradigms.\n4. The analysis on scaling laws and the diminishing returns of increasing model size offers valuable insights for future neural network training strategies.\n5. The proposed task-agnostic metric for evaluating context usefulness is a great contribution.\n\nWeaknesses:\n\nThe paper could provide more discussion on the computational complexity and scalability of the proposed methods.", "other_comments_or_suggestions": "1. Overall the paper is well-written.\n2. Consider discussing potential limitations of the contextures framework in handling highly heterogeneous data distributions.", "questions_for_authors": "1. What is the computational cost of estimating the top singular functions for large-scale datasets? A discussion on efficiency and scalability would clarify the feasibility of implementing contextures in practical settings.\n2. How does the proposed task-agnostic metric compare to existing evaluation methods for representation learning (e.g., probing classifiers, alignment metrics)? A direct comparison would help justify the usefulness of the new metric.\n3. How does the contextures framework perform when the context is noisy? Would the learned representations still be useful, or would they degrade in quality?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: There was a healthy discourse amoung the authors and reviewers for this paper. The primary concerns raised in the review process where that there no experiments on larger datasets, some concerns about related work, and some further analysis. The authors have responded to these comments in their rebuttal and the reviewers are in agreement that this paper merits publishing. Reviewer asLG did not respond to the rebuttal but I believe their concerns have been addressed.
Decision: Accept (poster)