Generalization vs Specialization under Concept Shift
Authors: Alex Nguyen
David J. Schwab
Vudtiwat Ngampruetikorn
Venue: NeurIPS 2025 poster ()
Keywords: concept shift
distribution shift
ridge regression
thermodynamic limit
high-dimensional learning
solvable model
out-of-distribution generalization
Primary area: theory
TL;DR: We study concept shift in ridge regression in the thermodynamic limit, and found a non-monotonic data dependence of test performance that generalizes to realistic machine learning tasks (MNIST classification and transformer ICL regression).

Abstract: Machine learning models are often brittle under distribution shift, i.e., when data distributions at test time differ from those during training. Understanding this failure mode is central to identifying and mitigating safety risks of mass adoption of machine learning. Here we analyze ridge regression under concept shift—a form of distribution shift in which the input-label relationship changes at test time. We derive an exact expression for prediction risk in the thermodynamic limit. Our results reveal nontrivial effects of concept shift on generalization performance, including a phase transition between weak and strong concept shift regimes and nonmonotonic data dependence of test performance even when double descent is absent. Our theoretical results are in good agreement with experiments based on transformers pretrained to solve linear regression; under concept shift, too long context length can be detrimental to generalization performance of next token prediction. Finally, experiments on MNIST and FashionMNIST further validate our theoretical predictions, suggesting these phenomena represent a fundamental aspect of learning under distribution shift.

Generalization vs Specialization under Concept Shift
Alex Nguyen
David J. Schwab CUNY Graduate Center
Vudtiwat Ngampruetikorn'
Princeton University
University of Sydney
Abstract
Machine learning models are often brittle under distribution shift, i.e., when data distributions test time differ from those during training. Understanding this failure mode is central to identifying and mitigating safety risks of mass adoption machine learning. Here we analyze ridge regression under concept shift form of distribution shift in which the input-label relationship changes at test time. We derive an exact expression for prediction risk in the thermodynamic limit. Our results reveal nontrivial effects of concept shift generalization performance, including a phase transition between weak and strong concept shift regimes and nonmonotonic data dependence of test performance even when double descent is absent. Our theoretical results are in good agreement with experiments based on transformers pretrained to solve linear regression; under concept shift, too long context length can be detrimental to generalization performance next token prediction. Finally, experiments MNIST and FaSHionMNIST further validate our theoretical predictions, suggesting these phenomena represent a fundamental aspect of learning under distribution shift.
Distribution shift
is unsurprising that a model trained on one distribution does not perform well when applied to data from different distribution. Yet, this
setting is relevant to many practical applica- tions from scientific research [1, 2] to medicine and healthcare [3-6]. A quantitative understanding of ditto---istribttion generalization is key to developing safe and robust machine learning techniques. model that generalizes to arbitrary distribution shifts of course does not exist. The generalization scope of a model, however, needs not be limited to the training data distribution. A question then arises as to how much a model's I's scope extends beyond its training distribution.
Answering this question requires assumptions the test distribution. For example, covariate shift, a well-studied setting for distribution shift, assumes a fixed input-label relationship while allowing changes in the input distribution (see, e.g., Refs [7-11]).
We consider concept shift- relatively less-studied setting, in which the input- label relationship becomes different at test time [13, 14], see Fig 1. While many works have studied how to detect and mitigate concept shift [15. 16), characterizing how concept shift affects generalization behavior in neural networks has remained underexplored. In our work, we formulate a minimal model which enables continuous modulation of the input-label function, based on high dimensional ridge regression a solvable setting that has helped develop intuitions for some of the most interesting phenomena of modern machine learning (see, e.g., Refs [17-25]). We derive an analytical expression for prediction risk under concept shift in the high-dimensional limit and demonstrate that its behavior can change from monotonically decreasing with data for the in-distribution case to monotonically
DIIS and VN contributed to this work equally.
Concept shift or concept drift sometimes defined to be equivalent distribution shift [12]. Here, we adopt a narrower definition which concept shift describes only the change in the input-label relationship [13, 14].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

y
668
Px x
Px
Px
Covariate shift
Concept shift
Figure I: Two flavors of distribution shift. Distribution shift describes the scenarios which the joint input-label distribution of training data {(x1. y1), (x2, y2), ... (filled circles) differs from that of test data {(1, F1). (2)....... (empty circles). Covariate shift (left) assumes a fixed input-label relationship but the input distribution differs at test time, i.e., regression, this condition means that the regression coefficient / is unchanged. Concept shift (right) allows the input-label function change test time, which corresponds to shift in regression coefficient / # ß in the linear regression setting. See $2.
but Px # Px. For linear
increasing, and to nonmonotonic, depending on the degree of concept shift and the properties of the input distribution.
To differentiate these effects from double descent phenomena which can also cause nonmonotonic data dependence of prediction risk [26, 27], focus on optimally tuned ridge regression which completely suppresses the risk divergence at the interpolation threshold and for which in-distribution prediction risk decreases monotonically with more data [17, 18, 28]. The nonmonotonic behavior we observe is also distinct from effects due to model misspecification [18]. While misspecification- property model relative to the true data-gener process can produce similar generalization behavior, distribution shift represents fundamentally different setting in which the data- mata-gennrating process changes at test time. Our work isolates and characterizes the effects of concept shift, type of distribution shift, in correctly specified, optimally regularized models.
Our theoretical results in the thermodynamic limit agree well with experiments, based on transformers trained to solve finite-dimensional regression using in-context examples. We show that more in- context examples help improve model performance when concept shift weak, but can lead to overspecialization for strong concept shift. Finally, we illustrate similar qualitative changes in generalization behavior in classification problems, using MNIST and FashionMNIST as examples. Our work contributes new theoretical framework for analyzing concept shift that complements an extensive body of work on concept shift detection (see, e.g., Ref [29] for recent review).
Our main contributions are:
1. We develop an analytically solvable framework that isolates the effects of concept shift- an important yet often overlooked mode of distribution shift.
2. We explain precisely why and how more training data can hurt generalization performance under concept shift, illustrating its effects several specific settings, including coefficient shrinking, rotation, and feature robustness.
We identify and characterize a sharp transition, separating weak and strong concept shift regimes. 4. We show that feature anisotropy creates qualitatively different patterns of risk nonmonotonicity depending on whether concept shift affects high or low-variance features.
Our experiments on transformers and simple classification tasks are in good qualitative agreement with our theoretical findings, hinting at universal behavior that generalizes beyond our relatively simple theoretical settings.
2 Regression setting
Data. The training data consists of N iid input-response pairs (I)..)))... (vN. yN)). The input x E R is vector of Gaussian features and the response R. is a noisy linear projection of x, i.e.,
y == x with (x.E) ~ N(. (O.:N(EE))N(...
(I)

---

where BER* denotes the coefficient vector, & ER Gaussian noise with variance oz and ΣERPXP the covariance matrix. Similarly, test data point is input-response pair (&, f), drawn from the same process as the training data, Eq (1), but with a generally different set of thra is,
Σ
IB
Training data:
N
B
B
B'IB+OL
(2)
Σ
ΣB
Test data:
N
BT + o2
where in general E and o2 + # Here also define hignal-to-noise ratio SNR =B'
Model. We consider ridge regression in which the predicted response to input reads (x;xX,1 x Ba(X,Y) Y), with the coefficient vector resulting from minimizing Lz-regularized mean squared error,
(x. arg min || x xTb112= ||bb1| (xx* + ANII)) -XY beR! N
(3)
Here d O controls the regularization strength, and Y= y'(',....,,ww) ERRN and X X R PPN denote the training data.
Risk. We measure generalization performance with prediction risk,
R(X) E [s; X, y) E(ÿ 10) x| = B(X) V(X), where the last equality denotes the standard birs-variance decomposition with
(4)
B(X)= E |I X, E(² =x)1 x]
v(X)= [I|5(E:X,)) E(f(;; X) x, x]
Substituting the predictor from ridge regression into the above equations yields
B(X)=
4 AIp B
B- and v(X) 2 Tr
E
(5)
AIp
( + 21p)
where XX /N is the empirical covariance matrix.
It is instructive to consider the idealized limits of = and N co. First, when = inductive biases (e-g., from model initialization and regularization) dominate. For ridge regression, Eq (3), all model parameters vanish, B2=0, and the resulting predictor outputs zero regardless of the input, i.e. = for any x. As a result, RN=n( = E[E = see Eq (4). Second, when N - oo, the empirical covariance matrix approaches the true covariance matrix 4 Σ and, taking the limit - obtain - X) (B B) "BB B) When $= B.,
that RN=o (X) BB= 0 whereas - 00 (see also Fig 2). That is, infinite data is better than no data, as expected.
This intuitive picture breaks down under concept shift, B B. Consider, for example, B=0 which indicates that none of the features predicts the response at test time. this case, RN X) =0 and no = Σ 0 (see also Fig 2); that is, even infinitely more data decreases test performance in the presence of concept shift.
3 High dimensional limit
To better understand this counterintuitive phenomenon in the context of high dimensional learning, we focus on concept shift without covariate shift, i.c., B B and Σ=Σ, and take the thermodynamic limit N, P- and P/N y (0. co). In this limit, prediction risk becomes deterministic R(X) - with the bias and variance contributions given by (see Appendix for derivation; see also Ref [30]).
8T 22G2(-11) 1 T=1663((-2))
B(X) - B = 2-A(
- 2 -))
(6)
v(X) - -V (v(-a)-Av'(C -v
-
We follow the convention Ref [18] where the risk the expected squared error between the predictor and
the mean of test-time conditional distribution. effectively subtracting the irreducible noise variance. As N fixed P. the variance term vanishes, V(X)~O(N and the optimal regularization is 2 =.=
3

---

0=
b
x=l
cos
4
0=
1.0
2
0.0
1.0
R/ 8
0.8
0.2
0.8
0.6
0.4
0.6
0.4
0.6
0.4
0.2
0.2
0.0
0.0
0.1
1
10
0.1 1
10
0.1
10
N/P
NIP
N/P
Figure More data hurts performance when concept shift strong. We depict the data dependence of asymptotic prediction risk for isotropic features, Eq (9), under three concept shift settings of varying degree, parametrized by coefficient alignment cos 0= (=s0--///((llll@) and scaling factor x= |I] but varies. с Mixture of robust and nonrobust features: Bi = =B; feature i robust, otherwise =:= This setting is parametrized by x=x²= =cos2 B. We set SNR = in a-c, and the regularization is in-distribution optimal 1 yyy/SNR.
//Bll/ll// ||lB|| (see legend). Shrinking coefficients: ==xB. b Rotating coefficients: ||
Minimum risk
cos
-2 Rmin/ ||Ali
(normalized)
Figure 3: Concept shift induces a phase transition in generalization behavior. We depict minimum prediction risk Run of optimally tuned ridge regression (d=y/sNR) as and cos vary, see Eq (10) for definitions. No concept shift corresponds to к cos 0= (star). The thick curve, cos 0 = 1/2, separates the weak and strong concept shift regimes. More training data improves generalization only when concept shift is weak, cos 0> 1/2. Outside this region, any data hurts generalization.
where we define (z(2) = (m(()) z/p) (m)))) (() m()== the self-consistent equation,
and v(z) is the unique solution of
(2) 1 T+E62(()) with x(z) C*.
We note that concept shift enters prediction risk only through the last terms of the bias, Eq (6), whereas the variance, Eq (7), is completely unaffected.
3.1 Isotropic features
When Σ=/p, the bias contribution to prediction risk reads (see Appendix for closed-form expression for v(2)
B = ||:1] (3
22(1 2x(1+y((=1) (1 cos 0) 1 2x cos 0 x2
(9)
T+ 2(1 (y(-)))))))
where we quantify concept shift via two parameters
cos = B
Coefficient alignment:
||
(10)
Scaling factor:
= 11,811/ [|

---

K4
=1
b
K4 c0s
d
cos 6_
cos 0,
R/ EB
0.0
0.1
0.2
0.4
0.5
0000
0.5
0.6
0.6
0.7
0
1.0
1.0
0.1
10
0.1
10
0.1
1
10 0.1
10
N/P
N/P
N/P
N/P
Figure 4: Anisotropic features can lead to risk nonmonotonicity. We illustrate prediction risk for the two-scale model, Eq (11), with aspect ratio s-/s, 0.1, spectral weights p, p- = 1/2 and signal fraction
= 1/2. Concept shift is parametrized by coefficient alignments cos 0 B. B /B//|/Blll and scaling factors K₂ || where the subscripts indicate the variance S1 of the affected features. We consider the settings in which concept shift affects either low or high-variance features and either alignment or scale; that is, we vary only one out of four parameters, 0₂ and Kz. at time (see legend). Shrinking coefficients of low-variance features. b Rotating coefficients of low-variance features. c and Same as Panels and b, but concept shift affects high- variance features via K+ and cos 04, respectively. Here SNR = and regularization is in-distribution optimal.
Figure depicts thermodynamic-limi prediction risk for isotropic features under concept shift. We focus on in-distribution optimal ridge regression which corresponds to setting d=y/SNR, ensuring that double descent is absent (see, e.g., Refs [17, 18]). This choice motivated by its practical relevance (as can be approximated via cross- validation) and because it allows us isolate the effects of concept shift from the confounding non-monon non-monotonicity of double descent.
In Fig 2a, consider the effects of shrinking coefficients the coefficient vector becomes smaller at test time without changing direction, 0 and x S 1. When x = 1, concept shift is absent and prediction risk monotonically decreases with more training data, expected for optimally-tun -unne ridge regression. As x decreases and the features become less predictive of the response at test time, prediction risk starts increase with training sample size. This transition occurs at x= 1/2, which R/118|]=1/44
In Fig 2b, we observe similar crossover when the magnitude of the coefficient vector is fixed but its direction changes at test time, == I and 0. Examples of this concept shift setting include the case where some features have the opposite effects at test time, described by B, =-B; for the affected coefficient. Here we see that more data hurts when cos 0 < 1/2 (or equivalently 0> x/3).
In Fig 2c, we consider a mixture of robust and nonrobust features with B =B; if feature is robust, otherwise =0. That is, the robust features have the same effects at test time whereas the nonrobust ones become uninformative of the response variables. In this case, we have cos 0 Vq where 0 denotes the fraction of robust features. We see again that adequately strong concept shift changes the data-dependent behavior of generalization properties from improving to worsening with more training data.
Indeed we can make a more general statement about the transition between weak and strong con- cept shift. First, we observe that for in-distribution optimal ridge regression, A y/SNR, the thermodynamic-linii risk is always monotonic in y. To determine whether more data improve generalization, we only need to compare the limits no data and infinite data: =x and RN- (1 2x cos 0 + x 3/11 It follows immediately that two qualitatively distinct regimes exist. When x cos 1/2, concept shift is weak and more data improves generalization. When cos 0 < 1/2, concept shift is strong and more data hurts, see Fig 3. In the thermodynamic limit, the minimal prediction risk, analogous to phase transition in statistical mechanics. When cos 0= 1/2, prediction risk becomes completely
this sharp boundary manifests as a nonanalyticity
5

---

independent of training sample size-i.e., R = = = RN =R21111?. Quite remarkably, this analysis does not depend on SNR."
3.2 Anisotropic features
To study the effects of anisotropy, we consider a two-scale model which the spectral density of the covariance matrix is mixture of two point masses at and s. with weights p- and p. p.=1-p-, respectively. Without loss of generality, we let s+ 2 s- throughout this section. These two modes define subspaces into which we decompose the coefficients, / B_ B+ with IB=- s=B=. and similarly for B. We write down the bias contribution to prediction risk
B B= S Σ n 12 (--1)=?(-1)
EVEEPUSDS 2dg- Kr cos 0-) + 2x, cos 0, + x (11)
where g+ (2) = (m(2)s, 2)- and π. = B.. / denotes the signal fraction at each scale. Similarly to the isotropic case, we quantify concept shift using coefficient aligmments cos 0₂ B [=-1_/1|&=1I and scaling factors K₂ = =|| both of which now depend also on variance.
Figure 4 illustrates prediction risk under concept shift for two-scale covariates. We numerically tune the ridge regularization strength 1 such that the in-distribution risk is minimized and the divergences associated with multiple descent phenomena are completely suppressed. We consider the cases where concept shift affects only low-variance features via either K- or cos 0_ (Fig 4a and b), or only high-variance ones via either
cos 8, (Fig 4c and d). In all cases, we see that test performance develops nonmonotonic data dependence as test data deviates from in-distribution settings; however, its behavior is qualitatively distinct for low and high-variance concept shift. To isolate the effects of covariate statistics, we set the signal fraction to πI = = so that low and high-variance features carry the same signal strength.
In Fig 4a and b, we depict the effects of concept shift on variancee features. Panel a corresponds to the case where the low-variance coefficients B_ shrink at test time, K- s 1, thus suppressing the signal associated with low-variance features. Similarly to the isotropic case (Fig 2a), prediction risk decreases with N for weak concept shift. However, as || becomes smaller, we see that generalization performance exhibits nonmonotonic behavior, too much training data can worsen generalization. Panel turns to the case in which B- rotates at test time, cos 0_ 1 (cf. Fig 2b). We observe a similar crossover in the data dependence of prediction risk as concept shift becomes stronger. While generalization improves with more data in the low-data limit, this improvement continues monotonically only for adequately weak concept shifts.
In Fig 4c and d, concept shift affects only high- vigh-nariance features via shrinking coefficients (panel a) and rotated coefficients (panel b). We see again that strong concept shift leads to nonmonotonic data dependence of prediction risk. However, strong concept shift on high-variance features results in risk maximum at intermediate training data size N. This behavior contrasts sharply with low-variance concept shift, depicted Fig 4a and b.
Indeed we could have anticipated the intriguing differences between concept shift affecting low and high-varian high-variance features. The data dependence in Fig 4 results from the fact that it takes more data to learn low-variance features and their effects. At low N, high-variance features dominate prediction risk; more data hurts when these features do not adequately predict the response test time, Fig 4c and d. On the other hand, low-variance features affect test performance only when the training sample size is large enough to influence the model. Sufficiently strong concept shift on these features thus results in detrimental effects of more data at larger N (compared to high-variance concept shift), Fig 4a and We emphasize that the nonmonotonic data dependence of prediction risk due concept shift is unrelated to double descent phenomena (which describe risk nonmonotonicity in suboptimally tuned models).
4 Transformer experiments
To test the applicability of our theoretical framework to realistic scenarios, we train transformers to perform in-context learning (ICL) of (noisy) linear functions, which were argued to perform optimal
*sNR controls how prediction risk depends on training data size N (Fig 2), but not the transition between weak and strong concept shift regimes (Fig 3).

---

x²
b
cos 0
1.0
1.00
Theory
0.00
0.9
Pretrained
0.13
0.22
R R/1
0.70
transformer
0.31
0.6
0.5
0400
0.00
0.10
0.91
0.0
1.00
0.5
1.5
0
0.5
1.5
NIP
N/P
Figure 5: Longer context can hurt in-context learning performance. We depict prediction risk of transformers (circles), trained to solve linear regression tasks using in-context examples (see for details). We compare in-context regression We consider two concept shift settings. Coefficient shrinking parametrized by (see legend; cf. Fig 2a). Coefficient rotation parametrized by (see legend; cf. Fig 2b). We compare the asymptotic prediction risk of transformers the predictions from our theory (Section 3) under two concept shift settings. Circles depict MSE loss attained by the transformer whereas lines depict the optimally regularized asymptotic limit. We observe strong agreement between experimental results and thermodynamic limits. Here, SNR=128.
isotropic data with
ridge regression [31-33]. As many-shot prompting, enabled by recently expanded context windows, has shown promise in improving performance [34], it is timely to investigate how distribution shift affects whether longer context is beneficial. These experiments serve as a validation of our theory, confirming that the key phenomena derived from our analytical model persist in more complex, data-driven setting.
In our experiment, a transformer takes as its input series of points (xz, Yz) on an unknown function y; fx;; for i 1,2...... ... 1, terminating with a 'query' Xn whose funetion value yn is the prediction target. The model is trained to minimize the mean square loss L=B[(y, - }. This setup has proved valuable developing intuitions about ICL [31-33, 35-38J. Here, we focus on noisy linear functions (x) B x with & denoting the noise term and investigate the generalization properties of the learned ICL solution, implemented by a transformer, under concept shift at test time.
We consider linear regression in P=32 dimensions. The training tasks, parametrized by B, are drawn iid from a standard normal distribution and kept fixed during training. We generate a total of 22 training tasks, which are sufficient to elicit general- general-purpose purpose ICL. [33, 37]. An input sequence is drawn from y=B"xx+ x with (x, N(O, Ip) x N(O, We choose 2² 00.5. The input-response pairs are newly generated each time the model takes an input sequence. We adopt the nanoGPT architecture [39] with eight layers, an embedding dimension of 128, learnable position embeddings, and causal masking. The model is trained to minimize the next token mean squared error (MSE) using Adam with learning rate of 0.0001. At test time, we sample 10,000 new tasks and compute the in-distribution prediction risk simply as the MSE of the transformer on test tasks. To compute risk under concept shift, the transformer is presented with context, (x1, YI...,,.... Kn-1,, R),, in which = X; + &. But the query x is related to the final prediction target ý via linear function x= with f B in general.
Figure 5 depicts ICL prediction risk for linear regression under concept shift as a function of in- context sample size N. We parametrize the degree of concept shift using the scaling factor and cosine similarity, described in Eq (10). We consider two specific settings: in panel a. S I and cos 0 = 1, and in panel b, x= and cos 0 S (cf. Fig 2a and b). In both cases, we compare ICL prediction risk (symbols) with our thermodynamic-lime theory (lines). We see that they are in good quantitative agreement. In particular, the context-lenge length dependence ICL prediction risk changes from improving to worseming with longer context as concept shift becomes more severe.
To further investigate how transformers handle concept shift with anisotropic features, we conducted experiments comparing models trained on isotropic versus anisotropic data. Figure 6 illustrates how feature anisotropy influences the generalization behavior of transformer-bant based in-context regression under varying degrees of concept shift. When exposed two-scale anisotropic data (see $3.2) where
7

---

11100
b
-0666
=0333
Pretrained transformer
Isotropic training data
Anisotropic training data
R/ Σ
Theory (optimal ridge)
Isoroopcc ridge (A=A/p)
0.5
Structured ridge (A (A=
0.1
1
10
0.1
10
0.1
10
N/P
N/P
N/P
Figure 6: Feature anisotropy modulates concept shift effects in transformer-based in-context regression. We depict prediction risk on two-scale, anisotropic data (see $3.2) for optimal ridge regression (curves) and transformers (symbols) trained to solve linear regression tasks using in-context examples. The theoretical results are for optimally tuned isotropic and structured ridge penalties (dashed and solid curves, respectively). The transformers' results are for isotropic and anisotropic pretraining data (circles and plus symbols, respectively). Concept shift affects only high high-variance features via the scaling factor x = 1.0, 0.6, 0.3 in a, b, and respectively (k_= =os 0 throughout). Transformers trained on anisotropic data (pluses) closely follow the solid curve, suggesting that they effectively implement structured regularization that adapts to feature anisotropy--ffeectivee isotropizing the features and averaging the impact of concept shift (x² (K² /2) + x2) Transformers trained on isotropic data (circles) exhibit nonmonotonic risk under concept shift, but with much more pronounced peaks compared to theoretical predictions (dashed), Here s=/s,=0.1.
p. p+=p- 1/2, n4, = =1 112 and SNR=1.
concept shift affects only high-variance features, transformers exhibit markedly different behaviors depending their training data distribution. Transformers pretrained anisotropic data closely align with theoretical predictions for optimally tuned ridge regression with structured penalties, indicating that they can apply different penalties to features based their variances. This strategy is equivalent to whitening the features, resulting in an isotropized regression problem with an effective concept shift scale parameter x2 + <2) In contrast, transformers trained isotropic data struggle to adapt optimally to anisotropic test data, exhibiting more pronounced nonmonotonic risk curves than theoretically predicted for isotropic ridge penalties. This difference becomes particularly evident as concept shift strengthens (panel c), suggesting that the priors learned during pretraining significantly influence how transformers adapt to concept shift in features with different scales. These results highlight the importance of data structure in determining how models respond to distribution shifts and demonstrate that transformers can implicitly learn sophisticated regularization strategies when exposed to appropriately structured training data. We note that this agreement between theory and experiments not trivial, as transformers perform in-context regression on finite data, finite dimensions, and via learned, data-driven algorithm, whereas our theoretical analyses are based on optimally tuned ridge regression in the thermodynamic limit.
Classification experiments
0.9
MNIST
FashionMNIST
Test accuracy
60k
So far we have focused on regression prob- lems. In this section, we experimentally
0.8
50k
40k
test whether the key insights from our the- ory extend to the qualitatively different set- ting of classification, demonstrating that they likely capture more general phenomena of learning under concept shift. We consider standard multinomial logistic regression for MNIST [40] and FashionMNIST [41], using Adam optimizer with a minibatch size of 500 and learning rate of 0.001 for 2,000 epochs.
0.7
30k
20k
0.6
10k
Fraction of robust principal components
Figure 7: Concept shift qualitatively changes data dependence test accuracy in (a) MNIST and (b) FashionMNIST experiments for various training data size N (see legend). See for details.

---

To vary training sample size N, we choose training data points at random (without replacement); all of the training data used when 60,000.
We modify the test datasets by performing principal component analysis (PCA) on the images and designating the resulting features either 'robust' or 'nonrobust' (ef. Fig 2c; see also $3.1). We shuffle nonrobust features across data points to decorrelate them from the labels while preserving marginal statistics, whereas robust features are unchanged. In Fig we let lower-variance features be nonrobust and parametrize concept shift strength with the variance threshold that separates robust from nonrobust features. We depict the test accuracy as a function of concept shift strength, and observe that when concept shift is strong and few features are robust, more data hurts test accuracy, in qualitative agreement with our results for linear regression problems (cf. Figs 2 and 5). The qualitative agreement between our theoretical results and experiments on transformers, MNIST, and FashionMNIST suggests that the phenomenon we identify is not specific to linear regression, providing further validation for the broader relevance of our theory.
Related works
There is a significant body of work on out-of-distribution generalization with covariate shift, i.e., domain generalization (see, e.g., Refs [8, 42]), but significantly less work has been done on concept shift generalization. However, even within covariate shift, an understanding of ofitt--ddisriiuutin generalization remains clusive [11]. Within concept shift, work has tended to focus instead on detec- tion and mitigation strategies rather than generalization (see, e.g., Ref [29]). Our work contributes to distinct paradigm. We use tools from statistical mechanics and random matrix theory derive an exact, analytical characterization of generalization behavior under a discrete shift. This approach allows us to uncover new phenomena, such as the sharp phase transition between weak and strong concept shift regimes.
Linear regression has proved a particularly useful setting for investigating learning phenomena. Random matrix universality makes it well-suited for theoretical investigations, yielding much-needes
insights into nigh-dimenssioal learning [18, 19, 21-25, 43], including covariate shift generaliza- tion [44]. Linear models have also contributed substantially to our developing understanding of in-context learning in transformers [31-33, 35-38]; work leverages this setting for experimental validation of our theory. Other works use linear attention mechanisms to analyze transformer models (see, e.g., Ref [45]).
The nonmonotonic behavior we identify is distinct from double descent phenomena [26, 27]. Indeed, our theoretical results in Figs 2-4 are for optimal regularization, for which double descent is absent [17, 18, 28), thus demonstrating that concept shift induces risk nonmonotonicity through fundamentally different mechanisms.
We build on analytical techniques developed for high-dimensional ridge regression, but address a phe- nomenon distinct from prior works. In particular, optimally tuned models can exhibit nommonotonic risk due to model misspecification [18]-i.e., mismatch between model class and data- generating process, irrespective of test-time shift. This mechanism differs from distribution shift, in which the data-generating process itself changes at test time. Our framework specifically isolates the effects of concept shift in correctly specified models.
In-context learning of transformers has been studied under covariate shift [36, 46, 47] but not with respect to concept shift. Song el al. [48] studied how transformers generalize to catto--distribution tasks through symbolic manipulation via induction heads, but the scenario of within-contes concept shift remains underexplored. While Agarwal et al. [34] showed that longer context windows typically benefit in in-context learning, they also observed performance degradation in tasks such as MATH.
7 Conclusion and outlook
We introduce ridge regression model for concept shift. Our model is exactly solvable in the high dimensional limit. We show that concept shift can change the qualitative behavior of generalization performance; for sufficiently strong concept shift, ridge regression fails to generalize even with infinite data (Figs and 3). In addition, we demonstrate that input anisotropy can lead to nonmonotonic data dependence of prediction risk. In particular, too much data can harm generalization (Fig 4a and b) and

---

more data may only improve generalization above certain threshold (Fig 4c and d). We emphasize that our results are for optimally tuned ridge regression and thus differ from risk nonmonotonicity due to double and multiple descent phenomena which are absent under optimal regularization [28].
Taken together, our work provides fresh perspective on lesser-studied mode of distribution shift. Our model offers a relatively simple setting for building intuitions and testing hypotheses about concept shift generalization. Although our theoretical analyses are exact only for ridge regression in the high-dimensional limit, the qualitative conclusions generalize beyond this idealized setting. Indeed, our theoretical prediction agrees quantitatively with experiments on regression in finite dimensions from finite samples, using the in-context learning ability of transformer model as learning algorithm. Additionally, our classification experiments suggest that the insights from our theory may apply in more general settings. Finally, our theoretical results for anisotropic features have implications for understanding data dimension reduction techniques- -uuuu as principal component regression in which low-variance features are discarded-unde under concept shift. While we acknowledge that real-world concept shifts occur in more complex contexts, our approach allows us to isolate and precisely characterize key phenomena, establishing theoretical foundation in tractable models and providing a critical step toward understanding more complex settings.
Our work has limitations that open avenues for future research. First, our theoretical results are derived for linear models in the nigh-dimensional limit and may not quantitatively hold for complex nonlinear systems. Second, we analyze a discrete concept shift, leaving the study of gradual or continuous shifts for future work. Finally, our work characterizes a failure mode but does not propose an algorithm for mitigating its effects. We hope our work provides a foundation for addressing these important questions and encourages further systematic investigations of the rich learning phenomena induced by concept shift
Acknowledgments and Disclosure of Funding
Alex Nguyen is supported by NIH grant RFIMHI25318. DIS was partially supported by a Simons Fellowship in the MMLS, Sloan Fellowship, and the National Science Foundation, through the VN acknowledger research funds
Center for the Physics of Biological Function (PHY-1734033 from the University of Sydney. This work was supported in part by the National Science Foundation and by DoD OUSD (R&E) under Cooperative Agreement PHY-2229929 (The NSF Al Institute for Artificial and Natural Intelligence).
References
[1] S. V. Kalinin, C. Ophus, P. M. Voyles, R. Erni, D. Kepaptsoglou, V. Grillo, A. R. Lupini, M. P. Oxley, E. Schwenker, M. K. Y. Chan, al., Machine learning in scanning transmission electron microscopy, Nature Reviews Methods Primers 2, 11 (2022).
[2] X. Zhang, L. Wang, J. Helwig, Y. Luo, C. Fu, Y. Xie, M. Liu, Y. Lin, Z. Xu, K. Yan, et al., Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems (2023),
arXiv:2307.086 088423 [cs.LG].
[3] S. Azizi, L. Culp, J. Freyberg, B. Mustafa, S. Baur, S. Kornblith, T. Chen, N. Tomasev, J. Mitrović, P. Strachan, et al., Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging, Nature Biomedical Engineering 7, 756 (2023).
[4] H. Zhang, N. Dullerud, L. Kalanttr,,,,, Q. Morris, S. Joshi, and M. Ghassemi, An empirical framework for domain generalization in clinical settings, in Proceedings of the Conference Health, Inference, and Learning (Association for Computing Machinery, New York, NY, USA, 2021) pp. 279-290.
[5] R. J. Chen, J. J. Wang, D. F. K. Williamson, T. Y. Chen, J. Lipkova, M. Y. Lu, S. Sahai, and Mahmood, Algorithmic fairness artificial intelligence for medicine and healthcare, Nature Biomedical Engineering 7, 719 (2023).
[6] Y. Yang, H. Zhang. J. W. Gichoya, D. Katabi, and M. Ghassemi, The limits of fair medical imaging AI in real-world generalization, Nature Medicine
(2024).
[7] Quiñonero-Candee M. Sugiyama, Schwaighofer, and N. Lawrence, Dataset shift in machine learning (Mit Press, 2022).
10

---

[8] I. Gulrajani and D. Lopez-Paz, In Search of Lost Domain Generalization, in International Conference on Learning Representations (2021).
[9] N. Tripuraneni, B. Adlam, and J. Pennington, Overparameterization Improves Robustness to Covariate Shift in High Dimensions, in Advances in Neural Information P Processing Systems, Vol. 34, edited by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Inc., 2021) pp. 13883-13897.
[10] D. Idnani, V. Madan, N. Goyal, D. J. Schwab, and S. R. Vedantam, Don't 't forget the nullspace! Nullspace occupancy as a mechanism for out of distribution failure, in The Eleventh Interna- tional Conference on Learning Representations (2023).
[11] R. Vedantam, D. Lopez-Paz, and D. J. Schwab, An Empirical Investigation of Domain General- ization with Empirical Risk Minimizers, in Advances in Neural Information Processing Systems, Vol. 34, edited by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Inc., 2021) pp. 28131-28145
[12] J. Lu, A. Liu, F. Dong, F. Gu. J. Gama, and G. Zhang, Learning under Concept Drift: A Review, IEEE Transactions on Knowledge and Data Engineering 31, 2346 (2019).
[13] G. Moreno-Toree T. Raeder, R. Rodriguz,, N. V. Chawla, and F. Herrera, unifying view on dataset shift in classification, Pattern Recognition 45, 521 (2012).
[14] A. Zhang. Z. C. Lipton, M. Li, and A. J. Smola, Dive into Deep Learning (Cambridge University Press, 2023) https://D2L. ai.
[15] J. Gama, I. Zliobaité, A. Bifet, M. Pechenizkiy, and A. Bouchachia, A survey on concept drift adaptation, ACM computing surveys (CSUR) 46, (2014).
[16] K. Nishida and K. Yamauchi, Detecting concept drift using statistical testing, in International conference on discovery science (Springer, 2007) pp. 264-269.
[17] E. Dobriban and S. Wager, High-dimensional asymptotics of prediction: Ridge regression and classification, The Annals of Statistics 46, 247 (2018).
[18] T. Hastie, A. Montanari, S. Rosset, and R. J. Tibshirani, Surprises in high-dimensional ridgeless least squares interpolation, The Annals of Statistics 50, 949 (2022).
[19] P. I.. Bartlett, P. M. Long, G. Lugosi, and A. Tsigler, Benign overfitting in linear regression, Proceedings of the National Academy of Sciences 117, 30063 (2020).
[20] M. Emami, M. Sahrace-Ardee Ardakan, P. Pandit, S. Rangan, and A. Fletcher, Generalization Error of Generalized Linear Models in High Dimensions, in Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 1 119, edited by H. D. III and A. Singh (PMLR, 2020) pp. 2892-2901.
[21] D. Wu and Xu, On the Optimal Weighted f Regularization in Overparameterized Lin- ear Regression, in Advances in Neural Information Processing Systems, Vol. 33, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Curran Associates, Inc., 2020) pp. 10112-10123.
[22] G. Mel and S. Ganguli, A theory of high dimensional regression with arbitrary correlations between input features and target functions: sample complexity, multiple descent curves and a hierarchy of phase transitions, in Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 139, edited by M. Meila and
T. Zhang (PMLR, 2021) pp. 7578-7587.
[23] D. Richards, J. Mourtada, and L.. Rosasco, Asymptotics of Ridge(less) Regression under General Source Condition, in Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 130, edited by A. Banerjee and K. Fukumizu (PMLR, 2021) pp. 3889-3897.
[24] V. Ngampruetikorn and D. J. Schwab, Information bottleneck theory of high- nigh-dimensional regression: relevancy, efficiency and optimality, in Advances in Neural Information Processing Systems, Vol. 35, edited by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Curran Associates, Inc., 2022) pp. 9784-9796.
[25] W. Rocks and P. Mehta, Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models, Phys. Rev. Res. 4, 013201 (2022).
11

---

[26] M. Belkin, D. Hsu, S. Ma, and S. Mandal, Reconciling modern machine- learning practice and the classical bias-variance trade-off, Proceedings of the National Academy of Sciences 116, 1 15849 (2019).
[27] P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang. B. Barak, and I. Sutskever, Deep Double De- scent: Where Bigger Models and More Data Hurt, in International Conference on Learning Representations (2020).
[28] P. Nakkiran, P. Venkat, S. M. Kakade, and T. Ma, Optimal Regularization can Mitigate Double Descent, in International Conference on Learning Representations (2021).
[29] J. Yang, K. Zhou, Y. Li, and Z. Liu, Generalized Out-of-Distribution Detection: A Survey, International Journal of Computer Vision 10.7////s 1223-224--2222211222222 (2024).
[30] Patil, J.-H. Du, and R. Tibshirani, Optimal Ridge Regularization for Prediction, in Proceedings of the s1 International Conference on Machine Learning, Proceed- ings of Machine Learning Research, Vol. 235, edited by R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (PMLR, 2024) pp. 39908-39958
[31] S. Garg, D. Tsipras, P. Liang, and G. Valiant, What Can Transformers Learn In-Context? A Case Study of Simple Function Classes, in Advances in Neural Information Processing Systems, Vol. 35, edited by S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Curran Associates, Inc., 2022) pp. 30583-30598.
[32] E. Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou, What learning algorithm is in- context learning? Investigations with linear models, in The Eleventh International Conference on Learning Representations (2023).
[33] A. Raventós, M. Paul, F. Chen, and S. Ganguli, Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression, in Advances in Neural Information Processing Systems, Vol. 36, edited by A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Curran Associates, Inc., 2023) pp. 14228-14246.
[34] R. Agarwal, A. Singh, L.. M. Zhang, B. Bohnet, L.. Rosias, C. Chan, B. Zhang, A. Anand, 7. Abbas, A. Nova, J. D. Co-Reyes, E. Chu, F. Behbahani, A. Faust, and H. Larochelle, Many- Shot In-Context Learning, in The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024).
[35] J. Von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov, Transformers Learn In-Context by Gradient Descent, in Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, edited by A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (PMLR, 2023) pp. 35151-35174.
[36] K. Ahuja and D. Lopez-Paz, A Closer Look In-Context Learning under Distribution Shifts, in Workshop on Efficient Systems for Foundation Models @ ICML2023 (2023).
[37] C. Goddard, L. M. Smith, V. Ngampruetikomm, and D. J. Schwab, When can in-context learning generalize out of task distribution?, in Forty-second International Conference on Machine Learning (2025).
[38] I. M. Smith, C. Goddard, V. Ngampruetikom, and D. J. Schwab, Model Recycling: Model component reuse to promote in-context learning, in NeurlPS 2024 Workshop on Scientific Methods for Understanding Deep Learning (2024).
[39] A. Karpathy, NanoGPT, / gi gig.......... cww comm/KaarpatthyyynnooPPT (2022).
[40] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86, 2278 (1998).
[41] H. Xiao, K. Rasul, and R. Vollgraf, MNNS::: a Novel Image Dataset for Benchmarking Machine Learning Algorithms (2017), arXiv:1708.077 1708.07747 [cs.LG].
[42] J. Wang. C. Lan. C. Liu, Y. Ouyang, Qin, Lu, Y. Chen, W. Zeng. and P. S. Yu, Generalizing to Unseen Domains: A Survey on Domain Generalization, IEEE Transactions on Knowledge and Data Engineering 35, 8052 (2023).
[43] A. Nguyen, J. Schwab, and Ngampruetikorn, Data coarse graining can improve model performance (2025), arXiv:2509, 14498 (cond-mat.stat-meede
12

---

[44] A. Canatar, B. Bordelon, and C. Pehlevan, sion, in Advances in Neural Information Processing Systems, Vol. 34, edited by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Inc., 2021) pp- 12600-12612.
Generalization in Kernel Regres-
[45] Y. Lu, M. Letey, A. Zavatone-VVele A. Maiti, and C. Pehlevan, In-Context Learning by Linear Attention: Exact Asymptotics and Experiments, in NeurlPS 2024 Workshop on Mathematics of
Modern Machine Learning (2024).
[46] Z. Han, G. Zhou, R. He, J. Wang, T. Wu, Y. Yin, S. Khan, L. Yao, T. Liu, and K. Zhang, How Well Does GPT--V((sioo)) Adapt to Distribution Shifts? A Preliminary Investigation, in ICLR 2024 Workshop Mathematical and Empirical Understanding of Foundation Models (2024).
[47] Q. Wang, Y. Wang. X. Ying, and Y. Wang, Can In-context Learning Really Generalize to Out- of-distribution Tasks?, The Thirteenth International Conference on Learning Representations (2025).
[48] J. Song, Z. Xu, and Y. Zhong, Out-of-distributiin generalization via composition: A lens through induction heads in Transformers, Proceedings of the National Academy of Sciences 122, e2417182122 (2025).
[49] Rubio and X. Mestre, Spectral convergence for a general class of random matrices, Statistics Probability Letters 81, 592 (2011).
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have carefully worded our abstract and introduction to reflect the paper's contributions. We have also added section detailing our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? [Yes]
Answer:
Justification:
We discussed the limitations of our analytic results, which are for linear ridge regression.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Full derivations provided in Appendix. Guidelines:
The answer NA means that the paper does not include theoretical results.
14

---

All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All details to reproduce numerical experiments are presented in their relevant sections.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully specific model and empirical evaluation, may
might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: While we do not have open-source code for now, we will open source our code once the anonymity period for submission is over.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
15

---

Please see the NeurIPS code and data submission guidelines (https://nips.co Sewuidee/ooeSuumiisseeseeeseseee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //iis..co.. httpccpppppccccccccccccccccccccc httgg/ddde//oeuubmmiiiec /ot////ooeee/eeeeeeeeeeeeee wwwpbbmmssiooooioioooooooooooooo onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details to reproduce numerical experiments are presented in their relevant sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our error bars are Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2- 2-sigma emror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
16

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: All details to reproduce numerical experiments presented in their relevant sections.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt///nurrpppp wwt.erpp...ppuupiiiiieeeeepeeeee Etthi csGuidel ines?
Answer: [Yes]
Justification: Our research conforms to the NeurIPS Code of Ethics. Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
17

---

11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]
Question:
Justification: NA
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/condemmodel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
18

---

Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration not required. Answer: [NA]
Justification: NA
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https /ttprreee..////// for what should or should be described.
/LLM)
19

---

A Prediction risk in the thermodynamic limit
To derive prediction risk in the thermodynamic limit, we rewrite the nonasymptotie bias and variance contributions [Eq (5)] in terms of the resolvent operator (4 Alp)
B(X) (B EB - B) 22 BIB - B) w4Zlp
+ 22 Tr (BB 1 E wellp 2
(12)
V(X) = o t Tr E )मेP N Tr (T-dlpy
(13)
In the thermodynamic limit-N. P + oo and P/N - y E (0, col-ohceeeeeeeeeee the above traces become deterministic (see Appendix B), and the bias and variance converge to (see also Patil et al. [30])
- - (8 E(B 218 G2(- B) ( e (
B Gz(-N)
Gx(-1)B
(14)
T-EEG((-))
v(X) - V= - r(-1) TH(E6g(-11 (-1)
Tr(E, (-1)]
(15)
T+|EGz(-11)
Tr (-1))
where Gz(Z)= mlæ)E-zドr mz) Ityv(z) and x(=)= Tr Tr(EGz(2)) with v(z) e C+ We note that the last term of the bias vanishes for == and covariate shifts affect the variance term but concept shift does not.
B Spectral convergence for random matrix traces
Let Y, @ and A denote P x P matrices, Ip the identity matrix in P dimensions and z a complex scalar outside the positive real line. Assume the following: (i) A IR PxP is symmetric and nonnegative definite, (ii) A PxP has bounded trace norm Tri 6) 1/2 [0, oo) and (iii) y = where the entries Z PPN are iid random variables with zero mean, unit variance and finite 8 + moment for some > O, and Σ e E PxP is covariance matrix. In the limit P, N - and P/N-ye (0, co), we have [49]
1/2
Tr θ
Tr 0
(16)
A zlp
Toyela:a
E + z/p
where C* is the unique solution of
c(2; A) = P
Tr
(17)
E+ A zlp
T+yo(Z:A)
First we consider the trace of terms linear in the resolvent z/p) -1, Setting A= in Eq (16) gives
appearing in Eqs (12) ((3)).
1
Tr
Tr 0
(18)
z/r
TRyv(E)
z1p
where ((z)=<(2;)) is the solution of
v(2)
P Tr
Σ
with v(z) C
(19)
T+yr
Σ
zlp
In general, this self-consistent equation does not have closed-form solution. One exception the isotropic case E=/p for which
VI=IP (z)
1- y- - = (1 2)2 4yz
(20)
2yz
20

---

Next we obtain the asymptotic expression for the trace of terms quadratic in the resolvent, such as those in Eqs (12) (13). We A uB with μ Differentiating Eq (16) with respect μ and taking the limit μ - 0* yields
B Tr T zlp v =lp
7 Tr θ
(d(z: B)Σ + B) E z/p/
(21)
Σ z/p
Ityr(2)
1+yv(z)
Here we define
d
d(z; B)
(22)
du + yc(zz HB) 1-00"
yp Tr B (1-(1+y((2)))1p)
(23)
1 1 Tri (-((yy((E))
p Tr B 21p)
(z)
(24)
(I+rr(z))
Tr
p Tryaa Σ-2)²
where the last equality follows from
1
P
Tayotrs E-2)²
(z)
(25)
1 y Tr[( E-((+y((2)2d)
For B=/p, the spectral function d(2; B) reduces to
(2)
d(2; Ip) (1+ (1+))
(26)
Our result for prediction risk in the thermodynamic limit based on the asymptotic traces in Eqs (18)& & (21).
21


Reviews:
[{"review_id": "V2UnxtcxhW", "reviewer": "Reviewer_qGpA", "summary": "The paper analyzes the mean-field behavior of generalization under **concept shift** in ridge regression, where the input–label mapping changes at test time. The authors derive a closed-form expression for the test error and study its properties on simple data models under *optimal* in-distribution regularization. They uncover a phase transition that separates a **generalization phase** (more data helps) from a **specialization phase** (more data hurts). These insights are then applied to in-context learning (ICL) and classification tasks, and validated in realistic experiments.", "questions": "See above", "limitations": "The study is limited to simple Gaussian data.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. The paper is well written and easy to follow; the problem setting and theoretical results are presented clearly.\n2. Asymptotics of regression under concept shift are not well studied; the authors make a solid contribution by providing an intuitive mathematical exposition.\n3. They identify a novel “double-descent” behavior that interestingly appears only at test time.\n4. The theory is applied to ICL and shown to hold in non-trivial experimental settings.\n\n### Weaknesses\n\nThe theory focuses on *optimal* ridge regression to isolate generalization behavior under concept shift. With sub-optimal ridge, in-distribution generalization always degrades, but it is unclear whether the same holds for the OOD case. Optimal ridge may overspecialize the predictor to the training task and thus potentially contribute to poorer OOD performance. Couldn't there be an alternative ridge parameter which marginally worsens ID risk yet substantially improve OOD risk?\n\nAlso, the previous literature mainly considered optimizing ridge parameter with respect to OOD generalization (ref.\\[30,43]). How would that change the story? If optimally tuned ridge regression for ID yields an increasing OOD test risk, would tuning to optimal OOD ridge parameter have a similar effect on ID generalization?\n\nBelow are a few comments/suggestions\n\n1. **Line 85.** The risk need not approach to zero, but rather to an irreducible error (see ref. \\[43]) for generic feature maps. Likewise, the statement in line 88 is confusing; it would make more sense to me if $\\tilde\\beta_1 = \\beta_1$ and $\\tilde\\beta_{i>1} = 0$. It is just a technicality, but it confused me the first time I read it.\n2. **Line 117.** *nonrobut* → *non-robust*.\n3. Please clarify how your results differ from those of **Yue et al. (2023)** ([https://openreview.net/forum?id=Jw3ck7FWZh](https://openreview.net/forum?id=Jw3ck7FWZh)).\n4. What is the nature of the weak-to-strong concept-shift phase transition? Does it constitute a genuine phase transition in the strict statistical-mechanics sense—for example, does the risk exhibit a non-analyticity in κ cos θ—or is it merely a sharp crossover? A brief discussion would help readers interpret the result.", "was_revised": true, "final_justification": "This paper theoretically studies the generalization of linear models under general concept shifts, which are relevant for modern transformers. Their results are clearly stated and supported by extensive numerical experiments.", "rebuttal": "Thank you for the positive and thorough review! We are grateful for your insightful questions and detailed feedback, which helped us improve our paper. Please see below our point-by-point response with quotes rearranged for clarity.\n\n&nbsp;\n\n> The theory focuses on optimal ridge regression to isolate generalization behavior under concept shift.\n\nWe deliberately chose this setup to isolate the effects of concept shift from the confounding effects of double descent, another phenomenon where prediction risk can increase with more data (Belkin et al, PNAS 2019; Nakkiran et al, ICLR 2020). Focusing on optimal regularization, where double descent is absent, allows us to unambiguously demonstrate that concept shift is a distinct mechanism that can cause more data to hurt generalization performance.\n\n&nbsp;\n\n> Optimal ridge may overspecialize the predictor to the training task […] Couldn't there be an alternative ridge parameter which marginally worsens ID risk yet substantially improve OOD risk?\n\n> …optimizing ridge parameter with respect to OOD generalization […] How would that change the story? If optimally tuned ridge regression for ID yields an increasing OOD test risk, would tuning to optimal OOD ridge parameter have a similar effect on ID generalization?\n\nThank you for this excellent question. Your intuition is correct. Indeed, tuning the regularization parameter λ away from the in-distribution (ID) optimal value, λ* = γ/snr, can improve out-of-distribution (OOD) performance. Concept shift with *κ*cos*θ* > 1 benefits from λ < λ* and *κ*cos*θ* < 1 from λ > λ*. Intuitively, the stronger concept shift is (ie, when *κ*cos*θ* is small or negative), the less you should trust the input features; improving OOD generalization thus requires a greater λ, which decreases the relative influence of training data on the learned predictor. This behavior reflects the fundamental competition between the inductive bias induced by ridge regularization and the signal from the data. \n\nThis competition also underpins our key finding that a sharp transition exists between weak and strong concept shift regimes (Fig 3). Marginal deviations from λ* do not change the qualitative nature of this transition.\n\nWhile previous works considered OOD-optimal regularization, our focus on ID-optimal regularization is motivated by its practical relevance. Using optimal hyperparameters is a common practice, and optimizing regularization for unseen ID data is possible via methods such as cross-validation. Our transformer experiments, which are trained only on ID data, exemplify how it is possible to achieve virtually optimal ID generalization performance in practice.\n\nWe have revised our manuscript to clarify this important point.\n\n&nbsp;\n\n>  how your results differ from those of Yue et al.\n\nThank you for bringing this interesting work to our attention. We see our work as complementary to theirs, as both demonstrate the power of solvable models in generating new insights. However, we address fundamentally different questions.\n\nFirst, we focus on the phenomenon of concept shift within the framework of ridge regression, whereas Yue et al study how in-context learning with linear attention mechanisms can solve in-distribution linear regression tasks and don’t focus on concept shift. Second, our work uses ridge regression as a theoretically tractable model to derive exact, analytical results on concept shift. Our transformer experiments serve primarily as experimental validation, demonstrating that these insights generalize to practical architectures and optimization settings. In contrast, the transformer model in Yue et al is their primary object of study.\n\nWe added a reference to and a discussion of this work to our Related Work section to better highlight these distinctions.\n\n&nbsp;\n\n> What is the nature of the weak-to-strong concept-shift phase transition? Does it constitute a genuine phase transition in the strict statistical-mechanics sense…\n\nThank you for pointing out the opportunity to improve the clarity of our work.\n\nWhile the mappings between the transitions in learning problems and those separating phases of matter are instructive, they can be inexact, as many observables in the former do not have direct analogues in the latter and vice versa. In our work, the boundary between weak and strong concept shift is defined by an abrupt change in the utility of additional training data, from improving to degrading OOD generalization. This abrupt change manifests as a nonanalytic kink in the minimum OOD risk at the boundary (Fig 3), and this kink is sharp only in the strict thermodynamic limit. We have included this discussion in our updated paper.\n\n&nbsp;\n\n> Line 85. The risk need not approach to zero, but rather to an irreducible error\n\nYou are correct that, in general, prediction risk includes an irreducible term due to intrinsic label noise. In our work, we follow the convention of Hastie et al (Ann Stat 2022) where the risk is the expected squared error between the predictor and the mean of the test-time conditional distribution (Eq 4 in our paper). This definition effectively subtracts the irreducible noise variance, allowing for a cleaner analysis of the error attributable to the learning algorithm. We have clarified this definition in our revised paper.\n\n&nbsp;\n\n> Line 117. nonrobut → non-robust.\n\nThank you for catching this typo. We have corrected it."}, {"review_id": "DxRjEVDwGV", "reviewer": "Reviewer_fv8w", "summary": "This paper presents a theoretical study of how concept shift—a change in the input-label relationship—affects generalization in ridge regression. The authors provide exact expressions for prediction risk in the high-dimensional limit and identify a sharp phase transition: when concept shift is strong, adding more training data can actually worsen test performance. Experiments using in-context learning with transformers and classification on MNIST/FashionMNIST support the theory.", "questions": "See the Strengths And Weaknesses section.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper identifies a clear and interesting phenomenon: more training data can hurt generalization under strong concept shift. \n2. The theoretical analysis is rigorous and based on a solvable model, offering precise characterizations. \n3. Experimental results on transformers and classification tasks are generally consistent with the theoretical predictions.\n\nWeaknesses:\n1. The theoretical results rely heavily on idealized settings: linear models, Gaussian features, and optimally tuned ridge regression. These are useful for analysis but may not reflect practical machine learning scenarios. Would similar effects appear with nonlinear models, neural networks, or non-Gaussian data? \n2. The shift is modeled only as scaling or rotating the regression coefficients. In real-world problems, concept shift is often more complex, involving sparse, structured, or gradual changes. Discuss how realistic this modeling is, and whether the findings generalize to more practical types of shift. \n3. While the paper highlights its novelty, it does not sufficiently compare with prior theoretical work on concept drift, streaming learning, or temporal domain generalization. \n4. The transformer experiments focus only on simple synthetic tasks (linear regression) and PCA-modified MNIST. These setups are still far from real-world settings.", "was_revised": true, "final_justification": "The authors have give a detaield response to clarify the coontributions of the paper and most of my concerns have been addressed.", "rebuttal": "Thank you for your thoughtful comments. We appreciate the opportunity to clarify the motivation, scope and contributions of our work.\n\n&nbsp;\n\nFirst, we emphasize that our main contribution is in deriving an exact, analyzable framework for characterizing concept shift and its effects on generalization properties. We view our work as establishing an important theoretical foundation upon which future research can study other models of concept shift in different settings. The analytical precision achieved through our approach enables precise characterization of phenomena, such as the transition between weak and strong concept shift regimes, that would be difficult to quantify in more complex settings.\n\nWe will revise our manuscript to better highlight the broader implications of our findings. Specifically, we will include our arguments below in the Introduction and Related Work sections to more clearly articulate our main contributions and how our theoretical framework can inform understanding of concept shift in more complex settings.\n\nPlease see below our point-by-point response to specific comments with quotes rearranged for clarity.\n\n&nbsp;\n\n> The theoretical results rely heavily on idealized settings: linear models…\n\nWe agree that our theoretical settings are simple. However, this simplicity is precisely what enables rigorous theoretical analyses and controlled experiments that are required for developing an understanding of complex learning phenomena.\n\nIn particular, _linear models_ have proved crucial in advancing our understanding of learning dynamics of deep networks (see, eg, Saxe et al, ICLR 2014; Lampinen & Ganguli, ICLR 2019; Ji & Telgarsky, ICLR 2019; Arora et al, ICLR 2019, Lin et al, NeurIPS 2024; Marion & Chizat, NeurIPS 2024; Braun et al, ICML 2025; Nam et al, ICML 2025). _Linear regression_ has provided a crucial, theoretically tractable setting for developing insights into emergent abilities of LLMs, such as in-context learning (see, eg, Garg et al, NeurIPS 2022; Raventos et al, NeurIPS 2023; Goddard et al, ICML 2025; Lu et al, PNAS 2025). _Ridge regression in high dimensions_ has repeatedly yielded unexpected insights into machine learning behaviors such as double descent and benign overfitting (see, eg, Bartlett et al, PNAS 2020; Wu & Xu, NeurIPS 2020; Hastie et al, Ann Stat 2022; Richards et al, AISTATS 2021; Mel & Ganguli, ICML 2021; Patil et al, ICML 2024). Our work follows in this fruitful tradition.\n\nOur work provides a solvable framework that reveals nontrivial behaviors under concept shift—such as nonmonotonic data dependence of prediction risk and phase transitions between weak and strong concept shift regimes. These effects would be difficult, if not impossible, to isolate in more complex models.\n\n&nbsp;\n\n> ...simple synthetic tasks (linear regression) and PCA-modified MNIST [...] are still far from real-world settings.\n\nWe do not design our experiments to solve a complex real-world task, but to validate the insights from our theory in settings beyond which we can access analytically. We show that the key phenomenon—the weak-to-strong concept shift transition—persists in more complex settings, including transformer in-context learning and classification tasks on image datasets.\n\nThe strong agreement between our theory and these experiments hints at universal aspects of learning under concept shift that our theoretical model effectively captures. This highlights the value of our theory as a principled minimal model for investigating concept shift and its effects on generalization.\n\n&nbsp;\n\n> The shift is modeled only as scaling or rotating the regression coefficients. In real-world problems, concept shift is often more complex…\n\nWhile we acknowledge that real-world concept shifts occur in more complex contexts, establishing a theoretical foundation in tractable models and verifying it in basic settings is a critical step toward understanding these phenomena. Crucially, for the linear data-generating process we study, these two primitives—changes in magnitude (scaling) and direction (rotation)—fully characterize any possible discrete concept shift. This approach allows us to isolate and precisely characterize key phenomena in a way that would be otherwise difficult.\n\n&nbsp;\n\n> ...does not sufficiently compare with prior theoretical work on concept drift, streaming learning, or temporal domain generalization.\n\nThank you for this suggestion. Our work contributes to a distinct paradigm from much of the literature on concept drift and streaming learning. \n\nWhile many prior works focus on developing algorithms for _detecting_ and _adapting_ to drift, or on providing worst-case generalization bounds, we use tools from statistical mechanics and random matrix theory to derive an exact, analytical _characterization of generalization behavior_ under discrete concept shift. This approach allows us to uncover new phenomena, such as the sharp transition at the heart of our work. We will revise our Related Work section to better articulate this important distinction and provide representative references for other paradigms."}, {"review_id": "5EGdE374wV", "reviewer": "Reviewer_y7QV", "summary": "The paper develops an exact theory of concept shift scenario in which the relationship between inputs and labels changes at test time, and studies how this shift alters the generalization behaviour of learning algorithms. They verify the analytic predictions with experiments, finding close quantitative and qualitative agreement. The paper provides a detailed theoretical and experimental characterization of how concept shift affects generalization.", "questions": "1.\tThe paper focuses on ridge regression. How do the findings extend to other machine learning models?\n2.\tThe experiments are limited to MNIST and FashionMNIST. Could more diverse datasets be used to validate the findings? How do the results vary across different types of data distributions and problem domains?\n3.\tWhat advantages does this work offer over previous solutions?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Srengths:\n1.\tThe paper delves deep into the issue of machine learning models under concept shift, centering on ridge regression to derive exact expressions. The methodology is rigorous.\n2.\tThe paper is well-structured and logically coherent. It starts with the problem of distribution shift, narrows down to concept shift, and elaborates on the research background and significance. It then proceeds to theoretically analyze ridge regression under concept shift. The experimental section follows naturally.\n3.\tUnderstanding the generalization of machine learning models under distribution shift is crucial for developing safe and robust techniques. This paper focuses on the relatively less-studied concept shift, offering valuable insights.\n## Weaknesses\n1.\tWhile the paper provides a detailed analysis of concept shift under ridge regression, the scope of its research is relatively limited.\n2.\tSome mathematical derivations and notations in the paper are quite complex, which may pose challenges for readers without a strong mathematical background.\n3.\tThe practical applicability of the findings remains to be further demonstrated.", "was_revised": true, "final_justification": "Thank you for your detailed response to my review. I appreciate the time and effort you have put into addressing the comments. I have read your rebuttal carefully and have no further questions. After consideration, I will be maintaining my original score. I wish you the best with your submission.", "rebuttal": "Thank you for highlighting the significance and rigor of our work. We appreciate the opportunity to clarify our contributions and the scope of our findings. Please see below our point-by-point response to specific comments with quotes rearranged for clarity.\n\n&nbsp;\n\n> …the scope of its research is relatively limited.\n\nThank you for pointing out the opportunity to articulate our contributions.\n\nWe emphasize that our main contribution is in deriving an exact, analyzable framework for characterizing concept shift and its effects on generalization properties. We view our work as establishing an important theoretical foundation upon which future research can study other models of concept shift in different settings. The mathematical clarity achieved through our approach enables precise characterization of phenomena that would be difficult to quantify in more complex settings. \n\nTo the best of our knowledge, we are the first to:\n*  develop a theoretical framework that isolates the effects of concept shift—an important and relatively less studied mode of distribution shift;\n*  explain precisely why and how more training data can hurt generalization performance under concept shift, illustrating its effects in several specific settings, including coefficient shrinking, rotation, and feature robustness;\n*  identify and characterize a sharp transition that separates weak and strong concept shift regimes;\n*  show that feature anisotropy creates qualitatively different patterns of risk nonmonotonicity depending on whether concept shift affects high or low-variance features.\n\nWe will revise our Introduction to emphasize these points.\n\n&nbsp;\n\n> The practical applicability of the findings remains to be further demonstrated\n\n> What advantages does this work offer over previous solutions?\n\nThank you for raising this important point. Other work focuses primarily on detecting the presence of a concept shift. In contrast, our work does not propose a new solution or algorithm to detect or mitigate concept shift. Instead, we provide a fundamental characterization of model failure under concept shift, revealing counterintuitive phenomena and laying the foundation for the future development of robust solutions.\n\n&nbsp;\n\n> The paper focuses on ridge regression. How do the findings extend to other machine learning models?\n\nWe appreciate the opportunity to highlight our experiments. While our theory is derived for ridge regression, we provide experimental evidence that the key insights extend to a qualitatively different learning problem. In classification settings, our MNIST and FashionMNIST experiments demonstrate that under a strong concept shift, test accuracy can decrease with more training data. This result strongly suggests that the phenomenon we identify is not limited to regression models, thus providing convincing validation for the broader relevance of our theory.\n\n&nbsp;\n\n> The experiments are limited to MNIST and FashionMNIST. Could more diverse datasets be used to validate the findings? How do the results vary across different types of data distributions and problem domains?\n\nThis is an excellent point. We address this with our transformer experiments, which test our theory in a modern setting. The agreement between our theory and this complex model is not trivial. Our theoretical analyses are based on optimally tuned ridge regression in the thermodynamic limit. In contrast, transformers perform in-context regression on finite data (ie, their context) in finite dimensions via a learned, data-driven algorithm (as opposed to hardcoded ridge regression). Our transformer experiments thus serve as validation for our theoretical results by demonstrating that the phenomena we identify extend beyond idealized settings. In particular, we show that similar behavior—such as a transition between weak and strong concept shift regimes—emerges in a data-driven learning algorithm without (a) explicitly implementing ridge regression, (b) working at infinite dimensions, or (c) manually tuning regularization.\n\nThese results validate our analysis method of studying tractable concept shift models where explicit expressions can be derived and demonstrate that studying these models can reveal qualitative generalization behaviors that are also observed in large-scale data-driven learning. Together, our experimental results on transformer models and classification tasks hint at universal aspects of learning under concept shift that our theoretical model effectively captures. This highlights the value of our theory as a minimal model for investigating concept shift and its effects on generalization.\n\nWe will revise our experiment section to better motivate these experiments and discuss how they provide a nontrivial validation of our theoretical results.\n\n&nbsp;\n\n> …mathematical derivations and notations in the paper are quite complex…\n\nWe appreciate this feedback on the accessibility of our paper. While the mathematical details are essential for the rigor of our analysis, we agree that the high-level insights should be as clear as possible. In our revised manuscript, we will edit the Abstract, Introduction, and Conclusion sections to better distill the key takeaways and their implications in more accessible language, without diluting the core technical results."}, {"review_id": "eAMsjqOIVP", "reviewer": "Reviewer_hzmg", "summary": "The paper studies regression/classification performance with ridge regression under concept shift and show that concept shift can change the qualitative behavior of generalization behavior. Authors report experimental results for regression tasks using pretrained transformers to solve linear regression and for classification tasks using well known MNIST and FashionMNIST datasets. Empirical investigations show that ridge regression fails to generalize even with infinite data under a strong enough concept shift. They also demonstrate that too much data can harm generalization when dealing with anisotropy in features.", "questions": "- How can researchers and machine learning practitioners leverage the learning from this work in their work? In particular, detecting or predicting concept shift in test data under real world settings can be hard or impossible which makes it challenging whether or not to consider that takeaways from this work at training stage.", "limitations": "- Please refer to the first two bullets under the **weaknesses** section of this review.\n- The high-dimensional limit assumption made in the paper does not necessarily match the practical settings.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The topic addressed by this paper is of great significance since concept shift remains an active challenge for many ML systems in practice.\n- The theoretical aspect of the paper is solid and it provides a rigorous understanding of model performance under concept shift. The studied setting, i.e. concept shift, is well defined.\n- Experimental results shed light into less explored aspect of generalization performance under the studied settings e.g. empirical studies of how anisotropic features can cause more data hurting generalization performance.\n\n**Weaknesses**\n- Authors study concept shift's impact on ridge regression, an inherently linear model, which may make the observed model behavior specific to this oversimplified settings.\n- The impact of concept shift on generalization performance is not well studied for classification tasks since the studied datasets for classification are rather simple and do not pose real world concept shift challenges.\n- The paper is not easy to read. e.g. instead of using intuitive and easy to understand X and Y axis titles, authors have decided to use mathematical statements. e.g. Figure captions do not highlight the main observations/takeaways.\n- Limitations of the work are not clearly and comprehensively listed by the authors.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and for highlighting the significance of our work. We appreciate the opportunity to clarify our methodology and improve our presentation.\n\n&nbsp;\n\n> ...ridge regression, an inherently linear model, which may make the observed model behavior specific to this oversimplified settings.\n\n> The high-dimensional limit assumption made in the paper does not necessarily match the practical settings.\n\nThank you for raising these important points about our methodology. We agree that our theoretical settings are simple. However, this simplicity is precisely what enables rigorous theoretical analyses and controlled experiments that are required for developing an understanding of complex learning phenomena. \n\nIn particular, _linear models_ have proved crucial in advancing our understanding of learning dynamics of deep networks (see, eg, Saxe et al, ICLR 2014; Lampinen & Ganguli, ICLR 2019; Ji & Telgarsky, ICLR 2019; Arora et al, ICLR 2019, Lin et al, NeurIPS 2024; Marion & Chizat, NeurIPS 2024; Braun et al, ICML 2025; Nam et al, ICML 2025). _Linear regression_ has provided a crucial, theoretically tractable setting for developing insights into emergent abilities of LLMs, such as in-context learning (see, eg, Garg et al, NeurIPS 2022; Raventos et al, NeurIPS 2023; Goddard et al, ICML 2025; Lu et al, PNAS 2025). _Ridge regression in high dimensions_ has repeatedly yielded unexpected insights into machine learning behaviors such as double descent and benign overfitting (see, eg, Bartlett et al, PNAS 2020; Wu & Xu, NeurIPS 2020; Hastie et al, Ann Stat 2022; Richards et al, AISTATS 2021; Mel & Ganguli, ICML 2021; Patil et al, ICML 2024). Furthermore, the study of _high-dimensional limits_ has led to powerful frameworks such as the Neural Tangent Kernel (NTK; Jacot et al, NeurIPS 2018), and the Neural Network and Gaussian Process correspondence (NNGP; Lee et al, ICLR 2018).\n\nOur work follows in this fruitful tradition, providing a solvable framework that reveals nontrivial behaviors under concept shift—such as nonmonotonic data dependence of prediction risk and sharp transitions between weak and strong concept shift regimes. These effects would be difficult, if not impossible, to isolate in more complex models.\n\nCrucially, our experiments are specifically designed to test whether these insights extend beyond this idealized setting. The strong agreement between our theory and our transformers and classification experiments suggests the phenomena we identify are not mere artifacts of our theoretical assumptions but hint at more universal principles.\n\n&nbsp;\n\n> The impact of concept shift on generalization performance is not well studied for classification tasks…\n\nThis is an excellent point and an important direction for future research. Our work provides the first theoretical characterization of concept shift in a regression setting, and we agree that a similar, rigorous analysis for classification is an important next step. Indeed, our experiments on MNIST and FashionMNIST, which demonstrate that more data can hurt test accuracy under strong concept shift, provide compelling preliminary evidence that similar phenomena exist in classification problems. We hope our work inspires future theoretical investigation into this area.\n\n&nbsp;\n\n> The paper is not easy to read…\n\nThank you for this valuable feedback on readability. We agree that we can make our figures more accessible without sacrificing technical precision. In our revised manuscript, we will:\n\n1.  Expand all figure captions to explicitly state the key takeaway of each plot in plain language.\n2.  Add descriptive text to the axis labels of our plots to supplement the mathematical notation.\n\nWe believe these changes will significantly improve the clarity and impact of our results for a broader audience.\n\n&nbsp;\n\n> Limitations of the work are not clearly and comprehensively listed…\n\nThank you for this important suggestion. We identify three primary limitations, which also represent exciting directions for future work.\n\n1.  Our theoretical results are derived for linear models in the idealized limit and may not _quantitatively_ hold for more complex nonlinear systems and in practically relevant limits.\n2.  We analyze discrete concept shift, leaving the study of gradual or continuous shifts for future work.\n3.  Our work characterizes a failure mode but does not propose a new solution or algorithm for mitigating its effects.\n\nWe will add a dedicated _Limitations and Future Work_ section to our revised manuscript to further clarify these points.\n\n&nbsp;\n\n> How can researchers and machine learning practitioners leverage the learning from this work in their work?\n\nThank you for pointing out the opportunity to discuss the implications of our research. \n\nThe main takeaway for practitioners and researchers is a new perspective on the modern understanding of generalization. The double descent phenomenon challenged the classical intuition that _more data is always better._ While subsequent work established that optimal regularization can mitigate double descent, our paper identifies a distinct setting where this principle is insufficient.\n\nWe show that even with optimal regularization, concept shift can create a new regime where additional data is detrimental to generalization performance. For researchers, our work provides a novel, tractable theoretical framework and identifies a rich set of phenomena (eg, sharp transitions and risk nonmonotonicity) worthy of further investigation."}]

Author rebuttal:

Meta-review: The paper studies generalization under label shift in a simplified model of ridge regression and in the limit of data dimension going to infinity.
The reviewers appreciate the theoretical study and while concerned about the over simplified setup, hope that this work could serve as a starting point for analysis of closer to reality setups.
Decision: Accept (poster)