Scaling laws for learning with real and surrogate data
Authors: Ayush Jain
Andrea Montanari
Eren Sasoglu
Venue: NeurIPS 2024 poster (2024)
Keywords: Machine learning
Synthetic data
Surrogate data
Scaling laws
Linear regression
Primary area: learning_theory
TL;DR: We study the integration of surrogate data from different sources alongside real data in training, via weighted risk minimization and provide theoretical and empirical evidence for a new scaling law that allows to optimize the weighting scheme.

Abstract: Collecting large quantities of high-quality data can be  prohibitively expensive or impractical, and a bottleneck in machine learning. One may instead augment a small set of $n$ data points from the target distribution with data from more accessible sources, e.g. data collected under different circumstances or synthesized by generative models. We refer to such data  as `surrogate data'. We study a weighted empirical risk minimization (ERM) approach for integrating surrogate data into training. We analyze mathematically this method under several classical statistical models, and validate our findings empirically on datasets from different domains. Our main findings are: $(i)$ Integrating surrogate data can significantly reduce the test error on the original distribution. Surprisingly, this can happen even when the surrogate data is unrelated to the original ones. We trace back this behavior to the classical Stein's paradox. $(ii)$ In order to reap the benefit of surrogate data, it is crucial to use optimally weighted ERM. $(iii)$ The test error of models trained on mixtures of real and surrogate data is approximately described by a scaling law. This scaling law can be used to predict the optimal weighting scheme, and to choose the amount of surrogate data to add.

Scaling laws for learning with real and surrogate data
Ayush Jain' ayush. jansh..jinnggamica..a
Andrea Montanari!? contrnaarm www..aotiiaaaaaanninaaa.ammm @granica.
Eren Sasoglu' eren. www.ogluggranicc..aa.a ai
Granica Computing Inc. granica.ai Sttanford University
Abstract
Collecting large quantities of high-quality data can be prohibitively expensive impractical, and bottleneck machine learning. One may instead augment a small set of n data points from the target distribution with data from more accessible sources, e.g. data collected under different cireumstances or synthesized by generative models. We refer to such data as "surrogate data'. We study a weighted empirical risk minimization (ERM) approach for integrating surrogate data into training. We analyze mathematically this method under several classical statistical models, and validate our findings empirically on datasets from different domains. Our main findings are: (i) Integrating surrogate data can significantly reduce the test error on the original distribution. Surprisingly, this can happen even when the surrogate data is unrelated the original ones. We trace back this behavior to the classical Stein's 's paradox. (ii) order to reap the benefit surrogate data, it is crucial to optimally weighted ERM. (iii) The test error of models trained on mixtures real and surrogate data is approximately described by a scaling law. This scaling law can be used to predict the optimal weighting scheme, and to choose the amount of surrogate data to add.
Introduction and overview
1.1 Motivation and formulation
Consider a standard learning setting where we are given n i.i.d. points 2; from a target distribution D. Given family of parametric models governed by the parameters' vector 0, the goal is find that minimizes the expected test loss Re. incurred by the model predictions, where expectation is taken over the distribution 'D. In many application domains, the available data Z (2;) from the target distribution, referred to as either real or original data, may be difficult or expensive to acquire. One may then attempt to supplement these data with a different, cheaper source. Examples of such cheaper sources are (i) publicly available datasets; (ti) datasets owned by the same research group or company but acquired in different circumstances, e.g. different location; (iii) synthetic data produced by a generative model.
We will denote the data points obtained from this source by 2;, and assume have of them. We assume the 'surrogate' data 2 = (=f])mm to be i.idd samples with distribution D*. In general, will not assume the distribution D of synthetic data to be close the original data distribution D. However we assume that these distributions are over the same domain. number of questions the surrogate data in training? (ii) How many surrogate samples should we add to the original data? (iiz) Can we predict the improvement in test error achieved by adding surrogate samples the training?
arise: (i) How should
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A natural approach would be to add the surrogate data to the original one in the usual training procedure, and indeed many authors have explored this approach (see Section 1.3). Namely, one attempts to minimize the overall empirical risk Ry-m (0) = >i= ((0; z:) + > {(0; 2;), where {(2, B) is train loss function.
However, moment of reflection reveals that this approach has serious shortcomings. Consider a simple mean estimation problem. whereby 2; N(O.. Iu), x ~ N(O". Ia). {(O: x) ||0 - =1? and Rea (0 = ||0 0. 1|2 A straightforward calculation yields that the test error of the empirical risk minimizer 0 guive
a arg min An+m (0) is
Rien (
n+ = =(x m 110; 0-1
Blive
d m
As m increases the variance (the second term) decreases, but the bias due to the difference ||0: 0.1 increases, and the error approaches ||0 0.11². i.e. the model will be only as good as if training only on surrogate data.
In order to overcome these limitations, we study weighted ERM approach, and will show that the weight plays a crucial role. Namely, we consider the following regularized empirical risk:
Rm (0:a) =
E(0: zi)
E(0; =;) + ((0),
1=1
where [o. 1] is the weight of the surrogate dataset and Q : Rd + Rzo is regularizer, e.g. ridge (20)) 0 We denote by
m (a) == arg ming Rm (0:
(3)
the corresponding empirical risk minimizer, and by Risa ( (a)) the corresponding test error.
For supervised learning tasks, a sample z is represented as z (g. 2). where x E IR is covariate vector and y E R is response variable and parametrizes a family of models f(: 0) that predict the response y given covariate vector æ. We consider losses of the form { z) L(v, (æx O)) and Russ (0) == R (ne (ax 0)) for some functions L. and Treat We allow for the test loss Trest
to be different from the train loss T., but we will omit the subscript "test' from the risk R and the loss whenever clear from the context.
eipaliiin
Clavv Clavilicataiiu lose
0.22
500500
Class www.itattioonn l00k
1000
Cleviitaaton Inns
2000
Figure 1: IMDB and Rotten Tomatoes data and neural networks. Test error when trained on mixtures of original and surrogate data. Black curves: prediction from Eq. (4).

---

0.250
unweighted
optimally weighted
0.245
Classification loss
0.240
0.235
0.230
0.225
0.220
0.215
0.210
2000
4000
6000
8000
Figure 2: Performance of unweighted vs weighted ERM approach for the setting in Figure 1
Figure provides preview of results, for sentiment analysis task. (Technical details provided in Section 4 and Appendix A.3). Each frame corresponds to a different combination of n and m, and we report the error of approach lines report the prediction of scaling law that will be one of the main results presented below.
function of the weight parameter (red circles). Solid
We observe that the weighted ERM approach systematically achieves better test error than either training only on original data (a - 0) or on surrogate data (a - 1). Further the error for optimal a always monotone decreasing both in and and the approach outperforms the naive unweighted approach. This is shown more clearly in Figure 2, which also shows that the performance of unweighted ERM degrade with more surrogate data. Also, while scaling laws typically do not capture the dependence on hyperparameters, the scaling law presented below predicts the dependence on reasonably well. This is particularly useful, because such scaling law can be used to tune optimally and to predict the amount of surrogate data needed.
1.2 Summary results
We study the method outlined above both mathematically and via numerical experiments. Our mathematical results are developed in four different settings: (i) The Gaussian sequence model (Section 3.1): (ii) A non-parametric function estimation setting (Section 3.2): (iii) Low-dimensional empirical-risk minimization (Section 3.3); (iv) High dimensional ridge regression (Section 3.4);
We carry experiments with the following data sources. (1) Simulated data from linear Gaussian mixture models: this allows us to explicitly control the distribution shift between the original and surrogate datasets, well as check our theoretical results in controlled setting. (2) Real natural language processing (NLP) data for sentiment analysis, with the role of original dataset played by IMDB reviews and the role of surrogate datasets played respectively by Rotten Tomatoes review and Goodreads book reviews.
(3) Progression- dataset with female patients data and male patients data original and surrogate data, respectively. (4) Real image classification data, with CIFAR-10 and CIFAR- 100 datasets respectively playing the role of original and surrogate data. Our results support the following conclusions:
survival analysis using Lasso on TCGA PanCancer
Surrogate data improve test error. Including surrogate data in training generally improves the test error the original data, even if the surrogate data distribution is far from the original one. In agreement with the interpretation surrogate data as a regularizer (see also Sec. 2), the improvement generally positive, although its size depend the data distributions.
Tuning of a. The above conclusion holds under the condition that a can be tuned (nearly) optimally. For each of the theoretical settings already mentioned, characterize this optimal value. We verify

---

that nearly optimal can be effectively selected by minimizing the error on validation split of the original data, An attractive alternative is to use the scaling law we discuss next.
Scaling law. We propose scaling law that captures the behavior of the test error with n. m, ot:
R( (a)) - R. = a² R (20) + [a² [Rg m)) - Rggl (00)) 1/3 + - Ro (C) (n)=/1j²
(4)
Here R. is the minimal (Bayes) error, (m) == R(Oo.m (1)) R., is the excess test enror when training on the surrogate data (and testing on original), Rer (n) == R(O,, ni(0n,,000 n0)( R. is the excess test error' when training on original data (and testing on original), and is a scaling exponent as described in Section 4. The above scaling admits natural generalizations; see Section 5.
Practical uses of the scaling law. Given data {zi and a source of surrogate data, would like to predict how much the test error can be decreased by including any number m of surrogate samples in the mix. The scaling law (4) suggests simple approach: (1) Learn models on purely original data to extract the behavior test loss (()),.(0)).: (2) Learn models on purely surrogate data to extract the behavior of R R( (1)). (A relatively small sample is sufficient for this step.) (3) Use the minimum over a of Eq. (4) predict the test error at any given pair n, m.
We can further leverage the scaling law to achieve the desired error by: (T) Using the scaling law to determine the number of surrogate samples needed to achieve the desired performance. (IT) Acquiring the surrogate samples and train the model using weighted ERM with optimal weighting predicted by scaling law.
1.3 Related work
The use of surrogate data enhance training has attracted increasing research effort, also because of the recent progresses in generative modeling.
This line of work has largely focused on the techniques to generate synthetic data that are well suited for training. A wide variety of methods have been demonstrated to be useful in generating data for computer vision tasks, ranging from object classification to semantic segmentation [RSM+ 16, JRBM 17, AAMM+ 18, TPA 18, CLCG19, HSY+22, MPT+22, 22, YCFB + 222]. refer to [SLW20] for review. More recently, synthetic data have been used for training in natural language processing [HNK 22, MHZH22].
Scaling laws have been broadly successful in guiding the development of large machine learning mod- els [HNA+
+ 17, RRBS19, HKK 20, KMH 20, TDR+21, HKHM21, HBM 22, ANZ22, MRB 233. We expect them to be similarly useful for integrating heterogeneous data into training. The change in scaling laws when training on synthetic data was the subject of recent empirical study [FCK 233.. On the other hand, no systematic attempt was made at integrating real and synthetic data.
In data augmentation [KSHI2, SKI9J, the original samples are supplemented with transformed or noisy version of the same. In contrast, we assume that surrogate data is obtained from different source than the original one, and the surrogate samples are independent of the original samples.
The problem we consider was also studied within 'domain adaptation', a subarea of transfer learn- ing (MPRP16, TJJ20J. Among others, [BDBC* 10] establishes bounds on the generalization error of weighted ERM via uniform convergence. However these bounds do not reveal the full advantage achieved by this approach and are not precise enough to justify the scaling laws that derive. Recent works in domain adaptation study the hehavior of test error error [Has21, KJSJ24, YLS++24) and its scaling laws [Has21], but only consider vanilla ERM, a special of weighted ERM considered here.
2 Regularization, Gaussian mean estimation, Stein paradox
The role of the parameter a can understood by considering the limit mn - oo:
(0, El= ((0;2:) + R²) + ((())
We assume here that lim., (0))..(0)) = R.. i.e. that we achieve Bayes risk with infinitely many original samples. See Section S.

---

and (²) BITDD= >²) is the population risk for surrogate data. This suggests to think of the surrogate data as an additional (highly non-trivial) regularizer, with parameter a. This leads to a simple yet important insight: adding surrogate data to the original data is beneficial if a is chosen optimally, and large m reduces statistical fluctuations in this regularizer. This contrasts with the unweighted approach whose test error in general deteriorates for large m.
As a toy example, reconsider the mean estimation problem mentioned in the introduction: 2; N(O.. Ia) and o N(O Ia), ((0:z) z) = |@ =|1 and Rea( (0) = ||@ 0.1l?. We have n (a) (1-a) Eign iisn zi/n + a Zigm z m./m. In other words. the weighted ERM shrinks the mean of the original data towards the mean of the surrogate data. For a given a, the resulting test errors are
R(On,m(a) (a)) a R (00) +
[ m
+ "
d, (20) 100 0.112.
(5)
and for the optimum value a, = arg min, RiO,,. n,m (a)), this yields
R (00) + d/m m
R(Om,. (c.)))
(6)
Rey + d/m + d/n
72
Note that 1 //n is the enror of training only original data and the prefactor is always strictly smaller than one. Hence, weighted ERM always achieves better error than training only on original data, regardless of the distance between original and surrogate data, although the improvement is larger for small Rgg (oc). This might seem paradoxical at first. As mentioned above, we are shrinking towards an arbitrary point given by the empirical mean of the surrogate data: how can this help?
In fact, this is a disguised version of the celebrated Stein paradox [EM77, Ste81]: in estimating Gaussian mean, a procedure that shrinks the empirical mean towards an arbitrary point by a carefully chosen amount outperforms the naive empirical mean. In our toy example, the naive empirical mean corresponds to estimation purely based on the original data, and we shrink it towards the mean of the surrogate data. Of course, the improvement over empirical mean is only possible if a chosen optimally. Equation (6) assumes
a, is chosen by an oracle that knows the value of Rg (c0). Stein's analysis implies that the Gaussian mean problem, a can be chosen empirically as long as the dimension of 0 is d 2 3. In the settings we are interested in, a can be chosen via cross-validation.
3 Theoretical results
3.1 Gaussian sequence model
The sequence model captures the behavior of many models in con-aaamettric statistics while being simpler to analyze [Tsy09, GN21]. It also known to approximate the behavior of overparametrized linear regression [CM22]. The unknown target is O, E Rd (with potentially d = oo), and we observe
y; 0. + 9: i is n, y = 0 + o. G. i S m, where 0f is also unknown, and 9:, 9: ~ N(O, I are i.i.d. We study the penalized estimator
@ (0) == arg ming (1-a) E 1 0 | + Σ | 0| + >101ka]
(8)
where ||0|| f2 (0, and Ω Y 0 is regularization weight matrix. We will be concerned with the
expected risk
Rn, (a, A) =)(==110, (a))
(9)
The proof of the next result is presented in Appendix C.
Theorem Let W1 I W2 ... be the ordered eigenvalues of St and denote by v; the corresponding eigenvectors. Further denote by O., O,,zk, O O²,,, the projections of 0., O.,s onto span(v;; > k), and similarly for gu,gk, 0² 0- Assume that Wk = kd p > 1/2, ||0,,> 2|2 - Cok- p & μ,, and let De be such that (for all k): D == wi (00 5 0 S Cyk ((ump)) Then the following hold:
(a) There exists an explicit (a) such that, letting 3 = 2(H A p)/(1 + +( p)),
Rm,m (a, Xe(ai)) S a² (e + C. [1- 2² + + 20 20 8
(10)
5

---

(b) If > 2p-12 1/2, there exists C" > O and there exist 0,, 0º, satisfying the assumptions in point (a), such that,
minx Rn,m (a, A) a² Re (o0) C² (1 -x)2 + a² 20 8
(11)
m
Note that since the theorem also implies Rgu (m)) (00) (o2/m)² and Re (o² l this result confirms the scaling law (4).
3.2
Mon-parametric regression in Sobolev classes
In this section we consider the classic non-parametric regression model. We assume that Q4 for some integer Q 2 2, and the original data (æ,, Vi)ign are defined through
= fo (z;) + Es. ~ N(0,
(12)
where E; are independent of æ; and of each other, and {æi equally spaced grid points in the d-dimensional unit-cube, i.c. Xr = {q/Q : q E [Q]"]. Surrogate data have similar distribution, with m Q equally spaced points æ the unit cube, and yf = (af))))) where of N(0, o?). We assume that f. has small Sobolev norm, that is,
1 == Jo.y (15-00) + 1 (1)13) S 1.
Recall that | is special reproducing kernel Hilbert space (RKHS) norm: we expect some of the considerations below to generalize to other RKHS norms.
Following our general methodology, we use the estimator
fn,maa arg miny 1-a E (y: f(x:)) + E (V-- f(af))² + Alll |,:2
(13)
We interested in R(f) [[(f(x)) (f(=))))(((=)))) which is the excess squared loss for test point Unif((0, 1]4).
In order avoid technical burden we will carry the analysis for a continuous model, the so- called white noise model, where we observe the function at all points æ E =0,,]] perturbed by d-dimensional white noise:
dy = fxx dæ + Jz dB(x),
(14)
and similarly for Y We use an estimator that naturally generalizes (13) to the continuous case. Our results for the white noise model are follows.
Theorem 2. Let B = (2p A 4r)/(d + (2p A 4r)). Ifr > d/4 and X (8 (6Knnm0))3r/(at(220m)))
then for every 8 (0,1) there exists constant C C(d, 8) such that
R(fu,m,) I (1+ R Re( + C { - a)² o² + of 0
(1-1
(15)
with high probability, where Kn,m
2)22n a² 6²/m.
Remark 3.1. The white noise model (14) is known to be equivalent to the original model (12) (with deterministic equispaced designs) in the sense of Le Cam, for r >> d/2 [BL96, Rei08]. While suggestive, this equivalence does not allow us to formally deduce results for the data (12), because does not apply to the specific estimators of interest here.
With the given choice of X, r, the derivation of (15) also implies Rg Rey (oo) 2 C' Rer (n) 2 (a/onn) (for the least favorable / [Tsy09]). Hence (15) consistent with the scaling law (4).
3.3 Low-dimensional asymptotics
We study the estimator of Eqs. (2), (3) under the classical asymptotics m - 2c at d fixed. Since this type of analysis more standard, we defer it to Appendix B. The main result of this analysis is that the scaling law (4) holds in this setting, with the classical parametric exponent 3 = 1. for [0, maxx] for suitable Omax E (0, 1). Importantly, the interval [0, amax] includes the optimal choice of the weight a.

---

3.4 High-dimensions linear regression
In this section, we study ridge regression in the nighddimessioal regime in which the number of samples is proportional to the number of parameters. Denoting the original data by (y. X) (with
E RD the vector of responses and X E RRad the matrix of covariates), and the surrogate data by (y, x*) (with y² E Th and X Rmxd), we minimize the regularized empirical risk
1 1- a
Rm (0;a)
x
11y
101
(16)
2n
2m
We assume simple distribution, whereby the rows of X, X (denoted by æs xτ) are standard normal vectors and
y x0. te, X +
(17)
for E N(O, o² In), ex N(O, m)). Note that the two data distributions differ in the true coefficient vectors 0, versus 0º as well as in the noise variance. We will denote by θ (a) the ridge estimator,
(a) arg mingeR m (0; a).
The excess test error (for square loss) is given by R(θ) == ( 0.) (ar, 0)) = 110 0-112. The next result characterizes this error in the proportional asymptotics.
Theorem 3. Consider the ridge regression estimator (a). r := 11. 112. T. == 10 and
7 = cos ((00,, / (1|00|22)00 (2]])) Assume n. m, d oc such that n/d d - 8, m/ 6s, with + 5, > 12. For # (.) defined in Appendix F.1, let
(8, o)) be the unique minimizer. Then for any E, > there exist >> O Ouch that, for all
(a),, (a), w0 (a)
P suPoE(eo.1-eo) RR(On,m (a:) Prsu (a)) [e) 2 2m
where (a) (a) r)2 (E) (w (o)) Further, we can take en 0f 1.
Remark 3.2 (Optimizing a over the validation set). Note that the concentration of R 12(0 (a)) around the theoretical prediction (a)) Theorem 3 is uniform over [Eo, 10). This means that we can find the optimal a by computing On.m (ax) over a grid of a values, estimating R(0 n.m (a)) over the validation set and choosing the optimal a. The uniform guarantee insures that this procedure will achieve risk min (a)
(ax+ + (1).
Remark 3.3 (Relation to scaling laws). An analysis of the equations for (5",EI,"") reveals that, for large 5, 8.. the predicted excess risk behaves as Risx (ax) = a²
8.00 + a² a/C,/8, + (1 - C2/ + o11/5, 1 1/5,) (for some constants S 20000 C1, C2). This matches the low-dimensional asymptotics and our scaling law (4) with 3 = In practice, we find that, for moderate 8, 6,, the behavior of Res (a)
is better approximated by a different value of B (see Appendix A.)
4 Empirical results
In this section, we present experiments validating that the scaling law (4) is a good approximation both for simulated and real-world data. For simulated data, select two different distributions for the original and surrogate datasets. The test and validation sets are generated from the same distribution as the original dataset. In case of real-world data, we choose two different datasets the original and surrogate datasets. We split the original dataset into train, test, and validation sets, while all examples in the surrogate datasets are allocated solely to the train split.
For each dataset and model discussed in this section, we carry out the same experiment: (i) We use models trained on original data to fit the scaling curve (())..(0)) = Agr + Ban- for and obtain Agr and Por (ii) We use models trained purely surrogate data to fit the scaling curve R( θ f (1)) = Agu + Bm - Buu to obtain Agu and 8au- (iii) Since assume R. ((,o,,0(0)), we let
"The same proof, with some additional technical work, yields characterization for ó + 8, < as well. We omit it here for brevity.
7

---

R. Aur and excess risk estimates (n) = R R(O),.(0)) - Aor, (m) = R( (1)) Agr and Re (oo) A Age and we use 3 Sar, the fit exponent obtained from original data); (iv) For each combination of n. m,, we use our estimates of [g (m). (n) (as measured empirically on the test set), 3, (oc), and R. to plot the predieted R(O mm (a))
function of using scaling law (4). (v) We then train the model using n original and m surrogate examples with weights (1 a) and, a for the two datasets, respectively. We average the results of 10 independent runs to compare it against those predicted by the scaling law. For ridge regression, we also compare with exact high-dimensional asymptotics from Theorem
Let us emphasize that these plots probe the dependence on the hyperparameter a. These are much more demanding tests that the usual ones in scaling laws. We generally observe that the scaling law captures well the behavior of the test error for data mixtures. Furthermore, we perform experiments for variety of loss functions to show these scaling laws hold more widely than the theoretical settings we considered.
Binary classification with Gaussian mixture data This is a simple simulated setting. The original dataset consists of independent and identically distributed examples (ys, x;)) E R x R d = 200,
where ys is uniform over {+1, -13. and æi ~ N(v.O., Ia), where O. E R |0.1 = 1. Surrogate data have the same distribution, with a different unit vector O.... This data distribution is parametrized by d and the angle y between the original and surrogate parameters, COS y (0.. 0..)). We use 7/10 in our experiments. For each (n, m, a),, we averaged the results over 10 independent runs.
We use two different models for classification: (1) Logistic regression; (2) A one-hidden layer neural network with 32 hidden ReL.U neurons. The results for both models are presented in Appendix A.
Linear regression with Gaussian mixture data For the Gaussian mixture data generation setup described above, we also perform ridge regression. The results (presented in Appendix A.2) demon- strate that classification loss and square loss often have similar qualitative behavior as a function of weight a, as seen by comparing the classification loss in Figure 7 and the squared loss in Figure 11 for the same setup. Although theoretical results do not apply directly to classification loss, we believe that our qualitative conclusions generalize. This is confirmed by the similar behavior between the two losses and the successful prediction of actual risk by scaling laws in our classification experiments.
m- 2000
(11 4000
5000
expervari
n- 1000
2000
4000
Figure 3: CIFAR1 1 and CIFAR 100 data. Test error when trained on mixtures of original and surrogate
data. Black curves: prediction from Eq. (4).
8

---

6.58
6.22
10002
64
Figure 4: Lasso-based Cox regression on TCGA PanCancer dataset. Test error when trained on mixtures of original and surrogate data. Black curves: prediction from Eq. (4).
Sentiment analysis in movie reviews As original data, we use the IMDB dataset (link) which has 25k reviews for training, each labeled as positive or negative. For validation and testing, we split the IMDB test dataset of 25k reviews into validation set of 10k reviews and test set of 15k reviews.
We experiment with two different surrogate datasets: Rotten Tomatoes dataset movie reviews (link): these are data with different distribution but within the same domain. This dataset contains movie reviews and the corresponding sentiments, Goodreads book reviews (link): these are data from substantially different domain. This dataset has reviews and their ratings. We choose 10k reviews each with a rating of 5 and and label them as positive and negative, respectively.
We convert reviews into feature vectors with = 884 dimensions as explained in Appendix A.3. We logistic regression and neural network models with the same set of parameters in the Gaussian mixture experiments (except for the input dimension).
Results with neural nets and Rotten Tomatoes synthetic dataset are presented in Figure and the remaining results are Appendix A.3.
Image classification with CIFARIO and CIFARI00 We use 50,000 CIFARIO training images as original data, its 10 classes for the classification task, and test on the 10,000 CIFARIO 10 test images. We use 50,000 CIFAR 100 training images as surrogate data. We train a 9-layer ResNet model for classification. Appendix A.4 presents details the data pre-processing and mapping of labels. Results are shown in Figure Note that CIFAR1O and CIFAR datasets are quite different from each other, they have no overlap either in the images in their label sets. Yet, the test eтror training on their mixture is well predicted by the scaling law (4).
Lasso-based Cox regression on TCGA PanCancer dataset We use the public domain TCGA pancancer dataset [GCH*2] 20] (link), with gene expressions covariates and progression-free survival (PFS) as response. After filtering and feature selection, we are left with 3580 female patients, which use original data, and 3640 male patients, which model (link) with 500 selected genes and use "1-concordance score" as our loss function. The results shown in Figure 4. The details of pre-procsssing and experiment parameters' in Appendix A.5.
surrogate data. We fit CoxPHFitter
High-dimensional ridge regression We simulate the data distribution in Section 3.4, i.c.. Vi (0., 2.) + Et, < n; yf (0r,e, x;) + <i, I m; with D; x ~ N(O, Ia), E, ~ N(O, o2),
We observe that training at = yields somewhat singular behavior: we use a = 0.95 as a proxy of see appendices.

---

m 2000
m000000
Treary exgalierr)
y
12
82 04
66
Figure 5: Ridge regression on simulated data. Here d = 500, n 1000, 0² = 0 1, |00 =
1 Il = regul. par. À 2- and m varies by column. Top row 7 T/2, bottom row 7 π /
Et ~ N(O, o.), and fit a simple linear model using ridge regression. The results are shown in Figure 5. use d = 500, x² = o = 1. 00. = |10..all and regularization parameter = 2 2-10 Under these settings, the model is parametrized by the angle between 0, and 0.... where cos == (0., 0..)). We used y = / and / in our experiments."
In our experiments,
The theoretical predictions of Theorem
for these curves in high- dith--mmmniionlll asymptotics n, m, d - oc, with n//d - 6, m/d -> 5- are reported as blue lines, and match remarkably well with the empirical data. The simple scaling law (4) nevertheless provides good approximation of these (more complicated) theoretical formulas.
Note in particular that in the top row of Figure 5, we have (0., O-.a) 0, i.e. the surrogate data are as far as possible from the original ones. Nevertheless, the induced regularization effect leads to smaller test error on the original distribution.
We observe proposed scaling law (4) predicts well the behavior of the experiments, across of the datasets above, and for most combinations of original and surrogate examples we have tested.
Finally, emphasize that the scaling law is only empirical approximation of reality. This is clearly illustrated by the example ridge regression: in this case, we use Theorem to precisely predict the discrepancy between precise asymptotics and scaling law, see Appendix A.6.
5 Discussion
We conclude by discussing possible generalizations of the scaling law (4), and its applicability. First, throughout this paper we assumed that 1 (oc) 0, namely that we can achieve the Bayes error by training on infinitely many original samples. In practice this will not hold because of the limited model complexity. Following standard scaling laws [KMH 20, HBM 22], this effect can be accounted for by an additional term C N where N is the model size (number of parameters), Second, the scaling law (4) implies as special cases that (n) R= Ag 3,, R (m) so (oo)) + Ag In particular, the exponent 3 the same when training on real or surrogate data. In practice, we observe often two somewhat different exponents 3rr # B.pu In these cases, we set g = Bor. and this appears to work reasonably well. However, we can imagine cases in which the difference between Bor and Bag significant enough (4) will stop being accurate
Acknowledgements
We are grateful Joseph Gardi, Germain Kolossov, Marc Laugharn, Kaleigh Mentzer, Rahul Ponnala, and Pulkit Tandon, for several conversations about this work. This work was carried out while Andrea
4 For ridge regression simulations, directly plot the excess risks, as the parameter 0 for original data known. For any 0 the excess test risk in this model is simply ||e 0112.
10

---

Montanari was on leave from Stanford and a Chief Scientist at Granica (formerly known as Project N). The present research is unrelated to AM's Stanford research.
References
[AAMM* 18] Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger, and Carsten Rother, Augmented reality meets computer vision: Efficient data generation for urban driving scenes, International Journal of Computer Vision 126 (2018), 961- 972.
[ANZ22]
Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai, Revisiting neural scaling laws in language and vision, Advances in Neural Information Processing Systems 35 (2022), 22300-22312.
[BDBC+10] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan, A theory of learning from different domains, Machine learning 79 (2010), 151-175.
[Bir06]
Steven Bird, Nitk: the natural language toolkit, Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, 2006, pp. 69-72.
[BL%]
Lawrence D Brown and Mark G Low, Asymptotic equivalence of nonparametric regression and white noise, The Annals of Statistics 24 (1996), no. 6,
[CLCG19]
Yuhua Chen, Wen Li, Xiaoran Chen, and Lue Van Gool, Learning semantic segmenta- fion from synthetic data: A geomelrically guided input-oulpul adapiation approuch, Proceedings the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 1841-1850.
[CM22]
Chen Cheng and Andrea Montanari, arXiv:2210.08571 (2022).
Dimension free ridge regression,
[EM77)
Bradley Efron and Carl Morris, Stein's paradox in statisties, Scientific American 236 (1977), no. 5, 119-127.
[FCK+23]
Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian, Scaling laws of synthetic images for model training... for now, arXiv preprint arXiv:2312.04566 (2023).
[GCH+20] Mary Goldman, Brian Craft, Mim Hastie, Kristupas Repečka, Fran McDade, Akhil Kamath, Ayan Banerjee, Yunhai Luo, Dave Rogers, Angela N Brooks, et al., Visualiz- ing and interpreting cancer genomics data via the xena platform, Nature biotechnology 38 (2020), no. 6, 675-678.
[GN21]
Evarist Giné and Richard Nickl, Mathematical foundations of infinite-dimensional statistical models, Cambridge University Press, 2021.
[Gor85]
Yehoram Gordon, Some inequalities for gaussian processes and upplications, Israel
Journal of Mathematics 50 (1985), no. 4, 265-289.
[Has21]
Tatsunori Hashimoto, Model performance scaling with multiple data sources, Interna- tional Conference on Machine Learning, PMLR, 2021, pp. 4107-4116.
[HBM+22] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, et al., Training compute-optimal large language models, arXiv preprint arXiv:2203.I 15556 (2022).
[HKHM21] Danny Hermandez, Jared Kaplan, Tom Henighan, and Sam McCandlish, Scaling laws for transfer, arXiv preprint arXiv:2102.01293 (2021).
HKK*200) Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom Brown, Prafulla Dhariwal, Scott Gray, et al., Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.147: (2020).
[HNA+17] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou, Deep learning scaling is predictable, empirically, arXiv preprint arXiv: 1712.00419999 (2017).

---

[HNK* 22]
Xuanli He, Islam Nassar, Jamie Kiros, Gholanreza Haffari, and Mohammad Norouzi, Generate, annotate, and learn: Nip with synthetic text, Transactions of the Association for Computational Linguistics 10 (2022), 826-842.
[2SY+22]
Ruifei He, Shuyang Sun, Xin Yu, Chubui Xue, Wenqing Zhang. Philip Torr, Song Bai, and Xiaojuan Qi, is synthetic data from generative models ready for image recognition?, arxiv preprint arXiv:2210.07574 (2022).
[JRBM+ 17] Matthew Johnson-Roberseee Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan, Driving in the matrix: Can virtual worlds re- place human- sumaan-generate annotations for real world tasks?, 2017 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2017, pp. 746-753.
[KIS124]
Feiyang Kang, Hoang Anh Just, Anit Kumar Sahu, and Ruoxi Jia, Performance scaling via optimal transport: Enabling data selection from partially revealed sources, Advances Neural Information Processing Systems 36 (2024).
[KMH*20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei, Sculing laws for neural language models, arXiv preprint arXiv:2001 .08361 (2020).
[Kri09]
Alex Krizhevsky, Learning multiple layers of features from tiny images. Tech, report, 2009.
[KSH12]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, Imagener classification with deep convolutional neural networks, Advances in neural information processing systems 25 (2012).
[MDP*11] 11] Andrew L.. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts, Learning word vectors for sentiment analysis, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (Portland, Oregon, USA) (Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, eds.), Association for Computational Linguistics, June 2011, pp. 142-150.
[MHZH22]
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han, Generating training data with language models: Towards zero-shot language understanding, Advances in Neural Information Processing Systems 35 (2022), 462-477.
[MM21]
Léo Miolane and Andrea Montanari, The distribution of the lasso: Uniform control over sparse bulls and aduptive parameter luning, The Annals of Statistics 49 (2021), no. 4, 2313-2335.
[MPRP16]
Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes, The benefit of multitask representation learning, Journal of Machine Learning Research 17 (2016), no. 81, 1-32.
[MPT+22] Arthur Moreau, Nathan Piasco, Dzmitry Tsishkou, Bogdan Stanciulescu, and Arnaud de La Fortelle, Lens: Localization enhanced by nerf synthesis, Conference on Robot Learning. PMLR, 2022. pp. 1347-1356.
[MRB+23] Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel, Scaling data-constrained language models, arXiv preprint 1rXiv:2305.16264 (2023).
[PL05]
Bo Pang and Lillian Lee, Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales, Proceedings of the ACL, 2005.
[Rei08]
Markus ReiB, Asymptotic equivalence for nonparametric regression with multivariate and random design, The Annals of Statistics (2008), 1957-1982.
[RRBS19]
Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit, A construc- live prediction of the generalization error across scales, International Conference on Learning Representations, 2019.
1RSM+16]
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez, The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes, Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 3234-3243.
12

---

[SK19] Connor Shorten and Taghi M Khoshgoftaar, A survey on image data augmentation for deep learning, Journal of big data 6 (2019), no. 1, 1-48.
[SLW20] Viktor Seib, Benjamin Lange, and Stefan Wirtz, Mixing real and synthelic data to enhance neural network training-a review of current approaches, arXiv preprint Sili:20007.08755 (2020).
[Ste81] Charles M Stein, Estimation of the mean of multivariate normal distribution, The annals of Statistics (1981), 1135-1151.
[TAH18]
Christos Thrampoulidis, Ehsan Abbasi, and Babak Hassibi, Precise error analysis of regularized m-estimators in high dimensions, IEEE Transactions on Information Theory 64 (2018), , no. 8, 5592-5628.
(TDR+21] Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung. Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler, Scale efficiently: Insights from pretraining and finetuning transformers, International Con- ference on Learning Representations, 2021.
[T]J20]
Nilesh Tripuraneni, Michael Jordan, and Chi Jin, On the theory of transfer learning: The importance of task diversity, Advances in neural information processing systems 33 (2020), 7852-7862.
[TOH15]
Christos Thrampoulidis, Samet Oymak, and Babak Hassibi, Regularized linear regres- sion: A precise analysis of the estimation error, Proceedings of Machine Learning Research 40 (2015), 1683-1709.
[TPA+18) 18]
Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield, Training deep networks with synthetic data: Bridging the reality gap by domain randomization, Proceedings of the IEEE conference computer vision and pattern recognition workshops, 2018, pp. 969-977.
[Tsy09]
Alexandre Tsybakov, Introduction to nonparametric estimation, Springer, 2009.
[vdV00]
Aaad W der Vaart, Asymptotic statisties, Cambridge University Press, 2000.
[Ver18]
Roman Vershynin, High-ddmensionee probability: An introduction with applications in data science, vol. 47, Cambridge university press, 2018
[WM18]
Mengting Wan and Julian McAuley, Item recommendation on monotonic behavior chains, Proceedings of the 12th ACM Conference on Recommender Systems. RecSys 2018, Vancouver, BC, Canada, October 2-7, 2018 (Sole Pera, Michael D. Ekstrand, Xavier Amatriain, and John O' Donovan, eds.), ACM, 2018, pp. 86-94.
[WMNM19] Mengting Wan, Rishabh Misra, Ndapa Nakashole, and Julian McAuley, Fine- grained spoiler detection from large-scale review corpora, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers (Anna Korhonen, David R. Traum, and Lluis Mårquez, eds.), Association for Computational I Linguistics, 2019, pp. 2605-2610.
[YCFB+22] Lin Yen-Chen, Pete Florence. Jonathan T Barron, Tsung- Yi Lin, Alberto Rodriguez, and Phillip Isola, Nurf-supervisiin: Learning dense object descriptors from neural radiance fields, 2022 International Conference on Robotics and Automation (ICRA), IEFE, 2022, pp. 4496-6503.
[YLS+24] Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu, Data mixing laws: Optimizing data mixtures by predicting language modeling performance, arXiv preprint arXiv:2403. 16952 (2024).
13

---

A Details of empirical results
A.1
Binary classification with Gaussian mixture data
actual best fit
best fit
Classification loss
0.24
0.22
0.20
5000 10000 15000 20000 25000 30000 5000 10000 15000 20000 25000 30000
m
Figure 6: Gaussian mixture data and logistic regression. Test error when trained on original (left plot) and surrogate (right plot) data only (red dots). Best fits shown in black. These gives the estimates 3 = 0.72, R. = 0.157, and (x(()) = 0.013.
We provide details for the models used in the simulations.
Logistic regression: We use the scikit-leamn implementation with the Ibfgs solver, fitting the intercept, with maximum iterations set to 10k. For each run of each (n. m, a) combination, we set the & penalty (parameter
5, -4, 10.-3, 3, 4, 5, 6, and only report the test result for the value that achieves the best validation error. The results of the individual scaling law estimates and the comparison of joint training results with the scaling law predictions are shown in Figures 6 and 7.
scikit-learn) to 2 -8.... 8 and 10',
The network has one hidden layer with 32 ReLU neurons, and an output neuron using sigmoid. For training, we use the binary cross entropy loss, a constant learning rate of 0.05, and batch size 64. We train the network for 1,000 epochs. Similar to the procedure logistic regression, we use & regularization (weight decay) and use the validation set to choose the best regularization parameter from the (0, 10 10 1003, 2- 10-3
Neural network:
2 2 10 2.4 10 10-1, 10-1.4 4. 10 10-1). The results of the individual scaling law estimates and the comparison of joint training results with the scaling law predictions are shown in Figures and 9.
3, 4 10 10.2,,22
A.2 Linear regression with Gaussian mixture data
For the Gaussian mixture data, described in the previous section, we perform weighted ridge regres- sion experiments according to equation (16) and plot the square loss. As before, we choose the best
regularizer for the ridge regression of the set 2 -8, ...., 8 and 10*, i = -6, -5, -4, -3, 3, 4, 5, and report the test result for the value that achieves the best validation error. The results presented in Figures 10 and 11.
A.3 Sentiment analysis in movie reviews
To convert the movie reviews and book reviews to vectors, we use a combination of two different embedding: We use all the reviews in the training data and then use nltk tagger [Bir06] find the most frequent 500 adjectives appearing in the samples used for training. Then we use the common Tfidf vectorizer (we used seikit-learn's 's implementation of tfidf vectorizer) for which we use the list of these most common 500 adjectives as vocabulary. This gives us vector of length 500 dimension for each review. addition, we also apply Manaphrase-MinilMM--- based on BERT with 6 Transformer Encoder Layers, and return 384 dimension vector representation of the reviews. For each movie review we concatenate the results of tfidf vectorizer and sentence transformer get a 884 dimensional representation that use input vector.
sentence transformer which
14

---

Classification loss
n=500
2.78
0.27
Classification loss
0.39
n-1000
2.28
0.37-
0.21
Classification loss
0.20
2.29
0.18-
2.27
9.21
Classification loss
2.20
0.36
0.30
0.37
0.0
32
2.4
0.6
0.8
3.0
0.5
02
0.6
0.5
3.8
3.0
0.0
02
5.A
0.5
0.0
1.0
03
3.2
2.4
0.6
0.0
3.5
Figure 7: Gaussian mixture data and logistic regression. Test error when trained mixtures of original varying by row) and surrogate (m varying by column) data. Black curves: scaling formula (4).
We use logistic regression and neural networks with the same set of parameters as in the Gaussian mixture experiments (except for the input dimension). We plot the average loss over independent runs.
Results omitted from the main text are presented in Figures 12-16.
A.4 Image classification with CIFAR10 and CIFAR100
We largely use the model and the training procedure described at http::/jovian..//aaaasnne cifarl0-resnet We normalize the images for mean and standard deviation. We train 9-layer ResNet model for classification, using Adam for optimization, weight decay, and gradient clipping, trained over 16 epochs with one-cycle learning rate scheduling policy, minimizing cross entropy loss. For each combination of m, n. and a, we report the average test error over 10 runs. Since there is no overlap between the label sets of CIFAR1O and CIFAR 100. the latter dataset needs to be relabeled. We do this by training a separate 9-layer ResNet model on 10,000 randomly chosen CIFAR10 images from the training set of 50,000 examples (without creating separate split for them), and use its predictions on CIFAR 100 images as labels.
Scaling curves are presented in Figure 17 and 3.
15

---

actual
actual bist ft
0.325
best fit
0.300
Classification loss
0.275
0.175
2500 5000
7500
10000 12500
15000
2500 5000
7500
10000
12500 15000
n
m
Figure 8: Gaussian mixture data and neural network. Test error when trained on original (left plot) and surrogate (right plot) data only (red dots). Best fits are shown black. These gives the estimates 0.79, R, 0.160 and (oo) 0.010.
A.5 Lasso on TCGA PanCancer dataset
We used public domain TCGA pancancer dataset. After, filtering samples with incomplete values we are left with 9220 patients, each having 20,531 gene expression values and the outcome was PFS (progression-free survival). Out of these used a group of 2000 patients, splitted into train and test set of 1000 each to select 500 genes having the largest absolute Cox PH score. We also used the mean and standard deviation of gene expression values of these 2000 patients normalize the gene expression columns for the remaining 7220 patients. Among the remaining of 7220 patients 3580 were females. We treated the female patients data as original data, and split them into train (50%), test (25%) and validation split (25%). The remaining 3640 patients data was used surrogate dataset. We CoxPHFitter model (link) with 500 selected genes and use "I "I-concordance score" loss function. We used the validation split choose best value of l penalty parameter from 2', = 2,0, -2, -4 6, -, 100- 12, 14, -6 the model. We observed discontinuity at a = 1. To avoid this discontinuity, we approximated R(@ (1)) by R(On,m - c)) n > and by m/ - <)) if O, where we choose 0.05. We plot the average loss over 10 independent runs. The results are presented Figures 18 and 4.
A.6 High-dimensional ridge regression
We present additional ridge regression experiments here in Figs. 19-30. We plot the average loss over 10 independent runs. In these experiments, as in the main paper, we set 500, g² 0 1, ||o||| = 1, ||0,,el| 1, except for the last four Figs. 27-30, where we use | = 1/2. We used angle 7 T/6 and π 2/2 in our experiments.
We consider datasets select À that achieves the best validation performance. For the latter method, we try À = 2'. where 10, -8 -6.... ..... 8, 10. For ridge regression simulations, we directly plot the excess test risks, as the parameter for original data is known and for any the excess test risk in this model ||e 2
methods: (1) Fix À a very small value 2-10 and (2) For each random draw of
B Low-dimensional asymptotics
B.1 Formal statements
In this appendix, we present our results on the estimator of Eqs. (2), (3) under the classical asymptotics + oo at d fixed. For simplicity, assume regularizer used in this regime.
16

---

m-500
m=1000
m-2000
m=4000
timu allor
Classification loss
scating une
0.20
500
0.09
8.38
0.27
Classification loss
2.20
n=1000
9.29
0.30
0.27
Classification loss
0.23
2.20
n-2000
0.30
0.38
3.37
0.22
Classification loss
0.21
0.20
4=4000
0.39
Figure 9: Gaussian mixture data and neural network. Test error when training mixture of original (n varying by row) and surrogate (m varying by column) data. Black curves: scaling law (4).
Beyond classical regularity assumptions of dimenimnnional asymptotics, in this section will make the following assumption which guarantees that original and surrogate distribution are 'not arbitrarily far.' Recall that It (0) denotes population error on surrogate data.
Assumption (Distribution shift for low-d asymptotics). There exists constant K. such that for all E R
(0) R(0) & K. (1,(1++ (1+ R(0)).
(18)
The regularity conditions are similar to the ones in [vdVO0]. Here and in the following B(0., the ball of radius centered at 0..
Assumption 2 ("Classical' regularity).
(a) The original population risk R(O) is uniquely minimized at a point θ..
(b) 0 E(0; z) is non-negative lower semicontinuou. Further, define the following limit in
[0, oo) for gd-1,
loo (u; =):= = lim inf ((0:2)
(19)
0/10
Then assume inf Bll (u; > R((O)) + for some 0.
17

---

actual
actual
squared loss
0.75
0.70
0.55
0.50
10000
20000
30000
10000
20000
30000
Figure 10: Gaussian mixture data and ridge regression. Test error when trained on original (left plot) and surrogate (right plot) data only (red dots). Best fits shown in black. These gives the estimates 3 = 0.60, R. = 0.49, and (2o) 0.03.
m=500
m=1000
m=2000
m=4000
0.64
loss
0.62
0.80
0.58
n=500
0.56
0.54
experiment
0.52
scaing 1317
0.64
squared loss
0.60
0.58
n=1000
0.54
0.52
0.64
Squared loss
0.62
0.60
0.58
n=2000
0.56
0.54
0.52
0.64
Squared loss
0.62
0.60
n=4000
0.56
0.54
0.00
0.25
0.50
0.00
100
0.00
0.75
1.00
0.25
1.00
0t
Figure 11: Gaussian mixture data and ridge regression. Test error when trained on mixtures of original and surrogate data. Black curves: prediction from Eq. (4).
18

---

0.32
actual
actual best fit
best fit
0.30
Classification loss
0.28
0.18
2500
5000
10000 12500 15000
2500 5000
7500 10000
m
Figure 12: IMDB and Rotten Tomatoes data and logistic regression. Test enror when trained on original (left plot) and surrogate (right plot) data only (red dots), together with scaling law fits (black lines). Best fit parameters are B 0.27, R. 0.101 and Rg 0.148.
m-500
m-1000
m-2000
m m=4000
rimu aflor scelnn
Classification loss
500
0.00
0.38
Classification loss
n=1000
0.20
5.78
Classification loss
n=2000
0.22
0.20
3:38
Classification loss
2.24
4000
2.22
0.70
2.28
03
Figure 13: IMDB and Rotten Tomatoes data and logistic regression. Test error when trained on mixtures of original and surrogate data. Black curves: prediction from Eq, (4).
19

---

0.32
actual
actual best fit
0.30
best fit
Classification loss
0.28
0.26
0.24
0.22
2500 5000 7500 100001250015000
25.00 5000 750010001125011000
Figure 14: IMDB and Goodreads book reviews (as surrogate dataset) and logistic regression. Test error when trained on original (left plot) and surrogate (right plot) data only (red dots), together with scaling law fits (black lines). Best fit parameters are B = 0.27, R. 0.101 and (p0) = 0.101.
m-500
m-1000
m-2000
m m-4000
vimm
Classification loss
2.20
500
Classification loss
1=1000
Classification loss
2-2000
0.08
Classification loss
4000
0.20
2.28
0.8
0.4
0.0
0
Figure 15: IMDB and Goodreads book reviews and logistic regression. Test error when trained on mixtures of original and surrogate data. Black curves: prediction from Eq. (4).
20

---

0.32
actual
ac aual best fit
best fit
Classification loss
0.28
0.26
0.24
0.22
0.20
0.18
2500
5000
10000 12500 15000
2500
5000
7500
10000
12500 15000
m
Figure 16: IMDB and Rotten Tomatoes data and neural networks. Scaling law fits for models trained on original (left plot) and surrogate (right plot) data only (red dats)(as in Fig. 12), together with scaling law fits (black lines). Best fit parameters are B 0.37, R. 0.145 and (oo) 0.095.
0.7
Classification loss
0.5
0.3
0.1
20000
40000
60000
80000
100000
20000
40000
60000
80000
100000
Figure 17: CIFARIO and CIFAR 100 data: (left) Test error scaling of original data (left) and surrogate data (right). Best parameters 0.404, R. 0.0013, and Rez 0.199.
0.9
actual best
actual
best
1-concordance index
0.7
0.6
0.3
0
1000
2000
3000
4000
1000
2000
3000
4000
m
Figure 18: Lasso-bur based Cox regression TCGA PanCancer dataset with female patients data as original data and male patients data as surrogate data. Scaling law fits for models trained on original (left plot) and surrogate (right plot) data only (red dats) in Fig. 12.) Best fit parameters 3 = 1.55, R. = 0.29 and (o0) = 0.29.
21

---

3.0
actual
actual
best
best
exc excess squared loss
5000 10000 15000 20000 25000 30000
5000 10000 15000 20000 25000 30000
m
Figure 19: Ridge regression with x/ /2, and regularization parameter 10 Test error scaling of the original data (left), and surrogate data (right). Best curve fits give the estimates 3 = 1.57 and R2g (00) 2.0
1000
3.0
Steery
excess squared l055
1000
3.5
3.0
excess Squared loss
2.0
2000
3.0
excess squared loss
2.5
4000
3.0
3.5
0.6
3.6
0.8
3.0
0.
0.2
0.8
0.5
0.0
3.0
0.0
0.2
3.8
0.
0.R.
Figure 20: Ridge regression with T/2, and regularization parameter À 2-10
22

---

actual best fit
actual best
excess squared loss
0.8
0.4
0.0
5000
10000 15000 20000 25000 30000
5000 10000 15000 20000 25000 30000
m
Figure 21: Ridge regression with π / between and 0,, and regularization parameter À = 2 2-10, Test error scaling of the original data (left), and surrogate data (right). Best curve fits give the estimates 1.57 and Ag (0c) 0.29
Eheory
2.4
excess squared loss
5.8
1000
2.8
excess squared loss
3.6
0.8
2-00
2.4
0.2
3.2
eXcess squared loss
3.4
4=4000
2.8
3.5
0.6
3.6
0.8
1.0
0.
0.
0.5
0.0
3.0
0.
0.2
3.4
0.5
0.
13
Figure 22: Ridge regression with 7/6 between 0 and 0g. and regularization parameter 2 -0
23

---

actual best
excess squared loss
0.8
0.6
0.0
best fit
5000
10000 15000 20000 25000 30000
5000
10000 15000 20000 25000 30000
m
Figure 23: Ridge regression with π/ //2 between 0 and 0,, and the best regularization parameter: Test error scaling of the original data (left), and surrogate data (right). Best curve fits give the estimates 0.94 and (oo) 1.0
Steery
excess squared loss
1000
3.4
excess squared loss
2.8
2.8
2-000
3.4
3.2
excess squared loss
4000
3.5
0.6
0.6
0.8
1.0
0.3
0.2
0.8
0.5
0.0
3.0
0.0
0.2
3.5
0.5
0.R.
Figure 24: Ridge regression with n/2,, and the best regularization parameter
24

---

0.6
actual
actual
excess squared loss
0.3
0.0
10000
25000
5000 10000
15000
20000
25000
30000
Figure 25: Ridge regression with 76 between 0 and 0,, and the best regularization parameter: Test error scaling of the original data (left), and surrogate data (right). Best curve fits give the estimates 0.94 and Agg (Do) 0.24
m 1000
2000
m
Iicory
scallig
excess squared loss
2.4-
100000
0.3
2.5
excess squared loss
n=2000
2.4
excess squared loss
2.4
4000
0.4
0.2
2.5
0.5
0.
3.0
0.2
2.4
Figure 26: Ridge regression with 7/6 between and 0s. and the best regularization parameter
25

---

actual best
excess squared loss
0.9
0.2
actual best ftt
0.0
10000 15000 20000 25000
5000
10000 15000 20000 25000 30000
m
Figure 27: Ridge regression with //2 between I and 8,, ||||| 1, |0.11 1/2 and the best regularization parameter: Test error scaling of the original data (left), and surrogate data (right). Best curve fits give the estimates B 0.94 and (g(())) 1.00
m - 1000
2000
m 4000
1.5
!teery
coalis lou suulicion
excess squared loss
2.8
5.4
excess squared loss
n
4.4
3.0
excess Squared loss
0.8
3.8
4000
2.4
Figure 28: Ridge regression with 7/2 between and O.. ||0|| ||0,]| 1/2 and the best regularization parameter
26

---

actual best
actual best
excess squared loss
0.0
10000 15000 20000 25000 30000
5000 10000 15000 20000
25000 30000
m
Figure 29: Ridge regression with y= 7/2, ||0|| ||0,11 1/2, and regularization parameter 2-10: Test error scaling of the original data (left), and surrogate data (right). Best curve fits give the estimates B 1.57 and Rg (oo) 1.27
m 1000
2000
m 4000
!reery
colliiisw lew
excess squared loss
3.5
2.8
7.0
excess squared loss
2.8
n= 2000
excess squared loss
4000
2.8
Figure 30: Ridge regression with 7 = T/2, ||0|| ||0.1| 1/2, and regularization parameter 20 10
27

---

(c)
(c) θ + 0 E(0;z) is differentiable at O- almost surely, both under z ~ IP and under 2 ~ IP. Further, there exists > such that, letting := B(),.r), the following holds for constant C:
(e(01;z)
2(02; =
E sup
SC<O.
(20)
#0
|00|
0211
(d) The functions 0 + R(O). 0 (0), are twice differentiable in a neighborhood of 0.. with Lipschitz continuous Hessian. Further V R(O.) > (strictly positive definite).
Proposition B.1. Under Assumption I and Assumption 2, define the following d x d matrices
H == =
(21)
K == (o)
(22)
:
(23)
where Cov, Cov, denote the covariances, respectively, with respect to the original data (i.e., with respect to z P), and with respect the surrogate data (i.e., with respect z" P.). Further define the d-dimensional vector
g = VR. VR(O.).
(24)
Then there exists Omax E (0, 1] (depending only on the constants in the assumptions) such that, for all a E [0, Amax), the excess risk of the estimator θ (a) satisfies (for D == |lg'|| bounded by constant)
R R(On.m(a)) R(0.) a² (g) HHH 9") (.H''"))
-TH(H KK)
(25)
n
TT. + +0(( m V + DaO) 2
1 (m (m m) + De))
(Here the big O hides dependence on the constants in Assumptions / and 2.)
Remark B.1. For economy of notation we stated Proposition B.1 in the case in which the excess risk is measured by using the same loss as for training, i.c. lys = l. However the same result Eq. (25) applies with minor modifications the case las F l (and thus, with R replaced by R provided Rues is also twice differentiable with Lipschitz Hessian, and V RA.()) = O. In this case, (25) has to be modified replacing H by H V
(0.)H- (O.)
Remark B.2. The error terms in Eq. (25) negligible under two conditions: (i) and are large, which is the classical condition for low- diwdimenniooal asymptotics to hold; (ii) |g"1|2 = V is small. In particular, the latter condition will hold in two cases. First, when V (0. Il2 is of order one (i.e. the distribution shift is large), but a is small (surrogate data are downweighted), Note that, when the distribution shift is large, and the sample size n is large enough, we expect small a to be optimal and therefore Eq. (25) covers the 'interesting' regime.
Second, when || (R.().))) Il2 is small (i.c. the shift is small) and a is of order one. If in addition have 2 Rd (0.) ) R V² (0)), can be shown that the range of validity of Eq. (25) covers the whole interval E [0, 1].
Remark B.3. Note that the distribution shift is measured in Eq. (25) by the first term (g" H g²). The original and surrogate distribution can be very different in other metrics (e.g. in total variation or transportation distance), but as long as g" is small (as measured in the norm defined by H-!) surrogate data will reduce test error.
Note that, within the setting of Proposition B. the excess error of training only original data is Rar (n) (0)) R(O.) Tr(H K) /nn o(1/n), while (g)) R((,,... (0)) R(O.) = (g", H (1=")) + Tr H K. (.)/m + o(1 m.) Hence Eq, (B.1) can recast in the form of general scaling law (4), namely:
(() R(0.) x a² ( + aa (R
(co)) (1 - a: ( (m))
which (as expected) corresponds the parametric scaling exponent
An immediate consequence Proposition 1.1 is that surrogate data do not hurt, and will help if their distribution is close enough to the original one (under the assumption of optimally chosen a).
28

---

Corollary B.2. Under the assumptions of Proposition B. let Ro = Tr(H K)/n, and
Ra((m) = (g" g²' + Tr(H-* Tr. / For 0n Rac(n)/ (au(m) + (o)(n)), have
m0 R. (Ro(nn)-+ (m)(m) + An.m.
with Anm of the same order as the error in Prop, B.1.
B.2 Proofs
Lemma B.3. Under the assumptions of Proposition B.1 (Assumption I and Assumption 2) there exists Omax E (0, 11, depending only the constants appearing there such that the following holds;
The function 0 R( (1-a) R(0) - R (0) has unique minimizer 0x(a) R Further 0x (a)) E B(,,,T), r), and (a) - 0, as a 0.
(i We have nn (a) 0, in probability as n, oc.
Proof. Fix ro E r0, By Assumption 2.(a), infogr(o..)) R(O) R(0.) + g0 for some constant 5p. Hence, using Assumption 1, for any 0 g B(0.;r)
R(O,al a) > R(0) K,a [1 + R(0)]
2 (1 R(0) K.
2 (1 K.a) + 0o) K.a.
In the other hand a)) s (1+ + K.a) + K,a, whence R(0;a) - R(0.:a) a) 2 (1 - k - 2K,
-K,a,
which is strictly positive for < amax (ro) 6o/. + R(O.)). Hence the minimum must be achieved in ((0.:ro) (note that since R(0), (0) are lower semicontinuous, the minimum is achieved).
By Assumption (d).). for ro sufficiently small, 0 + VR(0; a:) is strictly convex in B(0.: ro) and therefore the minimizer is unique. This proves point (f).
Point (ii) follows from a modification of Theorem 5.14 in [vdV00]. Namely, for a diverging sequence {(n(k), m(k)) : & E N]. we consider to (u,((u) = (c(())))())(((()) a),, where c(u) == (1+ + ||/]|/)-1//_ This function is lower semicontinuous on the compact set B(0; 1) and converges almost surely to its expectation for every fixed this set, and hence the argument of Theorem 5.14 [vdV00] applies here.
Proof of Proposition B.I. By modification of Theorem 5.39 in [vdVO0] (here (a() is defined as in Lemma B.3)
0 (a) = + 1 H(a)
2
(1)
(26)
n
1
+ ((a)
(((0 E, V( Op(m + n n
(27)
m
where (a) = (1
+ V ( (a)). Note that in the present setting the error is of order m + because we assume the Hessian be Lipschitz continuous.
The population minimizer (o) solves
0 ())
VR(0,;a) + (²R(O.:a)(0 0.) + L (1.]) 0.) dt,
29

---

where 0, = (a) + (1 t) 0. Denoting by Le the Lipschitz constant of the Hessian (in operator norm), and recalling that VR().) O, we have
v (0. - 0.) = aVR.((.)) + u,
| - (a)) - 0.11". Recalling that, by Lemma B.3, B.3, (a) 0, as a - 0, this implies
(~(a) 0, = - VR, ( |VV Substituting in Eq. (26), we get
(28)
0 mn, 0. -0=--P - (. 1-a ((a) [E((O_(a):=)) -VE((;;=))
(29)
a
Haa) Σ
(V((0 - V.Ve(0; 2)] + A
(30)
=11
A (C( IIV.. a² C
(31)
mAn
The claim follows by substituting the above
ER(O m. (a)) R(0)= 1.(0,.((a) 0,, ((, - -0,) +O O (E)(@nm((a) - 0.113) (32)
and using H(a) = H + O(a).
C Gaussian sequence model: Proofs for Section 3.1
C.1 General ridge regression
We define & = x X
n, E. x-XX X X // m, and E. (1- a) aaE. We then have
(1 a)-0) Vn, (a, A)
a²0²
(a, A) =B (a, X)
V (a, A),
(33)
n
n
Bn,m (a, A) 12 -1)- (200, (0, 0.))
(34)
Vn.m (a, A) Tr (( -1 -(
(35)
Vr, (a, = Tr ( - E1
(36)
C.2 Proof of Theorem
Without loss of generality, we can assume Ω = diag( ((w))z2)) with wi. non-decreasing. A simple calculation gives the following general expression for the test error:
Ra,m (a, A) =B A) + 5n,m Vn,m Vn,m(a, (),
(37)
1
)²
2 [(ax + Awak - -0.,, x]
Bn,m (a,
(38)
1 Aaas
k=1
(a,
(39)
+ Xuia
1=1
8n,m (a)
a= +
(40)
We define (with bą 0 if the condition is never verified)
k = max k: Awak < {1]
(41)
30

---

Note that
0 ocksk k
0 <<& Awn S 1,
(42)
h1 < k 1 < Awk.
(43)
We now estimate various sums by breaking them by the value of k
kn
Ban
aa + )wx) abi,u]+
k= (Jwre) [[a + )wn))., at" }
1=1
h1
<
[02
07,1)² 200 1, DendO.n + (2wix)
1=1
8
a²
2a
(2wx)) ) (0,,1-
02,6) -2,,)
(0,,k - 0g 00, + 02.00
Awik
k=
2a
S a 2 0,,k,, - -0" 5 112
10, - 0² م S )ol
100,5k 11
Wki
k1
a +
||0,,>k
0 >, 112=2 + 20xw + 00,,2k
0 0, 0 b. 12 1/ 110,,>k, 11?
and
W +1
Vn, I k1 +
I (ki + c4),
w
k>ki
since under the assumption Wi x k", f > 1/2, we have EkSki (Wks wikk+//wx) S c#. Recalling the definitions in the theorem, and letting
8g == max [Wk+1|(0,.0) 0º . O 0,.nk)
122+110x.>4 0 02)2)
we have
1
Bnn S a²110_ 1100--011 [|]2++100,xx 12
100,5k 182 + +30k, + +2Ab,
w
whence
(a, S aa -0211 + 1|0.,>> ||//++10.,20
112 11
11. 1122 + (ki + c#) sn,((a) + 36k, + 24
w
= =a²Re(oo)) + +100,>a > 112 + w
1 ski lla + (k) + c()) (n)((a) + 36p, + 24R,
Next we specialize the case 00,>2|1² 11 Cok-2p, wi ktt p # p. In this case we have w 2 sll S Ck- C2-2(10)), and therefore, by suitably adjusting the constant
Rn,m (a, A) a a (c0) Ck, 2(PAp) + (ka + c#) sn,((a) (a: + 35k, + 2Ak
We now bound 8R. By Cauchy- Schwarz and monotonicity of w. O 0
wk+1 Wk+1| (0..>k - 0" -"
< |10,,> 0 12.>11el10,,0, S 2Cakk 2p
and further
2 w 0² 02,2 22 +2 =1, S 40ck 2p
=2
(44)
Therefore,
Ro (a, A) a² ReY Ck₃ 2(uAp) + (ki + c#) sn,m (a) + 24p
Proof of claim (a). The stated assumption on A imply that (eventually adjusting the constant C): < R (a, A) S a² (Oc) + Ck 2(unp) + (ki + +c)) (a).
31

---

We now select (a) so that hy Sn,m (a) - where / = 2( A p)/(1 + 2(μ A p)). (this is possible for n, m large enough under the assumption on wx., straightforward calculation yields:
Rm (a, Ao(a)) & a² 100( + +0 sn,m(a)
which proves claim (a).
Proof of Claim (6). We choose Wk = k"t 0z,k = 1/2-112 0² 8.,k + agk p-1/2, -p11/- with ag ~ Unif({ - A. +A)) We will choose A < 1 sufficiently small numerical constant. Note that, for μ > 2p 1/2
1/2 S 20-1/2te',
A = TM H
Σ t=1
agl
2p-1
S C'AK ptc
Pto
Σ 21
eh 4p-2
where, for any ε >> can therefore select the ag, so that 4 S C" Ak-2p- for some C" < cc.
the first inequality holds with probability least 1/2 for all k ko(E). We 22-p-
Following the calculation at point (a) decompose the bias term
[0= 00,,)² 2 + (Awx) 2a
Bn.m=
Awnk
) (0x, +0
En,m=
1 \wnk
k=1
Note that |E,.ll S 8k1
+ Dki S CAKT 2(un)) Therefore
Bn,m a² a²210. 0111
Awk
1
) (0,,k 0²2x)² -CAK -CAK 2(ump)
0 - a² Σ
+ Xwa
1 Awkk
Rm
k=1
0 00,5h 11 +
10
10 Sk.sk, / 100,,2h 1 -CAKT -(AA2((AAD)
Aw 4wk+1
2 k 2(wAp)
By similar calculation, we also obtain
Vnm 2 Cki
and therefore
Ra, A) 122a a² R (c0) + CK -(ump)))) + Ck1 (a).
The proof is completed by minimizing over kj.
D Analysis of the nonparametric model: Proofs for Section 3.2
This appendix is devoted to proving Theorem 2. Recall that this is established within the white noise model of Eq. (14), which we copy here for the readers' convenience
dY (x) dæ + Vn dB(x),
(45)
The adaptation of the estimator (13) to this continuous setting is given explicitly below
fa,ma arg min { a)llY f f|2 allY, - fll >
(46)
The proof of Theorem is based reduction suitable 'sequence model' via the Fourier transform, defined as
((g) = Sa f(x)) e- mg dx,
(47)
32

---

for q E Qa := {2mqq }"), where = V-1. 1. The inverse Fourier transform is defined as
((x)
Σ qEQa
((g)
e
(48)
(2xx)
We let Oz a, and 0x p.p.p.m respectively denote the Fourier transform of fu, fr.e. and fxpmma The Fourier transforms of the observations are given by
((g)= (g)g) + Vii
pz G(g), Y,(g) (g) + da Gz(g),
(49)
vm
where G(q) and (g) are i..dd. standard Gaussian. It then follows that
n (a) = arg min {11 (1-a))] a)ll 0112 + a allY, 011 + x1101
(50)
where we abuse the notation to define
e 01 = Σ Cp.g 10(g))
(51)
g=Qa
with Sp.q + |r Minimizing (50) we get
On.m(a a) = 1+ Aep. - (1-a) ((g) + a &,()))
(52)
Taking the inverse Fourier transform and plugging it into the excess risk formula we get
R(fn.m))= = Σ
[a(On, - -))(()))))()))))
(1 Acp.
qEQ.
+ 0. + Vn.m Σ
(53)
(1+ + )cp.g)2
QEQa
where
n.m == == -1)-) 0
(54)
The convexity of x x² implies
(9 a + (1 7) 1-7
2
b²
(a + b)) =
(
(55)
1-
for 7 E (0, 1) and therefore we can upper bound the first sum in (53) by taking 7 = 1/(1 + 8) for any 8 > O, which yields
1 1+6 6
Acp.a Acpag
2 10-(9))++V..,, k
R(f S (1+6)a2110, 0-1
Σ
I
(1 + ACp.a)²
qEQu
qe
(56)
D.1 Proof of Theorem
We now upper bound the first sum above. We note that, defining go via Ac,((o) = 1 (with an abuse of notation (t) = 1 + f2r), whence go 2 (A/2) 1/2r for all X < 1:
ACp.a
2
(9)
Σ
(1)
Σ
(0)
+ Xcr.
qe geQuill S
qe ge 2>q0
2 max
Σ
(9))
(g))))0_())
max
| (g)
Ilall>>0 Co (g(g)
qEQa.Ilall.5
qe gEQu.lil1a>9
(4) 2 max
max
(q) ||al|>>00 co(g)
< 2 max (1, (90)) 2
1
Cs (o())) (o(())
s O'mx(X? x2/r) CXP/T < 222/0)))
33

---

where in (a) we used the fact that 1 = Eg (=( on d, we have
Letting Ci(d) be constants depending
Σ q
Ci(d)
RR (1+ + )cr.g) dq
(1 + )c
S <i(dd)
(1 11>N1g//3111 dq
Rd
gd-1
< <2((d)
(1+ ++A22ry3 dl
-1122 ttt dt + C2(d)A
S C2(d)
10-1-4r dt.
1/2r
For convergence we requite > d/4, in which case
Σ
)rrg)) 1 S Cc(d))--1121 Q
(57)
E
Analysis of high-dimensional regression: Proofs for Section 3.4
E.1
Auxiliary definition for Theorem
Our characterization is given in terms of a variational principle. For 6, 8, E c0), define L(;a):
R3
via
:= :
p2
PV8((22 + + + p.. +
(58)
8p²
dap2
+ (5 + +w?)
2(1 0))
2a
2
where 7, Ts defined by
72 = (5 r)2 + 5 + w²
(59)
T? = (E rs cos 7)² + (E1 - r, sin 7)2 + w²
(60)
and p= p=pp///++/ppppmm pl/ /1 + 12, with solving the polynomial equation
p² ~0r²+0²)
(,(++)))
(61)
6/ 6/(1-a)+w/p) (5./atw/7)
and is given by
t w+5p/(1-at
6x(r3+03)
(62)
245.7/0
a(r2+a4)
Theorem states that the asymptotics of the test error is determined by the minimizer of A.
E.2 Proof of Theorem 3
The proof is based on Gordon Gaussian comparison inequality [Gor85, Ver18), and follow standard route, see e.g. [TOH15, TAH18, MM21]. We will limit ourselves to outlining the main steps of the calculation. Throughout, 5, 5, > 1) is analogous and less interesting.
consider the case Eo
0, 5 5, because the other one (Eo 0 and
We begin by rewriting the ridge cost function in terms of a Lagrangian
R
0; a) max max Σn.m (0, a²:a), "E ER
(63)
Ln. (0, u. u² == u, x 0z)) + (u" X - 0..))) (u, E) (u", E*)
(64)
||ll m
1 1
A
0
0
2 a)
2a
2
34

---

Let A u. u²) := normal random variables, independent of X, X By Gordon's inequality [Gor85], we can compare the Gaussian process m0, u, aa) + A (0, u, u to
1+1 14 ||2|| 00-0,.al22.... where G, G. are independent standard
Pn,m (0, u, u²; a0) := 1| (g. A - 0.) + 0 0. h,, u) + | (((",, 0 - -0.,)) + 110 - 0,,s (he, u²)
(65)
n u 1
m a 12
A
(2, E) - (ur, E²)
11012.
2(1 a)
2a
2
Next we define the orthonormal vectors
0-
Po. 0.,-
U. 100.12
(66)
P 0. l12
where Po. = I - Pe. == I - v _ is the projector orthogonal to 02. We then decompose
0 Ev. E v. + +0+,
(67)
where (v.. (+. ) (w. . 0. and define w == 110 Defining 72 = -10 0.112 T?
||0 0 12. Eq. (60) follows.
With these notations, and letting 8² = 1| + +l1B/n n 0² = =1|Tah" + +E* 5²112/m /m - T/. we get
GG n.m (0, (sj == maxx u.u
GG n (0, u, u² ;a)= : ||
p || Ps.
(68)
Vd
val
To n.m (0, (oic))
P
(g 0- 0.) + Ps
[g 0 0.,a) + PV 0 +
+ Ps 0, + 62)
Va
vd
(69)
8p² Saps
X (5² + 5 + w²)
+
2(1 a)
2a
2
We finally decompose g = 9 +g_ where g E span(u., (U. w.:) and ||| 1 span(u., v. and similarly for g.: and define
( min
T n.m (,, p. ps:aa: 0= =v + +E o, + + |10-1 w
(70)
Defining i via [aallgg/// p.g /ta.////tttgg//./// (1+ p3. we obtain
(G.n(B.EE11.:ppPPP..) = = (1+ P3
w
A + PV P.
0T +2)
(71)
8p²
5up2 x
(5² 5 +w²)
2(1 aa) a)
2a
2
where A is the contribution of the perpendicular components. Simple concentration estimates imply that for any & > 0 there exist c(E) > 0 such that
P(( al S S < 0², la. a.l s.lS 2 <(e)
(72)
P(A) S 102 + p3)( + {1) 2 - e <())
(73)
P(4 2 - <(eja
(74)
We can then estimate LL (E, w. p. as) by
LC = vp
+ p
w
PV + Po (
(75)
8p²
6ep2
(52 +
2(1-a)
2a
35

---

Differentiating with respect to p and Ps and setting the derivatives to 0 yields = p/vI+ Ps pt/ vV with p, f given by Eqs. (61), (62). By computing second derivatives, one obtain that this local maximum. Since (E, E,, W,,, P=Pa: - 20 as p² + / - oc, the maximum over P. Ps is either achieved at this point at the boundary {o = 0} U {p. 0). By checking the signs of partial derivatives along this boundary, the only other possibility is p = Ps 0.
For economy of notation, write P(), = ( p. For any unit vector v = (V1, v2) 2 the directional derivative
v.F(r) P= = - u VI-VVV 0(T2 + 0z² + U2 6TT +
2w1-1+0V0 11 u22 02vo
By maximizing over the direction, we see that v can be chosen so that F() 2 w|-1+ Vō+ Hence p /s cannot be the global aximum for 8 + 8, > 1. Hence, get
max J (,, Po Pa:aa)) P.P.20
(76)
We further note that, for fixed p. Ps > O, the function (L, ( + LG fi,,,, p. P.;a))) is jointly strictly convex for À > O. Hence (w) + (E, L w; is also strictly convex for À >> 0. Therefore, it has unique minimizer, which we denote by (E", "G,"" Proceeding as in [MM21], obtain the following result.
Proposition E.1. Under the assumptions of Proposition 3, for any 6,60 > there exists
a(c, (o) such that, if a E (Ea, 1 Ea] (letting P = =I- v.v. b.. (v
P (v. mm - -5 Se, ko On.m) -7 S EE. P θ m - w" S E 21 - on
(77)
In particular, the last proposition implies (a weaker form of) Theorem 3 whereby the supremum is taken over finite net. Namely for 1 > 0, we define
N(Eo: r7) := [ 1 - Eul nnz.
Recalling that, in the present case, R(0) ||@ -112. we obtain (after adjusting the constant have therefore:
max (CE(()))))
R(On,m Basu(a)) 20) 2 21-20 on
(78)
Finally, let X E (m)) be the matrix obtained by stacking X and X,, Given constants C1 C2, Cg. define the good event
9 = {Cin S Amin (XI S Amax (x]x,) S C2n; X ull S Can, |xIv.ll S C3n } /
(79)
By a standard bound cigenvalues of Wishart matrices [Ver18], for 8 + 8, 1, we can choose
C1, C2, C3 such that
P())= 2 1 - 2e -n
(80)
Further on G, On (ax is bounded (in & norm, and Lipschitz continuous a). As a consequence, for sufficiently large constant T.,
P( (((00,.
R(O,,. (o2)) < L|a - a2/Vai, 02 [ - sal) 2 1 2e- (81)
The claim follows by using this estimate together with Eq. (78).
36

---

Datasets information
Imdb reviews datatset:
Paper: [MDP*11]
Link
Licence: Creative Commons Aurthurioo MonccCmmeeaaaaaSeeeee 3.0 International License
Rotten Tomatoes reviews:
Paper: [PL05]
Link
Data has
been
scraped from the publicly
available
website
https//wwwwrrotennmmmeess
of 2020-10-31.
Licence: CCO: Public Domain
Goodreads bookreviews
Papers: [WMNM19], [WM18]
Link
License: Unknown
CIFARIO and CIFAR100:
Paper: [Kri09]
Link
License: Unknown
TCGA Pan-Cancer Clinical Data
Link
Publicaly availbale, free use
Licence: None
G
Compute resources information
We all experiments single machine with 2 RTX 4090 GPUs and 24-core Intel Xeon E5 CPU. All experiments completed in less than 24 hours.
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
These claims are justified by our theorems and the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
38

---

Justification:
All assumptions and proof' are clearly mentioned in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide detailed setup of our experiments ion appendix and the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
39

---

Answer:
[No]
Justification: All the datasets used are public datasets. The results can be reproduced using the details we provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ public/gus Moblic/guides/Cee iataddds/CodeeSumisssiiiioo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips.aa htlps.cccpubblc//c weciic/guiiiee guides/ httpai/i//ggiideeeodeeuiittiii "oooeoobmiiss onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide these details in the experiment sections in appendix and the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Since we observed good match between the smooth theoretical curves and the empirical results across several dozen different dataset/model combinations, we chose to keep the figures clean by plotting only the average results and omitting the confidence intervals. If preferred, we are happy to replace the existing plot with ones with confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
40

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: provide the details of the compute resources in Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the csGuidel NeurIPS Code of Ethics https: http///nurripaee cc htcppuulcc///eeeeecceeeeeeo.lltt d ines?
Answer: [Yes]
Justification:
We have read the code ethics and our submission abide by that.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is advance the field of Machine Learning and efficient use of data. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
41

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not release any new dataset or model.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All of the third-party dataset used in the paper are well-known, publicly available datasets, In Section F, we cite the original papers, link to the datasets, and when available, cite the licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
42

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
43

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
44


Reviews:
[{"review_id": "aYDEZLk0hU", "reviewer": "Reviewer_zfeC", "summary": "In the context of linear regression, this paper proposes to complement training data with surrogate data (generated / synthesized, or from an unrelated task / domain). For an optimal combination of real and surrogate data, new scaling laws are derived which show that that surrogate data can can help reduce the text error.", "questions": "- In practice, how is the optimal value of the weighting paramater $\\alpha_*$ to be set ? Is there a consistent estimator for it which only depends on the input training data ? Without such a prescription, the proposed scheme might fall short of practical usefulness.\n\n- My understanding is that the proof of the main result uses the Gordon comparison theorem. If a nontrivial covariance matrix for the features $x$ is introduced, major modifications might be required, or an entirely different approach (e.g via RMT) might be required. I'd be glad if the the authors can comment on this.\n\n- I wonder what this work could tell us about the \"model collapse\" phenomenon [1,2,3], where a model trained on synthetic data eventually breaks down (becomes biased towards a trivial / useless estimator).\n\n[1] Shumailov et al. \"The Curse of Recursion: Training on Generated Data Makes Models Forget\" (2023)\n[2]  Alemohammad et al. \"Self-Consuming Generative Models Go MAD\" (2023)\n[3] Dohmatob et al. \"A Tale of Tails: Model Collapse as a Change of Scaling Laws\" (ICML 2024)", "limitations": "The setup considered is extremely: linear regression with Gaussian data with no covariance structure (i.e isotropic covariates)", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is very well-written. The context and motivations are clearly presented. The paper makes nontrivial contribution to understanding commonplace practice in training large models in modern AI era: namely, the use of synthetic  or surrogate data.", "weaknesses": "NaN", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "In our paper we provide two recommendations for selecting $\\alpha_*$\n1. Compute the error on a validation split from the original data for various values of $\\alpha$, and optimize among those. (This can be improved using cross validation)\n2. Use the scaling law expression, with free parameter fitted to empirical error at $\\alpha=0$ and $\\alpha=1$.\nThe first technique can be proved to be consistent under standard assumptions. The second one is computationally faster. Of course, we can think of intermediate strategies as well.\nWe will emphasize these recommendations.\n\nIndeed, the result for high-dimensional ridge regression uses Gordon inequality. This approach can be generalized to cover cases with non-identity covariance at the price of becoming technically heavier. (See, e.g. Celentano, M., Montanari, A. and Wei, Y., 2023. The lasso with general gaussian designs with applications to hypothesis testing. The Annals of Statistics, 51(5), pp.2194-2220.)\n\nThe connection to model collapse is indeed worth pursuing. One interesting remark is that there exist weighting schemes that will prevent model collapse.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0CISnPA54g", "reviewer": "Reviewer_XfMG", "summary": "This is a technical report on predicting training loss for a model trained on a weighted combination of real (in-distribution) and surrogate (out-of-distribution) data. The report proposes a parametric function for predicting how the training loss scales with the amount of real and surrogate data. The report first conducts theoretical analysis with a number of standard models on stylized cases, such as Gaussian sequence model, high-dimensional linear regression, etc., and derivated conditions for the proposed scaling relationship to hold. The report conducts empirical validation of the proposed scaling law on a range of classic tasks such as Sentiment analysis for the IMDB dataset, ResNet for CIFAR-10 classification, etc., and shows the training loss consistent with the proposed scaling function.", "questions": "See Weaknesses.", "limitations": "The work does not have a standalone discussion for its limitations.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper adopts the format of a technical report rather than a typical research paper. The structure of the report is clear. \n\nThe problem is clearly introduced and formulated at the beginning of the paper.\n\nAbundant theoretical analyses are provided and a range of models and cases are considered.\n\nExperiments are diverse.", "weaknesses": "I'm not sure whether this technical report format is suitable for a NeurIPS paper. The lack of context or background is not a critical problem, though the lack of context may make it less accessible to broader audiences. What I found most concerning is the complete lack of comparison with existing works and a serious literature review.\n\nThe practice of data augmentation or/and using synthetic data from generative models/simulation, etc. is prevalent in the field and has a long history. This also seems not also the first paper to propose the idea of weighted training with real and surrogate data.\n\nIt is essential for any paper contributing to this topic to first\n1. organize and list previous efforts on this topic;\n2. introduce the current progress and research landscape; identify and justify there exists a valid research gap;\n3. how this paper moves forward compared to previous efforts.\n\nAll of these are missing from this paper. Section 1.3 simply listed a number of potentially relevant references without commenting on each one or explaining how they relate to or are different from this paper. It is not actually informative.\n\n---\nThe problem definition does not appear rigorous. Though problem formulation and notations come early on in the paper, many are not clearly introduced or defined.\n\nSome vague arguments mismatch with the math-heavy writing style of the paper. For example, in the abstract, the authors emphasized \n\n> \"Integrating surrogate data can significantly reduce the test error on the original distribution. Surprisingly, this can happen even when the surrogate data is unrelated to the original ones. \"\n\nThe word 'unrelated' makes it a very strong argument. Surprisingly, I did not find it clearly defined in the main paper.\n\nActually, I don't think learning with unrelated surrogate data will always reduce model loss. Imagine training the model with mislabeled data. It will only degrade the validation performance. My guess is what the authors are trying to say might be \"training the model with not directly related data may also contribute positively due to regularization effect or knowledge transfer\". But this is well-known to the community. So I am confused.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We agree: we will expand the related-work discussion.\n\nAt the same time, part of this criticism is unfair. The distinction between “technical report” and “research paper” depends on the subcommunity. Mathematical papers tend to be more technical and less focused on positioning.  \n\n\nReview: ‘The word 'unrelated' makes it a very strong argument. Surprisingly, I did not find it clearly defined in the main paper.’\n\nResponse: The abstract and introduction try to convey our results in an intuitive way.\nThis sentence means that the surrogate data have a distribution very different from the original one.\nIn Section 2 we give a very simple mathematical example in which this claim is formalized and formally proven. Subsequent sections provide more sophisticated examples.\n\nSo the claim is both rigorously stated two pages after it is introduced, rigorously proven, and, judging from the referee report, very surprising.\n\nReview: Actually, I don't think learning with unrelated surrogate data will always reduce model loss.\n \nResponse: Actually Section 2 already gives a toy example in which adding surrogate data (and proper data-driven weighting) will yield a non-zero improvement, regardless of the distance between original and surrogate data! \n\nIn practice, this improvement can be negligibly small, when the distribution of surrogate data is very different from the original one. However, we find that the improvement is often non-negligible in practice. Further, a data-driven weighing strategy yields significant improvements even in cases where the common practice of naively mixing real and surrogate data with equal weight hurts rather than helps.\n \nReview:  The work does not have a standalone discussion for its limitations.\n\nResponse: Section 5 is a standalone discussion of the paper’s limitations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pdrUrYyL71", "reviewer": "Reviewer_Fovm", "summary": "This study addresses the challenge of computational scenarios where true data are scarce, and surrogate data are available to assist in building statistical models. The authors provide novel theoretical and empirical insights demonstrating that training models with both true and surrogate data, with appropriated selected coefficient $\\alpha$, could improve empirical risk reduction on test sets, even when the surrogate data is unrelated.", "questions": "- Equation (4) appears to lack a variable that quantifies the \"closeness\" between the surrogate and true data, such as the $\\gamma$ parameter defined in Theorem 3. Could the authors clarify the absence of such a metric in this model formulation?\n\nNote: I realize that $R_\\text{su}^\\text{ex}(\\infty)$ can represent the deviation from surrogate data to true data (indirectly). But let's keep this in case others share similar questions.", "limitations": "The limitation is included and no negative societal impact should be considered.", "rating": 9, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The scaling law claim is clearly presented and supported by theoretical results across various contexts, including Gaussian sequence model, non-parametric regression, low-dimensional asymptotic, and high-dimensional linear regression. This breadth of application suggests the claim's novelty and relevance.\n\n- The link established between training with surrogate data and the Stein paradox is both novel and intriguing, potentially influencing future work in data augmentation, multi-fidelity modeling, and out-of-distribution training.\n\n- Theoretical and empirical results are robust and well-articulated, lending significant credibility and depth to the findings.", "weaknesses": "- The discussion is limited to certain loss functions, specifically $\\lVert \\theta_\\ast - \\theta\\rVert^2$ and $\\lVert f_\\ast - f\\rVert^2_{L^2}$. Extending these results to include more common loss functions like L-1/L-inf norms or classification-driven losses could broaden the applicability of the findings.\n\n- Some notations are unclear or undefined, which could hinder understanding for readers unfamiliar with the symbols or concepts. For example, the symbol $\\asymp$ used in Line 152 and $\\Lambda(\\alpha)$ in Line 207 require definitions.\n\n- Some minor errors: \n    - Line 50 introduces the test error $R_\\text{test}$ but this subscript is not used in the following context, e.g. Equation (4). Please make sure the test error notations are consistent;\n    - Line 226: $R_\\ast=R(\\hat\\theta_{0, \\infty}(0))$ should be $R_\\ast=R(\\hat\\theta_{\\infty, 0}(0))$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Thanks for the positive feedback!\n\n2. We agree that studying classification and other losses would be an important next step.\n\n3. Thanks for pointing out the typos. We will correct them in the final version. We will also add the definitions of the symbols/concepts pointed out by the reviewer.\n\n4. The “closeness” enters implicitly in the first term on the right-hand side of Eq (4), namely the excess population error of training on (infinitely many) surrogate samples. In this formula we assume that the population error of training on (infinitely many) original  samples is 0. Hence this term is non-negative and equal to 0 if and only if surrogate distribution coincides with the original one.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J92zK59XDi", "reviewer": "Reviewer_3jgb", "summary": "This work investigates the effects of augmenting training datasets with lower quality data, under a weighted ERM scheme. It is shown that introducing this surrogate data to optimally weighted training can improve predictive performance on the original data distribution, as measured by test error, even when the surrogate data is unrelated to the dataset of interest. A corresponding scaling law is derived, giving insight into both the optimal weighting and the amount of surrogate data that can be used. Numerical examples are provided.", "questions": "Can you please address the above weaknesses?", "limitations": "While the phenomena is framed as analogous to a regularization technique, I do not feel that adequate care is taken to ensure that non-expert readers are not encouraged to add junk data to their training regimes without adequate care being taken.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is generally well written. \nA substantial numerical study is provided.", "weaknesses": "The theory of this paper is developed for a squared loss function. However, most of the numerics are done on classification tasks without mention of this discrepancy. This should be addressed, with justification provided if possible.\nThe role of the regularizer is barely mentioned, however it appears to play a crucial role in the relatively clean form of the relevant quantities given in the result statements. Analysis of sensitivity of the results to the regularization parameters would seem important and be appreciated. Further, growth estimates on its spectrum do not match the usual ridge regression setup (i.e. setting $\\Omega$ to be the identity) in a way that's independent of parameter dimension. This makes the assumption highly artificial, and somewhat redundant (compared to just using the largest eigenvalue), and impacts the statement of the theorems.\nClear discussion on the rationale for a scaling law and its specific form is not given.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We agree that it would be important to generalize our theory to classification loss. At the same time, a substantial number of results suggest that the two settings are not as different in high dimension. Among others\n1. Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai, Classification vs regression in overparameterized regimes: Does the loss function matter?, Journal of Machine Learning Research 22 (2021),\n2. Montanari, A., Ruan, F., Sohn, Y. and Yan, J., 2019. The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime.\n\nThese papers show that the risk of max margin classification is either identical (paper 1) or very similar (paper 2) to that of least square regression in high dimension.\nWe also would like to point out that some of our simulations are for square loss and some for classification loss. The results are qualitatively very similar. \n\nWe also agree that investigating the sensitivity to the regularization value is an important research question. At the same time, we would like to point out that:\n1. Theorems 1 and 2 keep holding (up to constants)  if the regularization parameter is changed by a multiplicative constant.\n2. Theorem 3 captures the entire dependence on the regularization parameter. We did not explore this dependence in the plots presented in the manuscript, but we can do it  straightforwardly (it is just a matter of changing a parameter in a function that evaluate the formula).\n\nThe setting in Section 3.1 is really motivated by nonparametric regression and other nonparametric settings (eg  the Sobolev regression problem of Section 3.2). In these cases, the theta_k are Fourier or wavelet coefficients and $\\Omega$ encodes the prior smoothness information about the signal. For instance, cubic smoothing splines correspond to the case $\\Omega_k$ proportional to $k^4$.\n\nThe reviewer mentions that the paper does not take “adequate care to ensure that non-expert readers” do not “add junk data to their training”.\nWe would like to emphasize that, to the contrary, this is exactly the problem that we address in our paper.  If the weight \\alpha is optimized through a consistent procedure (eg cross-validation), then the resulting learning procedure is very robust to the use of bad surrogate data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tAbDtm8eR9", "reviewer": "Reviewer_yPKz", "summary": "This work addresses the challenge of augmenting limited data with more accessible surrogate data to improve generalization. The authors proposed a weighted ERM approach for integrating surrogate data into training and analyzed its performance under various statistical models. It was shown that incorporating surrogate data provably reduces test error, even when the surrogate data is far from the original ones. The paper introduced a scaling law that predicts the optimal weighting scheme and the amount of surrogate data to use, supported by both theories and experiments across different domains.", "questions": "- In Eq (4), is there any reason for using $\\approx$ instead of $\\asymp$?\n- In Theorem 1 and 2, how to interpret $\\beta$ intuitively, the difficulty of the learning problem?\n- In the Sobolev class example (line 161), what's the particular reason for assuming $n=Q^d$? Is this essential or just for simplicity?", "limitations": "- Some limitations are listed in the Weaknesses section.\n- It would be helpful to provide more intuitions for Theorem 3 given the complexity of the notations and the distinct form of the result in the asymptotic regime (cf. the clear sample complexities in Theorems 1 and 2). Remark 3.3 on the connection with the scaling law provides some insights but doesn't seem to be clear enough.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The effect of weighted ERM on generalization is theoretically analyzed on various statistical models.\n- The discussion on the connection with the Stein paradox through the toy Gaussian mean estimation example is insightful and provides a nice intuition for why incorporating surrogate data can be beneficial.\n- The theoretical results are well-supported by experiments in different domains.", "weaknesses": "Weighted ERM is a simple and arguably the most intuitive way to incorporate surrogate data. This is a strength rather than a weakness on its own. However, given the ubiquity of such methods, the discussion on related works could be more comprehensive.\n- For example, in addition to synthetic data from generative models, data augmentations also seem to fall into the category of surrogate data considered in the paper. The results on sample complexities of data augmentation methods could be relevant. \n- Another potentially related topic is distributionally robust optimization. Despite the different optimization problems, the idea of finding a near-optimal weight for different groups of samples shares a similar high-level goal.\n- Instead of \"introducing weighted ERM\", the contribution of this work seems to be more about providing a theoretical underpinning for the effectiveness of weighted ERM with an insight on the choice of weight. This could be made clearer in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Essentially all weaknesses pointed by the referee concern literature review. \nWe will expand the comparison with literature.\n\nRegarding the specific suggestions provided:\n1. Data augmentation. We feel this is a significantly different setting. In data augmentation, the new data is typically obtained by perturbations/transformations of the original one. As such, the dependence between original and surrogate data need to be carefully modeled.\n2. Distributionally robust optimization. This is indeed an interesting connection that is worth exploring.\n3. We agree that  “introducing weighted ERM” is not accurate. We will replace it with “investigating weighted ERM” or similar.\n4. We agree that the content of Theorem 3 is not easy to parse. We will complement it with some simplified/approximate expressions that are easier to parse and interpret.\n\nQuestions:\n\nQ1: The symbol $\\asymp$ indicates equivalence up to (possibly large) multiplicative constants. In several of our mathematical examples (and in simulations) constants seem to be close to 1 (possibly with some small additive error). We use $\\approx$ to indicate that we interpret the scaling law in this stronger sense.   \n\nQ2: $\\beta$ is a scaling exponent that quantifies how the test error scales with the sample size. Scaling laws have been widely popular among AI practitioners and in those cases this exponent is fitted to empirical data. In these examples we compute the scaling exponent explicitly.\n\nQ3: Indeed, we assume $n=Q^d$ for mathematical simplicity. It is standard to assume such “regular designs” in nonparametric regression because this simplifies the analysis, without changing the final result.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the challenge of augmenting limited real data with surrogate data, particularly in scenarios where high-quality data is scarce or expensive to obtain. The authors propose a weighted empirical risk minimization (ERM) approach to integrate surrogate data into training processes. The primary contribution is the introduction of a new scaling law that optimizes the weighting scheme for combining real and surrogate data, validated both theoretically and empirically. The key finding is that integrating surrogate data, even when unrelated to the original data, can significantly reduce test error, a result linked to Stein's paradox.
The paper is technically sound, offering both theoretical analysis and empirical validation across various domains.

Reviewer XfMG gave the lowest score of 3, primarily citing concerns about the literature review, notation clarity, practical utility, and format. After reviewing the discussion chain, I believe these comments are not entirely fair.

Reviewer 3jgb gave the second-lowest score of 4, with concerns centered on the theory-simulation gap, the need for a more detailed discussion of regularization, and the assumptions underlying the theorems. I believe these issues can be addressed with minor revisions and should not prevent the paper from being published.

The other scores are 6, 8, and 9, very high indeed.
Decision: Accept (poster)