Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression
Authors: Yeichan Kim
Ilmun Kim
Seyoung Park
Venue: NeurIPS 2025 spotlight ()
Keywords: Transfer Learning
Benign Overfitting
Generalization
Minimum-Norm Interpolator
Primary area: general_machine_learning

Abstract: Transfer learning is a key component of modern machine learning, enhancing the performance of target tasks by leveraging diverse data sources. Simultaneously, overparameterized models such as the minimum-$\ell_2$-norm interpolator (MNI) in high-dimensional linear regression have garnered significant attention for their remarkable generalization capabilities, a property known as *benign overfitting*. Despite their individual importance, the intersection of transfer learning and MNI remains largely unexplored. Our research bridges this gap by proposing a novel two-step Transfer MNI approach and analyzing its trade-offs. We characterize its non-asymptotic excess risk and identify conditions under which it outperforms the target-only MNI. Our analysis reveals *free-lunch* covariate shift regimes, where leveraging heterogeneous data yields the benefit of knowledge transfer at limited cost. To operationalize our findings, we develop a data-driven procedure to detect *informative* sources and introduce an ensemble method incorporating multiple informative Transfer MNIs. Finite-sample experiments demonstrate the robustness of our methods to model and data heterogeneity, confirming their advantage.

Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression
Yeichan Kim
Immu Kim
Seyoung Park" Yonsei University ishspsy@yonsei. ac. kr
Yonsei University klmyeic980yonsei.
KAIST
1lmunk0kaist.ac.u
Abstract
Transfer learning is a key component of modern machine learning, enhancing the performance of target tasks by leveraging diverse data sources. Simultaneously, overparameterized models such as the minimum-6z-norn interpolator (MNI) high-dimensional linear regression have gamered significant attention for their remarkable generalization capabilities, a property known as benign overfutting Despite their individual importance, the intersection of transfer learning and MNI remains largely unexplored. Our research bridges this gap by proposing a novel two-step Transfer MNI approach and analyzing trade-offs. We characterize its non-asymptotic excess risk and identify conditions under which it outperforms the target-only MNI. Our analysis reveals free-lunch covariate shift regimes, where leveraging heterogeneous data yields the benefit of knowledge transfer at limited cost. To operationalize our findings, we develop data-driven procedure to detect informative sources and introduce an ensemble method incorporating multiple informative Transfer MNIs. Finite-sample experiments demonstrate the robustness of our methods to model and data heterogeneity, confirming their advantage.
Introduction
Transfer learning [49, 67, 72, 21] is scheme that aims improve the performance of a learning task of interest, namely the target task, by leveraging knowledge acquired from related, but possibly different, source tasks. As modern datasets grow in size and heterogeneity, the seamless integration of diverse sources of information has become increasingly critical, positioning transfer learning as valuable approach. Its success is particularly evident in -inhe-immnsional regression, as there has been recent surge of research demonstrating its advantage in tasks including, but not limited to, LASSO [61] and its variants [6. 34, 20, 55], generalized linear models [60, 35], and nonparametric regression [40, 64, 38, 9].
While these transfer learning methods rely explicit regularization [53, 17, 18] address poor generalization arising from overfitting in high-dimensional regimes, contemporary deep learning methods have challenged the conventional wisdom of bias-variance trade-off framework. In partieular, certain interpolators that achieve zero training erro have been found to generalize remarkably well to unseen data, despite the absence of any explicit regularization. This surprising phenomenon of benign overfitting [7, 8, 5, 4, 44, 54] upends the traditional notion of model capacity and generalization. Given the prevalence of overparameterized models modern machine learning, this stark contrast raises an intriguing question:
Can transfer learning further enhance the impressive out-of-samppe generalization capabilities of such interpolators in high-dimensional linear regression?
"Co-corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Related work. Motivated by the empirical success of overparameterized deep learning models [70, 23, 46]. a substantial body of research has sought to understand why benign overfitting occurs. focusing on linear regression as a tractable setting for theoretical investigation. The theoretical foundation of our research, along with that of subsequent studies, has been established by Bartlett et al. [5], who studied the non-asympiotic excess risk of the minimum-62-norm interpolator (MNI) with n < p (i.e., dimension exceeds sample size). They demonstrated that if the covariance eigenvalues decay rapidly up to some "intrinsic" dimension k- < (Assumption 2) and yet. the remaining p - k* components have effective ranks (Definition 1) greater than n, then the MNI trained on noisy data can achieve vanishing excess risk by capturing signal from the leading k* high-spectrum components and dispersing noise over the many low-spectrum tail components, as the tails span an "effective space" larger than the noise dimension n. In this situation, the MNI behaves similarly to ridge regression, exhibiting the effect known as implicit regularization [28]. Hastie et al. [19] explored the asymptotic excess risk of MNI as p/n 7 E (0, based random matrix theory [2). and Tsigler and Bartlett [62] extended this ridgeless interpolator to benign overfitting in ridge regression. Zhou al. [71] and Koehler al. [29] interpreted the benign overfitting of MNI under the lens of uniform convergence. Additionally, Muthukumar et al. [45] popularized the notion of harmless interpolation, demonstrating that under certain conditions, the MNI trained on noisy data does not severely degrade generalization, as estimation variance vanishes with increasing n/n.
In parallel, distribution shift in transfer learning for regression has been characterized by the two main pillars: (1) model shift, where the conditional distribution of response given covariates differs between the target and source tasks; (2) covariate shift, where the marginal distribution of covariates varies. Wang and Schneider [66] established generalization bound under model shift, provided that the shift is smooth conditional on covariates. Recently, Tahir al. [57] quantified model shift in high-dimensional regression by the cosine similarity between target and source model coefficients, showing that transfer performance depends critically on this alignment. Covariate shift has been studied along several lines, such as minimax risk formulations [32] and importance weighting [12], among others. Recent analyses of linear models under model and covariate shifts have examined optimal ridge regularization parameter, which can be negative [50].
Despite the potential significance of the aforementioned question, its exploration within transfer learning for MNIs remains severely limited. Mallinar et al. [42] studied an -utoo--distribution (OOD) [39] setting', where the MNI is trained exclusively on source (in-distribution, ID) data and tested on the target distribution. They provided conditions where a beneficial covariate shift yields lower OOD risk than ID risk, focusing shifts in the eigenvalues of spiked covariances [24] and degree of overparameterization. However, their approach does not use target samples during training, even though least limited target data are often available. In contrast, Wu et al. [69] proposed an online stochastic gradient descent (SGD)² algorithm pre-trained with the source and then fine- tuned on the target, encompassing OOD settings by allowing no fine-tuning. They showed that pre-training with O(n²) can be comparable to t-taaget-supervised learning under certain covariate shifts. Despite using target data, their work, like Mallinar et al. [42]. did not examine model shift, and its experiments are primarily focused on underparameterized regimes. Notably, Song et al. [56] developed the poOIed-MNI, explicitly accounting for model shift. They derived its non-asymptotic excess risk and proposed a data-driven estimate for the optimal target sample size under model shift. However, pooling multiple sources can be susceptible to distribution shifts, as shown in our numerical evaluations (Section 5). Additional literature review is deferred to Appendix A, which discusses minimum-norm interpolators beyond the Euclidean norm.
Summary of contributions. Our study bridges the gap by investigating how transfer learning can enhance the generalization of target-only MNI through a novel knowledge transfer scheme. The main contributions are as follows.
We propose a novel two-step Transfer MNI: pre-train a source-only MNI and then fine-tune it by interpolating target data while "staying close" to the pre-trained model. Under model shift with isotropic covariates, we identify when Transfer MNI outperforms target-only MNI, specify the optimal transfer size, and quantify the maximal improvement in excess risk. We provide non-ssympto excess risk bounds under both model and covariate shifts when each single-task
I 'Concureetlyy with our work, Tang et al. [59] analyze OOD generalization of ridge regression and MNI and show that principal component regression can attain a faster convergence rate under certain conditions. PSD initialized zero with suitable learning rates is implicitly biased toward the MNI [68].

---

MNI is benignly overfitted. We further uncover free- free-lunch covariate shifts that alleviate the cost of knowledge transfer while preserving its generalization gain for Transfer MNI.
Based on a data-driven procedure for detecting informative sources inducing positive transfer, propose an ensemble method aggregating multiple informative Transfer MNIs with data-adaptive weights determined by source informativeness.
Finite-sample experiments demonstrate robustness to distribution shifts and superior performance over transfer baselines, including the pooled-MNI [56] and SGD-based transfer [69], thereby confirming the empirical advantage of our approach under overaaramcterization
Notation. Bold upper- (c.g., M and A) and lower-case (c.g., v and (3) letters denote matrices and vectors, respectively, with |M and ||v|| denoting the operator and fz-norms. For E N. define the sets [Q] == {1.2 11,22...,2 Q} and [Q]o ==
.2....QQ Q}. Write a V = max(a, b) and 0, let [x] denote the floor function of x. Denote by (A4) the indicator A b = min b). For function of an event A. For sequences faz ineN and (bn) Inn) write a, = O(ba). an S bn or
on 2 On if lan/bn, /bl S for some constant e > and all sufficiently large n; write an bn if an O(b,) and bn = 0 (ax); write an o(ba), an < bn. or bn >> On if |n/bal + 0 as n + co.
2 Preliminaries
Task setup. Consider an overparameterized linear regression setting involving one target task and
Q source tasks. We have access to the training datasets {(X(9) } each with ny samples of p-dimensional covariate generated from the following well-specified linear model, with 0 corresponding to the target:
(4) x x(1) + e o (Q).
Here, y(s) E RR is the response vector; X (() E R' is the design matrix with i-th row (sample)
x(9)) E RP; g(a) E IRP is the fixed model coefficient; c(9) R is the random noise. Provided
ng I p. the MNI trained on each dataset is uniquely given by
3 = arg min [I[3] x (1) 3 = y( = T (0) X (
EE
[Q/o,
(2)
where M E pdxm denotes the Moore Penroee inverse [51] of M E mxd Appendix C.I provides proof of its uniqueness and minimality in fn-norm. The MNIs 3 and 3 correspond to our target and (g-th) source tasks, respectively.
Distribution shift. For well-specified linear models, model shift is attributed the model contrast [34] defined as
8 (0) = 3( ( q E [Q].
On the other hand, covariate shift is characterized by discrepancies in the covariance structure Σ = Exx While Kausik et al. [27], LeJeune et al. [33], Mallinar et al. [42] assumed simultaneous diagonalizability, which gives spectral decompositions (o) VA VT and ( VA T for some common orthogonal matrix V E RP*P, Assumption I and our numerical studies (Section 5) do not necessarily impose such a shared eigenbasis.
In Assumptions and 2 and Definitions and 2, the index g applies for all E [Q]o-
Assumption 1. Let the design matrices X (x 19=0 and random noises & (c(1)) j be mutually independent, and the following holds.
Each design is of the form X(a) = Z -Z(( (Z(0)112 where the rows 7(4) R are i.i.d. V
sub-Gaussian vectors? with i.i.d. mean-zero, unit-variance components, and (4) RpxP is
deterministic, symmetric positive definite with eigenvalues X 2 > 2 .... 2 A >
Each [0() (E is bounded above by universal constant Czio 0 with Cz. = 1.
Each noise c(1) has i.i.d. mean-zero components with finite variance o 0.
'A random vector z E IRd is Ssuu-Gaussiinn if for any fixed unit vector E 18" the random variable u 2 is
(with variance proxy v²), i.e., [exp (fu (z Eτ))] I exp (t2 /² /2) for all t R.
3

---

The sub- Gaussianity of covariates is standard in recent transfer learning research [32. 12, 42, 20]. When ( and ( are simultancously diagonalizable, the condition |> -- = O(1)
reduces upper-bounded spectral ratios, i.e., max 2 /x Q(1, in the multiplicative
spectral shift in Mallinar et al. [42]. Lastly, a finite second moment of the noise suffices as the
formulation of mean squared loss in (5) only requires that each (E(9)((9) is well-defined.
Effective ranks. We additionally introduce the key condition underpinning the benign overfitting of MNI, characterized by the effective ranks [30, 5] of (
Definition 1. If x 0 0, the effective ranks of (9) defined 1
Σjsx (9)
rx()())
Rx(2)a)
EjSn (A/O))²
1+11
Assumption 2. There exists universal constants by Cip 2 such that the minimal index
k min { 0: (x:x( byng
well-defined with 0 S kg S /
Under Assumption 2, Bartlett et al. [5] formalized the benign overfitting of MNI as follows.
Definition 2. The single-tase task MNI ((?) with covariance (1)) and nl < p.p. is benign if
To ((()
kg
ngg
lim
lim
lim
(E))) = 0.
Rlg
Rig
3 Single-Source Transfer Task and Out-of-Sample Generalization
We propose single-source source transfer method the form of late-fusion [56], where the q-th source task is learned independently and then integrated with the target task. In the first step, we pre-train the q-th source-only MNI 3 The two-step Transfer MNI (TM) ((4) then fine-tunes by interpolating the target dataset while minimizing the Fuclidean distance from the pre-trained A that is,
( == arg minger= {ll | x (0) B = y q E [Q]-
The TM estimate admits the following interpretable decomposition structure that clarifies the knowl- edge transfer mechanism (see Appendix C.2 for the derivation):
(Ip H (0) A
transfer fask
target task
late: fusion knowleilge transfer
Here, ((4) ( [ 1 x ( x (4) denotes the orthogonal projection onto the design row (1))
space S == span (1) | for g E [Q): conversely, Ip - (9) is the orthogonal projection onto the
null space S such that R = Sg i S Denoting by (v)s, = ())), v and (V)s, = (I, H)))
the projections of vector E R2 onto the target row and null spaces respectively, we obtain
(3
(()
(B%) = (I, )
(4)
S.
That is, the fine-tuning step for TM retains target-learned signal in the span of no target samples (i.e., So) where benign overfitting ensures high prediction accuracy for the target-only MNI, while transferring source information only into the null space S.1 where the target samples provide no information with (3"") 5 Op-
Given this "retain wwt-t-plussseteeeeeee transfer" mechanism exhibited by TM, we next analyze its generalization behaviors by comparing the excess risks of target and transfer tasks. Suppose are given an out-of-sample target instance Xa IRP The excess risk of an estimate å = (x( measured on the target distribution is defined by the following conditional mean squared loss:
R( == Exx,,) (xo 3 x (00)
x
Ee (3 3(0) (0) (} -3(0)) | x]. (5)

---

The canonical bias varianee decomposition shows that the excess risk is the sum of the (squared) bias B and variance i.e. R(B) B(G)) + v(š). The bias / and variance V of the target-only MNI are obtained by Hastie et al. [19] as follows:
V = de Ir (0))) )("),
B = (0) II (0) ((0)
(6)
11
where we write II (I, H D H and Σ (1/ng) ( X (9) for [Q]o- The following lemma extends the cas-variance decomposition to our proposed TM estimate.
(9)T
Lemma 1. Under the mutual independence and mean-zero condition of (X,8) E) in Assumption the excess risk of the TM estimate is the sum of bias B[9) and variance Vllg such that
B = =0 (Ip HH II (L. H()) 0 H
H (1 g(4)
25 H( (0) (I, H
o Tr Tr(Sltt (1) (T)
(9)) TM
(0)
ng
vl0 (variance inflation)
Remark (Bias reduction versus variance inflation). The variance inflation vL.) in Lemma is positive almost surely, so the knowledge transfer always requires a higher variance as its cost. If 3 reduces its estimation bias enough to outweigh the variance inflation, it altains a lower excess risk than the larget-only MNI, referred to as positive transfer.
3.1 Model Shift under Isotropic Covariates
We analyze the effect model shift under the isotropic case where (0))
((0) = Ip. Although not "benign," this case ensures at least "harmless" [45] performance for the single-task MNI, making it a worthwhile subject of investigation. In addition, our analysis provides an important insight: the dynamics between bias reduction and variance inflation (Remark 1) is determined by the interplay among p. ny, shift-to-signal signal ratio (SSR), and sognal-to-noise ratio (SNR), where SSR and SNR are defined for each q-th source
| 2
SNR, = ||300|1 o > O,
SSR, = ||300]]2 2 O,
QE 1E)
provided ((0) Op.
Theorem Under Assumption 1, and further assuming that (Z Z have i.i.d. standard Gaussian entries with p > (no + 1) V (ng + 1) and [0) E Ip. the expected bias and
variance (expectation over X) ) of the target-only MNI and TM estimate as follows:
ExB²0 p- T10
113(00
ExBla
p n0
p
Tly
3
ng (1880011))
p
p
p
P
Ex
agno
E ExVid
(P
n1
2ng
Ex (x)
- (no + 1)¹
p
p
(ng
From Theorem I, we observe trade-off' between p and n.. If [100 || []. increasing n., further reduces the bias of TM. An increase in p mitigates the variance inflation, but only at the expense of compromising this bias reduction effect. Based Theorem
formalize the regime where TM is expected to outperform the target-only MNI and specify the optimal transfer size maximizing the improvement in expected excess risk E
(0) R()
Corollary 1. Under the setup in Theorem 1, the TM estimate satisfies
F R()) < R (0)
+
SSR, <1 and SNR, - SSR,,)
p
p (n, + 1)
5

---

The improvement in expected excess risk (mn)) :- Ex (0) E R fanction of mig is strictly concave on ni [1,p- 1). If and only if SSR, < I and SNR, SSR,) 2 (p 2)
p(p-1) the optimal
transfer size ni maximizing ((mg) exists and equals ni = with A (m)))
V
plp E [1.P 1),
P-no
(n2)9(-SSR..))
|| > 0 being the maximal improvement.
plp-1)
In Corollary 1, negative transfer occurs when model shift dominates the signal with SSR, 2 1. The improvement ((n)) strictly increases in n, up to the optimal threshold n but strictly decreases for ny ng: that is, transferring more source samples helps only up to nig and beyond n. it always degrades the efficacy of knowledge transfer. While Song et al. [56] also investigated the same isotropic Gaussian setting under model shift, only Corollary 1 provides necessary and sufficient conditions for positive transfer and identifies the maximal improvement A (m,.).
3.2 Convergence under Benign Covariates
Taking both model and covariate shifts into account, we now consider general sub- Gaussian covariates satisfying Assumption 2 and analyze non-asymptotic excess risk bounds and their implication. Theorem 2. Let Assumptions and hold. For each E [Q]o and any 2 log(2), define
(t)+) [1)) +t
To To(
1hg Rx;
Tg
ng
Tly
nl
With probability at least 1 2e 2 and 4e " respectively, the biases are bounded above by
B < (6) (0) 3 ( 112
B Z (n(n) |||| 8 1 112 + (n)(C))))))) |
Furthermore, there exist universal constants co, Cq 2 such that with probability at least 1 Te Pyl 5 the variance inflation of the TM estimate is bounded above by
v(9) S v -1
and with probability at least - 10e -naca the excess risk of the TM estimate is bounded below by R To.
where (0) x To the same high-probabilityy event lower-bounding R
The bias dynamics in Theorem mirrors that in the isotropic case. If ro(E(()) < no. which is sufficient for vanishing target-only bias, and ro (()) To (E))), B also vanishes. Moreover, it can achieve faster convergence if || 112 < [| 2 and ng fg. As for variance, the target- and source-only variances are within a constant factor Y, and Y, respectively (see Corollary in Appendix), and hence the terms vanish when each single-task is benign. The upper bound on v(() a product of the two "benign" terms Y, and v'n and the reciprocal of eigenvalue that reflects the lack of simultaneous diagonalizabiiity in Assumption 1. While the reciprocal may loosen the bound, numerical experiments (Section 5) show rapid decay in variances as p no V ng- Exploring tractable covariance structure to refine this bound is left for future work. Finally, the lower bound on (9)) follows from Lemma 1, where the TM variance is smaller than the target-only variance; the bias reduction effect cannot further improve the lower bound.
3.3 Free-Lunch Covariate Shift
In light of the invariance of spectral decay rates to uniform upscaling, we propose type of covariate shift that yields a lower TM excess risk than without any covariate shift. To proceed, define the following minimal index for ()) satisfying Assumption which is always well-defined:
min {k < 10) = (0) {
We may expect 7¹ 2s kj for some benign covariance structures, where Kn is specified in Assumption 2 for ()); the beginning of Appendix C.7 illustrates a case where r" < and T" < no indeed.

---

Corollary (Free-lunch covariate shift). For each QQo, let the spectral decomposition of I be E( V(() A ( where V(a) R is orthogonal and A (o) Diag(A) ....., 21
Suppose A(a) aA (0) for some a > 1, and under (A), the leading T' eigenvector pairs in
(Vl9, V align; under (B), eigenvector pairs fully align with V( V i.e,, Σ = a
Compared to the homogeneous case Σ( = I0), the following holds for each covariate shift case:
(A) The upper bound on Bid remains identical up to a constant factor independent of a, while the upper bound on ( is multiplied by x- (i.e., reduced by a factor of ot).
(B) The exact bias remains identical, while the exact variance inflation is multiplied by a-!
The covariate shift (A) in Corollary shows that as long as each 2 is uniformly upscaled from X preserving their decay rates, any misalignment in "noisy" eigendirections between Σ(a) and ( beyond the leading -* high-signal components does not affect the convergence rate of bias, while still promoting faster convergence in variance inflation. With all p T* eigenvector pairs aligning between { and ()), covariate shift (B) offers greater advantage than (A) by specifying the exact impact on the bias and variance inflation, rather than on their upper bounds.
Remark 2 (Relaxed free-lunch condition). The T" allignmmennt condition for (A) in Corollary can be relaxed: even if not all of leading T' source eigenvectors align with the target counterparts, we can still achieve the same free-rruluume free-lunch effect as in (A) whenever
v (0)
where v E RRxx' (resp. v(9) E R comprises the leading T" eigenvectors in V (resp.
V(i)) as specified in Corollary 2.
Informative Multi-Source Transfer Task
In this section, propose transfer task that incorporates multiple sources identified informative, those that induce positive transfer. The index set of informative sources is given by
[Q] : R R0 <0}.
= E
(8)
which, however, is unknown in practice. Hence, it of crucial interest in transfer learning to develop data-driven procedure for detecting informative sources. Inspired by Tian and Feng [60], utilize the K-fold cross-validation (CV) [1] to detect source transferability by comparing a "proxy" of excess risks.
First, partition the target dataset into K folds of equal size, each denoted by (X (014) y (0)A) for k E [K]; a common choice suggests K 5 [17]. At each training step, we use the left-out folds
(X () y (0] ;= {(X (0)M) y (0)4+) k== (X y to train an estimate @l-A] and then
evaluate the squared loss on the k-th fold given by
g (3I-
X (00 X00)11
na/K
We repeat this across all K folds to obtain the terminal CV loss Z(0) = ΣK Zil ( (BI-A1)
/I)
which estimates the prediction risk R(3) + oă. We then estimate the oracle set (8) by
1 : {q E [Q] ((a%) [(( < D)))
(9)
with some detection threshold ()) > 0 to be specified later that depends on the target CV loss.
If î is non-empty, we train the TM estimate ( for each E î and form a weighted linear combination of ((4 Each weight w; is initialized by the inverse of the CV loss ((م and then 1. which serves as a data- data-adeptive measure of source informativennss. This allows us to leverage Ī sources for knowledge transfer, which we name Informative- Weighted Transfer MNI (WTM). The entire procedure, from detecting informative sources to computing the WTM estimate, is outlined in Algorithm I (Appendix D), specifying ()) in the set (9).
normalized so that ΣiET
7

---

Numerical Experiments
We evaluate the sinite-sample performance of our proposed TM and WTM estimates by comparing them to the target-only MNI as baseline. For the WTM estimate, set K 5 and En = 1/2 in Algorithm Additionally, we test the pooled-MNI (PM) proposed by Song et al. [56] as a benchmark transfer task, which has an analytical solution:
BpM :- arg min BER" { || X X(9) ( = y(4), Vq E [Q]}
To consider a tine-tuning-basee benchmark, we also test SGD, y [69] each pre-trained with the q-th source, adjusting the initial learning rate from their sourcecode ensure convergence.
Each setup is replicated over 50 independent simulations, and the average excess risk over the 50 simulations is plotted against each value of p {300, 400, ..... 1000}. The mutually independent noise has i.i.d. mean-zero Gaussian components with common variance x² = 1. We adopt the para- metric configurations described below to simulate distribution shifts. Since only {(Z()) e(o)) }g=0 in Assumption are subject randomness in our setup, all other parameters are generated with fixed seed across the 50 simulations to ensure their deterministic nature; see Appendix F.1 for details.
Model shift. Let ((0) 51/22 (0) and 8(4) = (SSR, S) for 0 and q E [Q]. where each (9) is randomly generated from the p-dimensional unit sphere SP-1 without any structural assumption the coefficients, e.g., sparsity. We write SSR (SSR), ,,,
y= mutually independent and have i..d. standard Gaussian entries. In Section 5.1 with Q = 3 sources, the "benign" spectral matrices
Covariate shift. Let ( ( ((0)) where (Z(4) Q
(z) Diag (A (A(a X are as follows: (0) 15j log ((+1)ee
A1 15j 2.2
15j (1-)oo((m)))))))))) a = 15 1 = 1) + [( > 1)8 140 cos (fx/(--1)
where we set
1+82 cos (#/(++1)
3 = 3/2, = 1/2, and ε = 10 While we always fix the target covariance diagonal so that [0) (0), under covariate shift, we assign for each source covariance the spectral decomposition ( = ( where (V()) g== are independently sampled from the orthogonal group test the robustness of our methods to covariate shift without simultaneous diagonalizability, as discussed prior to Assumption 1. We evaluate the effect of free-lunch covariate shift (Corollary 2) by adjusting the source covariances by (aA (9) V with > 1, so we do not alter any source eigendirection align with the target. In Section 5.2, we set Q and ()) = [1) = ( Ip. and under free-lunch covariate shift, (1) = (2) alp with 1.
Op (Q MPPP. Q QQ Ip By doing
5.1 Benign Overfitting Experiments
Baseline
TM1
TM2
TM3
WTM
PM
SCD1
SGD2
SCD3
(a) SSR (0.0.3.0.6)
(b) common SSR 0.3
(c) common SSR 0.3,
Figure 1: Let no 25 and n1 n2 ng 75 for overfitting with S 500. Figures (b) and (c) incorporate covariate shifts as detailed, with (c) additionally benefiting from the "free-lunchh effect.
'Accrrding to Bartlett et al. [5], A (0) and A (1) are benign if and only if 3 > and E (0, 1), and A (3) is benign if and only if p > na. and pc < ns.

---

While 3PM can outperform our estimates when it only learns source data without any distribution shift from the target (see Appendix 1, the aggregate impact of distribution shifts for 9pst is catastrophic throughout Fig. On the contrary, in Fig. (.(,), each Aist remains robust, outperforming the target- only MNI even under severe model shift (SSR = 0.6). Notably, the ensemble fWTM outperforms each individual ((2), validating our data-adaptive weighting scheme based on CV losses.
Figures 1.(b) and 1.(c) additionally introduce covariate shift in the broadest sense, not only the spectrum of each ( differs from that of " but also all eigenvectors of ( arbitrarily misalign with those of X As a result, 3 evidently suffers negative transfer Fig. 1.(b); nevertheless, PWTM efficiently leverages only informative TM estimates by filtering out 3 consistently outper- forming all competitors. When the free-lunch factor > 1 is additionally applied in Fig. 1.(c), G achieves significantly faster convergence and now performs comparably to the target-only MNI.
5.2 Harmless Interpolation Experiments
Baseline
TM1
TM2
WTM
PM
SGD1
SGD2
18
18
(a) common SSR 0.4
(b) common SSR 0.4,
(c) common SSR 0.4,
Figure 2: Let ng n1 = 50, and n2 = Jnj]. the optimal transfer size for ( with = 10. Figure (c) adjusts ná in Corollary via the modified signal-to-noise ratio SNR, := (0) /2 which allows ( to leverage even more source samples than the original [nā]] in (a) and (b).
In Fig. 2, the target-only MNI with isotropic covariance demonstrates harmless interpolation [45], with its excess risk stcadily converging to [] 12 = 10. Therefore, the primary evaluation is whether each transfer task can prevent, or least delay, its excess risk from converging to 10. In Fig. 2.(a), despite considerable model shift (SSR 0.4) and sub-optimal transfer size, / slightly outperforms 3 While BPM and SGD, eventually surpass the baseline as p grows, they still lag behind ((2) by significant margin. Notably, even though (لا performs worse than ( (while still inducing positive transfer), their ensemble Bwrs substantially further reduces the excess risk.
Our methods benefit from the free-lunch covariate shift in Fig. 2.(b). However, in this scenario, the transfer size [n]] is longer optimal because the variance inflation T' is reduced by factor of a, essentially reducing the source noise level from o3 = a² to of = a2/a. Thus, transferring even more than |nž] samples becomes advantageous. Taking this into account, we use SNR, == a (0)2 which results in new optimal transfer size that depends
SNR,, and is larger than [nš]] used in Fig. (b(a)-(b). By incorporating this new optimum, ((2) and prrT in Fig. 2.(c) further outperform their counterparts in Fig. 2.b We present in Appendix E.2 additional experiment results under model shift with isotropic covariates, where the target sample size no optimized for 3px as identified by Song et al. [56]. To enhance the clarity of plotted figures, we separately report in Appendix F.2 the empirical mean and standard deviation of excess risks computed over the 50 independent simulations.
Conclusion and Future Work
We propose a novel two-step Transfer MNI and a data-adaptive ensemble aggregating multiple informative Transfer MNIs. These methods address the posed question highlighting the underexplored nature of transfer learning for MNIs, supported by both theoretical characterizations and numerical

---

validations under various distribution shift conditions. We further identify free-lunch covariate shift regimes for Transfer MNI where the trade-off between bias reduction and variance inflation is neutralized, allowing us to "cheery-pick"" the benefit of knowledge transfer at limited cost.
Consistency of informative source detection. Beyond the scope of this paper, we aim provide a theoretical foundation for the empirical success of the WTM estimate prominently demonstrated throughout our experiments. Specifically, we aim to establish the consistency of informative source detection via cross-validation in Algorithm 1 by proving that the event I I holds with high probability, where I is the oracle set in (8) and Z is its CV-driven estimate in (9). This agenda connects to the literature on source transferabilityy detection, a pivotal aspect of transfer learning research. While classification settings are well studied [3, 47, 58, 22]. less is known for regression. Recently, Nguyen et al. [48] introduced linear MSE as regression transferability metric, and Tian and Feng [60] established cross-validation consistency for transfer generalized linear models. However, both analyses rely on explicit regularization, whereas our setting features benign overfitting with implicit regularization. Establishing consistency of transferability detection in this regime remains open and would meaningfully advance the relevant literature.
Extension to minimum-RRie RKHS--orrm norm interpolator. Another promising direction is to analyze the extension of Transfer MNI from finit-ddimensional linear regression to nonlinear, dimensoaall regression in a reproducing kernel Hilbert space (RKHS) [53] via the minimum RKRRS-KKS---- norm in- terpolator, including both fixed kernels and network- induced kernels such the Neural Tangent Kernel (NTK) [23]. Extensive literature has explored benign overfitting in single-task cases; see Appendix A for detailed review. However, to the best of our knowledge, transfer learning for the minimum- R WKKS-nom interpolator beyond OOD settings still remains underexplored.
For the RKHS H = Hg associated with a positive semi-definite kernel K X x + R and norm || ll,, let fas and f(()) be the minimum RKHS norm interpolators trained on the target and q-th source datasets, respectively; their analytical forms with empirical kernel (Gram) matrices K(g)= E 2",x", for q [Q]o are well-known, where the (i,j)-th entry is k!4) K K(x," (1) ( This naturally motivates the RKHS analogue of Transfer MNI:
Alq) == arg migmiineee SEN {11s- ((4) 1| (xx y; (0) Vi E [mol], E [Q),
A(g) i... interpolates the target dataset [(x y; (0) ) while minimizing the RKHS-norm distance from the pre-trained fl.). With the target evaluation operator Es H - Ro and its adjoint E R + H such that (Fos); = _(x,")), Eo = k(O), and f = = the projection operator Po onto the target-induced span Su,u == span [((,xx
given by Po = ( Fg. The operator acts as the RKHS counterpart to the projection matrix H for linear regression in that P = Po and Pd = Po (i.e., self-adjoint and idempotent). Hence, the analytical form of Transfer MNI in Equation (3) systematically extends to provide the unique closed form of A(a). JM-
A( f(0) + (In Po)f(a)
where TH is the identity operator. Indeed, the RKHS TM estimate ( fully inherits the "retain-plus- transfer" mechanism in (4): it matches 2(0) on So.H and the fine-tuning step transfers knowledge learned by f(g) only into the complement space, i.e., - Po) Po) = (Ix Po) f(4).
Given the above formulation, it arises an important direction analyze the generalization gains of RKHS Transfer MNI with respect to factors that govern benign overfitting in an RKHS-such as the ground truth function class, kernel type, ambient input dimension, and sample size-as well as distribution shifts in the ground truth function and covariates between target and source tasks. We anticipate that our proposed scheme can serve as versatile baseline for further studies on transfer learning with implicitly regularized minimum-nome norm interpolators that span a broad class of norms and, on a separate axis, extend beyond linear models, encompassing fixed RKHS kernels, NT'K-type network- indduced kernels, and data- aata-adeptive kernels from deep representation learning.
10

---

Acknowledgment
The authors are grateful to the anonymous reviewers for their constructive comments and suggestions, which helped improve the quality of this paper. Yeichan Kim and Ilmun Kim acknowledge support from the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2022RI A4A 1033384), and the Korea government (MSIT) RS-2023-002 1073. Seyoung Park's 's work was supported by the National Research Foundation of Korea (NRF) grant funded by the MSIT (No. RS 05-2025-005517999 and by the Yonsei University Research Fund of 2025-22-0071.
References
[1] David M. Allen. The relationship between variable selection and data agumentation and method for prediction. Technometrics, 16(1):125-127, 1974.
[2] Zhidong Bai and Jack W. Silverstein. Spectral Analysis of Large Dimensional Random Matrices. Springer Series in Statisties. Springer New York, 2009.
[3] Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng. Amir Zamir, and Leonidas Guibas. An information-theone approach to transferability task transfer learning. In 20/9 IEEE International Conference on Image Processing (ICIP), pages 2309-2313, 2019.
[4] Peter L.. Bartlett and Philip M. Long. Failures model-dependen generalization bounds for least-norm interpolation. Journal of Machine Learning Research, 22(204):1-15, 2022
[5] Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):300633300770 2020.
[6] Hamsa Bastani. Predicting with proxies: Transfer learning in high dimension. Management Science, (7(5):2964-2998) 2021.
[7] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias variance trade-off. Proceedings of the National Academy of Sciences, (16(32):1544--15855
bias-variance
2019.
[8] Mikhail Belkin, Daniel Hsu, and Xu. Two models of double descent for weak features. SIAM Journal Mathematics of Data Science, 167):1167-1180, 2020.
[9] Tony Cai and Hongming Pu. Transfer learning for nonparametric regression: Non-asymptotic minimax analysis and adaptive procedure. arXiv preprint arXiv: 2401.12272, 2024.
[10] Niladri S. Chatterji and Philip M. Long, Foolish crowds support benign overfitting. Journal of Machine Learning Research, 23(125):1-12, 2022.
[11] Scott Shaobing Chen, David L. Donoho, and Michael A. Saunders. Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing, 20(1):33-61, 1998.
[12] Yihang Chen, Fanghui Liu, Taiji Suzuki, and Volkan Cevher. High-dimensional kernel methods under covariate shift: Data-dependent implicit regularization. Proceedings of the 41st Inter- national Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 7081-7102. PMLR. 2024.
[13] Geoffrey Chinot, Matthias Löffler, and Sara van de Geer. On the robustness of minimum norm interpolators and regularized empirical risk minimizers. The Annals of Statistics, 50(4): 2306-233 2333, 2022.
[14] Konstantin Donhauser, Mingqi Wu, and Fanny Yang. How rotational invariance of common kernels prevents generalization in high dimensions. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2804-2814, PMLR, 2021.

---

[15] Konstantin Donhauser, Nicolò Ruggeni, Stefan Stojanovic, and Fanny Yang. Fast rates for noisy interpolation require rethinking the effect of inductive bias. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5397-5428. PMLR, 2022.
[16] Moritz Haas, David Holzmüller, Ulrike Luxburg, and Ingo Steinwart. Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension. In Advances in Neural Information Processing Systems, volume 36, pages 20763-20826 Curran Associates, Inc., 2023.
[17] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer series in statisties. Springer, 2009.
[18] Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical Learning with Sparsity: The Lasso and Generalizations. CRC Press, 2015.
[19] Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):94--986, 2022.
[20] Zelin He, Ying Sun, and Runze Li. Transfusion: Covariate-shif robust transfer learning for disensinnnii regression. Proceedings of The 27th International Conference on Artificial Intelligence and Statisties, volume 238 of Proceedings of Machine Learning Research, pages 703-711. PMLR, 2024.
[21] Asmaul Hosna, Ethel Merry, Jigmey Gyalmo, Zulfikar Alom, Zeyar Aung, and Mohammad Ab- dul Azim. Transfer learning: friendly introduction. Journal of Big Data, 91): 2022.
[22] Long- Kai Huang, Junzhou Huang, Yu Rong. Qiang Yang, and Ying Wei. Frustratingly easy transferability estimation. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9201-9225, 2022.
[23] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: convergence and generalization in neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, page 8580-8589. Curran Associates Inc., 2018.
[24] lain M. Johnstone. On the distribution of the largest eigenvalue in principal components analysis. The Annals of Statistics, 29(2):295-327, 2001.
[25] Peizhong Ju, Yingbin Liang, and Ness B. Shroff. Theoretical characterization of the generaliza- tion performance of overfitted meta-learning. arXiv preprint arXiv:2304.04322 2023.
[26] Peizhong Ju, Sen Lin, Mark S. Squillante, Yingbin Liang, and Ness B. Shroff. Generalization performance of transfer learning: Overparameterized and underparameterizeed regimes. arXiv preprint arXiv:2306. 04901, 2023.
[27] Chinmaya Kausik, Kashvi Srivastava, and Rishi Sonthalia. Double descent and overfitting under noisy inputs and distribution shift for linear denoisers. arXiv preprint arXiv:2305.17297, 2024.
(28) Dmitry Kobak, Jonathan Lomond, and Benoit Sanchez. The optimal ridge penalty for worl-world high-dimensional data can be zero or negative due to the implicit ridge regularization. Journal of Machine Learning Research, 21(169):1-16, 2020.
[29] Frederic Koehler, Lijia Zhou, Danica J. Sutherland, and Nathan Srebro. Uniform convergence of interpolators: Gaussian width, norm bounds and benign overfitting. In Advances in Neural Information Processing Systems, volume 34, pages 20657-20668. Curran Associates, Inc., 2021.
[30] Vladimir Koltchinskii and Karim Lounici. Concentration inequalities and moment bounds for sample covariance operators. Bernoulli, 23(1):110-133, 10-133, 2017.
[31] Gil Kur, Pedro Abdalla, Pierre Bizeul, and Fanny Yang. Mininum norm interpolation meets the local theory of banach spaces. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 25726-25754 PMLR, 2024.
12

---

[32] Qi Lei, Wei Hu, and Jason Lee. Near-optimal linear regression under distribution shift. In Pro- ceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 6164-6174 PMLR, 2021.
[33] Daniel LeJeune, Jiayu Liu, and Reinhard Heckel. Monotonic risk relationships under distribution shifts for regularized risk minimization. Journal of Machine Learning Research, 25(54):1 1555)):1377, 2024.
[34] Sai Li, T. Tony Cai, and Hongzhe Li. Transfer Learning for High-Dimensional Linear Regres- sion: Prediction, Estimation and Minimax Optimality. Journal of the Royal Statistical Society Series Statistical Methodology. 84(1):149 1739 2021.
[35] Sai Li, Linjun Zhang, T. Tony Cai, and Hongzhe Li. Estimation and inference for high- dimensional generalized linear models with knowledge transfer. Journal of the American Statistical Association, 1196546):1274-1333 2024.
[36] Yue Li and Yuting Wei. Minimum norm interpolators: Precise asymptotics and multiple descent. arXiv preprint arXiv:2110.09502, 2021.
[37] Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel "Ridgeless" regression can generalize. The Annals of Statistics, 48(3):1329-1333 2020.
[38] Haotian Lin and Matthew Reimherr Smoothness adaptive hypothesis transfer learning. In Pro- ceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 30286-30316. PMLR, 2024.
[39] Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distriuttion generalization: A survey. arXiv preprint arXiv:2108. 13624, 2023.
[40] Cong Ma, Reese Pathak, and Martin J. Wainwright. Optimally tackling covariate shift in RKHS-basee nonparametric regression. The Annals of Statistics, 51(2):738-761, 2023.
[41] Neil Mallinar, James Simon, Amirhesam Abedsoltan, Parthe Pandit, Misha Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: Toward refined taxonomy of overfitting. In Advances in Neural Information Processing Systems, volume 35, pages 1182-1195. Curran Associates, Inc., 2022.
[42] Neil Rohit Mallinar, Austin Zane, Spencer Frei, and Bin Yu. Minimum-norm interpolation under covariate shift. In Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 34543-34585. PMLR, 2024.
[43] Marko Medvedev, Gal Vardi, and Nathan Srebro. Overfitting behaviour of gaussian kernel ridgeless regression: Varying bandwidth or dimensionality. In Advances in Neural Information Processing Systems, volume 37, pages 22624-52666 Curran Associates, Inc., 2024.
[44] Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667-766, 2022.
[45] Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpolation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, 111):67-83, 2020.
[46] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations, 2020.
[47] Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. LEEP: A new measure to evaluate transferability of learned representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7294-7305. PMLR, 2020.
13

---

[48] Cuong N. Nguyen, Phong Tran, Lam Si Tung Ho, Vu Dinh, Anh T. Tran, Tal Hassner, and Cuong V. Nguyen. Simple transferability estimation for regression tasks. In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216, pages 1510-1521, 2023.
[49] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, ((10):1345-1359 2010.
[50] Pratik Patil, Jin-Hong Du, and Ryan Tibshirani. Optimal ridge regularization for out-of- distribution prediction. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 39908-39954 PMLR, 2024.
[51] Roger Penrose. generalized inverse for matrices. Mathematical Proceedings the Cambridge Philosophical Society, 51(3):406-413, 1955.
[52] Alexander Rakhlin and Xiyu Zhai. Consistency of interpolation with laplace kernels is a high-dimensional phenomenon. In Proceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pages 2595-2623. PMLR, 2019.
[53] Bernhard Schölkopf and Alexander J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Oplimization, and Beyond. The MIT Press, 2001.
[54] Ohad Shamir. The implicit bias of benign overfitting. Journal of Machine Learning Research, 24(113):1 10011-40222 2023.
[55] Shuo Shuo Liu. Unified transfer learning in high diggnnimmnnional linear regression. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 1036-1044. PMLR, 2024.
[56] Yanke Song, Sohom Bhattacharya, and Pragya Sur. Generalization error of min-norm interpola- tors in transfer learning. arXiv preprint arXiv:2406. 13944, 2024.
[57] Javan Tahir, Surya Ganguli, and Grant M. Rotskoff. Features are fate: theory of transfer learning in dineeniiensional regression. In Forty-second International Conference on Machine Learning, 2025.
[58] Yang Tan, Yang Li, and Shao-Lun Huang. Otce: A transferability metric for cross-domain cross-task representations. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15774-15783, 2021.
[59] Shange Tang, Jiayun Wu, Jianqing Fan, and Chi Jin. Benign overfitting in out-of-distribution generalization of linear models. In The Thirteenth International Conference on Learning Representations, 2025.
[60] Ye Tian and Yang Feng. Transfer learning under high-dimensionall generalized linear models. Journal of the American Statistical Association, 1186544):2644-2697 2023.
[61] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series (Methodological), 58(1):267-288, 1996.
[62] Alexander Tsigler and Peter L. Bartlett. Benign overfitting in ridge regression. Journal of Machine Learning Research, 247123):1-76,- 2023.
[63] Roman Vershynin. High-dimenssinal probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018
[64] Chao Wang, Caixing Wang, Xin He, and Xingdong Feng. Minimax optimal transfer learning for kernel-based nonparametric regression. arXiv preprint arXiv:2310. 13966.960..166 2023.
[65] Guillaume Wang, Konstantin Donhauser, and Fanny Yang. Tight bounds for minimum -norrm interpolation of noisy data. In Proceedings of The 25th International Conference Artificial Intelligence and Statisties, volume 151 of Proceedings of Machine Learning Research, pages 10572-10602. PMLR, 2022.
14

---

[66] Xuezhi Wang and Jeff Schneider. Generalization bounds for transfer learning under model shift. In Proceedings of the Thirty-Firss Conference on Uncertainty Artificial Intelligence, 15, page 922-931. AUAI Press, 2015.
[67] Karl Weiss, Taghi M. Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big Data, (1):1-40,, 2016.
[68] Jingfeng Wu, Difan Zou, Vladimir Braverman, and Quanquan Gu. Direction matters: On the implicit bias of stochastic gradient descent with moderate learning rate. In International Conference on Learning Representations, 2021.
[69] Jingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu, and Sham Kakade. The power and limitation of cevraininn--inetuiinn for linear regression under covariate shift. In Advances in Neural Information Processing Systems, volume 35, pages 33041-33053, Curran Associates, Inc., 2022.
[70] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
[71] Lijia Zhou, Danica J. Sutherland, and Nati Srebro. On uniform convergence and low-norm interpolation learning. In Advances in Neural Information Processing Systems, volume 33, pages 6867-6877. Curran Associates, Inc., 2020.
[72] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongho Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1): 43-76, 2021.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract accurately reflects the paper's scope by clearly stating its focus on transfer learning for benignly overfitted MNI in high-dimensional regression. It outlines the main contributions, including theoretical analysis under distribution shifts, an informative source selection algorithm, and empirical validation, without overstating the generality of the results.
Justification:
Guidelines:
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper discusses limitations in Section 6, particularly noting the lack of theoretical consistency guarantees for the proposed informative source detection algorithm. The authors also suggest that bridging this gap presents opportunity for novel theoret- ical advancement, demonstrating awareness of the scope and current limitations of their contribution.
Justification:
Guidelines:
15

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model cell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical results stated in the main manuscript, including Lemmas, Theorems, and Corollaries, are accompanied by complete proofs provided in Appendix. Additionally, all assumptions are clearly stated, and preliminary lemmas are presented with appropriate citations Appendix before the main proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- keferenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
The paper provides full details necessary to reproduce the main experimental results, including synthetic data generation process specified Section 5 and Appendix F.1. Additionally, the authors release anonymized R sourcecodes at submission time that reproduce all figures in the paper to ensure the transparency and reproducibility of the empirical claims.
Justification:
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-soneee open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The authors release anonymized sourcecodes submission time to repro- duce all reported figures. Each sourcecode script includes clearly written comments and instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.co cc/ Saaicc/guidee//ooeeeuee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips.co ppppuulccuuuuds..... /wtoeeeubesssiiiieoeaae wwwiiissssoooooo.. asiinoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
17

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The authors specify in Section 5, Appendix E, and Appendix F all necessary training and testing details, including data generation process, sample sizes, dimensionality, evaluation metrics, and the value of hyperparameter in Algorithm 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: To enhance the visual clarity of the reported figures, the authors separately report in Appendix F.2 the mean and standard deviation of 50 excess risk values computed across independent simulation runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
18

---

The authors specify in Appendix F the computational resources used and preliminary simulations conducted prior to the finalized experimental results reported in the paper.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https http//eeurrp../// wttpippp...ppppppeee wccppbbicc/thiccstteeeelleleeell
Answer: [Yes]
Justification:
The research fully conforms to the NeurlPS Code of Ethics. The authors maintain anonymity throughout the submission process and in the released sourcecodes, and there is no ethical concern related to data usage, experimentation, potential societal harm.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics. If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: While the authors expect their work to contribute to the theoretical development of transfer learning under benign overfitting, they do foresee any direct societal impact at the time of submission. As the paper focuses on foundational methodology without immediate application to real-world systems, the answer appropriately marked as [No].
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
19

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The authors release anonymized R sourcecodes submission time to repro- duce all reported figures. Each sourcecode script includes elearly written comments and instructions.
Guidelines:
20

---

The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
21

---

The answer NA means that the core method development in this research does not involve LLMs as any important. original, non non-standard components.
Please refer to our LLM policy (https : /ttp//eeerppppppeeee cc/Conferrecces/2225 /ttpe/e/nneeee////////// /LLM)
for what should or should not be described.
22

---

A Additional Literature Review
Minimum-lp-nore interpolators with p + 2. For the minimum-(1 -norm interpolator, i.c., Basis Pursuit (BP) [11], its consistency hinges on the ambient input dimension d, sample size n, and model sparsity level s. Li and Wei [36] analyzed the asymptotic excess risk as d/n + E (0, oo) with isotropic design and linear sparsity, illustrating multiple descent phases with increasing degree of overparamettriiation. Wang et al. [65] demonstrated matching excess risk bounds of order o²/ /log(d/n) for noisy BP with isotropic design and noise variance o², provided 8 E n/ log(d/n)² and X d I exp(n!/5). Complementary to these results, Donhauser et al. [15] showed an bias-variance tension across ep: interpolation achieves fast polynomial rates near 1/n for p > 1, but only logarithmic rates for p = 1. As instructive counterpoint for BP, Chatterji and Long [10] proved lower bounds under Gaussian design, showing that the excess risk of BP can converge exponentially more slowly than that of the minimum-Ez-nore interpolator even when the ground truth is sparse.
interpolation-specee
A separate, Algoritth--ggosste line controls the population error of any interpolator via uniform convergence: Koehler et al. [29] analyzed generalization in terms of Gaussian width and, instantiating the result on the & simplex, obtained BP consistency. Beyond Hilbert- et al. [13] developed a robustness framework for minimum-norm interpolators with potentially adversarial errors, where their bounds track localized complexity, the norm of interpolated noise, and the subdifferential at the ground truth. Finally, Kur et al. [31] placed minimum-norm interpolators in general Banach spaces: under 2-uniform convexity, the bias is controlled by Gaussian complexity, while under cotype 2, they established a reverse Efron-Stein variance lower bound and sharpness of lp regression with [1.2].
arguments, Chinot
Rorm- interpolator. Within reproducing kernel Hilbert space (RKHS) [53], a line of recent works has examined when the minimum minimum-RKHS -orm interpolator, often called kernel ridgeless regression (KRR)', generalizes well. Liang and Rakhlin [37] attributed implicit regularization in an RKHS to high input dimension, kernel curvature, and favorable spectral decays and provided out-of sample guarantees under such conditions.
In fixed dimension, interpolation can fundamentally fail: Rakhlin and Zhai [52] showed that Laplace- kernel-based KRR is inconsistent for any bandwidth selection, making consistency a high-dimensional phenomenon. Refining this fixed-d paradigm, Haas et al. [16] proved that for "smooth" kernels, benign overfitting impossible in fixed d. yet "moooh"smooth" kernels (with large derivatives) can achieve rate-optimal benign overfitting; they further transferred the mechanism to wide networks by modifying activations. For the widely used Gaussian kernel, Medvedev al. [43] showed that in fixed d, KRR is never consistent even when the bandwidth is tuned and, with sufficiently large noise, is often worse than the null predictor; with increasing d, they tracked transitions among benign, tempered, and catastrophic regimes, where sub- sub-polynomial-d example typically exhibits benign overfitting.
In high-dimensional regimes, Donhauser et al. [14] identified a structural barrier for a broad class of rotationally invariant kernels (RBF, inner product, and fully connected NTKs): KRR is consistent only for low-degree polynomial targets, so spectral decay alone does not guarantee benign overfitting. Finally, Mallinar et al. [41] proposed a unifying taxonomy of benign, tempered, and catastrophic overfitting linked to the kernel eigenspectrum and ridge regularization parameter À 2 O: overfitting O. whereas for KRR with X = 0, depends on spectral decay rates; Laplace kernels and ReLU NTKs typically lay in the tempered regime.
benign with a positive
'While the acronym KRR often refers to kernel ridge regression with regularization parameter À > 0 in the literature, we KRR to denote the ridgeless minimum-norm interpolator with À 0.
23

---

B Preliminary Lemmas
B.1 Matrix Properties
Lemma 2 (Expectation quadratic forms), Let 0 R' be a random vector with mean E covariance Cov(O), and M E RPxp be deterministic (or conditioned) symmetric matrix. The expectation of the quadratic form ('M is given by
B[ό] and
E 0 MÒ] = [[O] ME [0] + Tr Tr(MCov(O))
Proof. Since trace is invariant to eyclic permutation, we have
E
0
M0
IE TT M0]
Tr MI [00
'Ir M Cov (0) [[]EE 1)
Tr MCCv (0) + Tr ME E 01 )
= =Tr (MCov
0)) + E [[0] ME
The following lemma an adjustment of Lemma in Ju al. [26] and Proposition 3 and Lemma 16 in Ju et al. [25]. tailored to align with our setting of well-specified linear models. The result follows from the rotational invariance of Gaussian distribution.
Lemma 3 (Standard Gaussian orthogonal projection). Let X(9) RR have i.i.d. standard Gaussian entries with p > ny + 1. Since the matrix has full row rank almost surely, the orthogonal projection onto the nu dimensional row space of ((4) is given by
x T [X X ( x(1)
and Ip H(9) is the orthogonal projection onto the (p- -nn)-niiennsinn null space, with expectations
EH
H( =(1 1lg p B
I )
p
Thus, for any fixed vector a E RP, we have
E
ng ||l|?
(I, ( )all =(1==
ng
|||]
p
The following lemma can be generalized broad class of norms, with our focus on the norms of real matrix.
Lemma 4 (Hölder's inequality for Schatten p-norm). Given two real matrices A and B, Hölder's inequality yields
TT((A S ||AI,, |BI,
where 1 p
+ = and ||Al| is the Schatten p-norm of A defined as
A = (TT(IAPP)) 1/P
A = (ATA)
The above inequality also holds when p = 1 and g = 0o, where the infinity norm corresponds to the operator norm, i.e,
B [B||
In particular, if both A and B are symmetric and positive semi-definite, have
Tr(AB) < Tr(A) BlI |||-
24

---

Lemma' 5 (Sherman- Morrnion-- Morrison-Woodbury formula). Let A E Rux* be an invertible matrix and E Raxk be matrix with k n such that ZZ' A invertible. By the Chernan-Morrisoe Woodbury formula, it follows that
z 7ZZ T + A) Z
Z - (1, + z A 2) - - A A-
Z A Z ( 2 --)=
and
z (ZZ A) Z (I+ +
Z A A - Z A 2 +2 A A- 'Z)
B.2
Concentration Inequalities
For simplicity, we present the subsequent lemmas using the target design matrix X = x and covariance ( whose eigenvalues are denoted by
A 2 A2 2 ... 2 X 0
and assume without loss of generality that X has full row rank so that XX is invertible. We further define the target sample covariance Σ = (1/no)XX X, and the eigenvalues of a matrix M e Rdxd denoted by
#i (M) 2 μ2 (M) .... MM(M).
All subsequent results readily extend to the source counterparts X(1), E (()), and (g) under Assumptions and 2.
The following lemma provides a high- sroh-pbbbabiltty upper bound on the operator norm of a sub- Gaussian sample covariance, characterized by the effective ranks in Definition 1.
Lemma 6 (Exercise 9.2.5, Vershynin [63]). Let Assumption hold. There exists a constant u > that depends only on the sub- Gaussian parameter in Assumption I such that with probability at least - 2e for t Z l2g(2).
ro Σ +
ro(2)
+
m
Σ|
+
E|
no
n1
Next, we restates some key results from Bartlett al. [5] and Mallinar al. [42] for ease on the reader and completeness. The central focus of their analysis is the normalized variance of target-only (0) MNI M in Equation (6), where
V /00 = - (T(E = Tr ( [xx [xx
X
(10)
720
Following Bartlett al. [5], we fix X 1/2 the eigenbasis of E, where Z = ( "o" is the sub-Gaussian whitened design matrix as specified in Assumption 1. Then, the gram matrix and its aaduced-rank variants can be expressed as
: xx = ATzz
A- EAZZT i71
A : -
(11)
j>k
where {z/}/=1 C Re i.i.d. columns of Z; each j-th column 2j has i.i.d. Sa-sub-Gauusian mean-zero, unit-variance components. Consequently, the trace term Equation (10) can be written
Tr (xX) Xx =
Σ 1=1
AzZJJ
Σ AiZiZ 1=1
Zj
Σ X32J (2z2,2} + A,,)) 2z
132J A 33
Σ
(12)
J=1 (1 A5z, A 22,
where the last equality holds by Lemma Opening the discussion, the following lemma gives the concentration of the spectrum of the matrices in (11).
25

---

Lemma 7 (Lemma 5, Bartlett et al. [5]). Let Assumption 1 hold. There exist universal constants b,e c 2 such that for any & 2 O, with probability least 2e a/c,
1. for all j 2 1,
MK+1 ( MA+1 S (m((A Sc
Σ bS
X; + nodet1)
2. for all isisk,
Mmo (A) 2 Ung (A-)) 2 Mno (At) 2 - Σ A cnodR+1,
1
3. (Z) 2 buo, then
AN+ITH (2) S Mno (Ax) S (I(A)) S CAR+ITA(E).
B.2.1 Upper Bound on Single-Task Variance
The following lemma provides high-probability bounds on the norm of random sub-Gaussio vector with independent components. The result used to provide an upper bound on the trace term in Equation (12).
Lemma 8 (Corollary 1, Bartlett et al. [5] and Corollary B.5, Mallinar et al. [42]). There exists universal constant such that for random vector The with independent Uz-sub Gaussiar mean-zero, unit-variance components, any random subspace G C R of co-dimension k that is independent of and any > O, with probability at least 3e
---uu--Guusiinn
|z < no + cv2 + vnot).
[gzll 2 no 2v2 + vngt),
where Hg is the orthogonal projection onto G. Furthermore, let I. E (0, no/col and l E [0, na/ci] for 0j > 6o with sufficiently large Co Then, with probability least - -,
| < C2no.
[Gg] no/c3,
where O2 and C3 only depend on c, cn, and Vg For each E RnD and the corresponding subspace Gj defined analogously 2, and 9j. define the event
Fg = { ||2, 112 S 02n0
|| mo/c3}, j E
Then, the union bound argument implies
U j=1
PP
$ Σ P ((())
3=1
3e = 36e
j=1
> 1 38e + = 3E --11-3-((--ostt (1-log(t)
P
which necessitates 0 <t - log(!) I n/co complete the bound. Since each event E, is defined for e (0.n/co), by taking t = n/co pre-event and requiring that
log(e) S n/co + i $ n/00
all events {Eg}i=2 hold simultaneously with probability at least 1 n/00
26

---

We further introduce lemma concerning the weighted sum of sub-exponential random variables. Lemma (Lemma 7, Bartlett et al. [5]). Suppose that { jer is a non non- ion-innceasinng sequence of non- negative numbers such that Ljen Xj < co (as is the case of the eigenvalues of E) and that (Si)ien are independent centered
random variables. Then, there exists a universal constant a > O such that for any t >> O, with probability at least 1 2e
E, A5Es a max (e ( 13
We are now ready to provide a bigh-probability upper bound on the variance of single-task MNI. Lemma 10 (Lemma 6, Bartlett et al. [5]). Let Assumption hold. There exist universal constants b 1 such that if
0 S no/c,
Tx(2) 2 bmg
l kk
then with probability at least 1 - 7e ne/
o Tr
l +
no Esze X3 (Ax+ar* (E))
Σ
S
com
no
noo
Proof. It suffices upper-bound the trace term in Equation (12), which is given by
スa2 A } (1+ イリエ 23,2
Tr Tr(x (XX Xx
+
Σ}3 A 2z,
232J
j>
Fix b, C1 2 as specified in Lemma 7. Then, with probability at least - 2e nn/o1 then for all vectors in {z,}i= C R"o, we have
n/o1, rx (2) 2 bno
cc|lzz112
2/ A S (1 12, S Mmo (A- 11,,11 S
(
and on the same high-probabiin event, we have simultancously
x/ A A 2 (Ag, 2,) A A (II 0 {,,))
2 1l { T| g 3, 112
1
M
Mg g, 2
C1 C1 17k (Σ)
where II, is the orthogonal projection onto the span of the bottom (no k)) eigenvectors of A-j. independent of Zj. Therefore, for all j < e, it follows that
A
2
A 3zz
12
(1 A1 A 22
A 2,)2
II 2,
Here, we recall Lemma with union bound over & events. Let S na/co and E [0, na/d for sufficiently large co < c, where is as specified in the lemma. Since l < k, with probability at least 1 - 3e -n//co the intersection of all l events
n #=1
|1, 12 c2no O |,, no/ca
hold for constants 02 and ca that only depend on co, and c. Combining the results, with probability at least 1 - 5e n for some sufficiently large Co. we have
X2x,
Zz 22
(1
n10
27

---

Next, we consider the sum of the tail summands. On the same high--robability event in Lemma 7 we used, we have Mno (Ax) 2 A E2/c, /c/ rk rx()) 2 bng. Also, since μi (A :) S Mno 2 (A), this
implies that on the same event, if r& 2) 2 bno. then
2 A 23 S MI I Ano (A) 2 2|2111 S umo (Ax) 112,112 S (AR+ITA E)} |2,11
which implies
o EEiz A3112y112
Notice that each ||z,|| is sum of no independent -2-subeexponential random variables such that E||2,| = Yog. Therefore, by Lemma 9, there exists a universal constant > 0 such that with probability least 1 2e- for t no/co.
>3 S no Σ
13 + av² max
N241h,
tno Σ 15
328
13t
f32
S no
+ av² max
1
Σ
X2
vino
f>t
s>L
J26
$ ng
X
for some sufficiently large cg. Combining the above results yields
Ej>t X3
AjjzAA->2 I cono (AK+17H (2))²*
(+)))
Thus, putting both summations together and taking > max(Co. ca. cs) complete the proof.
B.2.2 Lower Bound on Single-Task Variance
To establish lower bound on the trace term in Equation (12), we first present preliminary result that extends a lower bound on individual random variables, each holding with equal probability, to unified lower bound on their entire sum.
Lemma 11 (Lemma 9, Bartlett et al. [5]). Suppose S oc. Let {nj be sequence of non- negative random variables and {1/}z=1 be sequence of non-negative real numbers (at least one of which is strictly positive) such that for some 8 (0, 1) and any j E [p], P(N; > (2)) 2 1 - 5. Then,
P
M
1
Σ44 j=1
P
1
Next lemma provides the lower bound by first establishing general bound that holds regardless of (2)) and then refining it for the case where (2)) is at least no, using Lemma 11. Lemma 12 (Lemma 10, Bartlett et al. [5]). Let Assumption hold. There exist universal constants b, c 2 1 such that if
0 0 k < no/c.
(x > bno.
then with probability at least
1 10e -n/ee
i b Σjze> X3 (AR+1TA (E))
Tr
S
Σ
o
min Esk
n4p
cb
no
Proof. Fix j 2 and no/C/ By Lemma 7, there exists universal constant C₁ such that with probability at least 2e no/ no/C,
Σ Ne A modeti). i>k
Pk+ (A (A-;) S
28

---

Let Gj be the span of the bottom (no k) eigenvectors of A-j and Igg be the projection onto the orthogonal complement of Gz. Then, we have
M6 1,,z2,1 11
A 2 (Ig, 2,) A
A (1) 2 (A+1((_)) 1g, 23112
C1 Ciix d =mo2++1)
By Corollary 8, with probability at least 30-
[gg 2 1| 2 no/ C22
provided that & na/Co for some sufficiently large Co C. Thus, with probability at least 1 5e Boa
7 2
720
C3 Disk d + nodx+1)
C3 (Eish x + nodk-1)
=
1 A |
1+
2z22 A-
nod,
and taking the squared reciprocals both sides and then multiplying them by 1322 A A-32, yield
332 _ A 2zz (1 +
2 |+
C3
(Eis d: + no
-2
2 A 2 2
nodj
(z) A 2,)
Also, by the Cauchy-Schwarc Schwarz inequality and the same piohaprobability event from Lemma 7, we have
2/
223 2, A
32
1
2
23
||
2 2,
Cano
Thus, taking C sufficiently large, for any 1 - 5e -nCC
2 1 and I E [0, no/CC, with probability at least
A321 A A
(1+ Eish + nods+1 -2
(I (1 A
Cna
nods
which establishes a general lower bound. We refine it by establishing a lower bound on the summation over j (p) as follows: by Lemma 11, with probability least 1 10e-no/e1,
X AZZJJ T
Tr Tr(xT (XX
(
X
A
2,)2
Eise di + nadc+1 -2
1
n0
nody
/=1
min
1 EE
C21
);)
1
bno
2
min
1,
22 n0
(r(E))
x
k++1
where b 2 is as specified in Lemma 7. Thus, if (x(()) 2 bng, then
br
Th Tt(x (XX
Σ min
o2b²
no (M)))
Pnoo E وuز! x2 i
min
+
o2
ESk
no
(A1+I7k(E))
where the equality follows from the fact that Xy are non-increasing. Taking > max(ci, 02) completes the proof.
29

---

B.2.3 Matching Upper and Lower Bounds
Combining Lemmas 10 and 12, if Fh constant factor of
Σ)
2 brn for some S n/c, the variance v is within a
l no E>>e> X3
min tSk
nip
(AK+ITK (2))²
Choosing k = kg. which is the smallest one among all qualifying k's, simplifies the bound as follows.
Lemma 13 (Lemma 11, Bartlett et [5]). Let Assumption I hold. For any 2 1, the minimum
ki = min k 2 O: r.()) 2 bmo}
well-defined, then
bro Σise 23
kū bno
bmo Ejson X3 (Nez+ (E))²
kō
bmo
min tSko
+
+
bno
(Aeg (z))
broo
Reg
(2)
Proof. Observe that
L + bno
Ej
bm
bno
(ARG+ (23)
bmo
jot (AEg+ITA; (2))² 2
hp
bno
b
min
I (E)) bno
(Aeg+ (Σ))
j=1
bmod
bno
( (2)) 2
j=1
i>t
where l" denotes the largest value of j S kō for which
bn
bmo
(Aeg+ 1rng (2))
since X3 are non-increasing. This condition holds if and only if
Aラ Ang+i
bno
By the definition of kip we have Tkg- (ag-1(22) < bno. which implies
Σs>1 d Σソ A3 Akg
Aug
Akg ARg+1
Σ
(rug (12))1(2)
(bng 1),
Apz+1
Neă+1
ARg+1
so the minimizing é is kn- This completes the proof.
Using Lemma 13, we obtain the upper and lower bounds that match up constant factor in the desired form that aligns with the formal definition of benign overfitting in Definition 2.
Corollary 3. Let Assumptions I and 2 hold; that is, there exist universal constants bo co 2 such that the minimal index
Ko mini
min ( 2 : (x) 2 bono }
is well-defined with 0 - kg S no/co. Then, with probability at least 1 - Te Tooooo
0h T (
< cooo min ESko
i no X
cooo
ke
72
71
no
(2)))
110
Rs; (E)
and with probability least noo/oo
0 Tr
( Σソ
of
min esk
12:o Ex>e X2
1
o
+
no
cubg
no
(AKg+1TK3 (E))²
co
n2
RAg
(Σ)
30

---

C
Proofs
Throughout our proofs, we assume without loss of generality as in Section B that each design X XP is of full row rank, i..., rank(X(")) ny < p. that X() is invertible. Thus, we write the matrices (X) and (x (X())(())) interchangeably.
C.1 Proof of Equation? 2
Proof. The identity in Equation (2) is well-known; we provide a proof here for completeness. Recall that each single-task MNI is given by BLC = =X (x (X(()x(()T) -1 "'1)) .Given an arbitrary
interpolator f trained on the same dataset (X(s), y(1)), we have
( (3 B y( - y(0) On
This implies
(B- - 3(00 T (B4) (3 B(2) T x((1) (x -1 y(())
x (1) (B - B(2)) (x )y)))
O,
i.e., (B - - and 3 are orthogonal. Thus,
| || - -3
||35
|||| |
||مل
and the equality holds if and only if ß B.6²; ttht is, the MNI has the minimum no-m among all possible interpolators and is uniquely determined. This completes the proof.
C.2 Proof of Equation
Proof. We present detailed derivation of the TM estimate in Equation (3). First, we may write
0(1)
arg min {l|@ 3 ( B y(0)
arg min { {| | X (0) (B -3(9 = X (0)())
arg min ||0| X x 0 y x (
Here, we have 0 = 3 --)()) and the minimizer 0 of f is given by 0" = A(9) where 3 -3(%)
the minimizer of B. By the definition of MNI in Equation (2), the minimizer 0* is given by
θ = (0)
[X x (0)) -xx)))((xxxx))))))))) (()))
Therefore, have
BL4 (O) (x X (0)) (y((0) )(
x B(9)
(.0 B 3:00 H (0) 3.9)
=0 + (I, H (0) B/
where O (0) (x()) 1 (0) This completes the proof.
C.3 Proof of Lemmal
Proof. By Equation (5), the excess risk of the TM estimate is given by
R Eg
[ TM ((0) (0) (3(9) - 30)) x).
where Ex are as specified in Assumption 1. For simplicity, we write
B1:1 = Esl | x)). Cov(-) = Cove(- X),
31

---

so that we always take expectations over the randomness of noises & conditional on the designs X, unless specified otherwise. Lemma 2 gives
R
(E
3(
E
E TM
E.
,(00)
Tr Σ 1 Cov (
Vila
L.., the excess risk decomposes into sum of bias and variance.
Bias of TM. Recall from Equation (3) that
3 = 3 + ) A
where H ( (X(O)
((1)
is the orthogonal projection onto the space of X
Here, the expectation of the single-task MNI 3/1 is given by
E x
T
(x (1) X
T) E X 2 3( + e
H H 3))
3(0) q E [Q]o
since (()) is mean-zero by Assumption Thus, the following expectation holds:
E = (0) ( + (Ip - - ) (()))
E (0) = (I, (0) (9))) 3( - (
Denoting (T) (I, (I,-HO (()) (0) [I, H (0) where the orthogonal projections ( and I, H(9)
are symmetric, we have
B =
(H 3(0) ((0) (H()) -B00, q a 0,
where
H9 -3(00 H ((0 + +11)) - -3 = (()) (Ip H
Thus, we have
B ( TM
(0)) [I, H (0) (I, H (1) g + H I (0) H (0) (()
2
( IT (I, H()) 3(0)
(
which completes the derivation of B( Lemma
Variance of TM.
As for the variance (:(a) we may write
Cov ((l()) = Cov (B!0) + (Ip ( )
Cov (o + (Ip H H( Cor( (I,-H())
where the second equation holds since 3 and ((( are independent. Here, the covariance of the single-task MNI ((4) is given by
Cov( Cov x ((9) ( (x())x(0) x())) ((10))
Cov Cor((x(1)) ( x x()) x ( yr(0))
= (x(1) x(1)) T Cov ( (5) ((9) (x(1) x(0))
= (x((0 ( x( E iQ)o,
where the last equality holds since Cov (0) L,n, by Assumption and M M'M M for any M by the definition of Moore-Penroee inverse. Denoting (n) (1/m,) ((()))))))))))))) (()), we have
oh D nn
Cov (2(()
q E
32

---

which yields
o
+ og
Cov (8%)
(I,
H (0) {64M (I, H)
no
ng
Plugging this into 1 yields
1io
Tr Cov (3.0) E()
of Tr
{(1)) 5(1)
oe Th (I,
H ( (Ip H H()))
no
ng
o Tr (5
o Th (
This completes the derivation of V in Lemma 1.
C.4 Proof of Theorem
Proof. Since assume iidd standard Gaussian design for (X X(()) in Theorem 1, have (())
(1) Ip and (I) (I, H H (I, HH) I H for idempotent H())
Bias of TM. Under the standard Gaussianity assumption, the bias of the target and transfer tasks, which are provided in Equation (6) and Lemma I respectively, are reduced to as follows:
B (0) ((0)T (I, - H())) 3(0)
M
( ((0)T (I, H()) (Tp - HO) H()) + 5(8)T H H
) H
H())(()
2 (I, H (I, H()
First, by Lemma 3, the expected bias of target task derived as
Ex B (0) = E [T, ( 3 (w-M) no ||000
Next, to derive the expected bias risk of the transfer task, we again use Lemma' 3. The expectation of the first term in (!9) obtained follows:
[ x(u)
((OT (I, HI, ( H(9) )(())
Exiv) Bx(0)
[3(0TT (I, H
H [I, H(2) ) x(4))
ng
E Exco | H
p
no
[ (p-ne
ng | |3,
q a 0,
p
p
where we use the law of total expectation the second line and the equality
1 [1
(0)
(2
120
given by Lemma 3 to obtain the third line. Applying the law of total expectation in the same way, derive the expectation of the second term in Blgi as follows:
E (xx0)) x(0) ((4) (9) (I, H g(
p 710
Exw
H
g(0) !|
p
p no
ny ||000)
* O.
p
33

---

ng Lastly, the expectation of the third term in B vanishes, since
E [x(0)) x(O))
26
H (1) (I, H(0)) (Ip H(()) )
-25
Ex E
H (Ip H())) (Ip - H(())) x(9) 3
= 2
p no p
8
Ex
H (Ip - H 8
and H(t)
H
Opxp almost surely as H(4) is idempotent. Combining the results, we have
Ex Bid
p no
p
ny
|13:0112
(" no my
|1080012 p
p
p
p no
p
fy p
g 1|2 10)
which completes the derivation of the expected bias in Theorem 1. Note that Lemma 3 requires
p > no and p > ng + 1, which is assumed in Theorem 1.
Variance of TM. Under the standard Gaussianity assumption, we can simplify the variances in Equation (6) and Lemma as follows:
2>(0))
(0)))
= of (x
n4n
o
Σ (n)t (I, H())) + Vi00
22g
variance inflation V(2
since, for each & E [Q]o. we have
) ( x
X [ x(4)
ny x
= n2 xxx (
(X())(())T) -2 -2((1)
nly
and trace is invariant to eyclic permutation. As we assume > (na + 1) V (ng + 1) in Theorem
we may use the result from Section 2.2 in Belkin et al. [8]: for each g E [Q]o. xx())
follows that each diagonal entry of (x
distribution with scale matrix In, and p degrees-of-freedon This implies
(x i i [ng]]
has a reciprocal that follows the x2 distriibutio with (ng - 1) degrees-of-freedonn Therefore, we have
-1]
E
i E [mg],
n p (ny + +1
and
Ex (1()) oeETT (xxx
of
(x (X))
o
= p (no 1)
Now it remains derive the expectation of variance inflation. The law of total expectation gives
E (x
)()))
o2 Tr
( (I, H())
0²
E
i Tr (Ex10) (SOt - (()) x x(1))
ng
nhg
o² p Rg
Ex ni (x (x
Thg
0%
p
no
nl
p
p
(ny + 1)
which completes the derivation of the expected variance in Theorem 1.
34

---

C,5 Proof of Corollary 1
Proof. From Theorem 1, we have
0nn
Ex (0)
no
|/300 +
p
p (no + +)²
Ex RR10) p no p ng
(0) 2 +
ng 1 + (P- n2
og
onno
p
p
(ng + 1)
p - (no + 1)
Thus, it follows that
Rid < Ex (0)
p no (p- ng 110 1|+
ne 1100012)
(P- - no
ogng
P no |
+
p
p
p
p (ng + 1)
,g
ng 1 12 p- (ng + +)
mg
11
p
p
o
= (11300)12 12- -18(600))13 >
(13)
p
(ng + 1)
Using the definitions SNR, = |1/3 1| 12 0 and SSR, / ( 0
0, we may write
|| o SNRg.
12 o SNR SSR,,
and therefore, inequality (13) can be written as
SNR, g (1 -SSR,) > - (ng + 1)
P
(14)
From inequality (14), observe that its right-hand side always positive, which makes SSR, < (0) Also, the hinh side minimized with respect to ng E [L,p 1) is p/ (pppp - 2), which makes SNR, (1 - SSRg) > p/(p - 2) another necessary condition.
necessary condition for ExR < Ex
Provided the condition SSR, < 1, define the following function of ng:
À (mg) = Ex
R(g)
Ex R())
n20
T4y
13(00 1
+ ng (18(0) 12) +
P
9 nlg
no
|
p
p
p
p
p
(gg
+
1)
p
p no
F2
(10(0) 1|2 -1|3(00) (2)
o²
+ n2 p
p
p
(nh +1)
p
no
[|(300) 1|2 na
SSR, -
1
p
SNR, .n. 1)
B(o)
where we write ((D)
(| 12 and define A(n)) = -m(p,) in Corollary 1.
Holding all quantities other than ny constant and differentiating Á with respect to ng. we obtain
σ
(0)
SSR, g -
+ SNR, (p my - 1) ) + -B(On ng SNR, (p -ng- 1)2
1
mry
P
SSR, -
B (0)
+ SNR, (p nng - 1) SNR, n4 1)2
p
35

---

For the first-order condition we obtain
03 0. multiplying both sides by p SN ny 1)2/B(0)) > 0.
8nq
(SSR, (1 - n 1)2 p(p nig 1) + pm 0
(SSR,- 1)SNNR2 (p= n2 1)² + p(p 1) =
p(p 1)
= p- n2 -1= V SNR, (1 SSR,)
> 0
p(p 1)
= mi =P-1- SNR, -SSR,)
Here, since g < 1 and SNR, > O, ny is strictly less than p 1. Furthermore, we have ny 2 1 if and only if
p(p-1) 1) S - SSR,)
SNR, SSR,) 2 p(p p(p (p- 2)2
2)-
To check the second-order condition at nz. consider the second derivative of Á given by
0
σ
(())
SSR,
1
+ SNR. (p ng 1)2
3m
dng
p
SNR. (p- nlg - 1)
(())
(p - nl 1)² + +Tg. 2(p f4g - 1)
+
SNR,
(p ng 1)2
(p 1gg 1)4
((O)
2p Rgg - 1)2
+ 2ny (p Mlg -1)
SNR,
(p- 14 1)²
(B(0)
- 1
> 0,
Vng E [1, [1,p-11 1).
SNR,
(p - n1 1)3
That is, ((n.) is strictly convex on the interval p- - 1); conversely, A(n,) = E Ex is strictly concave on the same interval, and thus the improvement in expected excess risk A A strictly increases on Fi E [1, m[] and strictly decreases on ni E (m 1).
Now it remains show that A (n,)) > Recall from the first-order condition that we have
oλ
SSR,
1
n
Ing
SNR (p ni 11
+
SNR, (p ny 1)2 =
SSR, 1
1 SNR, - n
B00 n
(0)
p
SNR,(P - nn 1)2
Plugging this into ((ry) at n2 = n. and flipping the sign yield the maximal improvement value
2 (1 SSR
pp
pip SNH, (1- SSIR,)
A(ng)
p 1-- no
|1
p
p(p- 1)
(IG)((-SSSR)) - SSR))
- no
11 O.
p
p(p - 1)
This completes the proof.
C.6 Proof of Theorem 2
Proof. We first recall the notations Σ(1) = (1/n,,)x x X H X (X(() -1 -xx(1)
and II (I, (0) ( (I, H (1).
36

---

Upper bound on bias. First, notice that the bias of the target-only MNI in Equation (6) can be written
B B (I, H -
H )))0) = B(0) (Ip H - ) - H
since
(I,
(
-
11)
1
[I x (x -Ix - x x Opxp: q [Q]o-
nl
This implies
B s
(I, - (0) (Σ (100 (I, - H HO) 11
[0) {0) (
where = P H (1)
H
= for all E [Q]o- By Lemma 6, there exists a constant v > 0 such
that with probability at least 1 - 2c-6 for 2 log(2).
To To(2 +
To (0) +
| 1000 S
||
(15)
140
which completes the proof for the upper bound on (0) note that the constant u >> 0 in Lemma depends only the sub-Gaussian parameter Va in Assumption 1, As in Bartlett al. [5], assume that V is fixed and therefore regard u as a universal constant.
Next, recall that the bias of the TM estimate in Lemma is given by
B T (9) II (OH (( + (()T (I, - H
(0) (I. H (1)) 3())
UU220 20
U2 2
27
H II (0) (I, H (0))
where the Cauchy-Schward Schwartz and AM-GM inequalities yield
2 2 ((9) II (I, H())) g(0)) S 2V S U1 + U2,
and therefore
B < 2 + UZ)
So suffices to upper-bound the terms Un and U2, and their concentration inequalities can be obtained similarly. Notice that U, is bounded above by
U 8(1) H(#) (I, H (0))
1 (0)) H H (7())( g(9)
( (0) (I, - H ()) {(0) (I, (0) H 8(00
(0) S (0) 5 11
where the concentration inequality for ()))) 00) has already been established in inequality (15).
Here, only substitute the variable 8 in (15) by m log(2): with probability at least 1 2e ",
ro
(T(() +
To (x +
U S uul
|> (0) |||
(16)
n0
Next, recall that by Assumption 1, there exists a universal constant Cres > such that
| Σ (0)) s Czio)
with which we define the following set matrices:
+ = { A
A A R
||A|| 5 Czia
37

---

Consider any A E O. We have
[| (9)
(Ip (1) " (0) (I, H(0)) (T, - ||
S (T, H(1) Σ (0) (I, - -H
[T H (0) AE (I (I
min Ace
[0) A
This implies
U2 = 3 (I, - (3) ) -1p- (0) (0) (1 H H())) 3(0))
$ |1 H (1) (I, H (0) (Ip - H (I, -H |0
| - 101
min Ace
[0(]] AΣ x || + | A A ||
< min Ae
( -1 ( - )00 s012 with the minimizer A' = (2(1))
b 00)
> 30 |||
())) ())) |
and the tail bound on || | ()) holds similarly by Lemma 6. That is, there exists a universal constant u > 0 such that with probability least 1 -e 72 for 1/2 2 log(2).
U2 S
ro
((1))
122
ro (EE)) 722
Czlo
Σ (2)
| ||
(17)
R4g
Now, let let (m) and (2(222 denote the events (16) and (17) respectively, emphasizing the dependency on 77 and 172 of the corresponding inequalities. Then, it follows that
PP (()))))
P P(U2(1a) 2 1 2e -12
n
By taking the union bound, we have
2
IP
(c((n))
2 1 - Σ
P p((())))
1=1
2e 2e 12
Taking 2 log(4), 72 2 log(4), and r7 = - -log
c - -2 n2
yields
P (uuun) 2 1 - 40
72 log(4).
Thus, with probability at least
40 for 1) 2 llg(4))
ra ((() + 1 +
ro (()) +
U1 + Un $
||>(0) (|]0|||8p |||
no
no
ru (e))) +
To ((1) + 1
Cz(o)) ()) ||)3000
+
ng
ng
(g which completes the proof for the upper bound on TTM-
38

---

Upper bound on variance inflation. By Lemma 4, the variance inflation >1(0) is bounded above by
I'
(1)
n
0g ((
( 1/2 (x
1/2 -1/2 II (0) (-/)))-1/)) -1/2)
0 Th
(101 (0) (Σ 1/2 (Ip (= E (Ip H (E!") - |
ng
o Tr
1 (0) || (Σ - 1//2 1 ||
ng
where || (
12 1 ( = (2x3) -1 By Corollary 3, there exist universal constants
ba Cy specified in Assumption such that with probability at least 7 7emna/oa,
T
S cgp
he
2g
ng
nly
Rx;
(2()))
and by Lemma 6, with probability at least 1 2 for & 2 log(2),
100 su (V (1r033000 ) + ro (0()) +
| -100
||
n2
nop
Combining the results, with probability at least
7e 7// - 2e - for & 2 log(2),
02 Tr
11g
ro(Ex)) + + n0
70( + (2(0)) -1 - =
Tg
o
nl
Rug
El4)
no
Lower bound on excess risk. Even when the TM estimate induces positive transfer, i.c., R < which is sufficient for ((2) < B its excess risk is bounded below by
RM, (0)
R = B( + v(a)
B + v(u) (0)
2 12(0)
and by Corollary 3, there exist universal constants bD co specified in Assumption 2 such that with probability at least 1oocoo
(1)) x
2
ki
0
n70
Rag
which implies R(0) 2 o proof.
Km
no
on the same high-probability event. This completes the
Eioy
C.7 Proof of Corollary 2
Before proceeding to the proof, illustrate a case where r* in Equation (7) satisfics 7' < 74p- A canonical example with a "benign" spectrum is given by the spiked covariance model [24], where the
target covariance (0) has eigenvalues 2 2... (0)
s+1 .... 2 2 such that
(0)
(0)
1 jšs
1 xE, j>3
39

---

Here, the first largest eigenvalues are referred to as spikes, while the rest p eigenvalues are non-spikes; similar structure is considered by Mallinar al. [42]. Suppose the covariance satisfies p > No. pe < fo,, and < no- Then, we have r" = and r" = kig indeed, where kü is as specified in Assumption 2. In this case, ())) is "benign" according to Definition 2, since
To (0)) Y40
8 + (p- 8)
T40
Y40
O,
- O,
X
- 0. p-s
no
Y1
R,
(EO)
We now proceed to the proof of Corollary 2.
Proof. Recall the high-probability upper bounds (ignoring constant factors) on the bias and variance inflation of the TM estimate obtained in Theorem 2:
B =
ro (2 + 1
ro(
+
Σ [
10p
no
ru (1)) +n
ro
(())) +
Ozto: Σ 9)) |||
(18)
R4
12g
kg
(0)) + ro ro( +
V 0g
nly
(A)0)) -2 - |
(19)
14g
Rag
no
no
Case (A) of Corollary 2. If there is covariate shift, i.e, E(e) ( the upper bounds B and V in (18) and (19) are reduced to as follows:
Bim=
ro (0) +
ro(
+
Σ (0) |||
120
no
rn (())) +
To (1) + 17
| |||| 1.
(20)
Tlg
nh
V HM = 0g
Kg
+
n2g
ro
(0) +
To (0) +
(2
(090)~1120
(21)
REg (Σ))))
nhg
110
1τ0
where Cz
--11 11)00)))00))
for B HM. Under the covariate shift (A) in Corollary 2, we have 00)
a2 EEl,, [p).
for some 1 and the top 7" cigenvectors (corresponding to the r" largest cigenvalues) of (4) and ( align where 7" is defined in Equation (7). Under this covariate shift, the upper bound B in (18) becomes
(v To
BuT =
To(E
+
To (1) +
[ ||)))
720
no
(V
(ro(E000 +
ro (O() +
+ aCyp(e)
||>000
|
(22)
T4
ng
since
(T(x(4)
(T))
(D())
ro (I
| a
[
[0)
To derive the upper bound Cz on |Σ (,(0)) we first recall the spectral decompositions of (0) and (() in Assumption I with simplified notations:
Σ 0)) VAVT,
(()) U(aAJUT,
40

---

where A = Diag( (A1 122 .,. and Vla) E Rexp and U E RexP are orthogonal matrices
whose columns consist of the eigenvectors of x and E respectively. We partition V and U by their leading 7* and the rest (p- - T") columns as follows:
(V V-r.),
= (U,. U- (V,- U_r.),
where V. E Rexx" (resp. U, E (Rxx)) consists of the leading r" eigenvectors of Σ (resp. ()) and
v U
due to the alignment of the leading T eigenvector pairs of (1 and Σ E For
Σ (>(()) = VAV U(aA) V
UA UA- ",,
a
we have
V V..U U.
V. (0, I.
0 xpprr*)
V
V U, V_,
0 (pp-
U
Denoting A, = Diag(A 111 ...., \ and A A_ = Diag(A (1 7'
yields
[ x()) -1 VA
I.
0--x(p-79)
A A-U
a
(p--)))77
V V_,.U U U-r
1 V
I.-
VT, U (A- U
07-x(1-7")
a
(x--
where
V
U_
(A
11
$
A ||| ||||
|||(--)))~
A1+11
Ap
Therefore, we have
E 00) 2
1
I.
A_,. U_, 1--AAA (A
0 0x-(0-77)
|T
(p--)))7-
1
||
||A_,, VI VI,. (A U..
8
1
(1.41/2/0))
0
and by taking Cyto = a (1 12 ) the upper bound Ber in Equation (22) becomes
ro
(o()))) + + ro (I))) +
Bar
|| ||
no
no
(0)
(v
ro ((() +
ru (() + +1
+
+
|
|
(23)
(0)
ng
Dlg
where
X x 1
+1
(0)
by the construction of T" in Equation (7). Therefore, comparing BTT in (20) under the homogeneous covariate case and Byy in (23) under the covariate shift case (A), it follows that
BIs
BUT.
i.c., the upper bounds are within constant factor independent of a.
41

---

In contrast, under this covariate shift, the upper bound 1/ in (19) is adjusted to VHr as follows:
Kg
Thy
ro ro(2)()) - +
ro(2 ) -
|
Vur = 9 12g
RRØ (E))))
n1
axp (0)
V
HM:
since E(0) and ( have the same effective ranks (Definition 1) for all A 2 0, and therefore they share the same minimum
kio
provided Assumption for ( Thus, the upper bound on the variance inflation reduced by factor of a > 1, when compared to the upper bound VAs in (21).
To summarize, the following holds for the covariate shift (A) in Corollary 2 when compared to the homogeneous case ( ()))
The upper bound on the bias of the TM estimate remains identical up to constant factor independent of a.
The upper bound on the variance inflation of the TM estimate is reduced by factor of
Case (B) of Corollary 2. Suppose X Z (E())) is the q-th source design matrix under the
homogeneous case where E(() = ( where ( is as specified in Assumption 1. Recall that the bias and variance inflation the TM estimate Lemma are given by
B (O)T
I H
II (0) (I. H()) g(0) 5
H
H (9) g(9)
(8 H
(I, (0) 1(00)
v t = Th T
(x x
( (I))
where
H x (0) X x (1) x(1)
Let x(1) Z(0) (5(1)) 1/2 be the source design under the covariate shift (B) in Corollary 2 where
= =
1,
so that
X = x()))
Compared the homogeneity case, the bias Bly) under this covariate shift remains unchanged, since
H
x T ( X
(()))
On the other hand, the variance inflation v(a) is reduced by factor of a, since
(e) x111x(11)
a -x(00) (X -2 (())
To summarize, the following holds for the covariate shift (B) in Corollary 2 when compared to the homogeneous case ( ()).
The exact bias of the TM estimate remains the same.
The exact variance inflation of the TM estimate in case is reduced by a factor of a 1.
This completes the proof.
42

---

D Computation Algorithm
Algorithm 1: Informative- Weightted Transfer MNI (WTM)
Input: all datasets {(X (0) .y(()) number of folds K, detection threshold Eo >> 0
Output: estimated index set of informative sources Σ, the WTM estimate @wrs
{(X [M y ) + randomly partition the target dataset into K folds of equal size no/ K
for q = to Q do
for k = 1 to K do
(X (x) y + {(X())) ()) (X)
MNI trained on the left-out target folds (X())) y (0))))))
@ -k| + g- TM pre-trained with (X(1), l and fine-tuned on (X (x (x
[ (0(0)) - no/K |y(0 (x(0 1 (000 per-fold loss of target-only MNI
I [g9) ) no/K K
|000 X (0)4 (( kll? per-fold loss of q-th TM
end
K k== [ik] (310)) (()))=))) terminal CV loss of q-th TM
1 Σ
TM
end
200) -
K Ei= 2 (330 (2101-k)) terminal CV loss of target-only MNI
ōC K-1 Ek= (3300 (0)1-k) 2
+ E [Q] E(a) E < solōc 0.01)}
r i.e. no informative source is detected then
@WTM - : no knowledge transfer
else
for I do
(1) +- i-th TM pre-trained with informative (X (x(1), ( (()) and fine-tuned on (x("), y(0)
uy ) -1, weight initialized by the inverse CV loss of i-th TM
end
Wi + wt/( Eie w:)) for all i E Σ: weight normalization
end
@WTM f- Σie 3i
E
Additional Numerical Experiments
This section presents additional experimental results along with their implications. The experimental setups are identical in Section 5. The average excess risk over 50 independent simulations plotted against each value of p (300, 400.... 1000). When implementing Algorithm 1, set K and én = 1/2. As for the ground truth Equation (1) given by
( = (1 3(9) (4)
( (1)))1/2) (B + -(()) + t(()) E [Quo
(z
where (0)) O,, all parametric configurations are specified in Section 5, See Appendix F.1 for more details on the generating process of the deterministic parameters (,((0, (814), 1
E.1 Benign Overfitting Experiments
Here, we replicate the experimental setup used for Figure 1.(a) under benign overfitting, but now compare to three single-source versions of the Poo [56]. Specifically, for each of Q
43

---

source datasets, we employ the following forms of pooled-MNI:
3
= arg min BeR
[|]] X X 3 = y
qE {0, {0.1)]
(2)) PM
arg aig
||3] X 3 = yl
qe (0,2)
P arg ginges?
||||| 3 3 y!
q (0,3)
That is, consider each pooled-MNI single-source transfer task, by having each 3 interpolate the target and only the q-th source.
Baseline
TM1
TM3
WTM
PM1
PM2
PM3
10
(a) SSR = 0.3,0.006)
(b) SSR = (0, 0.6,
Figure 3: We compare the performance of our proposed methods to single-soure source- sowrcc-pooeed-eeeeeeee poopod- -NNs under model shift, with the same setting Figure 1.(.). Figure (a) re-uses the exact seeds for Figure 1.(a), so the excess risk curves for _ ((4) and BWTM are identical across the two figures. In contrast, (b) uses distinct seeds, hence its curves differ from those in Figure 1.(a).
In Fig. 3, the first source has distribution shift, E() (0) A (0) where A (0) is as specified in Section and SSR, = While 3 pre-trained with the first source clearly outperforms the
target-only MNI, the fisstsoource-polee--Mi of the early-fusion approach Song et al. [56] for leveraging source data when the source and target distributions are exactly identical.
pooled-MNI ( even performs better. This confirms the strength
However, the effectiveness of our approach is much more pronounced in terms of its robustness model shift. Once model shift is introduced for the second and third sources with SSR, 0.3 and SSR; 0.6 respectively, the depicted curves change markedly. Both corresponding pooled-MNIs @ and 8(3), not only undergo negative transfer but also show unstable convergence in excess risk. On the other hand, both TM estimates ((2) and (( exhibit steadily decreasing excess risk with ( consistently inducing positive transfer. Even under model shift severe as SSR, 0.6, ( still matches (Fig. 3.(a)) or surpasses (Fig. 3.))) the target-only MNI. Overall, under the present experimental design, the model shift severe as SSR = 0.6 appears to mark threshold for the TM estimate in inducing positive transfer, whereas it deteriorates the performance of pooled-MNI below the target-only baseline.
F.2 Harmless Interpolation Experiments
We extend the experiments in Section 5.2 to the case where Q = 1 and the target sample size varies. We first set n to be the optimal transfer size n for (L3 as identified in Corollary 1:
p(p
nj
=
SNR(1 SSR,
(24)
where we use the floor function to ensure the optimum be an integer.
While ni 50 is fixed throughout Section 5.2, we now optimize no for the pooled-MNI Ares suggested by Song et al. [56]. Specifically, under the same isotropic Gaussian setting
44

---

experiment, Corollary 3,4 in Song et al. [56] gives
p² SNN. + nj SSR1
ngt tot no arg min
((Gos)) p ni
V 0.
(25)
That is, given each value of p. ni. SNR, and SSR1, the optimal target sample size that minimizes the excess risk of 3Pm is determined by ng. which can be zero. The constraint in Equation (25) ensures that the total training sample size no + ní is less than so Box always operates in an overparameterizee regime enabling interpolation.
Easeline
TM1
PM
SGD1
36
36
36
(a) SSR1 0.1
(b) SSR 0.4
(c) SSR 0.7
Figure 4: As in Figure (.,,), the target and source covariates are i..d. (V I,) with SNR 10. Once ni is determined by (24), the corresponding target sample size follows from (25).
Table 1: Optimal target and source sample sizes (n ni) for each value of p in Figure 4
Figure
Optimality
Sample size for each p {300, 400, ........
1000)
mi
8
9
11
13
14
16
18
(a)
ni
199
265
332
399
465
532
599
665
n2
38
46
55
84
93
(b)
mi
176
235
295
354
413
472
531
590
(c)
Yig
105
131
157
209
235
262
Y2
126
168
210
252
295
337
379
421
We observe an interesting pattern from Table for each value of p. the total sample size ni + nj remains identical across Figures (.))) (In some instances, the totals differ by due to the use of floor function when computing ni and ni.)) To provide context, once ni source samples are provided, Aex appears to select an appropriate number of target samples no to balance the overall quality of the training inputs. Specifically, in Fig. 4.(a). the pooled-MNI already has large amount of high-quality inputs with SSR = 0.1 obtained from the source, so it requires fewer additional target samples. In contrast, Fig. 4.(c), the training inputs are highly "corrupted" by source data of poor quality with SSR = 0.7, and consequently, the model pools much more target samples to "dilute" the corruption. This results in significant increase in ni compared to Fig. (.). This decision-makes the pooled-MNI appears natural, considering that the estimate is an early-fusion approach [56], where the source integrated at the input level and then jointly learned with the target data.
process of
When comparing ( with Brm. their performances are nearly identical in Fig. 4.(a), with our method performing slightly better. In fact, if SSR 0, i.e., no model shift, we have ni = 0 by Equation (25). In that case, both PM and TM estimates are reduced to the source-only MNI ((3 trained on ni samples, resulting in identical performance. However, once under model shift, ( is consistently superior even when the model shift is low that it is favorable for 3PM., as illustrated in Fig. 4.
where SSR = 0.1. The superior performance of ( is further highlighted Fig. 4.(c) where model shift is severe as SSR 0.7, as it is the only method that consistently induces positive transfer.
45

---

Throughout Fig. 4, the performance of SGD remains consistent regardless of the degree of model shift. This plausible because, while the total sample size vi + nī fixed for each value of p, the estimate pre-trains with fewer source samples and instead fine-tunes with more target samples in proportion to the degree of model shift.
F
Experimental Details
This section provides further details on the experiments presented in Section 5 and Appendix E. All numerical experiments are conducted by R using a standard laptop (ASUS ZenBook UX33IUN, Intel(R) Core(TM) 17-8550U CPU @ 1.80GHz, 16 GB 2133 MHz DDR3). All sourcecodes we release are scripted from scratch, relying on standard R packages such as MASS and mvtnorm.
For the evaluation metrie, we compute the excess risk of estimate 3r trained at the (-th simulation run by
(a.-3(0) Σ (0)) (B, (0)).
for & E [50]. where 50 is the number of independent simulations conducted. The deterministic
parameters (0() and (0) are generated once and kept fixed across the 50 simulations, guaranteeing they are agnostic to the replicate index t. We then report the average excess risk
50
B. B(0) T Σ (ß. -(:
50
To ensure the stable convergence of SGD [69] we test as benchmark transfer task, we tune the initial pre-training and fine-tuning learning rates for each experimental setup. While Wu al. [69] adopted an initial learning rate of 0.1 (both for pre-training and fine-tuning) in their experiments, we find this value leads divergence across all our experimental setups. Accordingly, we reduce the learning rates based preliminary implementations; for example, use 0.001 for both pre-training and fine-tuning Fig. (a).).
F. Parameter Generation Process
In the formulation of the ground truth model
y Zz (z (E())11) (3 + 5(11) + (
E [Qlo.
where (0)) Op. only the random quantities (Z)(), e(1) y vary across the 50 simulations. The
)
deterministic parameters (3((), (5(1), } remain fixed across all runs. While these fixed
quantities are sampled from specific probability distributions, we generate them using fixed random seed, thereby ensuring their deterministic nature.
Model coefficients. As specified Section 5, the target coefficient ((0) and the contrast vectors {8(a) q=1 are given by
(0) = 112
8( = (SSR, (0)
where (S, ( are pre-specified hyperparameters. Each p-dimensional vector ( is independently sampled from / (Op, In) and then normalized so that [ = for all g E [Q]o- Each u(4) is held fixed across the 50 replications.
Spectral decomposition of covariance, Random sampling for covariance generation is used only in producing Figures 1.(b) and 1.(c). In these figures, the source covariances are defined via the spectral decomposition
E ( V(9) A VO.T
E [
from the orthogonal group O, = (Q R Q Q QQ Jg=1 are independently sampled Ip}- Specifically, we first independently sample the matrix ( E Rexp with i..d. standard Gaussian entries (which is then fixed across the 50 simulations), perform its QR decomposition ( Q(0) and assign v()) (()). Consequently, the matrices ( 1==1 held fixed.
where (A 12 are fixed as specified in Section 5, and [V(4)
46

---

F.2
Variability of Experiment Results
Table 2: Mean and standard deviation (in parenthesis) of excess risk
Figure
Estimate
p
300
400
500
600
700
800
900
1000
11,577
9.846
6.978
6.321
5.389
4.721
4.719
3.920
Target-only MNI
(1.686)
(1.734)
(1.008)
(1.187)
(0.796)
(1.030)
(0.743)
(0.701)
7,491
6.021
5.105
4,453
4.140
3.690
3.563
3.249
TMI
(1.396)
(1.019)
(0.915)
(0.713)
(0.709)
(0.687)
(0,503)
(0.560)
1.(a)
9.151
7.318
6.191
5.356
4.586
4.216
4.148
3.810
TM2
(1.602)
(1.404)
(1.153)
(0,905)
(0.812)
(0.713)
(0.688)
(0.695)
10.006
8.289
6.760
6.313
5.855
4.724
4,543
4.282
TM3
(2.021)
(1.378)
(1.128)
(1.164)
(0.787)
(0.934)
(0.795)
(0.678)
6.370
5.462
4.865
4.024
3.735
3.244
3.249
2.986
WTM
(1,423)
(1,154)
(1,083)
(0.584)
(0.382)
(0.553)
(0,469)
(0.882)
71.942
39.543
12.258
12.788
11.343
4.805
5.795
5.638
Pooled-MNI
(16.986)
(3.745)
(1.772)
(1.184)
(1.484)
(0.665)
(0.857)
(0,582)
14.864
11.359
8.749
7.558
6,421
5.627
4.956
4,470
SGDI
(1.379)
(1.069)
(0.957)
(0.742)
(0.467)
(0.581)
(0.427)
(0.447)
16.129
13.576
9.251
9.444
7.095
5.521
4,986
5.519
SGD2
(1.751)
(1.983)
(1.024)
(1,322)
(1.020)
(0.608)
(0.471)
(0.937)
18.504
18.032
10.328
7.729
7.766
5.744
6,384
5.488
SGD3
(3,544)
(3.918)
(1,336)
(0.970)
(1,359)
(0.789)
(1.169)
(0.683)
11.577
9.846
6,978
6.321
5.389
4.721
4.719
3,920
Target-only MNI
(1.686)
(1.734)
(1.008)
(1.187)
(0.796)
(1.030)
(0.743)
(0.701)
9.744
7.975
6.638
5,591
5.024
4.531
4.198
3.858
TMI
(1.836)
(1.395)
(1.179)
(0.975)
(0.832)
(0.861)
(0.741)
(0.624)
I.b)
9.416
7.695
6.772
6,047
5.066
4.592
4.467
3.860
TM2
(1.695)
(1.469)
(1.217)
(1.073)
(0.771)
(0.949)
(0.717)
(0.616)
20.849
14.105
10.661
8.655
6.819
5.995
5,352
4.600
TM3
(4.591)
(2.828)
(1.985)
(1.765)
(1.165)
(1.067)
(0.938)
(0.778)
8.648
7.614
6,288
5,638
4.853
4.457
4,464
3.614
WTM
(1.404)
(1.858
(0.865)
(1.115)
(0.649)
(1.181)
(0.754)
(0.478)
75.355
22.625
14.484
9,085
7.697
6.592
S.476
4,884
Pooled-MNI
(22.828)
(4.058)
(4.170)
(1.799)
(1.164)
(1.320)
(0.948)
(0.738)
20.011
14,223
11.006
9.630
8.247
6,970
6.293
5.710
SGDI
(2.690)
(1,551)
(1.328)
(1.108)
(0.963)
(0.877)
(0.616)
(0.606)
19.957
15.045
11.291
9.973
8.543
7.317
6,500
5.824
SGD2
(2.149)
(1.341)
(1.136)
(1.102)
(0.817)
(0.835)
(0.602)
(0.531)
20.247
15.198
11.462
9,916
8.545
7.332
6.515
5.763
SGD3
(2,226)
(1.301)
(1.135)
(1.092)
(0.820)
(0.831)
(0.591)
(0.529)
11.577
9.846
6.978
6.321
5.389
4.721
4.719
3,920
Target-only MNI
(1.686)
(1.734)
(1.008)
(1.187)
(0.796)
(1.030)
(0.743)
(0.701)
9.735
7.977
6.638
5.589
5,021
4.531
4.198
3.857
TM1
(1.845)
(1.397)
(1.181)
(0.972)
(0.833)
(0.862)
(0.741)
(0.622)
1.Cc)
9,361
7.657
6.747
6.038
5.048
4,570
4,454
3.840
TM2
(1.680)
(1.469)
(1.219)
(1.072)
(0.774)
(0.940)
(0.708)
(0.621)
11.031
8.763
7.170
6.295
5.361
4.720
4.392
3.813
TM3
(2,251)
(1.668)
(1.170)
(1.104)
(0.873)
(0.832)
(0.721)
(0.564)
8.296
7.357
6.018
5,480
4.814
4,238
4.376
3.541
WTM
(1.351)
(1.671)
(0.846)
(1.076)
(0.667)
(0.956)
(0.783)
(0.521)
47

---

26,400
9,480
7.341
5,910
5.306
4.540
4.373
3.777
Pooled-MNI
(6.839)
(1.586)
(1.525)
(1.004)
(0.727)
(0.842)
(0.782)
(0.464)
20.181
14,498
11,164
9.715
8,336
7.060
6,353
5.772
SGDI
(2.567)
(1,478)
(1.275)
(1.083)
(0.918)
(0.844)
(0.597)
(0.623)
20.174
15,208
11,422
10.029
8.603
7.379
6.542
5.883
SGD2
(2.162)
(1,355)
(1.140)
(1.114)
(0.813)
(0.825)
(0.593)
(0.591)
20.405
15.328
11.561
9.983
8.606
7.391
6.554
5.833
SGD3
(2.225)
(1.320)
(1.140)
(1.106)
(0.818)
(0.822)
(0.585)
(0.594)
8,510
8.839
9.166
9.249
9.346
9,440
9,490
9.555
Target-only MNI
(0.259)
(0.299)
(0.169)
(0.119)
(0.135)
(0.130)
(0.124)
(0.091)
7.790
8,344
8.698
8,808
9.014
9.112
9.222
9.314
TMI
2.(a)
(0.426)
(0.291)
(0.292)
(0.252)
(0.251)
(0.186)
(0.167)
(0.170)
6.715
7.146
7.253
7.318
7.403
7.407
7.555
7.606
TM2
(0.555)
(0.466)
(0.445)
(0.335)
(0.318)
(0.324)
(0.320)
(0.318)
5.526
5.831
5,962
5,923
6.213
6,251
6.363
6.342
WTM
(1.040)
(0.900)
(0.797)
(0.951)
(0.936)
(1.006)
(0.959)
(1.051)
33.144
15.647
12.361
11.092
9.842
9.663
9,413
9.117
Pooled-MNI
(6,900)
(2.941)
(1.540)
(1.204)
(0.580)
(0.697)
(0.573)
(0.266)
9.424
9.445
9.506
9,496
9.531
9.549
9.603
9.619
SGDI
(0.103)
(0.110)
(0.118)
(0.103)
(0.108)
(0.080)
(0.098)
(0.093)
8.886
8.833
8.786
8.626
8.676
8.613
8.660
8.655
SGD2
(0.160)
(0.182)
(0.139)
(0.149)
(0.161)
(0.160)
(0.135)
(0.152)
8.510
8.839
9.166
9.249
9.346
9.440
9.490
9.555
Target-only MNI
(0.259)
(0.299)
(0.169)
(0.119)
(0.135)
(0.130)
(0.124)
(0.091)
7.640
8.219
8.607
8.741
8.958
9.059
9.168
9.273
TM1
2.(b)
(0.413)
(0.288)
(0.289)
(0.239)
(0.251)
(0.186)
(0.171)
(0.168)
5.654
6.016
6.083
6.147
6,253
6.255
6.343
6.387
TM2
(0.439)
(0.359)
(0.322)
(0.275)
(0.268)
(0.263)
(0.253)
(0.273)
4.858
5.290
5.313
5,239
5.535
5.520
5.774
5.901
WTM
(0.651)
(0.677)
(0.243)
(0.307)
(0.268)
(0.280)
(0.759)
(1.133)
24.454
12,022
9.331
8.486
7.776
7.576
7.481
7.266
Pooled-MNI
(5.173)
(2.586)
(1.132)
(0.822)
(0.541)
(0.550)
(0.334)
(0.287)
9,468
9.482
9.540
9.524
9.554
9.565
9.607
9.623
SGDI
(0.094)
(0.101)
(0.106)
(0.096)
(0.099)
(0.076)
(0.093)
(0.085)
9.015
8.957
8,905
8,745
8.764
8.679
8.692
8. 8.640
SGD2
(0.143)
(0.156)
(0.122)
(0.134)
(0.142)
(0.144)
(0.123)
(0.137)
8.525
8.830
9.178
9.251
9.345
9.435
9.491
9.551
Target-only MNI
(0.261)
(0.302)
(0.168)
(0.117)
(0.137)
(0.137)
(0.128)
(0.089)
2.(c)
7.647
8.192
8.592
8.747
8,960
9.059
9.166
9.261
TMI
(0.420)
(0.281)
(0.271)
(0.247)
(0.256)
(0.177)
(0.193)
(0.158)
4.813
5.073
5.142
5,222
5,366
5,297
5,368
5.451
TM2
(0.376)
(0.271)
(0.343)
(0.259)
(0.333)
(0.243)
(0.228)
(0.246)
3.943
4.178
4.279
3,974
4.328
4.153
4.188
4.058
WTM
(0.864)
(0.902)
(0.937)
(0.335)
(0.359)
(0.316)
(0.190)
(0.244)
11.577
9.846
6.978
6.321
5,389
4.721
4.719
3.920
Target-only MNI
(1.686)
(1.734)
(1.008)
(1.187)
(0.796)
(1.030)
(0.743)
(0.701)
7.491
6.021
5.105
4.453
4.140
3,690
3.563
3.249
TM1
(1.396)
(1.019)
(0.915)
(0.713)
(0.709)
(0.687)
(0.503)
(0.560)
3.(a)
9.151
7.318
6.191
5.356
4.586
4,216
4.148
3.810
TM2
(1.602)
(1.404)
(1.153)
(0.905)
(0.812)
(0.713)
(0.688)
(0.695)
10.006
8.289
6. 6.760
6.313
5.855
4.724
4,543
4.282
TM3
(2.021)
(1.378)
(1.128)
(1.164)
(0.787)
(0.934)
(0.795)
(0.678)
48

---

6.370
5.462
4.863
4.024
3.735
3.244
3.249
2.986
WTM
(1.423)
(1.154)
(1.083)
(0.584)
(0.382)
(0.553)
(0,469)
(0.882)
5.266
4,481
3.867
3,470
3. 3.199
2,913
2.672
2.519
Pooled-MNII
(0.540)
(0.435)
(0.459)
(0.389)
(0.318)
(0.380)
(0.283)
(0.299)
12.062
14,689
11.461
15.202
8.666
3.829
4,907
6.335
Pooled-MN12
(1.703)
(1.827)
(1.627)
(1,983)
(1.406)
(0.449)
(0.615)
(0.873)
35.867
43.077
12.266
10,009
17.337
6.611
8.298
7.215
Pooled-MNI3
(4.602)
(5.516)
(1.418)
(1.290)
(2.274)
(1.047)
(1.338)
(1.084)
13.150
11,451
9.587
7.813
7.186
6,870
6.328
5.511
Target-only MNI
(1.665)
(2.422)
(1.894)
(1,015)
(1.073)
(1.382)
(1.089)
(1.017)
7.329
6,477
5.936
5,263
4.871
4,269
4,138
3.829
TMI
(1,229)
(1,073)
(1,252)
(1,098)
(0.804)
(0.742)
(0.693)
(0.766)
3.(b)
9.073
8,249
6,711
5,800
5.532
4.857
4.836
4.544
TM2
(1.643)
(1.407)
(1.391)
(1.148)
(1.092)
(0.664)
(0.759)
(0.903)
11,002
8.839
7.858
7.154
6.263
5.519
5.317
4.714
TM3
(1.969)
(1,590)
(1.456)
(1.305)
(1.120)
(1.066)
(1.073)
(1.002)
6,403
5.516
5.260
4,462
4.282
3,541
3,994
3.515
WTM
(1.144)
(1.139)
(1.467)
(0.576)
(1.054)
(0.510)
(1.194)
(0.596)
5.371
4.887
4,262
3.935
3.611
3.377
3.110
2.883
Pooled-MNI1
(0.620)
(0,506)
(0.416)
(0.474)
(0.381)
(0.432)
(0.409)
(0.325)
16.679
14.829
7.570
7,246
10.062
4.507
11.547
9.949
Pooled-MN12
(1.930)
(1.912)
(0.869)
(1.117)
(1.460)
(0.477)
(1.624)
(1.487)
16.011
16.383
14.135
30,615
11.072
12.042
12.065
19.334
Pooled-MNI3
(2.094)
(2.609)
(2.164)
(3.426)
(1.285)
(1.826)
(1.423)
(2.228)
9.831
9.798
9.843
9.815
9.816
9.847
9.840
9.835
Target-only MNI
(0.106)
(0.116)
(0.075)
(0.085)
(0.067)
(0.061)
(0.067)
(0.056)
4.(a)
5.966
5.864
5,811
5,886
5,953
5.958
6.026
5.945
TMI
(0.500)
(0.467)
(0.380)
(0.380)
(0.364)
(0.372)
(0.304)
(0.299)
6.179
6.006
5.913
5,984
6.071
6.074
6.072
Pooled-MNI
(0.551)
(0.525)
(0.396)
(0.383)
(0.360)
(0.395)
(0.325)
(0.341)
9.016
8.916
8.791
8.613
8.535
8.464
8.418
8.367
SGDI
(0.149)
(0.134)
(0.120)
(0.145)
(0.148)
(0.129)
(0.154)
(0.154)
9.186
9.216
9.157
9.215
9.186
9,226
9.175
9.147
Target-only MNI
(0.246)
(0.201)
(0.180)
(0.159)
(0.164)
(0.135)
(0.137)
(0.141)
4.(b)
7,269
7,336
7.225
7.316
7.297
7.301
7.289
7.284
TMI
(0.638)
(0.552)
(0.401)
(0.332)
(0.394)
(0.380)
(0.293)
(0.293)
8.341
8.516
8.381
8.423
8.369
8.480
8.408
8.382
Pooled-MNI
(0.872)
(0.728)
(0.648)
(0.547)
(0.508)
(0.548)
(0.435)
(0.418)
9.091
8.885
8.796
8.647
8.626
8.526
8.535
8.522
SGDI
(0.129)
(0.160)
(0.149)
(0.164)
(0.155)
(0.196)
(0.151)
(0.167)
7.686
7.673
7.734
7.679
7.710
7.709
7.726
7.751
Target-only MNI
(0.381)
(0.299)
(0.276)
(0.276)
(0.299)
(0.258)
(0.212)
(0.218)
4.(c)
7.274
7.254
7.313
7.368
7.286
7.354
7,309
7.351
TMI
(0.618)
(0.523)
(0.463)
(0.376)
(0.350)
(0.351)
(0.319)
(0.338)
10.574
10.703
10.688
10.545
10.579
10.850
10.684
Pooled-MNI
(1.025)
(0.945)
(0.917)
(0.784)
(0.680)
(0.728)
(0.737)
(0.806)
8.971
8.826
8.785
8.615
8,599
8.481
8.555
8.508
SGDI
(0.178)
(0.160)
(0.132)
(0.179)
(0.171)
(0.182)
(0.196)
(0.149)
49


Reviews:
[{"review_id": "zqOwdcKFvi", "reviewer": "Reviewer_KZWN", "summary": "This investigates the bias and variance trade-off in transfer learning based on Minimum-l2-norm interpolator (MNI) in high dimensional linear regression models. A two-step transfer learning approach is proposed, and the non-asymptotic exccess risk is studied under some theoretical assumptions on model-shift and covariate-shift. Numerical performance of the proposed transfer learning procedure is evaluated through simulation studies. Overall, the paper is well written and clearly presented.", "questions": "1. The theoretical insights are based on rather restrictive assumptions. For example,\n\n (a) In point 3 of Assumption 1, the condition that all covariance matrices can be simultaneously diagonalized (as assumed in [19], [24], [29]) implies that $\\prod_{j=1}^p\\frac{\\lambda_j^{(0)}}{\\lambda_j^{(q)}}$ is bounded for all $q$. This assumption appears quite restrictive and somewhat unnatural in practice. Could the authors provide some meaningful real-world examples where this condition would realistically hold?\n\n(b) In Section 3.1, the paper analyzes the “non-benign” case that violates Assumption 2. However, in this setting, the MNI estimator does not even provide any benefit. How do the theoretical insights from this analysis offer practical value for developing or understanding transfer learning methods?\n\n(c) The results in Sections 3.2 and 3.3 are interesting from an intuitive perspective but seem to have limited theoretical implications. Notably, the actual estimator proposed in Section 4 is not derived from these theoretical results, but is instead based purely on using cross-validation errors. While this is intuitively reasonable, it is also fairly straightforward and does not appear to leverage the earlier theory in a meaningful way.\n\n2. The result in Lemma 1 is somewhat counterintuitive. Typically, one would expect transfer learning to increase the effective sample size, thereby reducing the variance. However, Lemma 1 suggests that the variance is universally increased by transfer learning. Could the authors provide further clarification or intuition for why this phenomenon occurs?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength: some insights are derived for transfer learning using MNI estimator.\nWeakness: the derived results has little practical implications.", "was_revised": true, "final_justification": "The authors' rebuttal has clarified my earlier questions and I would like to raise my score to 5.", "rebuttal": "Dear Reviewer KZWN,\n\nWe deeply appreciate your valuable feedbacks and positive summary of our submission. We address your questions as follows.\n***\n**Q1.(a) Clarifying point 3 of Assumption 1**\n\nUnder simultaneous diagonalizability, we may write $\\Sigma^{(0)}=VD^{(0)}V^\\top$ and $\\Sigma^{(q)}=VD^{(q)}V^\\top$ for a shared orthogonal matrix $V$. In this case, point 3 of Assumption 1 implies\n$$\n||\\Sigma^{(0)}(\\Sigma^{(q)})^{-1}||\\_{op}=||VD^{(0)}(D^{(q)})^{-1}V^\\top||\\_{op}=\\max\\_{j\\in [p]}\\frac{\\lambda_j^{(0)}}{\\lambda_j^{(q)}} = O(1).\n$$\nThat is, the **maximum** of eigenvalue ratios is bounded, rather than the **product** of ratios. This bounded eigenvalue ratio condition is commonly adopted by recent works on the generalization of linear model under covariate shift; please refer to, e.g., [1], [2], [3], and [4]. In addition, [5] assumes a uniform bound on the density-ratio $dP_0/dP_q$ in RKHS setting. This upper-bounds the operator norm of the kernel-integral operators, the infinite-dimensional analogue of the above $O(1)$ condition. Moreover, this condition is not confined to technical convenience. For example, in multi-site neuro-imaging, [6] found that scanner-specific variance multipliers are mostly contained in $[0.5,1.7]$, implying a bound of $\\lesssim 2$. Hence our assumption aligns with current covariate shift analysis and is well grounded in real data.\n\nWe hope this answer clarifies the implication and relevance of our assumption.\n\n***\n**Q1.(b) Motive for analyzing transfer MNI under isotropic covariates**\n\nWe especially thank the reviewer for raising this important question. First of all, we kindly note that Assumption 2 holds for the isotropic covariance $\\Sigma=I_p$ as well, since taking $k^*=0$ gives $r_0(I_p)=p>n$. This underpins the pattern where the estimation variance of single-task MNI vanishes as $p\\gg n$ even under isotropic sub-Gaussian designs. Consequently, the bias of MNI dominates its variance in high-dimension, and the excess risk is $O(||\\beta^{(0)}||_2^2)$. In this sense, MNI under isotropic design is often named **harmless interpolation**-while not \"benign\", at least excess risk does not catastrophically explode.\n\nAlthough MNI under isotropic design is sub-optimal as noted, **most papers that built today’s understanding of single-task MNI start from an isotropic (and primarily Gaussian) design**, because it is the only setting where one can interpret the bias/variance terms in closed form and read off qualitative phenomena such as double descent. Many references cited in our manuscript fall into this case, including, but not limited to, [7], [8], and [9]. Isotropic design is even more predominant in analyzing the min-$\\ell_1$-norm interpolant, with an additional sparsity assumption typically imposed; please see references [8]-[11] in our response to Reviewer E9gm.\n\nThat said, we believe that our Theorem 1 and Corollary 1 under isotropic design provide an adequate baseline for quantifying how the measures of source quality such as signal-to-shift ratio **(SSR)** $||\\delta^{(q)}||_2^2/||\\beta^{(0)}||_2^2$ and signal-to-noise ratio **(SNR)** $||\\beta^{(0)}||_2^2/\\sigma_q^2$ interact. Our analysis explicitly accounts for the role of model shift $\\delta^{(q)}$, which has not been addressed by recent works solely concerned with covariate shift (e.g., [1] and [3]). Moreover, our results draw a **favorable comparison** to [10], which proposes the **pooled MNI** approach and investigates transfer performance under isotropic Gaussian designs. As a pre-train and fine-tune approach, our TM estimate perfectly separates the contribution of target and source noise levels $\\sigma_0^2$ and $\\sigma_q^2$, whereas pooled MNI assumes a common noise level. In addition, Corollary 1 not only provides the necessary-and-sufficient condition for positive transfer, but also identifies the maximal excess risk improvement at the optimal transfer size $n_q^* \\propto \\mathrm{SNR}(1-\\mathrm{SSR})$, while Proposition 3.3 in [10] only gives  a sufficient condition and fails to quantify such maximal improvement. \n\nIn summary, analyzing Transfer MNI under the isotropic design furnishes the clearest lens through which one can quantify TM's bias-reduction mechanism. Moreover, in practice, **the the target-model signal magnitude $||\\beta^{(0)}||^2$ is itself typically an increasing function of ambient diemsion $p$** (e.g., more informative pixels, genes, or sensor channels that additively contribute to the square signal norm). Thus, as $p$ grows, the bias component-already dominant under isotropy-swells accordingly, **creating even a larger room for TM to reduce excess risk** compared to the target-only baseline. Our simulation studies in Sections 5.2 and F.2 corroborate these insights, as TM achieves marked reduction in total excess risk under isotropic designs.\n\n***\n**Q1.(c) Drawing a theoretical connection for WTM estimate**\n\nWe also appreciate the reviewer for this question, as it closely relates to our future research agenda suggested in Section 6 of our submission. To address the question, we suggest some sketch of the theoretical properties of WTM as follows. First, we recall that $I$ is the true (oracle) set of informative sources as identified in Eq. (7) of our paper. Accordingly, we define the oracle WTM (oWTM) as \n$$\n\\hat\\beta\\_{oWTM}:=\\sum\\_{i \\in I}w_i \\hat\\beta\\_{TM}^{(i)},\n$$\ni.e., oWTM ensembles truly informative TM estimates, while in practice our WTM is based on the set $\\hat{I}$ estimated by our cross-validation algorithm. Then, the following would hold for the oWTM:\n$$\n\\mathcal{R}(\\hat\\beta_{oWTM})=\\mathbb{E}\\left[\\left(\\sum_{i \\in I}w_i\\hat\\beta\\_{TM}^{(i)}-\\beta^{(0)}\\right)^\\top\\Sigma^{(0)}\\left(\\sum_{i \\in I}w_i\\hat\\beta\\_{TM}^{(i)}-\\beta^{(0)}\\right)\\right] \\leq \\mathbb{E}\\left[\\sum_{i \\in I}w_i \\left(\\hat\\beta\\_{TM}^{(i)}-\\beta^{(0)}\\right)^\\top\\Sigma^{(0)}\\left(\\hat\\beta\\_{TM}^{(i)}-\\beta^{(0)}\\right)\\right] \\leq \\sum_{i \\in I}\\mathbb{E}\\left[\\left(\\hat\\beta\\_{TM}^{(i)}- \\beta^{(0)}\\right)^\\top\\Sigma^{(0)}\\left(\\hat\\beta\\_{TM}^{(i)}-\\beta^{(0)}\\right)\\right] \\leq |I| \\max_{i \\in I} \\mathcal{R}(\\hat\\beta\\_{TM}^{(i)}),\n$$\nwhere the first inequality follows from Jensen's inequality, and $|I|$ is the cardinality of $I \\neq \\emptyset$.This implies that if $|I|$ is finite (which is true in general) and $\\max_{i \\in I}\\mathcal{R}(\\hat\\beta\\_{TM}^{(i)}) \\ll \\mathcal{R}(\\hat\\beta_{M}^{(0)})$, then oracle-WTM can achieve a robust positive transfer.\n\nThen, it remains to precisely characterize the consistency of source detection, i.e., to prove that the event $\\hat{I} = I$ holds with high probability, which is outlined as our future research agenda. We anticipate that such theoretical guarantee will require a \"separation condition\" that distinguishes informative from non-informative sources. For example, for some \"transferability threshold\" $h >0$, write $\\mathcal{I}_h := \\\\{q \\in [Q]: ||\\delta^{(q)}||_2 \\leq h\\\\}$. Then, we may need the condition such as $||\\delta^{(q)}||_2 \\geq h\\log(n_0)$ for $q \\notin \\mathcal{I}_h$ to infer detection consistency. Characterizing the minimal separation condition and matching it with bounds on excess risk when an uninformative source is transferred remains an open problem and is part of our planned research agenda.\n\n***\n**Q2. Why is the variance of TM always higher than the target-only variance?**\n\nBefore answering this question, we would like to first recall the decomposition structure of TM estimate, which is given by\n$$\n\\hat\\beta_{TM}^{(q)}=\\hat\\beta_{M}^{(0)}+(I_p-H^{(0)})\\hat\\beta_{M}^{(q)},\n$$\nwhere $I_p-H^{(0)}$ is the orthogonal projection onto the null space of target design, and $\\hat\\beta_{M}^{(0)}$ and $\\hat\\beta_{M}^{(q)}$ are independent MNIs exclusively trained on the target and source datasets respectively. Let $S:=row(X^{(0)})=Im(H^{(0)})$ and $S^\\perp=Ker(H^{(0)})$. Then, any vector uniquely decomposes as $\\beta = (\\beta)\\_S + (\\beta)\\_{S^\\perp}$ and our TM estimate satisfies\n$$\n(\\hat\\beta_{TM}^{(q)})\\_S=\\hat\\beta_M^{(0)},\\quad(\\hat\\beta_{TM}^{(q)})\\_{S^\\perp}=(I_p-H^{(0)})\\hat\\beta_M^{(q)}.\n$$\nThat is, TM \"preserves\" the signal learned by the target-only MNI on the target-supported subspace and \"carries over\" source information only in the null-space where the target offers no information. We kindly ask the reviewer to see how this “projection + carry-over’’ principle is further supported in our reply to **Reviewer bZxw (Q3)** and extends verbatim to the non-linear RKHS setting (**Reviewer E9gm, Q2**).\n\nWhile the mechanism explains bias reduction, it inevitably inflates variance. Since each interpolant is independent, the covariance (conditional on $X$) of TM satisfies\n$$\nCov(\\hat\\beta_{TM}^{(q)}) = Cov(\\hat\\beta_M^{(0)})+(I_p-H^{(0)})Cov(\\hat\\beta_M^{(q)})(I_p-H^{(0)}) \\\\: \\succeq \\\\: Cov(\\hat\\beta_M^{(0)}),\n$$\nso the estimation variance of TM measured on target domain inflates:\n$$\n\\mathcal{V}(\\hat\\beta_{TM}^{(q)}) = Tr[\\Sigma^{(0)}Cov(\\hat\\beta_{TM}^{(q)})] \\\\: \\geq \\\\: \\mathcal{V}(\\hat\\beta_M^{(0)}) = Tr[\\Sigma^{(0)}Cov(\\hat\\beta_M^{(0)})].\n$$\n\nNevertheless, the second term of TM lives entirely in the target null-space, so the variance inflation is confined to directions that already have \"small\" predictive power under the target distribution. Our main message shows that whenever the accompanying bias reduction dominates this controlled variance inflation, Transfer MNI achieves strictly lower excess risk than the target-only baseline.\n\nWe hope our answers clarify your question.\n\n***\n**References**\n\n[1] arXiv:2404.00522.\n\n[2] arXiv:2106.12108.\n\n[3] arXiv:2208.01857.\n\n[4] arXIv:2412.14474.\n\n[5] arXiv:2205.02986.\n\n[6] Neuroimage.2017.11.024.\n\n[7] arXiv:1903.07571.\n\n[8] arXiv:1903.09139.\n\n[9] 21-AOS2133.\n\n[10] arXiv:2406.13944"}, {"review_id": "b6vZCajGUQ", "reviewer": "Reviewer_E9gm", "summary": "The first paper I am aware of that demonstrates benign overfitting in transfer learning related models. The work provides a comprehensive overview of this model under the $\\ell_2$ norm interpolator and studies it in great detail.", "questions": "Can you extend these results to a minimum norm interpolator that is not the $\\ell_2$ one?\n\nCan you extend to isotropic sub-Gaussian (even uniformly bounded entires from above and below, or isotropic log-concave i.i.d. ) covariates at least in the case of $\\ell_2$ minimum norm interpolator (extending to sub-Gaussians was done recently to $\\ell_p$ norm?", "limitations": "These results are only valid for the $\\ell_2$ minimum norm interpolator with a low-rank matrix.\n\nAs I do think that the authors did a good and comprehensive work despite these limitations, I am willing to increase my score to accept, under a considerable discussion on $\\ell_p$ norms (see Donhasuer et al., Kur et al., and Lecue et al. works) related to objects and the RKHS settings, and cite the relevant work. In those settings, benign overfitting was shown, so I am curious how to connect your work to those settings.\n\nPlease use an anonymous link to address this topic.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengh:\n- \\emph{extremely} comprehensive analysis \n- The first paper to study this problem in this natural setting\n- Potentially high-impact baseline work\n\nWeakness:\n- No novel new technical ideas that do not appear in the works of Bartlett and Koehler.\n- only \\ell_2 norm interpolator, i.e., a closed formula.", "was_revised": true, "final_justification": "Solid work on this setting with potential impact", "rebuttal": "Dear Reviewer E9gm,\n\nThank you for your thoughtful comments and for recognizing the depth of our analysis and its potential as a high-impact baseline for interpolation-based transfer learning methods. We kindly note that we were not allowed to include any external links in our rebuttal, so we address your questions as follows.\n***\n**Q1. Can the result under isotropic Gaussianity extend to isotropic sub-Gaussianity?**\n\nOur answer is yes, due to the so-called *universality* phenomenon that has been justified in other learning problems with i.i.d. designs (e.g., [1],[2]). To elaborate beyond this heuristic, we note that the main property we used to derive Theorem 1 under isotropic Gaussianity in our paper is\n$$\n\\mathbb E H= \\mathbb E X^\\top(XX^\\top)^{-1}X = (n/p)I_p,\n$$\nwhich follows from the rotational invariance of Gaussian distribution. Such concentration of the projection $H$ to $(n/p)I_p$ would hold for a general sub-Gaussian distribution as well.\n\nSuppose $X\\in\\mathbb R^{n \\times p}$ has i.i.d. sub-Gaussian entries of zero mean and unit variance, so each row has covariance $I_p$. Let $z_i$ be the $i$-th column of $X$ for $i \\in [p]$ and assume $X$ is of full rank. Let $A:=XX^\\top=\\sum_{i} z_iz_i^{\\top}$, $A_{-i}:=\\sum_{j\\neq i}z_jz_j^{\\top}$, and $H_{ij}$ be the $(i,j)$-th entry of $H$. Applying Sherman-Morrison-Woodbury (SMW) formula gives\n$$\nH_{ij} =\\frac{z_i^\\top A_{-i}^{-1}z_j}{1+z_i^\\top A_{-i}^{-1}z_i}.\n$$\nSince $Tr(H)=\\sum_{i=1}^p H_{ii}=n$ and the diagonals are i.i.d., we have $\\mathbb E H_{ii}=n/p$ for all $i\\in [p]$. For the off diagonal terms $H\\_{ij}$ with $i\\neq j$, observe that conditioning $A_{-i}^{-1}$ and ${z_j}$ that are independent of $z_i$, we have\n$$\nH_{ij}|(A_{-i}^{-1},z_j)\\overset{d}{=}-H_{ij}|(A_{-i}^{-1},z_j)\n$$\nwhenever $z_i$ follows a distribution symmetric about the origin, i.e., $z_i \\overset{d}{=}-z_i$. This implies\n$$\n\\mathbb E[H_{ij}|A_{-i}^{-1},z_j]=0\n$$\nand taking expectation once more gives $\\mathbb EH_{ij}=0$ for all $i\\neq j$. Thus, $\\mathbb EH = (n/p)I_p$ and this result hold for **any** distribution symmetric about 0, but sub-Gaussianity (e.g., each entry of $X$ are i.i.d. $\\mathrm{Unif}(-\\sqrt{3},\\sqrt{3})$ or Rademacher random variable) would warrant a sufficiently fast concentration of $H_{ij}$ to 0.\n\nTo address a general, asymmetric sub-Gaussian distribution, let $G:=\\sum_{k\\neq i,j}z_kz_k^\\top$, $\\alpha:=z_i^\\top G^{-1}z_i$, $\\beta:= z_j^\\top G^{-1} z_j$, and $\\gamma:=z_i^\\top G^{-1}z_j$ for $i\\neq j$. SWM formula gives\n$$\nH_{ij} = z_i^\\top(G+z_iz_i^\\top+z_jz_j^\\top)^{-1}z_j=\\frac{\\gamma}{(1+\\alpha)(1+\\beta)-\\gamma^2}.\n$$\nSince $G$ is PSD, we have $(1+\\alpha)(1+\\beta)-\\gamma^2 \\geq 1$ and $\\mathbb E|H_{ij}|\\leq\\mathbb E|\\gamma| \\leq(\\mathbb E \\gamma^2)^{1/2}.$ Since $z_i$ and $z_j$ are independent and  $\\mathbb Ez_iz_i^\\top=\\mathbb Ez_jz_j^\\top=I_n$, we also have $\\mathbb E[\\gamma^2|G]=Tr(G^{-2})$. Then, by [3], sub-Gaussianity gives\n$$\n\\mathbb E\\gamma^2 = \\mathbb ETr(G^{-2})\\lesssim\\frac{n}{(\\sqrt{p}-\\sqrt{n})^4}\\asymp\\frac{n}{p^2},\n$$\nand thus, $\\mathbb E|H_{ij}|\\lesssim\\frac{\\sqrt{n}}{p}$. That is, the off-diagonals of $H$ concentrate to 0.\n\nIn response to your inquiry regarding bounded or log-concave distributions, we evaluated the performance of target-only MNI and our methods under i.i.d. isotropic non-Gaussian designs. We used the same setting as in Figure 2(a) in our paper, changing only the input distribution. Specifically, we considered $\\mathrm{Unif}(-\\sqrt{3},\\sqrt{3})$ and the Laplace distribution with location $0$ and scale $1/\\sqrt{2}$. Both have mean zero and unit variance; the former is sub-Gaussian, while the latter is log-concave and sub-exponential but not sub-Gaussian. The excess risks across $p=300,500,700,900$ are as follows:\n|Estimate|Distribution|Excess Risk|\n|-|-|-|\n|Target-only MNI|Uniform|8.55 / 9.05 / 9.39 / 9.53|\n||Laplace|8.51 / 9.05 / 9.38 / 9.52|\n|TM1|Uniform|7.88 / 8.65 / 9.04 / 9.26|\n||Laplace|7.86 / 8.63 / 9.04 / 9.26|\n|TM2|Uniform|6.75 / 7.21 / 7.37 / 7.50|\n||Laplace|6.78 / 7.21 / 7.36 / 7.55|\n|WTM|Uniform|5.14 / 5.97 / 5.94 / 6.12|\n||Laplace|5.28 / 5.84 / 6.21 / 6.03|\n\nCompared to the excess risk table for Figure 2(a) on page 40 of our submission, the performance of each estimate remains nearly identical to that under the isotropic Gaussian case. We hope our explanation and empirical evidence answer your question.\n***\n**Q2. Can we draw a connection to the RKHS setting or min-$\\ell_p$-norm interpolant for $p\\neq 2$?**\n\nWe thank the reviewer for highlighting the connection between our setting and the extensive literature on benign overfitting under general norm constraints.  While a rich body of work analyzes the generalization of the min-RKHS-norm interpolant (**kernel ridgeless regression, KRR**), to the best of our knowledge, systematic results for KRR in transfer learning context are still lacking.\n\n[4] demonstrated that the KRR can benignly overfit in high dimensions due to an *implicit bias* shaped by kernel curvature, high ambient dimension $d$, and spectral decay of the empirical covariance and kernel matrices. In contrast, [5] proved that if $d$ is fixed, KRR with Laplace kernel is inconsistent for any bandwidth, pinpointing the \"blessing of dimensionality’’ as essential for benign overfitting of KRR. [6] showed that for Gaussian kernels, ridgeless KRR with fixed $d$ is inconsistent even with optimal bandwidth tuning and can be *catastrophically overfitting*. They introduce a \"taxonomy\" overfitting depending on how $d=d_n$ grows with sample size $n$. [7] provide geometric upper bounds for KRR and establish a restricted isomorphism property that explains why certain data–dependent kernels (e.g., NTK) enter the benign-overfitting regime. Taken together, consistency hinges on a *spectral bias*, either implicit or explicit via geometry, and low-dimensionality demands additional regularization.\n\nFavorably, our TM estimate in linear setting admits a structure that can be extended to the RKHS; **we kindly refer the reviewer to our response to Q4 of Reviewer bZxw, where we present most mathematical expressions**. Let $\\mathcal{H}$ denote a RKHS associated with a PD kernel $K:\\mathcal{X}\\times\\mathcal{X}\\to\\mathbb{R}$. We fit respective KRRs $\\hat f_M^{(0)}$ and $\\hat f_M^{(q)}$ on target and source. Write $E_0:\\mathcal{H}\\to\\mathbb{R}^{n_0}$ for evaluation on the target inputs and let $K_0=E_0E\\_0^*$ be the target Gram matrix. The orthogonal projector onto the target representer span is\n$$\nP_0 = E_0^*K_0^{-1}E_0,\n$$\nwhich is a RKHS analogue of $H^{(0)}$ in our linear setting. Then, our \"TM-RKHS\" is of the analytical form\n$$\n\\hat f_{TM}^{(q)}=\\hat f_M^{(0)}+(I-P_0)\\hat f_M^{(q)}.\n$$\nLet $S=Im(P_0)$ and $S^\\perp=ker(P_0)$. Decomposing $\\hat f=(\\hat f)_S+(\\hat f)\\_{S^\\perp}$ gives\n$$\n(\\hat f\\_{TM}^{(q)})_S=\\hat f_M^{(0)},\\quad(\\hat f\\_{TM}^{(q)})\\_{S^\\perp}=(I-P_0)\\hat f_M^{(q)}.\n$$\nThat is, TM \"retains\" the target-only KRR in target-supported directions and transfers the source KRR only where the target provides no information, mirroring the linear TM decomposition (**please check our response to Q3 of Reviewer bZxw for its validity**.) **We also numerically validate TM's performance in RKHS under the benign setting considered by [4] in our response to Q4 of  Reviewer bZxw.**\n\nBeyond the $\\ell_2$ case, systematic theory exists mostly for $\\ell_1$, i.e., **basis pursuit (BP)**. These works share two common grounds-isotropic Gaussian design and sparsity level $s$-but differ in how $s$ scales with dimension $d$. [8] consider linear sparsity $s=\\epsilon d$ with $\\epsilon<1$ and $d\\asymp n$; [9] obtain consistency when $s\\lesssim n/log(d/n)^5$ with $d\\gg n$ but $d\\leq \\exp(n^{1/5})$; [10] require $s/\\log(d/n)\\lesssim n$. Notably, [11] identified regime where $\\ell_2$-MNI can outperform BP even under sparsity.\n\nA natural analogue \"Transfer BP (TBP)\" in  the $\\ell_1$-setting would be the anchored basis pursuit:\n$$\n\\hat\\beta_{TBP}^{(q)}:=\\arg\\min_{\\beta}||\\beta-\\hat\\beta_{BP}^{(q)}||_1\\quad s.t.\\quad X^{(0)}\\beta=y^{(0)}.\n$$\nUnder the usuaul restricted-eigenvalue (RE) condition on the target design, the interpolation constraint fixes the coordinates that the target data can identify,  while the $\\ell_1$ anchor would keep the remaining coordinates close to the source solution. So heuristically, TBP would mirror the similar behavior as in the $\\ell_2$-norm/RKHS transfer setting discussed earlier. It would be beneficial when target task is sparse enough, the source task itself is \"$\\ell_1$-benign\" under one of the sparsity-dimension regimes above, and $d$ is high, but not too high to warrant the generalization of single-task BP. Heuristically, we anticipate that if the target design satisfies the RE condition (as in the references), TBP would benefit by staying close to $\\hat\\beta\\_{BP}^{(q)}$ not captured by the target, provided that the source task itself is benign. This mirrors the algebraic decomposition obtained in the ridgeless (linear,RKHS) regimes.\n\nIn contrast to $\\ell_2$ and RKHS settings, the TBP lacks a closed-form expression; nevertheless, it is a convex LP problem solvable by standard methods.  Because the transfer behavior of BP remains virtually unexplored, deriving risk bounds for TBP is an open and highly promising research direction. More broadly, although our precise characterization focuses on the linear $\\ell_2$ geometry, we believe the underlying “project–then–carry’’ framework provides a versatile baseline that naturally extends to both sparsity-aware ($\\ell_1$) and RKHS regimes. We hope this work stimulates further investigation into when such structure-aware interpolators can deliver positive transfer under distribution shift.\n\n***\n**References**\n\n[1] ISIT.2017.8006947.\n\n[2] arXiv:2003.10431.\n\n[3] arXiv:1011.3027.\n\n[4] 19-AOS1849.\n\n[5] arXiv:1812.11167.\n\n[6] arXiv:2409.03891.\n\n[7] arXiv:2404.07709.\n\n[8] arXiv:2110.09502.\n\n[9] arXiv:2111.05987.\n\n[10] arXiv:2012.00807. \n\n[11] arXiv:2110.02914."}, {"review_id": "F0RTwl1IDV", "reviewer": "Reviewer_bZxw", "summary": "This paper studies transfer learning in linear models. The specific transfer learning set up is a form of \"late-fusion knowledge transfer\" with potentially multiple source tasks. The authors then study the effect of model shift and derive several results on the maximal risk improvement. Perhaps most interestingly, the authors propose a procedure to identify informative sources and a weighting scheme to use the contributions from multiple source tasks.", "questions": "What is the relationship between the free-lunch covariate shift and \"feature overlap\" suggested by recent works on transfer learning?\n\nHow scalable is the source detection algorithm in the non-linear setting? Are there efficient proxies for it?\n\nWhat minimal numerical experiment could illustrate the informative weighted transfer in a non-linear setting?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality:\n\nOverall, the quality of the paper is high. The explanations of the results are crisp and the numerical results provide a nice demonstration of the theoretical results.  \n\nClarity:\n\nThe paper is well-written and mostly developed logically. As is natural for a paper of this type, much of the technical detail is deferred to the Appendix. I would have prefered that the authors provide more commentary on the nature of the arguments in the main text. For example, some important objects are discussed in the main text without much context. Effective rank is a good example of this---it would be much clearer to explain what the role of effective rank will be in the context of the results and then give the definition. Overall, though, I think these are stylistic rather than substantive quibbles. \n\nSome specific suggestions include:\n\n- more motivation for late-fusion knowledge transfer when it is introduced \n- Lemma 1, say where it is proved in the appendix, comment on the meaning of the bias term, and define $\\delta^{(q)}$. \n\nSignificance:\n\nTransfer learning is a relatively important topic given the popularity of foundation models, which indeed are often specialized for some task. Better theoretical understanding of this topic is crucially important and I think, in particular, the question of selecting good source tasks is a particularly relevant focus. On the other hand, the significance of the results is of course limited by the fact that the authors analyze only the case of linear models. If the authors could illustrate their results, even numerically, in a more realistic setting, that might improve the impact. \n\nOriginality:\n\nThere are quite a few recent works that derive similar results in either linear setting or in the context of linear probing. One thing that stood out is the \"free-lunch covariate shift\" results. These are quite striking but also similar to the results in: https://arxiv.org/abs/2202.10054, https://arxiv.org/abs/2206.12314, https://arxiv.org/pdf/2410.08194. I think the nature of the results should be better contextualized within this literature that is more focused on \"feature learning\".", "was_revised": true, "final_justification": "The response resolves all my questions and I'm happy to maintain my accept score.", "rebuttal": "Dear Reviewer bZxw,\n\nWe deeply appreciate your thoughtful comments and high appraisal of the overall quality and clarity of paper. We address your questions as follows.\n***\n**Q1. Motivation for late-fusion knowledge transfer**\n\nThank you for raising this important question. The term **late-fusion** contrasts with the **early-fusion** (Pooled MNI) approach to transfer learning for MNI proposed by [1]. Unlike [1], our method aligns with the typical pre-training and fine-tuning paradigm common in deep learning. One immediate advantage of our late-fusion approach is that it cleanly separates the contributions of target and source noise levels, $\\sigma_0^2$ and $\\sigma_q^2$. We will address your Question 3 to further substantiate the soundness of our approach.\n***\n**Q2. Definition of $\\delta^{(q)}$ and interpreting the bias of TM in Lemma 1**\n\nPlease note that Line 111 in our manuscript defines the contrast vector $\\delta^{(q)}:=\\beta^{(q)}-\\beta^{(0)}$ for each $q$-th source, which induces the model shift between the source and target. In Lemma 1, the bias of TM consists of three terms ($U_1+U_2+U_3$):\n$$ \n||(\\Sigma^{(0)})^{1/2}(I-H^{(0)})(I-H^{(q)})\\beta^{(0)} ||_2^2 +||(\\Sigma^{(0)})^{1/2}H^{(q)}\\delta^{(q)}||_2^2 +U_3,\n$$\nwhere $H^{(0)}$ and $H^{(q)}$ are orthogonal projections onto target and source design row space respectively. We omit the expression of cross-term $U_3$, as it is dominated by the first two terms with $2|U_3| \\leq \\sqrt{U_1U_2} \\leq U_1+U_2$. The first term $U_1$ quantifies how much of the target signal $\\beta^{(0)}$ lives in directions that neither target or source design matrix captures, as the term involves projections onto each null space, so it represents an irreducible error. It is zero only when the source design perfectly spans the signal, i.e., $H^{(q)}\\beta^{(0)}=\\beta^{(0)}$, which is not true in general. On the other hand, $U_2$ is the model shift energy $\\delta^{(q)}$ living in directions that are seen by the source with $H^{(q)}$ but unseen by the target due to $I-H^{(0)}$. Obviously, a small model shift with low $||\\delta^{(q)}||_2$  introduced by the source can lower $U_2$ (and thus $U_3$). \n***\n**Q3. Relationship between the free-lunch covariate shift and \"feature overlap\"**\n\nWe especially thank the reviewer for drawing this connection to the broader transfer learning literature, as it enables us to clarify how our free‑lunch covariate‑shift complements and strengthens existing feature‑overlap and LP/LP‑FT perspectives, thereby supporting the validity of our transfer approach.\n\nThe analysis in [2] is framed around **concept/task drift** under a fixed input distribution: the geometry that governs transfer is the **feature-space overlap** between the source and target regression functions. In the linear model context, this overlap is parametrized by an angle $\\theta\\in[0,\\pi/2]$ such that $\\langle\\beta^{(q)},\\beta^{(0)}\\rangle =||\\beta^{(q)}||_2||\\beta^{(0)}||_2 \\cos\\theta$. The main takeaway from [2] is that if the target function is well represented in the pretrained feature space (which manifest to large $\\cos\\theta$), transfer is better than target-only. This is rather a **feature-overlap/bias** concept, whereas our **free-lunch covariate shift** is characterized by a spectral relation between $\\Sigma^{(q)}$ and $\\Sigma^{(0)}$ that controls the variance of transfer. The setup of [2] keeps covariate distribution fixed across the target and source domains and varies the alignment of model parameter with the learned features. That said, the effect of free-lunch covariate shift is **complementary to** the feature-overlap perspective of [2]. An ideal overlap with large $\\cos\\theta$ would help achieve the bias of TM lower than the target task bias, and the free-lunch covariate shift can inhibit variance inflation. The two effects can play a role independently. \n\nNext, [3] investigates how fine-tuning (FT) can underperform linear probing (LP) out-of-distribution (OOD) when the pre-trained representation is strong and the shift is large, and they advocate **LP-FT** as a practical remedy. Their theoretical mechanism is that gradient updates under FT modify the feature extractor only in the training row-space and leave the orthogonal complement unchanged, potentially \"distorting\" features where the target test distribution places mass. Our Transfer MNI approach encodes an analogous \"do not distort what the target does not constrain* principle in an analytical form and in a linear setting. To corroborate this intuition, let $S:=row(X^{(0)})=Im(H^{(0)})$ and $S^\\perp=Ker(H^{(0)})$. Any vector decomposes uniquely as $\\beta=\\beta_S+\\beta_{S^\\perp}$ with $\\beta_S:=H^{(0)}\\beta$ and $\\beta_{S^\\perp}:=(I-H^{(0)})\\beta$. Since our TM estimate is given by\n$$\n\\hat\\beta_{TM}^{(q)} = \\hat\\beta_{M}^{(0)} + (I-H^{(0)})\\hat\\beta_{M}^{(q)},\n$$\nour TM estimate exactly matches the target-only MNI $\\hat\\beta_{M}^{(0)}$ on $S$ and exactly preserves the source-pre-trained MNI $\\hat\\beta_{M}^{(q)}$ on $S^\\perp$. This \"projection + carryover'' structure is the linear analogue of the empirical prescription behind LP/LP-FT: fit only the target-supported directions; avoid altering directions unsupported by the target data.\n\n***\n**Q4. Can the source detection algorithm be extended to non-linear settings?**\n\nTo address your question, we evaluated the performance of our cross-validation procedure in kernel ridgeless regression (KRR) within an RKHS, operating in a nonlinear, interpolating regime. As in [4], we use the RBF kernel\n$$\nK(x,x'):= \\exp(-||x-x'||_2^{2}/d),~x,x' \\in\\mathbb{R}^{d}\n$$\nfor an ambient dimension $d$. Given a training dataset $\\{(x_i,y_i)\\}\\_{i=1}^n$, let $K\\in\\mathbb{R}^{n\\times n}$ be the Gram matrix with $(i,j)$-th entry $K(x_i,x_j)$. It is well-known that the minimum-RKHS-norm interpolator (KRR) is given by\n$$\n\\hat{f}_M(\\cdot) =\\sum\\_{i=1}^n \\alpha_i K(\\cdot,x_i)\\quad\\alpha_i = K^{-1}y_i.\n$$\nLet $\\hat{f}\\_M^{(0)}$ and $\\hat{f}\\_M^{(q)}$ be the KRR trained on target $(X^{(0)},y^{(0)})$ and $q$-th source $(X^{(q)},y^{(q)})$ respectively. Our TM finds the solution that minimizes the RKHS-norm distance from $\\hat{f}\\_M^{(q)}$ while interpolating target data, and a brief projection argument gives the analytical function form\n$$\n\\hat f\\_{TM}^{(q)}(x):=\\hat{f}\\_M^{(q)}(x)+k_0(x)^{\\top}K_0^{-1}(y^{(0)}-K\\_{0q}K_q^{-1}y^{(q)}),\n$$\nwhere $k_0(x)=(K(x,x^{(0)}_1),\\dots,K(x,x^{(0)}\\_{n_0}))^{\\top}$, $K_0$ and $K_q$ are the target and source Gram matrices respectively, and the $(i,j)$-th entry of $K\\_{0q}\\in\\mathbb{R}^{n_0 \\times n_q}$ is $K(x_i^{(0)}, x_j^{(q)})$.\n\nIn our simulation, we let $x_i\\in\\mathbb{R}^d$ be i.i.d. $\\mathcal{N}(0,\\Lambda^{(0)})$ with $d=500$, where $\\Lambda^{(0)}$ is the \"benign\" covariance as specified in Section 5 of our manuscript. The true target function is given as in [4] by \n$$\nf_0(x)=\\sum\\_{l=1}^{100} 10K(x,\\theta_l),\\quad\\theta_l\\overset{i.i.d.}{\\sim}\\mathcal{N}(0,I_d).\n$$\nThe three source functions are:\n$$\nf_1(x)=f_0(x),\\quad f_2(x)=f_0(x)+\\cos(5x_1),\\quad f_3(x)=-f_0(x),\n$$\nwhere $x_1$ is the first component of $x$. Intuitively, $f_1$ would be informative, while the cosine perturbation in $f_2$ can make it less informative than $f_1$. With the anti-aligning signal, $f_3$ would be harmful to pre-train. Each data generation is given by\n$$\ny_i^{(q)}=f_q(x_i^{(q)})+\\epsilon_i^{(q)},\\quad i=1,...,n_q,\\quad q=0,1,2,3,\n$$\nwhere $\\epsilon_i^{(q)}$ are i.i.d. $N(0,\\sigma^2)$ with $\\sigma^2=1$. We set $n_0=25$ and $n_1=n_2=n_3=75$ and use $K=5$-fold CV as in our paper. The CV algorithm follows that in Appendix C, where we measure on the $k$-th target fold the validation MSEs of target-only interpolator and each TM. Averaging over the $K=5$ folds yields the CV losses $\\widehat L\\_M^{(0)}$ and $\\widehat L\\_{TM}^{(q)}$, which are compared to identify the set of informative sources $\\hat{\\mathcal{I}}$. The WTM function is obtained accordingly based on the weighted average of informative TMs, i.e., $\\hat f\\_{WTM} = \\sum_\\{i \\in \\hat{\\mathcal{I}}} w_i \\hat f\\_{TM}^{(i)}$.\n\nFor each Monte-Carlo seed, we draw $1,000$ i.i.d. unseen test points $x_t \\sim \\mathcal{N}(0,\\Lambda^{(0)})$ and obtain the noiseless target response $f_0(x_t)$. We then record the RMSE of each estimate $\\hat{f}$ given by\n$$\n\\mathrm{RMSE}(\\hat{f})= \\frac{1}{1000}\\sum\\_{t=1}^{1000}(\\hat{f}(x_t)-f_0(x_t))^2,\n$$\nwhich approximates the true excess risk $\\mathcal{R}(\\hat{f})=\\mathbb E_x[(\\hat{f}(x)-f_0(x))^2]$. We repeat this entire process across 100 independent simulation seeds and report the average of CV loss $\\hat{L}$ and RMSE as follows:\n |  | CV Loss | RMSE |\n|---|---|---|\n| Target-only | 2.567 | 1.081 |\n| TM1 | 1.896 | 0.973 |\n| TM2 | 2.129 | 1.072 |\n| TM3 | 6.279 | 1.796 |\n| WTM |  | 0.935 |\n\nOverall, our previous intuition on $f_1,f_2,f_3$ matches with the loss values. As the CV loss estimates the prediction risk $\\mathcal{R}(\\hat{f})+\\sigma^2$ with $\\sigma^2=1$, subtracting 1 from the losses put them close to the corresponding RMSEs, confirming that CV is an adequate proxy in this non-linear experiments. For TM3 that pre-trains the negative of true target signal, the sign error shows up more evidently with CV loss. But once we refit TM3 on all 25 target points to compute RMSE, the projection cancels part of the opposite signal, resulting in RMSE of only 1.796. Finally, we note that WTM consistently incorporated TM1 and TM2, and the pattern in which the RMSE of WTM is lower than that of each individual TM is consistent with what we observed in the linear MNI setting.\n\nWe hope this experiment answers your question.\n\n***\n**References**\n\n[1] Generalization error of min-norm interpolators in transfer learning. arXiv:2406.13944.\n\n[2] Features are fate: a theory of transfer learning in high-dimensional regression. arXiv:2410.08194.\n\n[3] Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution. arXiv:2202.10054\n\n[4] Just Interpolate: Kernel “Ridgeless” Regression Can Generalize. AoS 2020."}, {"review_id": "uljX2iZHDg", "reviewer": "Reviewer_zKSu", "summary": "The paper presents a novel algorithm for transfer learning in an overparameterized linear regression setting where data is available from both a transfer and target task (“transfer MNI”). The authors provide extensive theoretical analysis characterizing the bias and variance of the transfer MNI estimator, the conditions for positive transfer, generalization bounds for transfer MNI, and so-called “free-lunch” covariate shifts.", "questions": "- How does violation of each of the conditions in Assumption 1 affect your results?\n- How does violation of the assumption that the linear model is well-specified affect your results? This seems particularly practically important given that overparameterization can be especially beneficial in cases of model misspecification (c.f., arXiv:2109.02355).\n- It was unclear to me whether and how the results applied to settings where the learner transferred knowledge from multiple sources. The analysis in section 3 appears to concern exclusively $\\hat{\\beta}^{(q)}\\_{\\mathrm{TM}}$, which I understand to be the estimator obtained with single-source transfer from the $q^{\\mathrm{th}}$ task. How does transfer from multiple sources affect the results in section 3? In particular, to what extent do these results apply to $\\hat{\\beta}\\_{\\mathrm{WTM}}$, the estimator described in section 4.2?", "limitations": "I would have liked to see more discussion of the restrictiveness of the authors’ assumptions (see my Questions to the authors).", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Overall, I find this to be a high-quality, significant, and original paper. The question of when, in an overparameterized learning setting, knowledge transfer from a source task improves performance in a target task is a practically significant question. The authors provide extensive theoretical analysis characterizing the bias and variance of the transfer MNI estimator, the conditions for positive transfer, generalization bounds for transfer MNI, and so-called “free-lunch” covariate shifts. They also propose and empirically assess an algorithm for identifying and incorporating informative sources.\n\nThe paper could be clearer. In particular, there are several typos in the paper. Some examples:\n- Line 148: “covariates” is misspelled.\n- Line 171: The sentence appears incomplete.\n- Line 281: “severe” is misspelled.\n\nAnother minor point: $\\mathcal{R}^{(q)}\\_{\\mathrm{TM}}$ and $\\mathcal{R}^{(0)}\\_{\\mathrm{M}}$ are used in Corollary 1 and Theorem 2 without definition.\n\nI would have also liked to see more discussion of the restrictiveness of the authors’ assumptions (in particular, Assumption 1 and the assumption that the linear model is well-specified), and the effect of violation of these assumptions on the authors’ results.", "was_revised": true, "final_justification": "The paper provides extensive theoretical analysis of and an algorithm to operate in an important although somewhat restrictive setting. As stated in my original review, I find this to be a high-quality and original paper that tackles a practically significant question. My main concern was lack of discussion of the restrictiveness of the authors’ assumptions.\n\nIn their rebuttal, the authors discussed the restrictiveness of the conditions in Assumption 1 and the effect on their results of violation of the assumption that the linear model is well-specified. Including such discussion in the paper would address my main concern, and demonstrate the potentially broader significance of the results.", "rebuttal": "Dear Reviewer zKSu, \n\nWe deeply appreciate your thorough analysis and acknowledging the significance and originality of our findings. We would like to address your questions as follows.\n\n***\n**Q1. Some minor typos and definitions**\n\nThank you for pointing these typos. We have fixed the typos and will ensure to incorporate the changes to the camera-ready version once accepted. In Line 171, our full sentence is as follows: \"Based on Theorem 1, we formalize the regime where the TM outperforms the target task and specify the optimal transfer size that maximizes the improvement in excess risk over the baseline.\"\n\nOn the other hand, the notations $\\mathcal{B}\\_{TM}^{(q)}$ and $\\mathcal{V}\\_{TM}^{(q)}$ follow from Lemma 1 in our mauniscript, which stand for the bias and variance of TM respectively. The notation $\\mathcal{X}$ is defined in Assumption 1 as the collection of all designs, so the expression $\\mathbb{E}\\_{\\mathcal{X}} \\mathcal{B}\\_{TM}^{(q)}$ in Theorem 1 and Corollary 1 is the expected bias of TM (expectation over designs) and thus $\\mathbb{E}\\_{\\mathcal{X}} \\mathcal{V}\\_{TM}^{(q)}$ is the expected variance of TM.\n\n***\n**Q2. How restrictive are the conditions in Assumption 1?**\n\nThank you for raising this question. We address each condition separately as follows.\n* **Sub-Gaussian covariates**: Encompassing a wide class of distribution, this is a standard assumption in relevant studies on transfer learning, including, but not limited to, [1], [2], and [3]. On the other hand, we also note that investigating the generalization of single-task min-$\\ell_1$-norm interpolant is primarily based on Gaussian covariates; please see references [8]-[11] in our response to Reviewer E9gm.\n\n* **Spectral decomposition**: Although written in \"Assumption\", this would rather be a definition. As noted just before Assumption 1 in our manuscript, our analysis and numerical validations do not necessarily impose simultaneous diagonalizability between the target and source covariances. Hence our condition is strictly weaker. \n\n* **Bounded operator norm**: This trivially holds when $\\Sigma^{(0)}=\\Sigma^{(q)}$ with $||\\Sigma^{(0)}(\\Sigma^{(q)})^{-1}||_{op}=1$. We kindly ask the reviewer to check our response to Q1 of Reviewer KZWN regarding the generality of this assumption. A counter-example is when $\\Sigma^{(0)}=I_p$ but $\\Sigma^{(q)}$ is a spiked covariance with decaying eigenvalues, where the mismatch would make transfer harmful. Nevertheless, our source detection procedure is designed to filter out such negative transfers. \n\n* **Sub-Gaussian noise**: Our assumption is again milder than the Gaussian requirement (e.g., [4] and [5]). In fact, this can be readily extended to a noise having i.i.d. non-sub-Gaussian components with zero mean and variance $\\sigma^2$. This is because the precise definition of excess risk (Eq. (4) of our paper) takes expectation over noise, and its expression involves the form $Tr(M \\mathbb{E}\\epsilon\\epsilon^\\top)]=\\mathbb E[\\epsilon^\\top M\\epsilon]$ for some conditioned (fixed) matrix $M$. While taking expectation over $\\epsilon$ simply reduces it to $\\sigma^2 Tr(M)$, sub-Gaussianity of $\\epsilon$ provides a high-probability upper bound (see Lemma S.2 in [6]) on $\\epsilon^\\top M \\epsilon$, thus guaranteeing a high-probability upper bound on the *instance-wise* variance of estimate.\n\n***\n**Q3. How would model-misspecification affect the performance of TM?**\n\nThank you for raising this intriguing question. While our current analysis focuses on well-specified models (as is the standard for much of transfer learning studies, e.g., [1], [2], and [7]), a precise theoretical characterization of TM under model misspecification remains beyond the present scope. Nevertheless, we would like to offer some intuition based on the structural transfer of TM and the findings discussed in your citation [8].\n\nWe first kindly recall the decomposition structure of TM estimate, which is given by\n$$\n\\hat\\beta_{TM}^{(q)}=\\hat\\beta_{M}^{(0)}+(I_p-H^{(0)})\\hat\\beta_{M}^{(q)},\n$$\nwhere $I_p-H^{(0)}$ is the orthogonal projection onto the null space of target design, and $\\hat\\beta_{M}^{(0)}$ and $\\hat\\beta_{M}^{(q)}$ are independent MNIs exclusively trained on the target and source datasets respectively. Let $S:=row(X^{(0)})=Im(H^{(0)})$ and $S^\\perp=Ker(H^{(0)})$. Then, any vector uniquely decomposes as $\\beta = (\\beta)\\_S + (\\beta)\\_{S^\\perp}$ and our TM estimate satisfies\n$$\n(\\hat\\beta_{TM}^{(q)})\\_S=\\hat\\beta_M^{(0)},\\quad(\\hat\\beta_{TM}^{(q)})\\_{S^\\perp}=(I_p-H^{(0)})\\hat\\beta_M^{(q)}.\n$$\nThat is, TM \"preserves\" the signal learned by the target-only MNI on the target-supported subspace and \"carries over\" source information only in the null-space where the target offers no information. Before procedding, we kindly ask the reviewer to see how this “projection + carry-over’’ principle is further supported in our reply to **Reviewer bZxw (Q3)**.\n\nWhen the true regression function lies outside every linear span we may fit, the usual squared bias of an estimate splits into two terms $B_1+B_2$:\n$$\n\\text{bias}^2 =\\text{mis-specification bias}^2 + \\text{in-class bias}^2,\n$$\nwhich precisely matches Eq. (22) in [8]. $B_1$ is unavoidable unless we enlarge the feature space; $B_2$ behaves as in our well-specified analysis. [8] show that, as we add extra (low-energy) features, $B_1$ and its associated variance fall faster than the new estimation error rises, so the test error curve drops again in the over-parameterized regime (double descent). Empirically they observe the misspecification components decreasing decisively beyond $p >n$.\n\nHere, we kindly request to check the three terms $U_1+U_2+U_3$ comprising the bias of TM, explained in our response to **Reviewer bZxw (Q2)**. Under the mis-specification, $U_1$ now splits into the usual leakage plus the true target signal that lies neither in the span of target or source space and lives outside the linearity. Therefore, an irreducible approximation error arises on $U_1$. On the other hand, if the source carries a good approximation of the mis-specified part, TM can still copy it into $S^\\perp$ and provides some bias improvement. If the source is also mis-specified, $U_2$ now imports the source’s model approximation error, which corresponds to the phenomenon “amplified misspecification’’ termed by [8] and makes TM prone to negative transfer. \n\nFavorably, we anticipate that our free-lunch covariate shifts can still play a role, as their requirements match the remedial properties suggested by [8]: there are only a few number of (such as $\\tau^\\ast$ in Section 3.3 of our paper) high-energy directions and the rest $p-\\tau^\\ast$ low-energy directions soak up noise. With the partial/full directional alignments in free-lunch covariate shifts, TM would benefit from reduced variance inflation independent of mis-specification bias. \n\n***\n**Q4. Drawing a theoretical connection for WTM estimate**\n\nYou are absolutely right; as noted, the result in Section 3 is based on the excess risk of TM, which is fundamentally a single-source transfer method. We appreciate this opportunity to elaborate further on a promising direction for future research, as briefly outlined in Section 6 of our manuscript.\n\nTo do so, we suggest some sketch of the theoretical properties of WTM. First, we recall that $I$ is the true (oracle) set of informative sources as identified in Eq. (7) of our paper. Accordingly, we define the oracle WTM (oWTM) as\n$$\n\\hat\\beta\\_{oWTM}:=\\sum\\_{i \\in I}w_i \\hat\\beta\\_{TM}^{(i)},\n$$\ni.e., oWTM ensembles truly informative TM estimates, while in practice our WTM is based on the set $\\hat{I}$ estimated by our cross-validation algorithm. Then, the following would hold for the oWTM:\n$$\n\\mathcal{R}(\\hat\\beta_{oWTM})=\\mathbb{E}\\left[\\left(\\sum_{i \\in I}w_i\\hat\\beta\\_{TM}^{(i)}-\\beta^{(0)}\\right)^\\top\\Sigma^{(0)}\\left(\\sum_{i \\in I}w_i\\hat\\beta\\_{TM}^{(i)}-\\beta^{(0)}\\right)\\right] \\leq \\mathbb{E}\\left[\\sum_{i \\in I}w_i \\left(\\hat\\beta\\_{TM}^{(i)}-\\beta^{(0)}\\right)^\\top\\Sigma^{(0)}\\left(\\hat\\beta\\_{TM}^{(i)}-\\beta^{(0)}\\right)\\right] \\leq \\sum_{i \\in I}\\mathbb{E}\\left[\\left(\\hat\\beta\\_{TM}^{(i)}- \\beta^{(0)}\\right)^\\top\\Sigma^{(0)}\\left(\\hat\\beta\\_{TM}^{(i)}-\\beta^{(0)}\\right)\\right] \\leq |I| \\max_{i \\in I} \\mathcal{R}(\\hat\\beta\\_{TM}^{(i)}),\n$$\nwhere the first inequality follows from Jensen's inequality, and $|I|$ is the cardinality of $I \\neq \\emptyset$. This implies if $|I|$ is finite (which is true in general) and $\\max_{i \\in I}\\mathcal{R}(\\hat\\beta\\_{TM}^{(i)}) \\ll \\mathcal{R}(\\hat\\beta_{M}^{(0)})$, then oracle-WTM can achieve a robust positive transfer.\n\nThen, it remains to precisely characterize the consistency of source detection, i.e., to prove that the event $\\hat{I} = I$ holds with high probability, which is outlined as our future research agenda. We anticipate that such theoretical guarantee will require a \"separation condition\" that distinguishes informative from non-informative sources. For example, for some \"transferability threshold\" $h >0$, write $\\mathcal{I}_h := \\\\{q \\in [Q]: ||\\delta^{(q)}||_2 \\leq h\\\\}$. Then, we may need the condition such as $||\\delta^{(q)}||_2 \\geq h\\log(n_0)$ for $q \\notin \\mathcal{I}_h$ to infer detection consistency. Characterizing the minimal separation condition and matching it with bounds on excess risk when an uninformative source is transferred remains an open problem and is part of our planned research agenda.\n\nWe sincerely hope our answers clarify your questions.\n\n***\n**References**\n\n[1] arXiv:2404.00522.\n\n[2] arXiv:2106.12108.\n\n[3] arXiv:2404.01153.\n\n[4] arXiv:1903.09139.\n\n[5] arXiv:1805.10939.\n\n[6] arXiv:1906.11300.\n\n[7] arXiv:2406.13944.\n\n[8] arXiv:2109.02355."}]

Author rebuttal:

Meta-review: The paper focuses on bias and variance trade-off in high-dimensional transfer learning by minimum $\ell_2$-norm interpolator. The authors propose a transfer learning strategy and provide both a theoretical analysis and a numerical study of its performance. The paper has been appreciated for its quality, clarity, and potential impact, collecting unanimous high ratings by all reviewers. The discussion period between reviewers and authors lead to the clarification of some points related to the possible limitations of the setup, and a further increase in the scores. The theoretical analysis of transfer learning in modern machine learning and benign overfitting is naturally extremely important in the current machine learning research, and the submission has been recognised as a solid, comprehensive and high-impact contribution in this direction. I therefore recommend its acceptance as *spotlight*.
Decision: Accept (spotlight)