Title: Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression
Venue: NeurIPS 2025
Accepted: yes
Keywords: Transfer Learning

Benign Overfitting

Generalization

Minimum-Norm Interpolator
Abstract: Transfer learning is a key component of modern machine learning, enhancing the performance of target tasks by leveraging diverse data sources. Simultaneously, overparameterized models such as the minimum-$\ell_2$-norm interpolator (MNI) in high-dimensional linear regression have garnered significant attention for their remarkable generalization capabilities, a property known as *benign overfitting*. Despite their individual importance, the intersection of transfer learning and MNI remains largely unexplored. Our research bridges this gap by proposing a novel two-step Transfer MNI approach and analyzing its trade-offs. We characterize its non-asymptotic excess risk and identify conditions under which it outperforms the target-only MNI. Our analysis reveals *free-lunch* covariate shift regimes, where leveraging heterogeneous data yields the benefit of knowledge transfer at limited cost. To operationalize our findings, we develop a data-driven procedure to detect *informative* sources and introduce an ensemble method incorporating multiple informative Transfer MNIs. Finite-sample experiments demonstrate the robustness of our methods to model and data heterogeneity, confirming their advantage.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel algorithm for transfer learning in an overparameterized linear regression setting where data is available from both a transfer and target task (“transfer MNI”). The authors provide extensive theoretical analysis characterizing the bias and variance of the transfer MNI estimator, the conditions for positive transfer, generalization bounds for transfer MNI, and so-called “free-lunch” covariate shifts.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, I find this to be a high-quality, significant, and original paper. The question of when, in an overparameterized learning setting, knowledge transfer from a source task improves performance in a target task is a practically significant question. The authors provide extensive theoretical analysis characterizing the bias and variance of the transfer MNI estimator, the conditions for positive transfer, generalization bounds for transfer MNI, and so-called “free-lunch” covariate shifts. They also propose and empirically assess an algorithm for identifying and incorporating informative sources.\n\nThe paper could be clearer. In particular, there are several typos in the paper. Some examples:\n- Line 148: “covariates” is misspelled.\n- Line 171: The sentence appears incomplete.\n- Line 281: “severe” is misspelled.\n\nAnother minor point: $\\mathcal{R}^{(q)}\\_{\\mathrm{TM}}$ and $\\mathcal{R}^{(0)}\\_{\\mathrm{M}}$ are used in Corollary 1 and Theorem 2 without definition.\n\nI would have also liked to see more discussion of the restrictiveness of the authors’ assumptions (in particular, Assumption 1 and the assumption that the linear model is well-specified), and the effect of violation of these assumptions on the authors’ results.\n\nQUESTIONS:\n- How does violation of each of the conditions in Assumption 1 affect your results?\n- How does violation of the assumption that the linear model is well-specified affect your results? This seems particularly practically important given that overparameterization can be especially beneficial in cases of model misspecification (c.f., arXiv:2109.02355).\n- It was unclear to me whether and how the results applied to settings where the learner transferred knowledge from multiple sources. The analysis in section 3 appears to concern exclusively $\\hat{\\beta}^{(q)}\\_{\\mathrm{TM}}$, which I understand to be the estimator obtained with single-source transfer from the $q^{\\mathrm{th}}$ task. How does transfer from multiple sources affect the results in section 3? In particular, to what extent do these results apply to $\\hat{\\beta}\\_{\\mathrm{WTM}}$, the estimator described in section 4.2?\n\nLIMITATIONS:\nI would have liked to see more discussion of the restrictiveness of the authors’ assumptions (see my Questions to the authors).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies transfer learning in linear models. The specific transfer learning set up is a form of \"late-fusion knowledge transfer\" with potentially multiple source tasks. The authors then study the effect of model shift and derive several results on the maximal risk improvement. Perhaps most interestingly, the authors propose a procedure to identify informative sources and a weighting scheme to use the contributions from multiple source tasks.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\n\nOverall, the quality of the paper is high. The explanations of the results are crisp and the numerical results provide a nice demonstration of the theoretical results.  \n\nClarity:\n\nThe paper is well-written and mostly developed logically. As is natural for a paper of this type, much of the technical detail is deferred to the Appendix. I would have prefered that the authors provide more commentary on the nature of the arguments in the main text. For example, some important objects are discussed in the main text without much context. Effective rank is a good example of this---it would be much clearer to explain what the role of effective rank will be in the context of the results and then give the definition. Overall, though, I think these are stylistic rather than substantive quibbles. \n\nSome specific suggestions include:\n\n- more motivation for late-fusion knowledge transfer when it is introduced \n- Lemma 1, say where it is proved in the appendix, comment on the meaning of the bias term, and define $\\delta^{(q)}$. \n\nSignificance:\n\nTransfer learning is a relatively important topic given the popularity of foundation models, which indeed are often specialized for some task. Better theoretical understanding of this topic is crucially important and I think, in particular, the question of selecting good source tasks is a particularly relevant focus. On the other hand, the significance of the results is of course limited by the fact that the authors analyze only the case of linear models. If the authors could illustrate their results, even numerically, in a more realistic setting, that might improve the impact. \n\nOriginality:\n\nThere are quite a few recent works that derive similar results in either linear setting or in the context of linear probing. One thing that stood out is the \"free-lunch covariate shift\" results. These are quite striking but also similar to the results in: https://arxiv.org/abs/2202.10054, https://arxiv.org/abs/2206.12314, https://arxiv.org/pdf/2410.08194. I think the nature of the results should be better contextualized within this literature that is more focused on \"feature learning\".\n\nQUESTIONS:\nWhat is the relationship between the free-lunch covariate shift and \"feature overlap\" suggested by recent works on transfer learning?\n\nHow scalable is the source detection algorithm in the non-linear setting? Are there efficient proxies for it?\n\nWhat minimal numerical experiment could illustrate the informative weighted transfer in a non-linear setting?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The first paper I am aware of that demonstrates benign overfitting in transfer learning related models. The work provides a comprehensive overview of this model under the $\ell_2$ norm interpolator and studies it in great detail.

STRENGTHS AND WEAKNESSES:
Strengh:
- \emph{extremely} comprehensive analysis 
- The first paper to study this problem in this natural setting
- Potentially high-impact baseline work

Weakness:
- No novel new technical ideas that do not appear in the works of Bartlett and Koehler.
- only \ell_2 norm interpolator, i.e., a closed formula.

QUESTIONS:
Can you extend these results to a minimum norm interpolator that is not the $\ell_2$ one?

Can you extend to isotropic sub-Gaussian (even uniformly bounded entires from above and below, or isotropic log-concave i.i.d. ) covariates at least in the case of $\ell_2$ minimum norm interpolator (extending to sub-Gaussians was done recently to $\ell_p$ norm?

LIMITATIONS:
These results are only valid for the $\ell_2$ minimum norm interpolator with a low-rank matrix.

As I do think that the authors did a good and comprehensive work despite these limitations, I am willing to increase my score to accept, under a considerable discussion on $\ell_p$ norms (see Donhasuer et al., Kur et al., and Lecue et al. works) related to objects and the RKHS settings, and cite the relevant work. In those settings, benign overfitting was shown, so I am curious how to connect your work to those settings.

Please use an anonymous link to address this topic.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis investigates the bias and variance trade-off in transfer learning based on Minimum-l2-norm interpolator (MNI) in high dimensional linear regression models. A two-step transfer learning approach is proposed, and the non-asymptotic exccess risk is studied under some theoretical assumptions on model-shift and covariate-shift. Numerical performance of the proposed transfer learning procedure is evaluated through simulation studies. Overall, the paper is well written and clearly presented.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: some insights are derived for transfer learning using MNI estimator.\nWeakness: the derived results has little practical implications.\n\nQUESTIONS:\n1. The theoretical insights are based on rather restrictive assumptions. For example,\n\n (a) In point 3 of Assumption 1, the condition that all covariance matrices can be simultaneously diagonalized (as assumed in [19], [24], [29]) implies that $\\prod_{j=1}^p\\frac{\\lambda_j^{(0)}}{\\lambda_j^{(q)}}$ is bounded for all $q$. This assumption appears quite restrictive and somewhat unnatural in practice. Could the authors provide some meaningful real-world examples where this condition would realistically hold?\n\n(b) In Section 3.1, the paper analyzes the “non-benign” case that violates Assumption 2. However, in this setting, the MNI estimator does not even provide any benefit. How do the theoretical insights from this analysis offer practical value for developing or understanding transfer learning methods?\n\n(c) The results in Sections 3.2 and 3.3 are interesting from an intuitive perspective but seem to have limited theoretical implications. Notably, the actual estimator proposed in Section 4 is not derived from these theoretical results, but is instead based purely on using cross-validation errors. While this is intuitively reasonable, it is also fairly straightforward and does not appear to leverage the earlier theory in a meaningful way.\n\n2. The result in Lemma 1 is somewhat counterintuitive. Typically, one would expect transfer learning to increase the effective sample size, thereby reducing the variance. However, Lemma 1 suggests that the variance is universally increased by transfer learning. Could the authors provide further clarification or intuition for why this phenomenon occurs?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}