Transfer Learning in $\ell_1$ Regularized Regression: Hyperparameter Selection Strategy based on Sharp Asymptotic Analysis
Authors: Koki Okajima
Tomoyuki Obuchi
Venue: Accepted by TMLR ()

Abstract: Transfer learning techniques aim to leverage information from multiple related datasets to enhance prediction quality against a target dataset. Such methods have been adopted in the context of high-dimensional sparse regression, and some Lasso-based algorithms have been invented: Trans-Lasso and Pretraining Lasso are such examples. These algorithms require the statistician to select hyperparameters that control the extent and type of information transfer from related datasets. However, selection strategies for these hyperparameters, as well as the impact of these choices on the algorithm's performance, have been largely unexplored. To address this, we conduct a thorough, precise study of the algorithm in a high-dimensional setting via an asymptotic analysis using the replica method. Our approach reveals a surprisingly simple behavior of the algorithm: Ignoring one of the two types of information transferred to the fine-tuning stage has little effect on generalization performance, implying that efforts for hyperparameter selection can be significantly reduced. Our theoretical findings are also empirically supported by \rev{applications on real-world and semi-artificial datasets using the IMDb and MNIST datasets, respectively.

Published in Transactions on Machine Learning Research (1/2025)
Transfer Learning in l Regularized Regression: Hyper- parameter Selection Strategy based on Sharp Asymptotic Analysis
Koki Okajima
dorjeeling@g..co.uuoooo www.://g.......oooloo..
Graduate School of Science, Depertment of Physics
The University of Tokyo
Tomoyuki Obuchi
obuchi@i.kyooo-u.cm
Graduate School of Informatics, Department of Informatics Kyolo Universily
Reviewed on OpenReview: https: // openreview. net/ forum? id= ccuOM3nml.
Abstract
Transfer learning techniques aim to leverage information from multiple related datasets to enhance prediction quality against a target datasct. Such methods have been adopted in the context of high-dimensional sparse regression, and some Lasso-based algorithms have been invented: Trans-Lasso and Pretraining Lasso are such examples. These algorithms require the statistician to select hyperparameters that control the extent and type of information transfer from related datascts. However, selection strategics for these hyperparancters, as well as the impact of these choices on the algorithm's performance, have been largely unexplored. To address this, we conduct a thorough, precise study of the algorithm in a high-dimensional sctting an asymptotic analysis using the replica method. Our approach reveals surprisingly simple behavior of the algorithm: Ignoring one of the two lypes of in- formation transferred to the fine-tuning stage has little effect on generalization performance, implying that efforts for hyperparameter selection can be significantly reduced. Our theoret- ical findings are also empirically supported by applications on real-world and semi-artificial datasets using the IMDb and MNIST datasets, respectively.
1 Introduction
The increasing availability of large-scale datasets has led to an expansion of methods such as pretraining and transfer learning (Torrey & Shavlik 2010 Weiss et al.] 2016 Zhuang et al., 2021), which exploit similaritics between datasets obtained from different but related sources. By capturing general patterns and features, which are assumed to be shared across data collected from multiple sources, one can improve the general- ization performance of models trained for specific task, even when the data for the task is too limited to make accurate predictions. Transfer learning has demonstrated its efficacy across a wide range of applica- tions, such as image recognition (Zhu et al.) (011). text categorization (Zhuang al.
2011). bioinformaties
[Petegrosso et al.] 2016), and wireless communications (Pan et al., 2011: Wang et al.
2021
It is reported
to be ellective in high-dinensiooal statistics (Banerjee et al. 2020 Takada Fujisawa,
2020 Bastani 2021
Li et al.) 2021), with clinical analysis (Turki et al.) 2017 Hajiramezanali et al.) 2018 McGough et al. 2023
Craig et al. 2024) being major application area due to the scarcity of patient data and the need to comply with privacy regulations. In such applications, one must handle high-dimensional data, where the number of features greatly exceeds the number of samples. This has sparked interest. in utilizing transfer learning to sparse high-dimensional regression, which the standard method for analyzing such data.
A classical method for statistical learning under sigh-dimensional settings is Lasso Tibsshirani 1996). which aims at simultaneously identifying the sparse set of features and their regression coefficients that are most

---

Published in Transactions on Machine Learning Research (1/2025)
relevant for predicting the target variable. Its simplicity and interpretability of the results, as well its convex property as an optimization problem, has made Lasso a popular choice for sparse regression tasks, leading to a flurry of research on its theoretical properties (Candes & Tao 2006; Zhao & Yu 2006; Meinshausen & Bühlmann
2006 Wainwright
2009 Bellec et al. 2018).
Modifications to Lasso have been proposed to incorporate and take advantage of auxiliary samples. The data- shared Lasso [Gross & Tibshirani 2016) and stratified Lasso (Ollier & Viallon 2017) are both methods based multi-target learning, where solves multiple regression problems with related data in single fitting procedure. Two more recent approaches are Trans-Lasso (Bastani 2021 Li et al. 2021 and Pretraining Lasso (Craig et al.) 2024), both of which require two-stage regression procodures. Here, the first stage is focused on identifying common features shared across multiple related datasets, while the second stage is dedicated to fine-tuning the model on a specific target dataset. To simplify the explanation, we introduce a generalized algorithm that encompassos both methods, which referred to as Generalized Trans-Tasso hercafter. Given a set of datasets of size K, D {D(*) j==1 each consisting of an observation vector of dimension Ms; ( E RRM, and its corresponding covariate matrix A(k) E MR generalized Trans-Lasso performs the following two steps:
1. Protraining procedure: In the first stage, all datasets are used to identify a common feature vector (1st) which captures shared patterns and relevant features across the datasets:
K
((st) = arg min (11st) (x:; D), (ast) (x; D)
yy (k) - (k) 2012 + +1 1
(1)
2
2. Fine-tuning: Here, the interest is only on a specific target dataset, which is designated by 1 without loss of generality. The second stage incorporates the learned feature vector's configuration and its support set into a modified sparse regression model. This is done by first offsetting the observation vector y by a common prediction vector deduced from the first stage, i.e. (1) Further, the penalty factor is also modified such that variables not selected in the first stage is penalized more heavily than those selected. The above procedure is summarized by the following optimization problem:
(2nd) = arg min (ndd (ar; (15t), D(U), EERN
(2)
[(2nd) (x; (1st), D()) = 11 (1A(1) (1)
+ AATA (1st) = =0) |0s1.
where is the indicator function. Here, K, AA 2 0 are hyperparameters that determine the extent of information transfer from the first stage.
It is crucial to note that the performance of this algorithm is highly dependent on the choice of hyperpa- rameters, which control the strength of knowledge transfer. Suboptimal hyperparameter selection can lead to insufficient knowledge transfer, or negative transfer (Pan et al. 2011], where the model's performance is degraded by the transfer of adversarial information. Thus, the choice of appropriate hyperparameters should be addressed with equal importance as the algorithm itself. In fact, a distinctive difference wet Trans-Lasso and Pretraining Lasso is whether one should inherit support information from the first stage or not. While this adds an additional layer of complexity to hyperparameter selection, it also provides an opportunity to inherit significant information from the first stage, if the support of the feature vectors be- tween the sources are similar. Empirical investigation of the qualitative effects of hyperparameters on the algorithm's 's performance can be computationally intensive, especially in high-dimensional settings.
To address the issue of hyperparameter selection without resorting to extensive empirical research, one can turn to the field of statistical physics, which provides set of tools to analyze the performance of high- dimensional statistical models in a theoretical manner. While these theoretical studies are restricted inevitably simplified assumptions on the statistical properties of the dataset, they often provide valuable
2

---

Published in Transactions on Machine Learning Research (1/2025)
qualitative insights generalizable to more realistic settings. The replica method MMzardd et al.
1986 Char-
bonneau et al. 2023); in particular, has been widely used to precisely analyze high-dimensional statistical
models including Lasso (Kabashima et al. 2009 Rangan et al.) 2009 Obuchi & Kabashima 2016 Ta kahahhi
& Kabashima 2018
Obuchi & Kabashima
2019 Okajima et al. 2023 and has been shown to provide
accurale predictions many cases.
In this work, we conduct sharp asymptotic analysis of the generalized Trans Lasso's performance under a high-dimensional setting using the replica method. Our study reveals simple heuristic strategy to select hyperparameters. More specifically, it suffices to use either the support information or the actual value of the feature vector ((1st) obtained from the pretraining stage to achieve ncar-optimal performance after fine- tuning. Our theoretical findings are complemented through empirical experiments on the real-world IMDb dataset (Maas et al. 2011 and semi-artificial datasets derived from MNIST images (Deng 2012
1.1 Related Works
Trans-Lasso first proposed by Bastani 2021 and further studied by Li et al. (2021). is a special case of the two-stage regression algorithm given by equations D and 2 In this variant, the second stage omits information transfer regarding the support of (1st) (i.e., AA 0) and sets x to unily. This method has demonstrated efficacy in enhancing the generalization performance of Lasso regression, and has since been extended to various regression models, such as generalized linear models (Tian & Fong, 2023 Li et al.) 2024), Gaussian graphical models (Li et al., 2023), and robust regression (Sun & Zhang] 2023). Moreover, the method also embodies a framework to determine which datasets are adversarial in transfer learning, allowing one to avoid negative transfer. While the problem of dataset selection is a practically important issue in transfer learning, the present study will not consider this aspect, but rather focus on the algorithm's performance given a fixed set of datasets.
On the other hand, Pretraining Lasso controls both k and AA using tunable interpolation parameter E [0, 1] as
1 - 8, A = - X2
This formulation allows for a continuous transition between two extremes: Setting 1 reduces the problem to standard single-dataset regression, while s = 0 reduces to regression constrained on the support of (1st). Note that there is no choice of which reduces Pretraining Lasso to Trans-Lasso.
Previous research has not yet established a clear understanding of the optimal hyperparameter selection for these methods. Moreover, the introduction of additional hyperparameters, such as those proposed in this study, has been largely unexplored in cxisting literature, possibly due to the computational demands involved. A more thorough investigation into the performance of these methods is necessary to reevaluate current, potentially suboptimal hyperparameter choices and to provide a more comprehensive understanding of the algorithm's 's behavior.
The analysis of our work is based on the replica method, which is non-rigorous mathematical tool used in wide range of theoretical studies, such as those on spin-glass systems and high-dimensional statistical models. The results from such analyses have been shown to be consistent with empirical results, and in some cases later proved by mathematically rigorous methods, such as approximate message passing theory [Bayati & Montanari 2011 Javanmard & Montanari] 2013), adaptive interpolation (Barbier et al. 2019: Barbier& & Macris| 2019), Gordon comparison inequalities (Stojnic 2013 Thrampoliddis et al.] 2018 Miolane
& Montanari 2021), and second-order Stein formulae (Bellec & Zhang 2021 Bellec & Shen 2022). The
sharpness of the formulae obtained in these approaches have also be exploited to gain insight into the effect of hyperparametors on the performance of learning methods, such as bagging estimators Takahashi (2023)
Koriyarn et al. (2024), ) stochastic algorithms (Takahashi] 2024 Lou et al.) 2024), and robust regression (Vilucchio et al. 2024). The application of the replica method to multi-stage procedures in machine learning
has also boen done the context of knowledge distillation (Saglictti & Zdcborová 22022). self-training
(Takahashi) 2024), and iterative algorithms (Oaajima & Takahashi 2024). Our analysis can be seen as an extension of these works to this generalized Trans-Lasse algorithm.
3

---

Published in Transactions on Machine Learning Research (1/2025)
2
Problem Setup
As given in the Introduction, we consider a set of K datasets, each consisting of an observation vector ( E MM and covariate matrices A E RM X The observation vector y is assumed to be generated from a linear model with Gaussian noise, i.e.
y (k) ( (k),
e ~ (0))
(4)
Here, e(4) E T. is a Gaussian-distributee noise term, while ( E IRNN is the true feature vector of the underlying linear model. We consider the common and individual support model (Craig et al. 2024], where {r(*)] ]k=1 is assumed to have overlapping support. More concretely, let {I(k))
k=0 be set of disjoint indices, No: for all k = 0,... K, and E No S N. Then, ((k) is given by
where I(k) C {1, ,
[I(K)] N,
r [() = x. No,
(k)
(0)
(b) T I(k)
= I.
(k)
ERR, T (K)
(I(()LI(()) = N-N-NNo
(5)
and
y A I(0)
I
x
+ A (R)
I(k) x + elk)
(6)
Here, th denotes the submatrix of A (R) with columns indexed by I, rz (k) denotes the subvector of (k) with indices I, and \I is the shorthand of 21,2, ...,..... (1,2,...,NV]I Therefore, each linear model always affected by the variables in the common support set [()), with additional variables in the unique support set I(k) for each dataset k = 1, 1,... ,....KK K. We refer to xx as the common feature vector, while xx as the unique feature vector for class k. The implicit aim of Trans-Lasso can be seen as estimating the common feature vector (0) in the first stage, treating the unique feature vectors in each class as noise. On the other hand, the second stage is dedicated to estimating the unique feature vector (1) given a noisy, biased estimator of the common feature vector (0) i.e. (1st).
We evaluate the performance of the generalized Trans-Lasso using the generalization error in each stage, which is defined as the expected mean square error of the model prediction made against a set of new observations = (1) { where (kk) E R and A (R) E RM XN;
(Ist)
1
E EDD
- (kk)
(7)
N
k=1
for the first stage, while the generalization error for the second stage, with specific interest on class k = 1 without loss of generality, is given by
(2dd)
1 E
D.D
(
A (Ko((s)) + a ((2d))
(8)
N
The objective of our analysis is to determine the behavior of the generalization error given the choice of hyperparameterss (A1, AA).
Note that the definition of ((sst) in equation 7 is appropriate for the case where the new data is redrawn, inchuding the covariate matrix, from the common and independent support model with the same hyperpa- rameters as the training data. In other scenarios, different definitions might be more suitable. For instance, if consider the case where only one datapoint is newly observed equally for each class k E (1, K), the generalization error should be defined to have equal weight over all the classes, which different from ours. In the analytical framework discussed below, that case also be treated: The resultant formula of the generalization error would become equation 21 with all (Ak) fixed equal. This would give quantita- tively different results from those based on equation 7 yielding another interesting situation. However, the investigation of that situation requires another detailed computation which we leave as future work.
High-dimensional setup. We focus on the asymptotic behavior in a high-dimensional limit, where the length of the common and unique feature vectors, as well the size of the datasets for each class, tend to infinity at the same rate, i.e.
No + (0) = O(1,,
N
M + =
+ - k=1,.. K, as N + oc.
(9)
N
N
N
4

---

Published in Transactions on Machine Learning Research (1/2025)
Also, define the proportion of the number of variables not included in any of the support sets x x(())) "k= ((4), The covariate matrices are all assumed to have i.i.d. entries with zero mean and variance 1/N, while the true feature vectors are also assumed to have i.id. standard Gaussian entries. While such a setup is a drastic simplification of real-world scenarios, it should be noted that these conditions can be further generalized to more complex settings. For instance, the analysis can be further extended to the case where the random matrix (A) inherits rotationally invariant structure (Vehkaperä et al. 2016; Takahashi & Kabashima 2018), or to the case where the observations are corrupted by heavy-tailed noise (Adomaityte
et al. 2023
The notations used in this paper are summarized in table 1
Notation
Description
I(0) C [1,22.... N)
Support of features common across all classes
I(k) c (1,
Support of features unique to class kt = I,..., K
N
Total number of features Size of common support, [I(O)]
No
N
Size of independent support of class 1,..., K; [I(K))
(0) E ER
Common feature vector shared among all classes
x. E B.
(k)
Unique feature vector exclusive to class I.....K
(4) E TRM
Observation vector for dataset == 1,..,, K Gaussian noise term with i.idd elements of variance (x)))
e(k) E M.
A (K) E RRMEX
Covariate matrix for dataset k: = I....,
D = (D(K) LK
Set of K datasets. ((k) = (8) Set of K test datasets with same distribution and size as First stage feature vector (pretraining)
(8), ( (k)
D (D(K) UR R=
1
(1.z)
(2nd)
Second stage feature vector (finc-tuiig)
A
First stage regularization parameter
22
Second stage base regularization parameter Transfer coefficient to second stage
AA
Additional regularization for features non-selected during pretraining
(0)
Proportion of common features No/N
T (6)
Proportion of unique features N₂/N Proportion of features absent from all linear models, 1 1-E (k) Sample size ratio M,/N
(neg.)
Ek=
(1)
Table 1: Notations used in this paper.
3 Results of Sharp Asymptotic Analysis via the replica method
Here, we state the result of our replica analysis. The series of calculations to obtain our results is similar to the well-established procedure used in the analyses for multi-staged procedures (Saglietti & Zdeborová 2022: Takahashi 2024 Okajima & Takahashi 2024). While a brief overview of the derivation is given in the next subsection, we refer the reader to Appendix A for a detailed calculation. Although our theoretical analysis lacks a standing proof from the non-rigorousness of the replica method, we conjecture that the results are exact in the limit of large N. This is supported numerically via experiments on finito-size systems, which will be shown to be consistent with the theoretical predictions.
In fact, since the first stage of the algorithm is standard Lasso optimization problem under Gaussian design, it is alrcady known that at least up to the first stage of the algorithm, the replica method yields the same formula derivable from Approximate Message Passing (Bayali & Montanari] 2011 or Convex Gaussian Minmax Theorem (CGMT) (Thrampoulidis et al. 2018 Miolane & Montanari 2021 J. Moreover,
CGMT can be further applied to multi-stage optimization problems with independent data each stage Chandrasskherr et al. (023).. The extension of this proof to the second stage of our algorithm, which contains data dependence among the two stages, is an unsolved problem that we leave for future work.
5

---

Published in Transactions on Machine Learning Research (1/2025)
3.1 Overview of the calculation
Here, we brielly outline the derivation of the generalization errors e (1st)) and e(2nd) using the replica method. Readers not interested in the derivation may skip this subsection and proceed to the next subsection for the main results.
Given the training dataset D (D(k) J probability measure for æ1 E TRN and 22 E
with D(k) = (A(h) (k), ()))), k =, K, define the following joint
k=1
RRN. as
83 (21, F2; D) = - w (21, 02; D),
(10)
31 ( (x - ( (222) æ1, 2.,
wa (2), x22 D) = exp
where 2 = I d wa (11, 10, 22; D) is the normalization constant. Under this measure, the generalization error (1st) and ((2nd) can be expressed as
K
(1st) = DDD z-1
2 dx 121, 22; D)
f
Σ
(
(A)
21 113
k=
1 D,
log / dx. dor (31,
22 D) e Ex. 11000 (11)
lim lim lim E 02-000 -0000 40
N
Öt,
e (2nd) Ep, z-1 /
dx; dxr 22; D))lg()) (Ka) + x22)
lim -00000
lim
lim E t2
Ep,
log /
dx (21, x2: 22: D) AA"
(12)
N
Otz
The first lines of equations 11 and 12 stem from the fact that, in the successive limit of 33 -> xo followed by 32 + oc, the joint probability measure P3 (2), (2); D) concentrates around (œ(1s), ((2nd)), the solution to the two-stage Trans-Lasso procedure. The second lines of equations [ and 12 can be derived by simply differentiating the logarithm, and exchanging the data average and the limit of t1 - or to + O. These expressions do not require the computation of the inverse normalization constant 2-1, which is difficult to perform in general. Still, it requires one to evaluate the average of the logarithm of the integral of " (I1, I2; Z2; (m) for appropriate choices of f. Techniques from statistical mechanics, however, allow one to perform heuristic calculations to evaluate such expressions, by alternatively expressing the logarithm of the integral as:
ED p log / de dx dleideesss.. (2), 22; (er
(13)
i On
log EDD If dzndzwws, (x,, D) D (ls
lim +
where the second line can be confirmed by a Taylor expansion in n. Although the second line of the above expression not directly computable for arbitrary n E Rz0. it turns out that under plausible assumptions, this can be evaluated for n E N in the limit N - co. The replica method Mezard et al. (1986); Charbonneau et al. (2023) is basod on the assumption that one can analytically continue this formula, defined only for N, E R On this continuation, the limit n +0 can be taken formally.
In the process of calculating equation 1 crucial observation is that from Gaussianity of the design matrices A (k), the products between the design matrices and the feature vectors æ are Gaussian random variables. This not only simplifies the high-dimensional nature of the expectation over D, but it also reduces the complexity of the analysis to consider only up to the second moments these Gaussian random variables. These moments are characterized by the inner product of the fcature vectors, which happen concentrate to deterministic values in the limit + co. One is then left with an integral over this finite number of concentrating scalar variables and other auxiliary variables introduced in this process, which then be

---

Published in Transactions on Machine Learning Research (1/2025)
evaluated using the standard saddle-point method, again in the limit N + This finite set of variables, as well as the saddle-point conditions imposed on them, are given in Definitions 1 and in the following section.
3.2 Equations of State and Generalization Error
The precise asymptotic behavior of the generalization error is given by the solution of set of non-linear equations, which we refer to as the equations of state. They are provided follows.
Definition 1 (Equations of state for the First Stage). Define the set of finite scalar variables θ (k)
{(m] mi Jk=o 41, Q1, X1, )1} as the solution to the following set of equations, which we refer to as the K (K)
equations of state for the first stage:
(tot)
01
Σ
(k)
E
E1 (x(())²)
+ (neg TE (x<(nex))
m
1 +
a (k)
1
(K)
E
Xx1 x
m1
(k)
1 1 X1
(14)
K
(1) - 2mm
+ m2
(k))
+ ((8)
81 =
+ x1)
k=1
Oxi (k)
Ox (neg.)
π (K)
x (neg.) E
X1
OvX121
b X121
where have defined ((8) (k) + 00)) + (o(R))2, and (tot) "E (k). Here, the random vari- ables (x(1) KK x (nez.) are the solution to the following random optimization problems dependent on x
((m) ( J o 811)
xi arg min (k) (x),
(k)
(6)
(x)
④1 q1 3-2-(Viizi + mm (1) x
+ \1
(15)
x
(neg.) arg min (neg)) (z),
neg
(x)
11 4 v
(16)
2
[K where Zı, {x4 k= are i.idd. standard Gaussian random variables. Finally, the average E₁ denotes the joint expectation with respect to these random variables.
Definition 2 (Equations of state of the Second Stage). Let O₁ be as defined in 1 i.c. the solution to the (k) (k)
equations of state 14 Define the set of finite variables O₂ ((m(4) ,mz Jk=nz 42, 92, gr drs X2, &2, Xr. {r) by

---

Published in Transactions on Machine Learning Research (1/2025)
the solution to the following set of equations, which we refer to the equations of state for the first stage:
K
a1
92 =
π
(4)
E2
(x2 (1)) + n (neR ( (xg )2]
f = 1 X2"
X22²
4. A
(R) 122 X/2
(R) (1x)
+ π (neg IR2 xx2 x
(neg.),
(neg.)
9r
+2
1=1
a
m2
(K)
π
E2
X2 Xx
(R) (141)
m
1 X2
K
0x (K)
Ox2 (neg.)
12)E2
+ π (neg)) 1E2
(17)
X2
1 Vx2ze
LovX2z2
k=
(1) 92
A²q1 91 + 2Aq
+ B2
-2Bmm (0)
+ m2))
2AA(m + mi
(0)
(1)
82
(1 + x2)²
K
Oxg
Ox2 (neg.)
X Σ x (2 k=
+ π ( .2
a 0Viizi
8 D
8, -a Bp(a) - mm2
(0)
- -m2 (1) A(m²
(0)
+ +mi (2))
(1 + x1) x2)
where A 1+x1 B x Here, the random variables (x2 (k) 1
(nes.) are the solution to the fol- 1k=0 X2
1-x1
lowing random optimization problems dependent on the scalar variables {fm) mz 1
(1),
Tk=o 41 X1, 42, 82. 4-):
x2 (R) = arg min (*) (x=
E
(x|x]
+ xxx{ (K) x r(xxx( (18) (8)) = 42 x² - -VVozz + m (1)
2
(neg.) arg min (es
E(mes
(ar |x{ (mBB.))
(neg)))
(xr(x(ne8))
(neg.)
42 1² (Viozz+ + q (m03))) (cr) (19)
X2
(k) KK where r<lly) = (A2 + AAIly = O]. The random variables 2ı, fx. (x)) and (xi Jk=o are as predefined in Definition and Zz Gaussian random variable with conditional distribution
Xr VXIX2
x²
22 21 ~ N
21,
(20)
X1X2
Finally, the joint expectation with respect to these random variables is denoted as Ez
Given these equations of state, the generalization error is given from the following claims: Claim 1 (Generalization error of the first stage). Given θ,, the expected generalization ervor (1st) converges in the large N limit as
lim (sst) N
Σ (k) [ 2(ml (0) + (k)) + (())
(21)
Claim 2 (Generalization error of the second stage). Given O and O2, the expected generalization enror ((2nd) converges in the large N limit as
2nqr + DD) 2(m (0) + mf)) 2x(m1 (0) mm( {?) lim (2nd) (1) Q2 + x N
(22)
Clains I and 2 indicate that for a given set of hyperparameters, the asymptotic generalization error can be computed by solving a finite set of non-linear equations in Definitions T and 2 Therefore, one can obtain the optimal choice of hyperparameters minimizing the generalization error via a numerical root- boot-findin and optimization procedure in finite dimension.
8

---

Published in Transactions on Machine Learning Research (1/2025)
3.3 Interpretation of 0 and 22
By choosing appropriate function f in the expression equation 13 can easily show that the following claims hold.
(2dd) Claim 3 (Expected inner product between the regressors). Recall that I(K)) and a I(k) are subvectors of
(1st)
((2st) and ((2dd) with indices corresponding to the unique feature vector for class k, x. (k) Given 11 22, the expected inner product between a 22())
(22d)) and (h))) are given by II(K)
lim E N-000 N
D.D
a
xz I(A)
(1st)
(k)
= my (k)
lim
D.D
2
I(K) xx mm2
(2nd)
[k)]
(23)
N N
Furthermore, we also have that
lim 1 No000 N E.p (1st) =1
lim N N E
[1]] = 92,
lim
Bo (2 (((2d)]] grs (24)
N-000 N
that is, 91992 denote the norm of the first and second stage regressors respectively, while 4r denotes the inner product between the first and second stage regressors.
Claim not only clarifies the interpretation of the some of the parameters in O₂ and O2 but it also provides an alternative derivation of the expressions in Claims and 2 For instance, (1st) can be expressed alternatively as
K
(1st) = lim
E ED
A(1) (x. (0)
(k)
a 2200)
+ A I(k))
(x.
(0)
2 211)) - Σ
A x 2
+ e (
No000
N
Iike
k=1
k'#0,k
lim
ED
x
+ $100 2 + || (0)
(1st)
12
+
|| I(Mr)
1| + (+)))
(25)
N +00
N
I(k)
k A
which results in the expression in Claim by using equations for the second stage generalization error e (nd)), in Claim
23 and
24
Similar calculations can be made
4 Comparison with synthetic finite-size data simulations
To verify our theoretical results, we conduct a series of synthetic numerical experiments to compare the results obtained from Claim 2 and the empirical results for the generalization error obtained from datasets of finite-size systems.
The synthetic ic data is specified as follows. We consider the case of K 2 classes, with ( 0.2, 0.4, 0.6, and (2) 0.8. Two types of settings for the common and unique feature vectors are considered; one with (0)) x(1 = 7(2) 0.09, and (0) 0.15, (11) = x2) = 0.04. These two settings are chosen to represent the cases where the common feature vector is relatively small compared to the unique feature vectors, and the case where the common feature vector is relatively large, but with the total number of non-zero features undorlying each class being the same (0.19N). Finally, the noise level of each dataset is set to (11) = (2) 0.1. The numerical sinulations are performed on random realizations of data with size N 16, 000. The second stage generalization error is evaluated by further generating 256 random realizations of (A(), ()) for each dataset and calculating the average empirical generalization error over these realizations. The perraarrameerss Aι and A2 are chosen such that they minimize the generalization error given in Claims and 2 for each stage respectively. The finite size simulations are also performed using the same values of A and 2 used in the theory. In figure I we plot the generalization error of the second stage as a function of AA for Æ = 0.0,0.2,00 0.5, and 1.0. As evident from the plot, the generalization error is in fairly good agreement with the theoretical predictions.
Case with scarce target data. Let us look closely at the left column of figure D where the target sample size is relatively scarce (1) 0.2). For both settings of (z)()) =(1), (21)), the generalization error
9

---

Published in Transactions on Machine Learning Research (1/2025)
0.10 10,
(( =0.0=00009
(a all (0.2,
(a 21i)
(a 2 (0.6,
0.030
K-02
x-02
K-655
0.0.5
0.030
0.030
0.029
0.026
0.028
0.028
0.024
0.027
0.022
0.026
0.025
0.020
0.924
10
107
10-
1.0
10'
10-
100
10'
M
A
A
81 0.15. اiا of 0.04
(.2.0.8)
(of (..4)..8)
2l (0.6.0.8)
0.0185
0.0
-.0
0.022
02
0.228
k=0.5
0.024
0.021
0.0180
0.022
0.020
6.0175
0.020
0.0179
0.0165
0.016
0.144
0.12
10"
6A
64
Figure 1: Comparison of generalization error obtained from Claim 2 (solid line) with finite size simulations (markers). Error bars represent standard error obtained from 64 realizations of cltta.
is insensitive to the choice of AA when is tuned properly. Another important observation is that Trans- Lasso's 's hyperparameter choice (K = 1, AA = 0) scems comparable with the optimal choice regarding the generalization error. The same trend is also seen in the case where (L = 0.4 and (zx(),, (0.15, 0.04, 0.04) (bottom center subfigure of figure
1=(0))=(1 x(11, x x(2) =
Case with abundant target. data. Next, we turn our attention to the right column of figure where the target sample size is relatively abundant (a 0.6). In this case, the generalization error is sensitive to the choice of AA but is insensitive to the choice of x around its optimal value (w 0.0), for both settings of (i("), (1),),)))) (2)). This indicates that under abundant target data, transferring the knowledge with respect to the support of the pretrained fcature vector (1st) is more crucial than the actual value of (1stt) itself.
The same trend can be seen for the case when a (1) = 0.4 and (I( x11), x(2)) (0.15, 0.04, 0.04) (top center subfigure of figure
5 Hyperparameter Selection Strategy
The experiments from the last section indicate not only the accuracy of our theoretical predictions but also the practical implications for hyperparameter selection. The observations made from the two cases suggest that there are roughly two behaviors for the optimal choice of hyperparameters: One where the choice of k is more crucial, and the other where the choice of AA more important. This observation suggests a simple strategy for hyperparameter selection, where one fixes Aλ 0 for the former case and for the latter case. To further investigate the validity of this strategy, we conduct comparative analysis of the generalization error among the following four strategies:
10

---

Published in Transactions on Machine Learning Research (1/2025)
= 02 0.8
01 = 0.3, 0; =.8
0.4, =.8
0.8
1.0
0.6
0.4
0.5
0.2
0.0
0.0
0.0
10a1
10-1
10a1
3 10
10a²
10a
10-3
10-2
10-1
2.0
2.0
2.0
1.5
1.5
1.5
1.0
0.5
0.0
Strategy O Strategy
Strategy
Strategy Strategy
A4 0 Strategy
LO Strategy
LO Strategy
LO Strategy
Strategy
Strategy
Strategy
Generalizatio Etc.
10-1
10-1
10"2
10:2
10-1
10
10-1
10
10-1
10
Figure 2: Generalization error and optimal hyperparameters for each strategy as a function of for (an, 02) (0.2, 0.8), (0.3, 0.8), (0.4, 0.8). The region shaded in red indicates the range of where the AA 0 strategy outperforms the strategy, while the region shaded in green indicates the opposite.
strategy. Here, we fix к to zero, while letting AA be tunable parameter. This reflects on the insight obtained in subscction for the abundant target data case. The regularization parameters Åı and λ are both chosen such that they minimize the generalization error in the first and second stages, respectively.
A = O strategy. Here, we fix AA to zero, while letting to be tunable parameter. This reflects the insight obtained in subsection 4 for the scarce target data case. The regularization parameters \ and A are both chosen in the same way as the k = strategy.
Locally Optimal (LO) strategy. Here, both and AA tunable parameters. The regularization parameters \ and X2 are both chosen in the same way the 0 strategy.
Globally Optimal (GO) strategy. Here, all four parameters (Aı, K, AA) are tuned such that the second stage generalization error is minimized.
The last strategy requires retraining the first stage estimator (1st) for each choice of (A1, A2, K, AX), which is computationally demanding. Even the LO strategy can be problematic when the datasct size is large. Basically, Whese two strategies are regarded as a benchmark for the other two stralegies: x == and AA 0 strategies. Below, we compare these four strategies in terms of the second-stage generalization error computed by our thcoretical analysis. Specifically, the case (I)(), z(1), x(2))= = (0.10, 0.09, 0.09) investigatod. In figure we plot the second stage generalization error, as well the optimal choice of hyperparameters for each strategy function of a, for (21,02) (0.2, 0.8), (0.3, 0.8), and (0.4, 0.8). Note that the shaded green
11

---

Published in Transactions on Machine Learning Research (1/2025)
d- 0.01, (nt nl)) (0.1)) 00.1,,009,0.000
Pretraining Lesso
Trans-Lasso
Generalized Trans-Lasso
2.0
1.06
1.0
1.7
1.25
0.35
0.35
0.35
1.24
5 0.255
1.4
1.03
14
1.02
0.15
1.2
0.15
0.15
1.01
1.1
0.05
0.05
0.00
01
0.3
0.5
0.7
0.9
0.3
0.3
0.5
0.7
0.9
0.1
0.3
2,5
0.7
0.9
0.1, (nt nfl m/1 (0.1,0.09, 0.09,0
Pretraining
Generalized Trans-Lasso
0.45
0.45
0.45
135
1.05
130
0.35
0.35
0.35
5 0.25
3.03
0.25
120
0.25
1.02
1.01
0.05
2.00
0.25
1.00
0.05
1.00
0,7
0.9
0.7 0.9
Figure 3: Ratios min Ax== =x=0 /LLO, Pretrriirerra and éTrans/ under setting (z(0),
(3(0)_(()))) (2) 12)))
10.1, 0.09, 0.09) and noise level a = 0.01 (top figure) and 0.1 (bottom figure) for various values of (a(1), al2)). Note that each heatmap has its own color scale bar; see Appendix for a direct comparison with shared color scale bars between the x = or AA = 0 strategy with Pretraining-Lasso and Trans-Lasso.
region indicates the range of where the strategy outperforms the AA strategy, while the shaded red region indicates the opposite.
Let us first examine the behavior of optimal AA across different strategies. With the exception of the case (0n,00) = (0.2, 0.8), the optimal AA values for each strategy (excluding the AA = 0 strategy) are approx- imately equal in the green region. Notably, for (a a²)) = (0.4, 0.8), the = strategy demonstrates gencralization performance comparable to the LO strategy in this region, while the AA = 0 strategy exhibits significantly lower performance. Above a certain noise threshold, the optimal value of AA for both the LO and GO strategies becomes strictly zero. At this point, the LO strategy and the Aλ strategy coincide, indicating that the AA 0 strategy can potentially be equivalent the LO strategy.
Next, let strategy outperfomms the A = 0 strategy, both the TO and GO strategies actually exhibit finite values of K. This means that the k 0 strategy is suboptimal. However, it still exhibits comparable performance with the LO strategy and gives a significant improvement from the AA = 0 strategy. This underscores the importance of solely transferring support information; transferring the first stage feature vector itself can be less beneficial. This may be due the crosstalk noise induced by different datascts: Using the regression results on other datasets than the target can induce noises on the feature vector components specific to the target. Consequently, it may be preferable to transfer more robust information, in this case the support configuration, rather than the noisy vector itself obtained in the first stage.
the behavior of the optimal к in each strategy. In the green region, where the 0
The above observations indicate that the performance closely comparable to the LO strategy can be achieved across a wide range noise levels by simply considering either the x 0 or AA = 0 strategy; the x = 0 stratogy preferred low noise scenarios while the AA strategy better high noise scenarios. For practical purposes, these two strategies should be examined and the best hyperparameters should be chosen based comparison. This is the main message for practitioners in this paper.
12

---

Published in Transactions on Machine Learning Research (1/2025)
Method
Drama
Comedy
Horror
Gen. Trans-Lasoo, A
5.58 + 0.07
5.07 = 0.11
Gen. Trans-Lasso,
5.67 = 0.08
0.10
5.48 + 0.13
Gen. Trans-Lasso LO
5.58 + 0.07
5.18
5.19 0.12
Pretraining Lasso
5.63 = 0.07
5.78
5.19 0.12
Trans-Lasso
5.71 + 0.08 5.78 0.10
5.17 + 0.12
Table 2: Comparison of lest errors for various Lasso-based methods and hyperparameter stralegies applied to the IMDb dataset. Error estimation on the test error is done by performing jackknife resampling on the test dataset.
Implications on the original Trans-Lasso. The AA = 0 strategy provides a lower bound for the gen- cralization error of the Trans-Lasso, as the latter constrains a to unity instead of treating it as a tunable parameter. Our results demonstrate that the Trans-Tasso suboptimal compared corporate support information in the sccond stage, particularly in the green region of figure 2 Notably, for 02) (0.4,088 and a 0.01, the AA 0 strategy reduces the generalization error by over 30% compared to the Trans-Lasso
strategies that in-
Ubiquity of the simplified hyperparameter selection. To further confirm the efficacy of the two simple strategies, we investigate the difference in generalization error between the LO strategy and x Aλ strategy. More specifically, let CLO. G==0 and сдλ=о be the generalization error of the corresponding strategies. Figure 3 shows the minimum ratio min <x=0)/ELO for various values of (a1, For comparison, the same values are calculated for the Protraining Lasso and Trans-Lasso, i.c. (Pretrain /f.0 and eTrans/eL.,, with Eretrrii and éTrans being the generalization error with the parameters chosen optimally (note that d1 and X2 are chosen in the same way as the LO strategy). The results indicate that, across a broad range of parameters, at least one of the simplified strategies (either AA = 0 or k. = 0) achieves a generalization error within 10% of that obtained by the more complex LO strategy. Notably, substantial relative differences are observed only within a narrow region of the parameter space (0()),((()) highlighting the effectiveness of the simplified approach across wide range of problem settings. We also report that for the case
0.5,
the AA = 0 strategy always coincided with the LO strategy in the region (a(1),(( E [0.05, 0.45] x [0.1, 10 0.9). Results consistent with the above observations were also seen for other settings of (z("), 7 (2) x(21); see Appendix B
The role of \1. For low noise scenarios, the GO strategy tends to prefer larger A1 in the first stage compared to the other strategics. This observation may be beneficial when the noise level of each dataset is known a priori: We may tune the A value to a larger value than the one obtained from the LO or its proxy strategies (x = 0 or AA = strategies). This offers an opportunity to enhance our prediction beyond the LO strategy while avoiding the complicated retraining process necessary for the GO strategy.
6
Real Data Application
6.1 Application to the IMDb dataset
To verify whether the above insights can be generalized to real-world scenarios, we here conduct experiments on the IMDb dataset Maas al. (201I). The IMDb dataset comprises a pair of 25,000 user-generated movie reviews from the Internet Movie Database, each being devoted to training and test datasets. The dataset is evenly split into highly positive or negative reviews, with negative labels assigned to ratings given 4 or lower out of 10, and positive labels to ratings of 7 or higher. Movics labeled with only one of the three most common genres in the dataset, drama, comedy, and horror, are considered. By representing the reviews as binary feature vectors using bag-of-wordds while ignoring words with less than occurrences, we are left with 27743 features in total, with 8286 drama, 5027 comedy, and 3073 horror movics in the training datasct, and 9937 drama, 4774 comedy, and 3398 horror movies in the test dataset. The datascts can be acquired from Supplementary Materials of Gross & Tibshirani 2016
13

---

Published in Transactions on Machine Learning Research (1/2025)
Drama
Comedy
Horror
5,50
5.7
6.00
5.95
5.70
5.3
5.65
5.85
5.2
0.0
00
0
$
06
03
00
09
10
Figure 4: Second stage test error for each IMDB genre plotted against varying values of hyperparameters s and AA/Az. The plots demonstrate that the effect of Aλ on generalization performance tiny across genres.
For sake of completeness, we calculate the test error for (x, AX/A2) E {0.0, 0.200.,..... 1.5) 0.10, .... ... ...0 to see if our insights are qualitatively consistent with experiments on real data. For each pair of (x, AA/22), Åι and λ2 are determined by a 10-fold cross-validation procedure, where the train- ing dataset is further split into learning dataset and validation dataset. Since λ (and consequently (((st)) must be selected oblivious to the validation dataset, we perform Approximate Leave-One-Out Cross- Validation (Obuchi & Ka Kabashima] 2016 Rad & Maleki 2020 Stephenson & Broderick) 2020) on the learning dataset, which allows one to estimate the leave-one-out cross-validation error deterministically without ac- tually performing the computationally expensive procedure. For the sake of comparison, we also compare with the performance Pretraining Lasso and Trans-Lasso with d and X2 chosen using the same method above.
In Figure 4 present the second stage test error for each genre across various values of (k, AA/12). The plot reveals that AA is essentially redundant hyperparameter having almost no inffuence on generalization performance. This observation strongly supports the elficacy of the AA = 0 strategy in the present dataset. We compare the test errors obtained from three approaches within our generalized Trans-Lasso framework: The AA 0 strategy, the к = strategy, and LO strategy. Additionally, we include test errors from Pretraining Lasso and the original Trans-Lasso for comparison, whose results are summarized in Table Recall that Trans-Lasso fixes to unity, while Pretraining Lasso employs (N, AA/X2) = (1 8, s)ss) for hyperparameter E [0, 1]. In the experiments, choose from (0.0, 0.1, 0.2, ...... 1.0) via cross-validation. Our analysis demonstrates that the hyperparameter choices made by both the original Trans- and Pretraining Lasso are suboptimal compared to the AA = 0 strategy of our generalized Trans-Lases algorithm. Note that the LO stratogy can potontially select hyperparameters exhibiting higher test error compared to the AA = stralegy (as seen in the horror genre), as they are chosen via cross-validation and the true generalization error not directly minimized.
6.2 Application to Compressed Imaging task the MNIST dataset
The performance of the Generalized Trans-Lasso algorithm also evaluated on a compresser imaging task Donohoo 2006 Takhar al.) 2006 Lustig et al. 2007 Romberg 2008 Candès & Wakin 2008 on the MNIST dataset Deng 2012). The MNIST dataset consists of grayscale images of handwritten digits with 28 28 pixels, totaling 784 dimensions. Given noisy, linear measurements of the first set of 3 handwritten images, "1", "7", and "g" appcaring in the MNIST dataset, the task is to refine the recovery performance of cach image in the wavelet. basis by transfer learning techniques. We consider the Gaussian measurement setup for vectorized images In," E 784 (n = 1, 7,9).. where measurements are given by Y*n" = H ".n""
"*," + e","" E RRM..... The matrices H consist of i.i.d. standard Gaussian elements, while es," represents additive Gaussian noise with i.i.d. centered elements calibrated to achieve uniform signal-to-noise ratio across measurements. The measurement. dimensions set to Maj- 200, Mag- 400, and Mage 600. The image recovery task is to
14

---

Published in Transactions on Machine Learning Research (1/2025)
reconstruct Z*x" under the assumption that its wavelet transform P'n- -W-la is sparse, where W is the 2-d wavelet transformation matrix. Therefore, all algorithms, generalized Trans-Lasso, Pretraining Lasso, and original Trsso-Lasso, are targeted to estimate the coefficients of each image in the wavelet basis, O-n", given observations y"u" and covariate matrix A' - W- The test error is evaluated by generating another random instance yon" with the same statistical profile and dimension as you", and using this as the test dataset.
We calculate the test error (k, A2/X2) grid, where (0.0,0.1,0.,..... 1.5), and AA/A₂ is taken from 21 logarithmically equidistanced points between 10-2 and 10. Other procedures are equivalent to those of the IMDb experiment.
In Figure 5 we present the second stage test error for different handwrilten digits across various values of (k. AA) under different SNR conditions. Consistent with our findings from syuthetic experiments, the support information plays crucial role in improving generalization performance, particularly at high SNR values. The test errors, with the hyperparameters chosen via cross-validation error minimization, are sunmarized in Table 3 For high signal quality (SNR = 20), the 0 strategy demonstrates superior performance. However, as the SNR decreases, the performance gap between different strategies diminishes, with differences in test error becoming statistically insignificant. This empirical observation aligns with our theoretical analysis (Figure 2 where the generalization error curves for the L.O, A = 0, and k = 0 strategies collapse into one curve at high noise levels.
SNR 20
Method
71 7
Gen. Trans-Lasso LO
0.0721
0.0069
0.0361 + 0.0029
0.0139 + 0.0009
Gen. Trans-Lasso, K = 0
0.0721
0.0069
0.0367 = 0.0029
0.0139 = 0.0009
Gen. Trans- Lasso, AA = 0
0.0816 0.0076
0.0404 + 0.0030
0.0144 = 0.0009
Pretraining Lasso
0.0847
0.0078
0.0381 + 0.0029
0.0140 + 0.0009
Trans-Lasso
0.1333 = 0.0122
0.0499 = 0.0037
0.0171 0.0010
SNR 5
Method
=
n = 7
9
Gen. Trans-Lasso, LO
0.163 = 0.013
0.103 + 0.008
0.0453
0.0028
Gen. Trans-Lasso, x
0.163 0.013
0.103 + 0.008
0.0450
0.0028
Gen. Trans- AA 0
0.169 0.014
0.106 1 0.008
0.0460
0.0028
Pretraining Lasso
0.165 0.014
0.103 + 0.008
0.0454
0.0028
Trans-Lasso
0.215 0.020
0.117 + 0.009
0.0504
0.0029
SNR
Method
π 7
Gen. Trans-Lasso LO
0.292 0.024
0.196 +: 0.015
0.099 + 0.006
Gen. Trans-Lasso, K = 0
0.290 0.025
0.196 = 0.015
0.101 = 0.006
Gen. Trans- AA = 0
0.303 0.026
0.203 + 0.015
0.101 = 0.006
Pretraining Lasso
0.292 0.025
0.196 +'2 0.015
0.099 + 0.006
Trans-Lasse
0.303 = 0.027
0.208 + 0.016
0.100 0.006
Table 3: Performance comparison of different methods across the three handwritten digits for SNR = 20, 5, and 2. Error estimation on the test error is done by performing jackknife resampling on the test dataset.
7 Conclusion
In this work, we have conducted a sharp asymptotic analysis the generalized Trans-Lasse algorithm, preciscly characterizing the effect of hyperparameters on generalization performance. Our theoretical calculations reveal that near-optimal generalization in this transfer learning algorithm can be achieved by focusing on just one of two modes of knowledge transfer from the source dataset. The first mode transfers
15

---

Published in Transactions on Machine Learning Research (1/2025)
0.0530
86
0018
1.00
1.09
0.2500
1.00
0.00
0.60
0.0475
0.30
0.39
0.99
0016
6.10
0.08
0.19
0.08
0.0475
0.05
0.05
0.03
6.63
0.0420
0.03
0.2375
00133
0.80
6.60
630
1.20
A50
0.30
0.50
150
6.90
1.3
1.59
SNR
Target
Target
Target
10.00
0.20
10.00
10.00
6.00
6.00
0.130
6.00
-0.054
3.00
0.26
3.0
3.00
0.052
1.00
0.24
1.00
1.00
0.60
0.60
0.50
0.30
0.3
0.30
3.050
0. 0.048
0.205
0 0.046
0.01
001
0.81
0:30
6.80
030
1.20
2.50
0.58
1.0
0.30
0.90
1,20
1.5
SNR =
Target
Target
Turget
10.00
10.00
6.00
0.33
8.09
5.00
3.00
3.09
3.00
1.00
0.2075
0.2050
0.104
0.2025
0.10
0.19
0.1
0.66
0.08
0.03
0.63
0.03
0.00
0.29
0.1975
6.01
6.0
0.61
0.00
0.50
0.90
1.5
6.88
0.30
0.60
8.00
1,20
1.50
0:0
0.30
128
Figure 5: Second stage test error for each MNIST image for SNR 20 (top), (middle) ,nd (bottom), plotted against varying values of hyperparameters and AX/A2. White markers are placed at the minimizer of the test error for sake of visualization. The plots demonstrate that the effect of AA generalization performance is nontrivial.
16

---

Published in Transactions on Machine Learning Research (1/2025)
only the support information of the features obtained from all source data, while the second transfers the actual configuration of the feature vector obtained from all sources. Experiments using the IMDb and MNTST dataset confirm that this simple hyperparameter strategy is effective, potentially outperforming conventional Lasso-type transfer algorithms.
Acknowledgements
This work was supported by JSPS KAKENHI Grant Nos. 22KJ1074 (KO), 22K12179 (TO) and Grant-in-Aid for Transformative Research Areas (A), "Foundation Machine Learning Physics" (22H05117) (TO).
References
Urte Adomaityte, Leonardo Defilippis, Bruno Loureiro, and Gabriele Sicuro. High-dimensional robust re- gression under heavy-tailed data: Asymptotics and universality. arXiv preprint FrXXi:2309.164 2023.
Trambak Banerjee, Gourab Mukherjee, and Wenguang Sun. Adaptive sparse estimation with side informa- tion. Journal of the American Statistical Association, 115 2067, 2020.
Jean Barbier and Nicolas Macris. The adaptive interpolation method: simple scheme to prove replica formulas in bayesian inference. Probability theory and related fields, 174:1133-1185, 2019.
Jean Barbier, Florent Krzakala, Nicolas Macris, Léo Miolane, and Lenka Zdeborová. Optimal errors and phase transitions in high-dimensional generalized linear models. Proceedings of the National Academy of Sciences, 116(12):5451-544)) 14400, 2019.
Hamsa Bastani. Predicting with proxies: Transfer learning in high dimension. Management Science, 67())) 2964-2984, 2021.
Mohsen Bayati and Andrea Montanari. The Lasso risk for gaussian matrices. IEEE Transactions on Infor- mation Theory, 58(4):1997-2011 2011.
Pierre C Bellec and Yiwei Shen. Derivatives and residual distribution of regularized m-estimators with
application to adaptive tuning. In Conference on Learning Theory, pp. 1912-1947. PMLR, 2022.
Pierre C. Bellec and Cun-Hui Zhang. Second-order Stein: SURE for SURE and other applications in high- dimensional inference. The Annals of Statistics, 49(4):1864 - 1903, 2021.
Pierre C. Bellec, Guillaune Lecué, and Alexandre B. Tsybakov. Slope meets Lasso: Improved oracle bounds and optimality. The Annols of Statistics, 46(6B):3603 3642, 2018.
Emmanuel J. Candes and Terence Tao. Near-optimal signal recovery from random projections: Universal
encoding strategies? IEEE Transactions on Information Theory, 52((1)):4406-544 2006.
Emmanuel J Candès and Michael B Wakin. An introduction to compressive sampling. IEEE signal processing magazine, 25(2):21-38 20, 2008
Kabir Aladin Chandrasekher, Ashwin Pananjady, and Christos Thrampoulidis. Sharp global convergence guarantees for iterative nonconvex optimization with random data. The Annals of Statistics, 51(1):179 210, 2023.
Patrick Charbonneau, Enzo Marinari, Mare Mézard, Giorgio Parisi, Federico Ricci- TercenTers,,hhi, Gabriele Si- curo, and Francesco Zamponi. Spin Glass Theory and For Beyond. WORLD SCIENTIFIC, 2023.
Erin Craig, Mert Pilanci, Thomas Le Menestrel, Balasubramanian Narasimhan, Manuel Rivas, Roozbeh Dehghannasiri, Julia Salzman, Jonathan Taylor, and Robert Tibshirani. Pretraining and the Lasso. arXiv preprint ar Xiv:24441 1291, 2024.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2916):141-142, 2012.
17

---

Published in Transactions on Machine Learning Research (1/2025)
David L Donoho. Compressed sensing. IEEE Transactions on information theory, 52(4):1289-1306 2006.
Codric Gerbelot, Alia Abbara, and Florent Krzakala. Asymptotic crrois for teachor-student convex gener- alized linear models (or: How to prove Kabashima's replica formula). IEEE Transactions on Information Theory, 69(3):1824-185, 2023.
Samuel M. Gross and Robert Tibshirani. Data shared Lasso: novel tool to discover uplift. Compulational Statisties and Data Analysis, 101:226 235, 2016.
Ehsan Hajiramezanali, Siamak Zamani Dadaneh, Alireza Karbalayghareh, Mingyuan Zhou, and Xiaoning Qian. Bayesian multi-domain learning for cancer subtype discovery from next-gencration sequencing count data. In Advances in Newral Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Adel Javanmard and Andrea Montanari. State evolution for general approximate message passing algorithms, with applications to spatial coupling. Information and Inference: A Journal of the IMA, 2(2):115 144, 2013.
Yoshiyuki Kabashima, Tadashi Wadayama, and Toshiyuki Tanaka. A typical reconstruction limit for com- pressed sensing based on f--nor minimization. Journal of Statistical Mechamics: Theory and Experiment, (00000)): 2009.
Takuya Koriyama, Pratik Patil, Jin-Hong Du, Kai Tan, and Pierre C. Bellec. Precise asymptotics of bagging regularized m-estimators, 2024.
Sai Li, T. Tony Cai, and Hongzhe Li. Transfer Learning for High-Dimenssioall Linear Regression: Predic- tion, Estimation and Minimax Optimality. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(1):149-15 173, 11 2021.
Sai Li, Tony Cai, and Hongzhe Li. Transfer learning in large-scale gaussian graphical models with false discovery rate control. Journal of the American Statistionl Association, 118(443):2171-21888 2023.
Sai Li, Linjun Zhang, T Tony Cai, and Hongzhe Li. Estimation and inference for high-dimensional generalized lincar models with knowledge transfer. Journal of the American Statistical Association, 119(546):1274- 1285, 2024.
Mengqi Lou, Kabir Aladin Verchand, and Ashwin Pananjady. Hyperparamoter tuning via trajectory predic- tions: Stochastic prox-linear methods in matrix sensing, 2024.
Michael Lustig, David Donoho, and John M Pauly. Sparse mri: The application of compressed sensing for rapid mr imaging. Magnelic Resonance in Medicine: An Official Journal of the International Sociely for Magnetic Resonance in Medicine, 58(6):1182 1195, 2007.
Andrew L. Maas, Raymond F. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (eds.), Proceedings of the 49th Annual Meeting of the Association for Compulational Linguistics: Human Language Technologies, pp. 142 44-050, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
Sarah F. McGough, Svetlana Lyalina, Devin Incerti, Yunru Huang, Stefka Tyanova, Kieran Mace, Chris Harbron, Ryan Copping, Balasubramanian Narasimhan, and Robert Tibshirani. Prognostic pan-cancer and single-cancer models: A large-scale analysis using a real-world clinico-genomic database. medRaie preprint 2023.12. 1923.12.18 18..2.1...00 19......8.23300166, 2023.
Nicolai Meinshausen and Peter Bühlmann. High-dinensional graphs and variable selection with the Lasso. The Annals of Statistics, 34(3):1436 1462, 2006.
M Mezard, G Parisi, and M Virasoro. Spin Glass Theory and Beyond. WORLD SCIENTIFIC, 1986.
Léo Miolane and Andrea Montanari. The distribution of the Lasso: Uniform control over sparse balls and adaptive parameter tuning. The Annals of Slatistics, 49(4), 2021.
18

---

Published in Transactions on Machine Learning Research (1/2025)
Tomoyuki Obuchi and Yoshiyuki Kabashima. Cross validation in LASSO and its acceleration. Journal of Stalistical Mechanics: Theory and Raperiment, 2016(5):053304, 2016.
Tomoyuki Obuchi and Yoshiyuki Kabashima, Semi-analytic resampling in Lasso, Journal of Machine Learn- ing Research, 20(70)11-33, 2019.
Koki Okajima and Takashi Takahashi. Asymptotic dynamics alternating minimization for bilinear regres- sion. Xiv prepriul or Xiv:24402.044 04751, 2024.
Koki Okajima, Xiangming Meng, Takashi Takahashi, and Yoshiyuki Kabashima. Average case analysis of Lasso under ultra sparse conditions. In Proceedings of The 26th International Conference Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 11317-11330. PMLR, 25 27 Apr 2023.
E. Ollier and V. Viallon. Regression modelling on stratified data with the Lasso. Biometrika, 104(():83-99, 2017.
Sinno Jialin Pan, Ivor W. Tsang, James T. Kwok, and Qiang Yang. Domain adaptation via transfer compo- nent analysis. IEEE Transactions on Newral Networks, 22(2):199 210, 2011.
Raphael Petegrosso, Sunho Park, Tae Hyun Hwang, and Rui Kuang. Transfer learning across ontologies for phenome-genone genome association prediction. Bioinformatics, 33(4):529 536, 11 2016.
Kamiar Rahnama Rad and Arian Maleki. A scalable estimate of the out-of-sample prediction error via approximale leave-one-out cross-validation. Journal of the Royal Slatisticul Sociely Series B: Stalistical Methodology, 82(4):965-99) 996, 2020.
Sundeep Rangan, Vivek Goyal, and Alyson Fletcher. Asymptotic analysis of map estimation via the replica method and compressed sensing. Advances in Newral Information Processing Systems, volume 22. Curran Associates, Inc., 2009.
Justin Romberg. Imaging via compressive sampling. IEEE signal processing magazine, 25(2):14 20, 2008.
Luca Saglietti and Lenka Zdeborová. Solvable model for inheriting the regularization through knowledge distillation. In Mathematical and Scientīfic. Machine Learning, pp. 809-846. PMLR, 2022.
William Stephenson and Tamara Broderick. Approximate valssvaaiiotion in high dimensions with guaran- tees. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 2424-2434. PMLR, 26 26-28 Aug 2020.
Mihailo Stojnic.
framework to characterize. performance of Lasso algorithms.
arXin preprint
arXiv:1303. 7291, 2013.
Fci Sun and Qi Zhang. Robust transfer learning of digensinnniio generalized linear model. Physica A: Statistical Mechanics and its Applications, 618:128674, 2023.
Masaaki Takada and Hironori Fujisawa. Transfer learning via & regularization. Advances in Neural Infor-
mation Processing Systems, 33:14266 14277, 2020.
Takashi Takahashi. Role of bootstrap averaging in generalized approximate message passing. In 2023 IEEE International Symposium on Information Theory (ISIT), pp. 767-772, 2023.
Takashi Takahashi. The role of pseudo-labels in self-training linear classifiers on high-dimensional gaussian mixture data. arxie preprint arXiv:2205.07739, 2024.
Takashi Takahashi and Yoshiyuki Kabashima. A statistical mochanics approach to de-biasing and uncortainty estimation in Lasso for random measurements. Journal of Statistical Mechanics: Theory and Experiment, 2018(7):073405, 2018.
19

---

Published in Transactions on Machine Learning Research (1/2025)
Dharmpal Takhar, Jason N Laska, Michael B Wakin, Marco F Duarte, Dror Baron, Shriram Sarvotham, Kevin F Kelly, and Richard G Baraniuk. A new compressive imaging camera architecture using optical- domain compression. In Computational Imaging IV, volume 6065, pp. 43 52. SPIE, 2006.
Christos Thrampoulidis, Ehsan Abbasi, and Babak Hassibi. Precise error analysis of regularized m-estimators in high dimensions. IEEE Transactions on Information Theory, 64(8):5592 5628, 2018.
Ye Tian and Yang Feng. Transfer learning under high-dimensional generalized linear models. Journal of the American Statistical Association, 118(544):2684 2697, 2023.
Robert Tibshirani. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society. Series B (Methodological), 58(1):267-28) 1996.
Lisa Torrey and Jude Shavlik. Transfer Learning. In: Handbook of Research on Machine Learning Applica- tions and Trends: Algorithms, Methods, and Techniques. IGI Global, 2010.
Turki Turki, Zhi Wei, and Jason T. L. Wang. Transfer learning approaches to improve drug sensitivity prediction multiple myeloma patients. IEEE Access, 5:7381-7393, 7393, 2017.
Mikko Vehkaperä, Yoshiyuki Kabashima, and Saikat Chatterjee. Analysis of regularized LS reconstruction and random matrix ensembles in compressed sensing. IEEE Transactions on Information Theory, 62(4): 2100-2124, 2016.
Matteo Vilucchio, Emanuele Troiani, Vittorio Erba, and Florent Krzakala. Asymptotic characterisation of the performance of robust linear regression in the presence of outliers. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li (eds.), Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pp. a11-819. PMLR, 02 04 May 2024.
Martin J. Wainwright. Sharp thresholds for high-dimensional and noisy sparsity recovery using & constrained quadratic programming (Lasso). IEEE Transactions Information Theory, 55(5):2183-220) 2202, 2009.
Meiyu Wang, Yun Lin, Qiao Tian, and Guangzhen Si. Transfer learning promotes 6G wireless communica- tions: Recent advances and future challenges. IEEE Transactions on Reliability, 70(2):790 807, 2021.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3:1-40, 2016.
Peng Zhao and Bin Yu. On model selection consistency of Lasso. Journal of Machine Learning Research, 7 (90):2541-2563, 2006.
Yin Zhu, Yuqiang Chen, Zhongqi Lu, Sinno Jialin Pan, Gui-Rong Xue, Yong Yu, and Qiang Yang. Hetero- gencous transfer learning for image classification. In Proceedings of the Twenty- Fifth AAAI Conference on Artificial Intelligence, AAAI'11 pp. 1304-1308 AAAT Press, 2011.
Fuzhen Zhuang, Ping Luo, Hui Xiong, Qing He, Yuhong Xiong, and Zhongzhi Shi. Exploiting associations between word clusters and document classes for cross-domain text categorization. Statistical Analysis and Data Mining: The ASA Data Science Journal, 4(1):100 114, 2011.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43 76, 2021.
20

---

Published in Transactions on Machine Learning Research (1/2025)
A Detailed derivation of Asymptotic analysis
Here, we provide a detailed dorivation for the asymptotic formulae for the generalization errons e(ss)) and ((2nd), Recall that calculate the generalization error, an appropriate choice of in equation 13
D) = t Σ (g A 1 + t2llg ( x2)
(26)
Therefore, the objective of our analysis is to calculate the data average of the n-th power of
$(D,D) D) = day dazze -B, (200 (z.: (waxe + E (1 2 (1+tallg)))))))) (xx) (27)
in the successive limit lim +0o lim For n E N, one can express this as
E DDD TT (D, (D.
=
II
da da22
exp - Σ
B1
|0 + 32r(222, ( a.
a=1
M
B1
K
C B
II
II exp
(1) A x1,a)) )
11
(T A =)
(28)
2
k=1
k=1
(y/h)
(2)
(K01A x22a))² 2 t2(
A (KE11 a + x22a))
Conditioned { 22.o 1 the distribution over {A 11
p.
a 1 A (.) 02 A 2, Bu 1p
is i..d. with respect to μ,, whose profile is identical to that of the random variables
11. = (k)
11,a
h2.a = (2) 32,05
N1. = (k))
11.as
h2.a = a(1) 22,05
(29)
(K)) = I(k) x
(k)
(k)
+ a
x. + (8)) I(0)
(R)
(0)
f = a (k)
I(k) x + I(0) x
(k)
(k)
(0)
g())
(30)
with (k), ((k) being set of N-dimensional centered Gaussian vectors with ii.d. elements of variance 1/N, and g(k), ((k) being independent centered Gaussian random variables with variance (x))) for k 1.... K. Note that the random variables with different index k are independent of each other. Following this observation, L can be expressed alternatively as : I (1) (LI())) where
61432 (H -kh h22a)2 81 (H -
(
exp
(31)
2
2
7,(0)
exp
- 1. (1)(0) - 1 S1 6111((1(1) xh - Tro.a)
(32)
Here, 6, = Illi ]] denotes the Kronecker delta, which is unity if = and zero otherwise. From the Gaussian
nature of (a(k) (ak) k== and (g()), E(k) Ik=1 the random variables { (1)()) nn
( ( and (H()) ]
21

---

Published in Transactions on Machine Learning Research (1/2025)
all centered Gaussians with the following covariance structure:
k
E[ (1] 1,b = 0t
Q = N
21,b
N
I
1,a
(k))
1,6
((K))
+ +
1,a
(neg.)
2 1,h
(neg.)
k==0
11 Q22
Q1
1
22
22,b
1
2,a 2.6
(K))
(K')
+ 22aa
(neg.)
22bb
(neg.)
N
N
k'=0
(1)
51
N
1
F1,
3 11.0
(k')
2.6
(K')
+ a: 1,0
(nog.)
(nng.)
(33)
E (k) HD) 5ra 55a(m) (ml (1)
+ m1 1
m
(k')
1
2
Li.a
(M)
(k))
(K' 0, 1, .K)
N
El 11 (8)]= = 5x1 (m²
(A:)
+ +m²
22,a
(K) = N 2,0
(K)
xx
(k)
(K' = 0, ... K).
1
(1)
| (0) N
1
11 + (0()) Seep!e)
ax
N
for 1 S k, S K and S a. S n. Here, 11 and a are defined as the subvectors of 2i,a and #2,a indexed by I(k) respectively, while ai, indices are not included in K I(k). The same covariance structure also follows for the random variables (00 nn.K [ k= (h2a)ā=1 and {
(neg.) and 22,
(neg.) are both subvectors of 31,a and T2 whose
1== Define the average with respect to the random variables given in (k) (R), equation 33, conditioned on Ω (Qth, Q mmia: "2,)) as Eng- Inserting the trivial identities based on the delta function corresponding to the definitions of Ω given in equation 33 we can rewrite equation 28 to a trivial multiplicative constant as
f
dd /
II
(k) dx L.a 2a
(k)
Σ (B1d1l331,a11 + B2r(211 ((20,
Σ ( (8)
(1))))
exp
a=1
k=0
K Σ
K
x II
NQU
(k')
(N')
(neg.)
(noz.)
NQQ
Σ
(k')
(K')
(neg.) (neg.)
1,b
11.0
ii,b
22a
2.b
22.0 #22.b
a5b=
===0
k==0
T2
NQ
x Fi,a
(k')
2,6
(K)
Li.
(neg.)
2,6
(neg.)
E.
II
5
(k)
11.0
(k)
xx (0)))((( 8(Nm2.
(k)
-22.
(N)
(
a,6= 11
==
k=
(34)
(k) where E, is the average with respect to the set of ground truth vectors (x. =k= whose elements are i.i.d. according to a standard normal distribution. Note that the delta functions can be expressed alternatively using its Fourier representation as
K (k')
8
NQ
1,a
1.b
(k')
x
11,0
(neg.)
1i.
(neg.)
d -N Ex o (k') (k') ) (..
=0=0
8
NQ
ab
K
2,a
(K')
x
2.b
(K')
zzza
2,a
(neg.)
22.h
(neg.)
aQga 2z" adppe (NO3" Ex mntt
(00
M==
8
NQub
r
K
1,a
x
2.b
(k')
11,a
(neg.)
22,b
(neg.)
[ dQpe e
6z" (NQ.)" ->K (A (00
- -i a (o0
(35)
===
5(Nml.)
(k)
(l))
xx
=
dm [
(k)
( (Nm =1/10 an(t))
8 6(NmL9)
(k)
-22 x
(k) (8)
dmzza
(k) thl -00 22:
22

---

Published in Transactions on Machine Learning Research (1/2025)
up to trivial multiplicative constant.
To obtain an expression that is analytically continuable to n + 0, we introduce the replica symmetric ansatz (Charbonneau et al. 2023). which assumes that the integral over Q is dominated by the contribution of the subspace where 52 satisfy the following constraints:
Q = + (1 - 80b) X1 B1
Q2 = 92 + (1-11-00 dab)
(1-bab)
X2
Q = gr +1 dab)
Xr B1
32
(36)
(k)
(k)
(k) (k)
m
m2,a
m2
for 1 S 0, S n and 0 S h: S K. Although a general proof of this ansatz itself is still lacking, calculations based on this assumption are known to be asymptotically exact in the limit N + co for convex generalized
linear models (Gerbelot et al., 2023) and Bayes-optimal estimation (Barbier et al. 2019). Consequently, the
conjugate variables 1Qth, Q2 Qnb (8) mg are also assumed to be replica symmetric:
Q = 31 - (1 - 88x)8881,
Q 32 - (1 3
Q = -2 + (1 -2 (37)
(k)
3mm (k)
(k)
32m2 (A)
m
m2,
Inserting equations 36 and
37
Lo equalion
34 and rewriting B12 as Eie given this simplified profile of 5 offers
E,
d0
d0
dx
(k)
dx
(K) 2,a
exp
M log(E
(k))
+
M log log( 1og(Bio
4=1
k=1
k=
x xexpm.1 N.N
31
XIXi
mi m1
(k) (4:)
+ 32
9202
X2X2
-9r4t Xcxr
Σ k=0
mi mi
(1) (1))
2
k=
K
x exp
B1
91
2
1,a
(k)
2
m
(k)
11.a
+ (11la -3 Σ
91
1,a
(neg.)
213 + A ||=
1,x_ !11)
0=1
k=0
(38)
82
72
42
2,a
(k)
2
(miz (k)
+ (()))
22,0 + rr r(a (002200100110) (.o))
(K)
k=0
82
Q2
2,a
(neg))
11
1.0
(neg.)
2,2
(neg.)
+ rxx
(neg:) | (mee)))))
+
81
F1.a
+
32
22a
+ 3132
Σ Eaia)
(Σ a2.a)
2
2
a=1
a=1
a=1
The last equation can be further simplified using the equality
83
n + 3.13288
cxp
Σ
11,
+
F2
Σ 11,a
( F2aa
(39)
2
2
a=1
a=1
a=1
a=
EK E2,,2 exp
Bi 81
21 L1,a + 1.2
82
Σ 22
222aa
(40)
a=1
a=1
where 21, F2 E V are random Gaussian vectors with elements i.i.d. according to ø1,1 ~ 21, 22,5 ~ Z2 for all == 1.... N. From this decomposition, the integrals over Ti, and Ø2 decouple over both vector coordinates = I.... NN and replica indices 1,
[ (0)d00/ / Ii M log Eja (0) >Kx, M log E 1(0)
{ 16 I/ dz (01)"")) 1 N00
K
II k=0
/ daidaze B BiE( (21) (0a121) m1) N. N(0
x {[Es
(41)
23

---

Published in Transactions on Machine Learning Research (1/2025)
In the successive limit of 31 and B2 + oo, the integrals 21 and 22 can be evaluated using Laplace's method, yielding the asymptotic form:
f d0
II k=1
M log Big + Be E
E2
(o [ "") v000
(42)
E2
Bi
A2E (oermr))) pu {m}
N ->Exo N00
where the definitions of Eg and (x1 X2 k= x1_
(nec.) (neg.) follow from Definitions and
X2
Let us proceed with the calculation of E Define the centered Gaussian variables (Hr(), 11 _,1 and au 42 3 as
H()
p(k)
(k)
(k)
nx) N
O3,
mi
(k)
mi
m2
Gr
(uia U2,a
N O2,,
X1131 Xr/31) X x2/B2)
(43)
h2
mig
Q2
where (ui,as )2 are independent for all a 1,... n. Then, we can see that the random variables admit the decomposition (k) (1)
11
h2,a
h12 + u2 offering the expression
(k)
B281k B1 (11 (k) 1110)-18 1 u1
()) = E, H000
Eur.
u2.a
exp
2
(x(h)) +u1
U2,0
a=1
(44)
Taking inlo account that 31 >> B2, the Gaussian measure over (ui,a; u2,a) can be wrillen as
Cg. exp
81 (1 + Ju,
B2
u2,a
Xr
u1.0
du duj,aduz,ar
(45)
2x1
2x2
X1
where Cg,,9, is a number subexponential in 81 and $2, and ca, is a real number which converges to zero B1 + co. Using this expression to equation 44 yields
{ dup duiduz exp
B1 (1 + cz,
81 (H( - -n(aa) -uz)²
E I,) =B =B,(00 (g .H(0))
2x1
2
(46)
Xru1) + log Cpr }
B2 2
3201k
(k)) k((((1) w1) 122
u2
2x2
X1
The same procedure can be applied to [00, this time accounting for t1, l2 < B2, offering
i() = Egg0 5,00) _.0 (e expm t (1)) - -n(()))()) )(())² togia (H()) - r2h(1) - T(( + log CBn
(47)
Note that the expressions for (R) and L (R) are simple Gaussian integrals that can be computed explicitly. Now that all the terms in equation 34 have been expressed analytical form with respect to n, the limit can be taken formally. Evaluating equations 42
46 and up to first order of n, and neglecting subleading terms with respect to and B2, we finally obtain an expression of the form
f d0, exp N ( θ2) O(n3)],
E ED (D(
(48)
where
0(0,,0%) -(.(()))))G((())) (.2((0)01)) 116 + 1262
(49)
24

---

Published in Transactions on Machine Learning Research (1/2025)
K
K Σ
() = 9191 - XIXI
Σ
m1 m1
(8)
(k)
Ej
E1 ( (x(²))
(neg E( (x1 (neg ))
2
k=
k=
a (R)
91 2(m1 (0)
+ m))
+ (()
(50)
2
1+ X1
k=1
K Σ k=0
G2(O2/(1)
9292 X2X2
9- - X.xr
m2 m2
(k) (k)
K
12 (k)) (x2
(k) (k))
(neg.)
Bg
E2
[(egg)) (xg (neg
(xg
|x/
km0
a (1) + 13 + -A4 + B²p(1) -2B + ) -2AB( (m) (0) +
+mi]" (1)
(51)
2
1 X2
and
K
61
π (k)
91
+ +m 2m
(0)
+ pl()]
(52)
k=1
92 + x²qq 2xg, + p11)) - 2(m2 (0) + mm 2x(m) (0) + mm (n/))]
62
(53)
For large N and finite the integral over θ₁ and O₂ can be evaluated using the saddle point method, where the integral is dominated by the stationary point of the exponent. This finally yields
a-0 - N On log Ep x" (D.
lim lim 1
Extr g(O,, 02). ,,
(54)
where Extr denotes the extremum operation of the function. The stationary conditions for θ and O2 in
the successive limit of 31 + oo and 32 + 30 are given by the equations of state in Definitions [ and 2 Note that the term t16 + f262 docs not contribute to the stationary condition the limit f, >> 0, and thus can be neglected until one takes the derivative with respect to 4 and l2, as in equations 11 and
B Additional numerical experiments
Here, we provide more numerical experiments highlighting the difference between the Generalized Trans- Lasso, Pretraining Lasso and Trans-Lasso. In figures Cand 7 we show the ratios of minf < ( and CTrans against eLO for the cases (z))), (1), x22) (0.15, 0.04, 0.04) and (0.05, 0.14, 0.14), respectively, The former case represents the problem setting where the underlying common feature vector among the datasets is large, while the latter represents the problem setting where the underlying common feature vector is small. In both cascs, the minimum of the x = 0 or AA = 0 strategy can obtain generalization errors close to the one from the LO strategy for both low or moderate noise levels. This is not. a property exhibited in the Trans-Tasso or Pretraining Lasso, where both algorithms have relatively low generalization performance when a 0.01 and (T) ((1), (2))) = (0.15, 0.04, 0.04). Note that when (T("), (1), (2)) = (0.05, 0.14, 0.14), all three algorithms yield comparable generalization performance.
AA O strategy with the Pretraining Lasso and Trans-Lasso, in figures 9 and 10 we also plot the same ratios with shared color scale bars. As evident from the plot, the k = 0 or A 0 strategy exhibits a clear improvement in generalization performance over Trans-Lasso, While the difference between Pretraining Lasso and x = 0 or AA 0 strategy is comparable for a = 0.1, we can see a consistent advantage for lower noise levels (a = 0.01).
To directly compare the
25

---

Published in Transactions on Machine Learning Research (1/2025)
-- 0.01, (n" If m1)) (0.15 0.04, 0.04)
Pretraining Lasso
Trans-Lasso
Generalized Trans-Lasso
4.0
1.12
3.5
1.30
0.35
0.35
0.35
3.0
2.5
2.0
1.04
0.15
0.15
0.15
1.52
0.05
0.05
81
0.3
0.5
0.7
0.9
0.1
0.5
0.7
0.9
0.1
0.3
2.5
0.7
0.9
0.1, (1f nf! (/l) (0.15, 00.15,0.04, 0.04)
Pretraining
Generalized Trans-Lasso
1.060
1.05
1.035
0.30
0.35
1.030
2.0
1.025
8
1.23
1.5
1.6
1,21
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
0.1
0.5
0.5
0.7
0.9
Figure 6: Ratios min o, o/moo/ PPrttrain/era /er.o and eTrans/eLo under setting (=(0), (2), x222) = (0.15, 0.04, 0.04) and noise level o = 0.01 (top figure) and 0.1 (bottom figure) for various values of (a aa2)).
a- 0.01, (x()) R!) R3 (0.05, 0.14, 0.14)
Pretraining
Trans-Lasse
Generalized Trans Lasso
1.12
0.35
0:35
1.30
0.35
1.000
1.08
1.08
1.015
1.26
1,000
1.04
0.15
3.15
0.15
1.52
1,50
1.00
1.000
0.1
0.3
0.5
0.1
0.3
0.5
9.1
0.3
0.5
0.7
0.9
82
d- 0.1, (nfft nlll n!!) - (0.05, 0.14, 0.14)
Pretrsining Lasso
Trans-Lasso
Generalized Trans-Lasso
3,025
1.025
1.08
0.75
0.35
1.06
0.35
1.05
1,010
1,000
1.53
0.15
3.15
1.02
0.15
1.005
1.005
1,200
1.00
1,000
6.1
0.5
0.1
0.3
6.5
0.3
0.5
0.7
92
Figure 7: Ratios min 6,=0 <Pretrain/ELO and eTrans/eLo under setting (1), (2))
(0.05, 0.14, 0.14) and noise level a 0.01 (top figure) and 0.1 (bottom figure) for various values of (1! (a(1),a(2) .Q(2)..
26

---

Published in Transactions on Machine Learning Research (1/2025)
0.01, (ott) at" x/4) 10.15,0.04, 0.04)
Pretraining
0.35
035
6.25
0.25
0.25
0.29
0.15
0.25
0.29
0.13
-15
0.05
9.28
0.09
0.09
9.5
0.3
0.1, GeMM of afl)) 10.15, 0.04,0044000000 0.000.0;
Pretraining Lesso
Genera fizee Trans-Lesse
Trans-Lassso
Generalized Trans-Lassop
0.45
0.45
2.6
2.0
1.33
3.0
1.42
1.4
1011
122
5.05
01
05
0.3
0.7
92
Figure Comparisons between ratios min(ca)=0.G6=0)/000 Preetriin /LO and CTrans /co under setting (T (0) x n ()) (0.15, 0.04, 0.04).
In?" 62 (2)) 10.1,
Pretraining Lasso
Trans-Lasso
1.8
6.35
0.35
6.35
14
1.6
5.25
0.25
14
0.25
0.25
3.4
0.15
0.15
12
0.15
5.15
1.2
0.05
0.05
0.05
0.05
3.0
0.5
0.2
0.3
0.7
0.5
0.9
0.3
0.7
0-0.1, (nflt (Pi) (lP) 10.1, 0.
Pretraining Lasso
Generalized Trnns Lasso
Tra Lasso
Generalized Lasse
0.45
3A5
0.45
0.45
0.35
0.25
1.04
1.25
1.00
3.20
1.00
0.15
1..01
3.55
0.5
1,00
3.5
3.00
Figure 9: Comparisons between ratios min(eax=o SPretrain/eLo and eTrans/eL.o under setting (T(0) 7 (1) (2)) 10.10, 0.09, 0.09).
27

---

Published in Transactions on Machine Learning Research (1/2025)
(n!) (Pa) 0.34)
Pretrahing
Tans-Lasso
Generalized
3.32
0.35
0.35
1.00
0.25
1.06
1.06
1.00
0.15
815-
0.15
0.15
1.02
0.05
9,5
0.05
0.05
0.1
0.3
0.5
0.7
3.6
0.1
6.3
0.5
0.7
0.5
0.1
5.3
0.5
0.7
0.9
0.1
0.3
0.5
5.7
o-
(11 ((9) - 10.05,
Pretraining Lasso
Generalized Trans- Lasso
Trans- La:50
Goneralized Trass-Lasse
0.45
2.5
1.005
0.25
0.45
0.35
035
0.35
0.25
1.06
0.25
0.55
0.25
0.25
1.04
1.010
0.15
5115
11,055
0.1
0.15
1.02
0.05
0.05
1,000
0.05
0.05
0.1
0.5
0.1
0.5
0.2
3.
0.7
0.1
5.7
Figure 10: Comparisons between ratios min(cAx=o- Cx=O//<LO, 6Pretraia/ /L0 and CTrass /co under setting (T (0) n (1) (2)) = (0.05, 0.14, 0.14).
28


Reviews:
[{"review_id": "2trRgN32m1", "reviewer": "Reviewer_XUb3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n\n-both theoretic analysis as well as experiments are provided;\n\n-authors work on a popular problem;\n\n-the quality of the writeup is good.\n\nweaknesses:\n\n-strong assumptions on the data, among others the true feature vectors are also assumed to have i.i.d. standard Gaussian entries;\n\n-in the real world dataset the hyperparameter $\\Delta \\lambda$ does not have a significant influence. Is there also a real world dataset where tuning $\\Delta \\lambda$ improves the error?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Authors study an algorithm, the Generalized Trans-Lasso, in a high-dimensional setting of Transfer learning via an asymptotic analysis using the replica method to address the issue that selection strategies for these hyperparameters, as well as the impact of these choices on other algorithm’s (such as Trans-Lasso and Pretraining Lasso) performance, have been largely unexplored.\nThey provide an asymptotic analysis as well as synthetic and real world data experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There are a lot of variables which makes it hard to follow especially if one is not familar with Transfer learning and it would be nice if there was a page with a definition and expanation of all variables (can be in the appendix).\n\nLast line page 10: Results \\*are\\* consistent...", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "v2Hz39TyVS", "reviewer": "Reviewer_nkcD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe paper is generally well-written. The authors do a good job of explaining different aspects of the replica method, specifically its limitations (e.g., the fact that it is a heuristic) as well as how its output (which is a set of equations whose behavior is not obvious upon inspection) can be used to make useful predictions. The topic should also be of interest to some members of the TMLR community. The pre-training/fine-tuning paradigm has gained significant popularity and influence recently, and while the present paper studies an admittedly highly stylized version of this problem, developing theoretical insights in simplified settings can be an important step towards understanding the more complicated practical settings.\n \n# Weaknesses\n\nI'm not fully convinced that the relevance of the theoretical results for the practical setting studied in the paper has been substantiated. In their case study on the IMDb dataset, the authors show that tuning $\\Delta \\lambda$ has very little effect on the test error. There are some settings where a similar effect is observed in theory (e.g., the bottom-left plot in Fig. 1), but it's not clear why we should expect the IMDb dataset to fall into this category. At present, it seems that the only connection between the theoretical results and practice is that for the IMDb dataset, the $\\Delta \\lambda = 0$ strategy happens to work well. In order to claim that the theory gives practically relevant insight, it seems to me that the authors should show how one can use the qualitative insights from the theory and some properties of the IMDb dataset to predict this behavior a priori, without first doing an extensive hyperparameter search.\n\nIn equations (11) & (12), the integral is taken with respect to the unnormalized measure $w_{\\beta_1, \\beta_2}(x_1, x_2; \\mathcal{D})$ rather than the probability measure $P_{\\beta_1, \\beta_2}$. Is this a typo, or is there some reason that we can use the unnormalized measure instead? Also, the intuition for why we take the limit of $\\beta_1, \\beta_2 \\to \\infty$ was clear; can similar intuition be used to explain the limits $t_1, t_2\\to 0$?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study transfer learning with the lasso in a high-dimensional linear regression setting via the replica method. Specifically, they consider a two-stage training setting (pretraining and then training on the target dataset) which generalizes two previous methods (Trans-Lasso and Pretraining Lasso) to allow for a more flexible transfer of two possible types of information from the first stage. The two types of information transfer are the support of the linear regression coefficients, or the coefficients themselves.\n\nUsing the replica method, they derive a set of equations whose solution gives the generalization error under different hyperparameter settings. They confirm a good agreement between the replica method predictions and empirical results on synthetic data which matches their theoretical setup, then use this to give guidelines on hyperparameter tuning and when one should prefer to transfer one type of information (support or coefficients) vs. the other. Finally, they explore the effect of the two different types of transfer on the IMDb dataset to see if their insights generalize to real data.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please provide more explanation of the connection between the first part of the paper/insights developed from the replica method and the practical results on the IMDb dataset. Is there some aspect of the theory and the properties of the IMDb dataset that allows us to predict the results of Fig. 4 a priori? If not, how would you suggest we apply the theoretical insights in practice?\n\n2. Please clarify the question on equations (11) & (12) from the weaknesses section.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KYgZnxnFNU", "reviewer": "Reviewer_jR6M", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n- (S1) The paper focuses on the critical computational limitation of hyperparameter selection with a theoretically motivated heuristic to reduce the number of hyperparameters to be searched over.\n- (S2) The empirical results in Section 4 demonstate how the generalization error estimated with finite data closely matches the asymptotic quantities, thereby nicely motivating the utility of an asymptotic analysis.\n- (S3) The paper provides nice discussion of the empirical results, and thus, proposes well motivated heuristics for efficient hyperparameter selection strategies, considering different problem setups such as scarce vs abundant data or low vs high noise.\n\n### Weaknesses:\n\nGiven my lack of expertise in statistical physics, and its application to machine learning, I believe that there are three main weaknesses of this manuscript:\n- (W1) The main analysis leading to the expression for the generalization error lacks appropriate explanations, motivations and discussions (even in the appendix), making the presentation extremely convoluted, and leaving the reader with no intuition as to why the analysis is meaningful and of interest. The following are some specific instances of the above issue:\n  - (W1.1) From the presentation of Section 3.1, there is no motivation for how we arrive at equations (11) and (12). Establishing a connection to (7) and (8) would be very useful. I can see the connection of $\\beta_1 \\to \\infty$ and $\\beta_2 \\to \\infty$ but the role of $t_1, t_2$ needs clear explanation\n  - (W1.2) It is not clear from the description why we should care about the equation (13) and how it is related to (11) and (12). I can see that when applying the $\\partial / \\partial t_1$ and then applying $t_1 \\to 0$, we get the term inside the integral but no $\\log \\left( \\int \\cdots \\right)$. So I think it needs some clarification since the ensuing discussion following (13) highlights the role of the replica method in handling the right hand side of equation (13). Even the detailed version in Appendix A begins by trying to compute the right-hand-side of equation (13) except that the equation (23) actually contains the exponentiated parts of equations (11) and (12) while the right hand side of (13) did not. So that is already a discrepancy that needs to be clarified.\n  - (W1.3) For the untrained eye, it might be useful to clarify why the equality in equation (13) is true.\n  - (W1.4) The \"equations of state\" in definition 1 need a lot of clarification. There are many unclear and somewhat confusing elements to it. Without proper motivation or understanding, I dont think readers will be able to understand why such a result makes sense and how they might be extended to situations where the assumptions of the analysis do not hold, which makes the utility of such analysis quite limited.\n    - (W1.4.1) First, the variables $\\Theta_1$ need some introduction.\n    - (W1.4.2) Second, we state that the $\\mathsf{x}_1^{(k)}$ etc are obtained as a solution to a random optimization problems (14) and (15) given $\\Theta_1$. However, then equation (16) defines the system of equations driving the variables $\\Theta_1$, which depends on (the expectation of) the solutions to (14) and (15). This bilevel nature of this system of equation makes the definition quite confusing without appropriate motivation and discussion.\n    - (W1.4.3) Given the bilevel structure of this system of equations (16), it is also good to understand the conditions under which we should expect a solution. Alternately, it would be good to understand the salient structure of this system of equations that ensures there will always be a solution. Similarly, it might be good to motivate why we are guaranteed to find finite solutions and why certain variables are ensured to be nonnegative (for example, why is $\\hat{\\chi}_1$ guaranteed to be positive in equations (14) and (15)).\n    - (W1.4.4) Note that some of the definitions require the gradient of an optimal solution (for example, $\\frac{\\partial}{\\partial \\mathsf{z}_1} \\arg \\min_x \\mathsf{E}_1^{(k)}(x)$) which is a nontrivial quantity in general.\n  - (W1.5) Similar lack of clarity and motivation for definition 2\n  - (W1.6) - Given the quantities defined in definitions 1 and 2, it is nice that the generalization errors are derived from simple expressions. However, without proper introduction and discussion of the quantities obtained via definitions 1 and 2, the results are not intuitive -- this would have been a great opportunity to highlight how the generalization error decomposes into different (possibly intuitive) terms, and how each of these terms are affected by the hyperparameters.\n- (W2) The definitions of the generalization error being considered in this manuscript seem unintuitive, and thus closed-form expression for such definitions makes the connection to the more standard definitions of generalization unclear.\n  - (W2.1) In equations (7) and (8), it is not clear why we are scaling by number of features $N$ to define the generalization errors. I think we should be scaling by $(1/K)$ (for just equation (7)) and the number of observations $\\tilde{M}_k$ per new dataset $\\tilde{\\mathcal{D}}^{(k)}$ (for both (7) and (8)). So it could be something like the per-dataset mean-squared error. Equation (7) could be \n\n$$\\epsilon^{\\text{(1st)}} = \\mathbb{E}_{ \\mathcal{D}, \\tilde{\\mathcal{D}} } \\frac{1}{K} \\sum\\_{k=1}^{K}  \n\\frac{1}{|\\tilde{\\mathcal{D}}^{(k)} |} \\left[ \\left \\| \\left \\| \\tilde{\\mathbf{y}}^{(k)} - \\tilde{\\mathbf{A}}^{(k)} \\hat{\\mathbf{x}}^{\\text{(1st)}} \\right \\| \\right\\|_2^2  \\right], $$ \nand equation (8) could be \n\n$$\\epsilon^{\\text{(2nd)}} = \\mathbb{E}_{ \\mathcal{D}, \\tilde{\\mathcal{D}} } \n\\frac{1}{|\\tilde{\\mathcal{D}}^{(1)} |} \\left[ \\left \\| \\tilde{\\mathbf{y}}^{(1)} - \\tilde{\\mathbf{A}}^{(1)} ( \\kappa \\hat{\\mathbf{x}}^{\\text{(1st)}}  + \\hat{\\mathbf{x}}^{\\text{(2nd)}}  )\\right\\|_2^2  \\right].$$ \n\nHow would the change in the definition of the generalization error affect the results and bounds (if at all)? Alternately, it is good to discuss why this is an appropriate notion of generalization?\n- (W3) Given the simplified hyperparameter search strategies proposed in this paper based on the analysis, it is not clear whether it has actually lead to smaller hyperparameter search spaces since there are baselines which reparameterize the hyperparameters to reduce one hyperparameter as well.\n  - (W3.1) Given the Pretraining Lasso formulation with a total of 3 hyperparameters, $\\lambda_1, \\lambda_2, s$ with $\\kappa \\triangleq 1 - s$ and $\\Delta\\lambda \\triangleq (1-s) \\lambda_2 / s$, what computational advantage does Generalized Trans-Lasso provide if any since it also has to tune 3 hyperparameters?\n  - (W3.2) It seems that a simple reformulation of the hyperparameters (similar to Pretraining Lasso) as $\\kappa = 1 - s, \\Delta \\lambda = s \\lambda_2 / (1 - s)$ span the spectrum of the $\\kappa = 0$ and $\\Delta \\lambda = 0$ strategies, with $s = 0$ giving us the $\\Delta\\lambda = 0$ strategy, and the $s = 1$ giving us the $\\kappa = 0$ strategy? This again reduces the number of hyperparameters then to 3 (instead of 4). And, with the strategies proposed by the theory in this paper, we still need to solve a 3 hyperparameter optimization problem (with either LO or GO). So what gain can Generalized Trans-Lasso provide over this?\n  - (W3.3) For the real-data experiments, what are the error bars around the metrics in Table 1? Without that, it is not clear if the differences are significant.\n\nBeyond the above, here are some less significant comments and questions:\n- (C1) Is a large $N \\to \\infty$ limit where $N_k / N \\to O(1)$ (so $N_k \\to \\infty$) an interesting usecase for practical sparse regression? Would the existing analysis be able to handle the case where $N_k / N \\to 0$ while $N \\to \\infty$? \n- (C2) The interpretation regarding noise levels and $\\kappa = 0$ vs $\\Delta \\lambda = 0$ strategies, it is a bit unclear why the $\\Delta \\lambda = 0$ strategy (share actual feature vector) performs better that $\\kappa = 0$ strategy (only share support information) in the high noise setting. As explained in 4th paragraph in Section 5 (page 9), \"Consequently, it may be preferable to transfer more robust information, in this case the support configuration, rather than the noisy vector itself obtained in the first stage.\" Hence, in higher noise setting $\\kappa = 0$ strategy would seem be the desirable one that just shares support information. What am I missing here?\n- (C3) Figure 3 can be improved where we have the same heatmap scale between the different columns since we are supposed to be comparing the ratios $\\epsilon / \\epsilon_{LO}$ where $\\epsilon \\in \\{ \\min\\lbrace \\epsilon_{\\Delta \\lambda = 0}, \\epsilon_{\\kappa = 0} \\rbrace, \\epsilon_{\\text{Pretrain}}, \\epsilon_{\\text{Trans}} \\}$. Otherwise, it is hard to compare the performances between the methods.\n- (C4) In page 2, step 1 (before equation (1)), should it be $\\hat{\\mathbf{x}}^{\\text{(1st)}}$ in place of $\\hat{\\mathbf{x}}_1$?\n- (C5) How are the two tasks different in the experiments in Section 4. Everything is Gaussian. Are the sampled non-overlapping $\\mathcal{I}(k), k \\in \\{1, 2\\}$ the only thing different? It would be good to clarify this in the experiment description.\n- (C6) Where are the results for Trans-Lasso that is mentioned in the \"Comparison with the original Trans-Lasso\" paragraph on page 10? Is it from Figure 3?  It would be good to refer to the appropriate figure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript focuses on the problem of transfer learning in sparse high-dimensional linear regression, and studies the hyperparameter selection aspect of some recent two-stage regression schemes. Proposing a generalization of two existing schemes, the paper pursues a direct relationship between the generalization error of this scheme (for both the stages) and the selected hyperparameters. For this purpose, the authors perform an asymptotic analysis under various assumptions (Gaussian design matrices and feature vectors) utilizing tools from statistical physics (the replica method), establishing a systems of equations whose solution lead to closed-form analytic expressions for the generalization error utilizing problem dependent quantities. Then the authors highlight how the estimated generalization error in a finite data setting closely matches the asymptotic estimate (given that the data satisfy the required assumptions and that various data dependent quantities are known), thereby motivating the asymptotic analysis.\n\nBased on the insights from the finite data simulations, the author motivate a couple of simplified hyperparameter selection processes that remove the need to tune one of the 4 hyperparameters and empirically demonstrate the ability of these simplified strategies to match the more computationally expensive strategy of optimizing all 4 hyperparameters simultaneously. The empirical evaluations consider both synthetic data as well as the non-synthetic IMDB dataset. The empirical evaluations with the synthetic datasets highlight the different scenarios (data scarce vs data abundant or low noise vs high noise or low support overlap vs high support overlap) where the proposed simplified hyperparameter selection strategy perform competitively to the more expensive standard strategy. The experiments on the IMDB dataset highlight that the simplified strategies outperform the existing baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Based on my discussion on the weaknesses, I think it is critical to address at least weakness W1.", "broader_impact_concerns": "There is no broader impact discussion in this paper, but there are no ethical concerns with this work in my opinion.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on the problem of transfer learning in sparse high-dimensional linear regression, and studies the hyperparameter selection aspect of some recent two-stage regression schemes. Proposing a generalization of two existing schemes, the paper pursues a direct relationship between the generalization error of this scheme (for both the stages) and the selected hyperparameters. For this purpose, the authors perform an asymptotic analysis under various assumptions (Gaussian design matrices and feature vectors) utilizing tools from statistical physics (the replica method), establishing a systems of equations whose solution lead to closed-form analytic expressions for the generalization error utilizing problem dependent quantities. Then the authors highlight how the estimated generalization error in a finite data setting closely matches the asymptotic estimate (given that the data satisfy the required assumptions and that various data dependent quantities are known), thereby motivating the asymptotic analysis. The authors address reviewers' concerns well in their rebuttal. Thus, I would like to recommend accept as is.
Decision: Accept (poster)