The Choice of Normalization Influences Shrinkage in Regularized Regression
Authors: Johan Larsson
Jonas Wallin
Venue: Accepted by TMLR ()

Abstract: Regularized models are often sensitive to the scales of the features in the data and it has therefore become standard practice to normalize (center and scale) the features before fitting the model. But there are many different ways to normalize the features and the choice may have dramatic effects on the resulting model. In spite of this, there has so far been no research on this topic. In this paper, we begin to bridge this knowledge gap by studying normalization in the context of lasso, ridge, and elastic net regression. We focus on binary features and show that their class balances (proportions of ones) directly influences the regression coefficients and that this effect depends on the combination of normalization and regularization methods used. We demonstrate that this effect can be mitigated by scaling binary features with their variance in the case of the lasso and standard deviation in the case of ridge regression, but that this comes at the cost of increased variance of the coefficient estimates. For the elastic net, we show that scaling the penalty weights, rather than the features, can achieve the same effect. Finally, we also tackle mixes of binary and normal features as well as interactions and provide some initial results on how to normalize features in these cases.

Published in Transactions on Machine Learning Research (11/2025)
The Choice of Normalization Influences Shrinkage in Regu- larized Regression
Johan Larsson
jola Caathm ku,
Deportment of Mathemationl Sciences, University of Copenhagen Deportment eoportment of Statistics, Lund Universily
Jonas Wallin
jonas. walas &staf su
Department of Statistics, Lund University
Reviewed on OpenReview: https: openrevie.. net/ met/ forum? id- 6aKyDBIuQ5
Abstract
Regularized models are often sensitive to the scales of the features in the data and it has therefore become standard practice to normalize (center and scale) the features before fitting the model. But there are many different ways to normalize the features and the choice may have dramatic effects on the resulting model. spite of this, there has so far been no research on this topic. In this paper, we begin to bridge this knowledge gap by studying normalization in the context of lasso, ridge, and elastic net regression. We focus binary features and show that their class balances (proportions of ones) directly influences the regression coefficients and that this effect depends on the combination of normalization and regularization methods used. We demonstrate that this effect can be mitigated by scaling binary features with their variance in the case of the lasso and standard deviation in the case of ridge regression, but that this comes at the cost increased variance of the coefficient estimates. For the elastic net, we show that scaling the penalty weights, rather than the features, can achieve the same effect. Finally, we also tackle mixes of binary and normal features well as interactions and provide some initial results on how to normalize features in these cases.
1 Introduction
When modeling high-dimensional data where the number of features (p) exceeds the number observations (n), it is impossible to apply classical statistical models such as standard linear regression since the design matrix X is no longer of full rank. A common remedy to this problem is regularize the model by adding a penalty term to the objective that punishes models with large coefficients. The resulting problem takes the following form:
minimize ((0, 9; x,X, h(B), B0ER
(1)
where y is the response vector, X the design matrix, By the intercept, and 3 the coefficients. Furthermore, g is a data-fitting function that attempts to optimize the fit to the data and h penalty that depends only on B. Two common penaltics are the norm and squared & norm penaltics, which g the standard ordinary least-squares objective, represent the lasso (Tibshirani, 1996; Santosa & Symes, 1986; Donoho & Johnstone, 1994) and ridge (Tikhonov) regression respectively.
These penalties depend on the magnitudes of the cocfficients, which mcans that they are sensitive to the scales of the features in X. To avoid this, it is common to normalize the features before fitting the model by shifting and scaling each feature by some measures of their locations and scales, respectively. For some problems it possible arrive at these mcasures by contextual knowledge of the data at hand. In most cases, however, they must be estimated. A popular strategy is use the mean and standard deviation of each feature location and scale factors respectively, which is called standardization.

---

Published in Transactions on Machine Learning Research (11/2025)
housing
ala
triazines
wla
Standardization
max, |3,1
0.5
0.0
1.0
3 max 13,
Max Max-Abs Abs
0.0
-05
20
40
60
0
20
10
60
0
20
10
60
20
10
60
Step
Step
Step
Step
Figure/ Lasso paths for real datasets using types of normalization: standardization and maximum absolute value normalization (max-abs). For each dataset, we have colored the coefficients if they were among the first five to become non-zero under either of the two normalization schemes. The r-axis shows the steps along the regularization path and the y-axis the estimated coefficients normalized by the maximum magnitude of the coefficients each case. See Section for more information about datasets used here.
The choice of normalization may, however, have consequences for the estimated model. As first example of this, consider Figure 1, which displays the regularization paths for the lasso' for four datasets: housing (Har- rison & Rubinfeld, 1978), ala (Becker & Kohavi, 1996; Platt, 1998), triazines (King et al., 1995; Hirst et al., 1994), and wla (Platt, 1998), and two lypes of normalization.
the figure, we have colored the lines corresponding to features that were among the first five to enter the model in either of the two type of normalization schemes. Note that the choice of normalization result different sets of features being selected well different, coefficient estimates. This is especially striking in the case of triazines and wla.
To illustrate this further, we show the estimated coefficients for the same datasets after having fitted the lasso with a penalty strength (A) set by 5-fold cross-validation repeated 5 times on a 50% training data subset. The estimated coefficients for the 50% held-out test data are shown in Table 1. We see that the estimated coefficients on the test set are different between the two cases, especially for the triazines datasets, where the two normalization schemes disagree completely and the max abs normalization results in all coefficients being zero.
In spite of this apparent connection bet normalization and regularization, there has so far been almost no research on the topic. And in its absence, the choice of normalization is typically motivated by computational concerns or by being "standard". This is problematic since the effects of normalization are unknown and because there exists no natural choice for many types data. In particular, there is no obvious choice for binary features (where each observation takes either of two values). Tn this paper we begin to bridge this knowledge gap by studying normalization in the context of three particular cases of the regularized problem in Equation (1): the lasso, ridge, and clastic net (Zou & Hastic, 2005). The latter of these, the clastic net, is
The estimated coefficients of the penalty strength is varied from large-enough value coefficients low value which the model almost saturated.
2

---

Published in Transactions on Machine Learning Research (11/2025)
Table 1: Estimated lasso coefficients on test sets, with set from 5-fold cross-validation repeated 5 times, Aata and amax-ahs show the coefficients based on normalizing the design matrix with standardization and maximum absolute value (max abs) normalization respectively. We show the five largest coefficients (in and the respective cocfficients for the max-abs
magnitude) for the standardization present the coefficients on the original scale of the features. See Section G for more information about these datasets.
In each case,
housing
ala
triazines
wla
Astd
Bunxx-abs
Bstd
f mmax-abe
Autd
Bimxx-aba abs
fotd
β mmax-abs
0.63
-0.68
0.54
0.54
0.17
0.0
1.8
0.0
-1.4
-0.78
0.33
0.52
0.069
0.0
1.8
0.78
0.27
0.0
0.39
-0.51
0.028
0.0
1.8
0.63
0.99
-0.34
0.31
0.32
0.071
0.0
1.4
0.080
2.8
3.1
0.18
0.23
0.029
0.0
1.7
0.0
a generalization the previous two, and represented by the following optimization problem:
X2 1111112.
BoER,BER 2 3o x,313 2 1111/111
minimize
(2)
where setting A 0 results in ridge regression and setting A2 = 0 results in the lasso. Our focus in this paper is on binary data and we pay particular attention to the case when they are imbalanced, that is, have relatively many ones or zeroes. In this scenario, we demonstrate that the choice of normalization directly influences the estimated coefficients and that this effect depends on the particular combination of normalization and regularization. For first illustration of this, see Figure 2, which corrosponds have generated two binary fealures with class balances 0.5 and 0.9, respectively, and fit full lasso and ridge regression paths. The coefficients when generating the data were set to one in both cases. The type of normalization affects the sizes of the resulting coefficients. This, for instance, leads to opposite conclusions being drawn about feature importance depending on whether L1 or Max-Abs normalization is used. For instance, at the point where the coefficient of the balanced feature is 0.5, the coefficient of the imbalanced features is roughly 0.75 with LI normalization compared to 0.25 with Max Abs normalization.
small experiment which
Standardization
Max-Abs
1.00
0.75
Ridge
Bernoulli(055)
Bernoull((.9)
0.25
0.00
1.00
2530
0.50
0.25
0.00
0
25
50
75
100
0
25
50
75
100
25
50
75
100
Step
Figure 2: Unstandardized coefficients for lasso and ridge regression along a regularization path. The models are fit to an experiment with two binary features, generated to have class balances 0.5 and 0.9, respectively. We have generated the response y; æT 8 (without noise), with 8 1. Note that the coefficients' paths overlap in for the L1 and Standardization normalization types in the lasso and ridge cases, respectively.
3

---

Published in Transactions on Machine Learning Research (11/2025)
Our key contributions are:
1. We reveal that class balance in binary features directly affects lasso, ridge, and clastic net estimates, and show that scaling binary feat ures with standard devialion (ridge) or variance (lasso) miligates these effects at the cost of increased variance. Through extensive empirical analysis, we show that this finding extends to a wide range of settings (Section 4).
2. In our theoretical work, we examine this relationship in detail, showing that this bias from class imbalance holds even in the case of orthogonal features (Section 3.1). For the clastic net, however, we show that this effect conmot be mitigated by normalization and instead must be dealt with by scaling the penalty weights rather than the features (Section 3.4).
3. For mixed data designs, we demonstrate how normalization choices implicitly determine the relative regularization effects on binary versus continuous features (Section 3.2).
4. For interaction features, show that common alternative to normalizing interaction features leads to biased estimates and provide an alternative approach that mitigates this problem (Section 3.3).
Collectively, our results demonstrate that normalization is not merely a preprocessing step but rather an integral component of the model that requires careful consideration based on data characteristies and the chosen regularization approach.
2
Normalization
To avoid possible confusion regarding the ambiguous use of terminology in the literalure, we will begin by clarifying what we mean by normalization, which we define as the process of centering and scaling the feature matrix.
Definition 1 (Normalization). Let X Raxp be the feature matrix and let e E R and s E R. be centering and scaling factors respectively. Then X is the normalized feature matrix with elements given by ã; = (Iig - /s)/ where i and] j index rows and columns, respectively.
Some authors refer to the procodure in Definition 1 as standardization, but here we define standardization only as the case when centering with the mean and scaling with the (uncorrected²) standard deviation.
There are many different normalization strategies and we have listed few common choices in Table 2. Standardization is perhaps the most popular type of normalization, at least in the field of statistics. One of its benefits is that it simplifies certain aspects of fitting the model, such as fitting the intercept. The downside of standardization is that it involves centering by the mean, which destroys sparsity in X since centering shifts zero values to non-zero.
When X is sparse, two common alternatives to standardization are min max and max abs (maximum absolute value) normalization, which scale the data to lie in [0, and [-1, respectively, and therefore retain sparsity when features are binary. These methods are, however, both sensitive to outliers. And since sample extreme values often depend on sample size, as in the case of normal data (Section B.1), use of these methods may sometimes be problematic. Another alternative is to replace the fo-norm in the standardization method with the fı-norm, which Icads to e--oomalizaaiin. We note here that this mothod is often used without centering, but this would make the method depend on the mean of the feature, as in the case of max abs normalization, and we therefore prefer the centered version here.
We have also included robust normalization in Table 2, which is version of normalization that uses the median and interquartile range (IQR) centering and scaling factors. Finally, we have also included the adaptive lasso (Zon, 2006), which is a special case of normalization that fits a standard ordinary least-squares regression (OLS) model to the data and uses the OLS estimates as scaling factors. 3 We have both of these methods in Table 2 for completeness, but we will not study it further in this paper.
2Standard deviation computed without Bessel's correction (use of n instead of the standard deviation formula). Tn the case when p 3 n, ridge estimator is typically used instead.
4

---

Published in Transactions on Machine Learning Research (11/2025)
Table 2: Common ways to normalize a matrix of features using centering and scaling factors oj and $51 respectively. Note that By the arithmetic mean of feature j and that Qa(x,) ) is the ath quartile of feature j.
Normalization
Cj
Standardization
Val ||æ, 12yl2
f--Normalization
zjj
va ||ay Tzlli
Max-Ar Abs
max; |وري||
Min Max
min (rig)
max;(()))) min;(())
Robust Normalization
Q2(1r))
Qs(2r)) - Qi (2zg)
Adaptive Lasso
0
180LS|
In the next section, will examine how the choice of normalization affects the estimates for the lasso, ridge, and clastic net regrossion.
3 Ridge, Lasso, and Elastic Net Regression
In this setting we begin to describe the connection between normalization and the elastic net estimator. We start by showing how the scaling and centering parameters of the normalization method factors into the clastic net estimator in a general case. We then narrow our focus to the case of binary features and present our main results bias, variance, and selection probability when the features unbalanced.
Throughout
the paper we assume that the response y is generated according to y .Bg + x + with X being the x design matrix with features (columns) zj. the vector of noise, with mean zero, finite variance o² and identically and independently distributed entries. We also assume X, 80 and 9 to be fixed and the features of the normalized design matrix to be orthogonal, that is, X = diag (#T xTpp). In this case, is a well-known fact (Tibshirani, 1996) that the solution to the elastic net problem given by
Sx, (aJu) 85(h
8(n) y
(3)
xJ +2
where Sx(=) is the soft-thresholding operator, defined as Sx = sign(2) max([=| X, 0), which the proximal operator of the & norm. We refer to Section B.2 for a derivation of the results above.
The assumption of orthogonal features may use it for our theoretical results to show the direct connection between normalization and the elastic net estimator, and prove that even in this simple case, normalization has pronounced effect on the estimates. In our experimental work (Section 4), however, we show that our findings extend to a much wider class of designs and also refer the reader to Section A, where we discuss this assumption in detail and provide additional theoretical and empirical results in relation to this.
strong and is indeed almost never realised in practice. Here
Normalization changes the optimization problem and the estimated coefficients, which will now be on the scale of the normalized features. But here we are interested in A: the coefficients the scale of the original problem. To obtain these, we transform the coefficients from the normalized problem, g(() back via 3, = ((m) /8, for .p). There is a similar transformation for the intercept, but we omit here since
E [p], where ip] = (1.2.....p) not interested in interpreting it.
Taken together, this means that the solution for & can be expressed as
Sx (zJv)
A
dj
5

---

Published in Transactions on Machine Learning Research (11/2025)
where
$Jy fznv, - æ
$j
(4)
dj = 5j
+ X2
= + A2),
bj
with Vy being the uncorrected sample variance of xy.
The bias and variance of 3 are then given by
EB, - -87
E.S(()J)) -8.-
(5)
dj
Var 3, Var S
(6)
See Section B.3 for derivation of the results above well as expressions for ESx(x) and Var Sx
These results hold in a general case. From now however, we will narrow our scope and assume that the entries of e are identically, independently, and normally distributed, in which case both the bias and variance of Bj have analytical expressions (Section B.3.1) and
JJy Normal (H, #] =
So far, we have assumed nothing about the features themselves, apart from being orthogonal to each other. The main focus of our paper, however, is binary features, which we will now turn to.
3.1 Binary Features
When Ty E (0, 1} for all i, we define 2, to be a binary feature, and the class balance of this feature as ones. It would make no difference the majority of our results if we were swap the ones and zeros as long an intercept is included, and "class balance" is then equivalent to the proportion of either. But in the case of interactions (Section 3.3), the choice does in fact matter.
0 0= = E Bay: the proportion
If feature j is binary then VU = (g) 03) (the uncorrected sample variance for binary feature), which in Equation (4) yields
m/y 3jn(g - -3))
x.
[n(g - 93) Xa )2)
dj 8j
8j
and consequently
17 (7)
and of oz - 97)
Mj 8j
We obtain bias and variance of the estimator with respect to q, by inserting zJy and d, into Equations (5) and (6).
The presence of the factor Ø q3 in Mjj 0j, and dj indicates link between class balance and the elastic net estimator and, moreover, that this relationship is mediated by the scaling factor sy. To achieve some initial intuition for this relationship, consider the noiseless case (a= = 0) in which we have
2 n(gs -971
B, (G) v) 83 (āJ X2)
SA
n(gs +3)
8j
A2
This expression shows that class balance (q,) directly affects the estimator through the factor 9j - of (the variance of the binary feature). Starting with the lasso (A2 0), observe that the soft-thresholdin part of the

---

Published in Transactions on Machine Learning Research (11/2025)
estimator (numerator) diminishes for values of qj close to 0 or unless use the scaling factor 8, =9 93. in which case Equation (7) simplifies to
SA
Sx, (33m)
93 03
n(:;; 9,)
(gj gf)
(1, (9--9;) + X12
which is independent of qj. For other choices of 8j, the soft- nofttthresholdin part of the estimator will be affected by class balance and also depend on the size of X1, with larger values of Y leading to larger effect of class balance.
Turning to the ridge case (Λ1 = 0), the soft-thresholding part simplifies to the identity function and the shrinkage instead comes from the presence of X2 in the denominator, which will scale the estimator towards as 9j approaches 0 1. Contrary to the lasso case, when then instead need to take sj (gz 0,) 12, in which case the expression becomes
[8] [8j[(g (in(z=-9))
SA1
(9, )
8jn n + A2"
nlg; -
(gj - 12)
+ X12
(o) 99))
which is again independent of gj.
Observe, however, that for the elastic net (A1 > 0, A2 > 0), there exists no Sy that can make the estimator independent of 4j- In other words, there is no type of normalization, at least under our parameterization that is able to mitigate the class balance bias in this case. In Section 3.4, however, we will show how tackle this issue for the clastic net by scaling the penalty weights. But for now continue to study the case of normalization.
Based on the reasoning above, we will consider the scaling paramcterization 8j = =g; --3) 82 0, which includes the cases that we are primarily interested in, namely å = 0 (no scaling, as in min max and max abs normalization),
1/2 (standard- deviiatio scaling), and 8 = (variance scaling). The last of these, variance scaling, is fact equivalent to scaling with the mean-centered 4nnnrm in this particular case of binary features. In Section B.4, we expand Equation (7) under this parameterization to clarify how the choice of ò affects claas-balance bias in the lasso and ridge cases and why it is impossible (under our parameterization) to remove this bias in the case of the elastic net.
Another consequence of Equation (7), which holds also in the noisy situation, is that normalization affects the estimator even when the binary feature is balanced (gj = 1/2). 5 = 0, for instance, scales 33 in the input to SA by n(g; of) 30=m// = n/4. 8 = 1, in contrast, imposes no such scaling in the Hass-balancee case. And for d = 1/2, the scaling factor is n/2. Generalizing this, we see that to achieve equivalent scaling in the Alass-balanced case for all types of normalization, under our parameterization, we would need to use 8, = 4 - o3);. But this only resolves the issue for the lasso. To achieve a similar effect for ridge regression, we would need another (but similar) modification. When all features are binary, we can just scale A and λ to account for this effect," which is equivalent to modifying 83- But when consider mixes of binary and normal features in Section 3.2, we need to exert extra care.
We now proceed to consider how class balance affects the bias, variance, and selection probability of the elastic net estimator under the presence of noise. A consequence of our assumption of a normal error distribution and consequent normal distribution of +Jy y is that the probability of selection in the clastic not problem is given by
(g3n(e 07)1/2 A1 02)6-112
Pr (B, +0)=
Jevn
1B83(o //2 - Xı -2)
+
(8)
oevn
We this strategy in all of the following examples.

---

Published in Transactions on Machine Learning Research (11/2025)
where ① is the cumulative distribution function of the standard normal distribution. Letting 8, -K X1 and 7j [4j Xı, we can express this probability asymptotically 4j 1 as
if
2
lim Pr(3, #0) % -1
2
A1 JLVR
if 0 21
(9)
if Ó 2"
In Figure 3, we plot this probability for various settings of 5 for single feature. Our intuition from the noiseless case holds: suitable choices of å can mitigate the influence of class imbalance selection probability. The lower the value of 8, the larger the effect of class imbalance becomes. Note that the probability of selection initially decreases also in the case when 2 1. This is a consequence of increased variance of // y due to the scaling factor that inflates the noise term. But as qj approaches 1, the probability eventually rises towards for E 1.5). The reason for this is that this rise in variance eventually quells the noft-thresholding effect altogether. Note, also, that the selection probability unaffected by λ2-
0e 0.25
Ge = 0.5
Ge = 1.0
Ge = 2.0
1.00
Pr (3 4
0.75
0.0
0.25
0.50
0.5
0.25
1.0
1.5
0.00
0.50
0.65
0.80 0.950.50 0.65 0.80 0.950.50
0.65 0.80 0.950.50 0.65 0.80 0.95
Figure 3: Probability of selection in the lasso given measurement noise Øs, regularization level Å and class
balance 9. The scaling factor set to 8, = (q q²) 6 2 0. The dotted line represents the asymptotic limit for 1/2 from Equation (9).
Now we turn to the impact of class imbalance on bias and variance of the elastic net estimator. We begin, Theorem 3.1, by considering the expected value of the elastic net estimator in the limit as qj + 1T Theorem 3.1. If æy is binary feature with class balance qj E (0, 1), A1 J0, 20), d2 [0, oo), de > 0, and 8, (gg 22 then
if0 S 1 if 6 = 2'
2
lim E,8
2n. n+
+
A1 vevm
1
9-1
Bj
if 8 1
Theorem 3.1 shows that bias of the elastic net estimator approaches -33 Qj - 1 when 0 < 8 < 1/2. When 8 = 1/2 (standardization), the estimate approaches a constant that depends on regularization strength, noise level, and the true strength of the coefficient. For 8 >> 1/2, the estimate is asymptotically unbiased as a by-product of variance dominating in the limit as - 1-1, which suggests that variance-scaling (41-norm normalization) could be problematic a scenario with much noise and highly imbalanced features.
In Theorem 3.2, we continue by studying the variance in the limit as qi + which shows that the variance of the clastic net estimator tends to so in the limit unless the scaling parameter 8, 1/2.
Theorem 3.2. Assume the conditions of Theorem 3.1 hold, ercept that X 0. Then
0 {o
if <
lim Var By
4 +1-
1822 z
Note that Theorem 3.2 applics only the case when \ 1. In Theorem B.2 (Section B.5), we state the corresponding result for ridge regression.
8

---

bias, but does so at the cost of Published in Transactions on Machine Learning Research (11/2025)
Taken together, Theorems B.2, 3.1 and 3.2, indicate that the choice of scaling parameter constitutes ilas-varianes trade-off with respect to d: increasing 8 reduces increased variance.
In Figure 4, we now visualize bias, variance, and mean-squared error for ranges class balance and various noise-level settings for a lasso problem. The figure demonstrates the results suggest and indicates that the optimal choice of ò is related to the noise level in the data. Since this level is typically unknown and can only be reliably estimated in the low-dimensional setting, it suggests there might be value in selecting 8 through syper-parameter optimization." In Figure 14 (Section B.7) we show results for ridge regression well. As expected, it is then 5 = 1/2 that leads to unbiased estimates.
trade-off that our asymptotic
C = 0.25
Gg = 0.5
0g = 1.0
6. = 2.0
MSE
2
0
0.0
-0.5
0.0
Bias
0.25
1.0
1.5
0.4
Variance
0.3
0.2
0.1
0.0
0.50
0.75
1.00
0.50
0.75
1.00
0.50
0.75
1.00
0.50
0.75
1.00
q
Q
q
Figure 4: Bias, variance, and mean-squared error for one-dimensional lasso problem, parameterized by noise level (oe), class balance (g), and scaling (8). Dotted lines represent asymptotic bias of the lasso estimator in the case when 8 = 1/2.
So far, we have only considered a single binary feature, but in Section F. present results power and false discovery rates for problems with multiple features. In the next section we will also step beyond the all-binary context and consider mixes of binary and continuous features.
3.2 Mixed Data
A fundamental problem with mixes of binary and continuous features is deciding how to put these features the same scale in order to regularize each type of feature fairly. In principle, we need to match one-unit change in the binary feature with some amount of change in the normal feature. This problem has previously been tackled, albeit from different angle, by Gelman (2008), who argued that the common default choice of presenting standardized regression coefficients unduly emphasizes coefficients from continuous features.
To setup this situation formally, we will say that the effects of a binary feature Φ1 and a normal feature 22 are comparable if 3 Ro where & > 0 represents the number of standard deviations of the normal feature consider to be comparable to one unit the binary feature. As example, assume 2. Then, if æ2 is sampled from Normal (1y, o² (1/2)2), the effects of æ and 22 comparable if B 20 83
5n Section E.4 demonstrate the utility doing so.
9

---

Published in Transactions on Machine Learning Research (11/2025)
The definition above refers to 8 but for our regularized estimates we need و = Kor to hold. If we assume that we are in noiseless situation (o: = 0), are standardizing the normal feature, and that, without loss of generality, Ij = O, then we need the following equality to hold:
Sx (a 51 (o क1 + x2)
RO SA (z)
SX
K KSx,
81 ===Ko KO
31
(10)
82 (+ + 2z)
n(g-9") X2
11 + A2
$1
$7
For the lasso (A2 = 0) and ridge regression (A1 0), we see that the equation holds for 81 =(qq q²) and 81 - (a- -a--g))// respectively. In other words, we achieve comparability in the lasso by scaling each binary feature with its variance times K. And for ridge regression, we can achieve comparability by scaling with standard deviation, irrespective of k. For any other choices of $1, equality holds only at a fixed level of class balance. Let this level be qu- Then, to achieve equality for X2 = O, we need $1 = k((u - - (g - g²)². Similarly, for Å = we need 81 = (go - 10)3-26( (q- - g²)0. In the sequel, we will assume that qo = 1/2, to have effects be equivalent for the class-balanced case.
Note that this means that the choice of normalization has an implicit effect on the relative penalization of binary and normal features even in the class- balanced case (q1 1/2). If we for instance use 8 0 and fit the lasso, then Equation (10) for a binary feature with Q1 = 1/2 becomes 4 (n,97/4 x Sx, (n.8//)) /k), which implies x- = 4. In other words, the choice of normalization equips our model with a belief about how binary and normal features should be penalized relative to one another.
For the rest of this paper, we will use K = and say that the effects are comparable the effect of a flip the binary feature equals the effect of two-standard deviation change in the normal feature. We motivate this by an argument by Gelman (2008), but want to stress that the choice of к should, if possible, be based contextual knowledge of the data and that our results depend only superficially on this particular setting.
3.3 Interactions
The clastic net can be extended to include interactions. There is previous literature on this topic (Bien et al., 2013; Lim & Hastie, 2015; Zemlianskaia et al., 2022), but it has not considered the possible influence of normalization. Here, we will consider simple pairwise interactions with no restriction on the presence of main effects. For our analysis, we let 1 and 22 be two features of the data and Ø3 their interaction, so that B3 represents the interaction effect.
We consider two cases in which we assume that the features are orthogonal and that æ1 is binary with class balance qı- In the first case, we let 22 be normal with mean μr and variance o², and in the second case æ2 be binary with class balance 42- To construct the interaction feature, we center the main features and then multiply clement-wise. The elements of the interaction feature are then given by F3,; = (21, - (12 - T2).
If 22 is normal and both features are centered before computing the interaction term, the variance becomes -²)), which suggests using $3 = alg--(99)) -2 along the lines of our previous reasoning. And if w2 is binary, instead, then similar reasoning suggests using $3 ((g) (g2 gz))². In Section 4.1.4, we study the effects of these choices in simulated experiments.
3.4 The Weighted Elastic Net
We have so far shown that certain choices of normalization can mitigate the class-balance bias imposed by the lasso and ridge regularization. But we have also demonstrated (Section 3.1) that there is no (simple) choice of scaling that can achieve the same effect for the elastic net. Equation (7), however, suggests natural alternative to normalization, which is to use the weighted elastic net, in which we minimize
1 2 y - 80 XB1 + A
A
Uy
33
2
j=1
6Se Section B.6 for motivation for why we center the features before computing the interaction.
10

---

Published in Transactions on Machine Learning Research (11/2025)
with u and being p-length vectors of positive scaling factors. This is equivalent to the standard clastic net for normalized feature matrix when U 8j and U/ = 28, which can be seen by substituting 938; = B3 in Equation (2) and solving for B. Note that we do not need to rescale the coefficients from this problem as we would for the standard elastic net on normalized data.
This allows to control class-balance bias by setting our weights according to uj v = (gj y and counteract it, at least in the noiseless case, with w= = which, we want to emphasize, is not possible using the standard elastic net. For the lasso and ridge regression, however, this setting of w = 1 is equivalent to using 5 = 11 and = 1/2, respectively, in the standard clastic net with normalized data. Results analogons to those in Section 3.1 can be attained with a few small modifications for the weighted elastic net case. Starting with selection probability, we can set sj = 1 and replace x with Aun, = \1 of) in Equation (8), which shows that w and ò have interchangeable effects for selection probability.
As far expected value and variance of the weighted elastic net estimator is concerned, the same expressions apply directly in the case of the weighted elastic net given 8j = 1 for all j and replacing λ as in the previous paragraph and X with \2(gj- dz) On the other hand, the asymptotic results differ slightly as we now show.
Theorem 3.3. Let xj be binary feature with class balance qi E (0, 1) and take X > A2 O, and og > O. For the weighted clastic net with weights új = 0j (gz oz g and w 2 0, it holds that
if w 1,
fosw< 80 ifW2
lim E 4 -1
9"
1w=1,,
lim 4-
Var B3 =
8 1f
This result for expected value is similar to the one for the unweighted but normalized clastic net The only difference arises in the case when w = 1, in which case the limit is unaffected by a in the case of the weighted clastic net. For variance, the result mimics the result for the clastic net with normalization. The results for bias, variance, and mean-squared error for the weighted elastic net are similar to those in Figure and are plotted in Figure 12 (Section B.7).
This wraps up our thcoretical contributions in our paper. In the coming section, we will turn to empirical experiments and demonstrate that our theoretical results both hold in practice and, at least empirically, extend beyond our current assumptions.
4 Experiments
In the following sections we present the results of our experiments. For all simulated data we generate our response vector according to y X3 + with e ~ Normal(O, binary Benasi-lernoull) and quasi-normal features. To generate binary vectors, we sample [nqy]] indexes uniformly at random without replacement from [n] and set the corresponding elements to one and the remaining ones to zero. To generate quasi-normal features, we generate a linear sequence w with n values from 10-2 1 to 1 10-1, set Zig- + (wi(,, and then shuffle the elements of æ; uniformly at random.
We consider two types of features:
We use coordinate solver from Lasso.jl (Kornblith, 2024) to optimize our models, which we have based on the algorithm outlined by Friedman et al. (2010). All experiments were coded using the Julia programming language (Bezanson et al., 2017) and the code is available at https:// /stps://gitta //githbb........../ thtb....co wtw.tt.o.ooooooooooooooooooooooo simulated experiments were run for at least 100 iterations and, unless stated otherwise, are presented as mcans + one standard deviation (using bars or ribbons).
All
4.1 Normalization in Lasso and Ridge Regression
In this section we consider fitting the lasso and ridge regression normalized datasets. To normalize the data, standardize all quasi-normal features. For binary features, center by mean and scale by 83 (gz o3)
11

---

Published in Transactions on Machine Learning Research (11/2025)
4.1.1 Variability and Bias in Estimates
In our first experiment, we consider litting the lasso to simulated dataset with n = 500 observations and = 1000 features. The first 20 features correspond to signals, with 3 = 1, and otherwise we set 83 to 0. Furthermore, we sct the class balance of the first 20 features so that increases goometrically from 0.5 to 0.99. For all other features coefficients using the lasso, setting À 20.v In addition, introduce correlation between the features by copying the first [pn/2] values from the first feature to cach of the following features.
pick qy uniformly at random in [0.5, 0.99). We estimate the regression
log 22 with de set achieve signal-to-noise ratio (SNR) of 2.
The results (Figure 5) show that class balance has considerable effect, particularly the case of no scaling (0 = 0), which corroborates our results in Scction 3.1. At 4 0.99, for instance, the estimate (920) is consistently zero when O. For larger values of d, we see that class imbalance leads to increased estimation variance accordance with our theory for the orthogonal case. The effect of correlation appears to have no effect on this balsnca bias.
0.0
= 0.5
p 0.9
2.0
1.5
0.0
0.5
0.0
0.5
2.0
1.5
0.5
0.5
0.0
0.5
2.0
1.5
1.0
0.5
0.0
15
20
25
10
15
25
30
20
25
30
Feature Index
Figure 5: Regression coefficients for lasso problem with binary data with 500 and p 1000. For j E (1. set $3 = 1 and let qj increase geometrically from 0.5 to 0.99. For the remaining features, we pick uniformly at random from [0.5, 0.99] and set 33 = O. We show only the first 30 coofficients.
4.1.2 Predictive Performance
In this section we examine the influence of normalization on predictive performance for three different datascts: rhee2006 (Rhee et. al., 2006), tunite2001 (Chen al., 2004), and triazines (Hirst et al., 1994; King et al., 1995).7 We present the results for lasso and ridge regression in Figure 6, which shows contour plots of the validation set error in terms of normalized mean-squared error (NMSE). We see that optimal setting of 5 differs between the different datasets: for eunite2001, both the lasso and ridge are quite insensitive to the type of normalization, and low error attainable for the full range of 6. For rhee2006, this holds for the lasso too, but not ridge regression, where a value in approxinately [0, 3] is optimal. Finally, for triazines, the problem quite sensitive to the choice of 8 well as the type of model used.
Also Section F.2, where show how the support size of the lasso solutions vary with 8 and and Section E.3, where we complement these results with experiments on simulated data under various class balances and signal-to-noise rat,os, again showing that normalization has strong impact on predictive performance.
7 See Section G for details about these datasets.
12

---

Published in Transactions on Machine Learning Research (11/2025)
rhee2006
eunite2001
triazines
0.8
1.8
0.5
Ridge
0.6
0.6
NMSE
0.4
0.4
1.4
0.3
0.2
0.2
1.2
1.0
1.8
0.8
0.8
Lasso
0.6
1.6
NMSE
0.6
0.4
0.4
1.2
0.2
0.2
0.2
1.0
10"
10"
10
A/ Amax
Figure 6: Contour plots of normalized validation set mean-squared error (NMSE) for ô and À in lasso and ridge regression on three real data sets: rhee2006, eumite2001, and triazines. The dotted path shows the smallest NMSE as function of À and the circles mark combinations with the lowest error.
4.1.3 Mixed Data
In Section 3.2 we discussed the issue of normalizing mixed data. Here, we examine this issue empirically. We construct quasi-normal feature with mean zero and standard deviation 1/2 and binary feature with varying class balance 9j. We set the signal-to-noise ratio to 0.5 and use 1000. These features constructed that their effects are comparable under the notion of comparability that we introduced in Section 3.2 using k = 2. In order to preserve the comparability for the baseline case when we have perfect class balance, we scale by 8j = 2 x (1/4) (q; o Finally, we set to Amax/2 2 and 2Amax for lasso and ridge regression respectively.
The results (Figure 7) reflect our theoretical results from Section In the case of the lasso, we need 5 = 1 (variance scaling) to avoid the effect of class imbalance, whereas for ridge we instead need 5 = 1/2 (standardization. As our theory suggests, this extra scaling mitigates this class-bel balance bias at the cost of added variance.
4.1.4 Interactions
Next, we study the effects normalization and class balance on interactions in the lasso. Our example consists of two-feature problem with an added interaction term given by Fh = 241 The first feature is binary with class balance q and the second quasi-normal with standard deviation 0.5. We use n 1000, A1 n/4, and normalize the binary feature by mean-centering and scaling by s(q consider two different strategies for choosing 83: in the first strategy, which call Strategy standardize the resulting interaction feature. This is a common strategy used, for instance in Bien et al. (2013); Lim & Hastie (2015). In the second strategy, Strategy 2, we center with mean and scale with 8182 (the product of the scales of the binary and normal features).
k(g- - g²), using & 2. We
simply
The results (Figure 8) show that only Strategy 2 estimates the effect of the interaction correctly. Strategy 1, meanwhile, only selects the correct model if the class balance of the binary feature close to 1/2 and in general shrinks the coefficient too much.
13

---

Published in Transactions on Machine Learning Research (11/2025)
6 0.0
0 0.5
8 1.0
1.00
Ridge
0.75
0.50
0.00
Bernouli(g)
Normal(0, 0.5)
1.00
0.75
Laaoo
0.50
0.25
0.00
0.5
0.7
0.9
0.5
0.7
0.9
0.5
0.7
0.9
q
Figure 7: Lasso and ridge estimates for two-dimensional problem where one feature binary feature with class balance qj (Bernoulligg)) and the other is quasi-normal with standard deviation 1/2, (N.rmal(0,055).
3
1
1
T
8=
1P 1]
3
1
8 [0 1|T
8 ===00 10
1.00
Strategy
Beruoullig))
31
0.50
Normal(0, 0.5)
Interaction
0.25
0.00
1.00
Strategy
0.75
3j
0.25
0.00
0.2 0.5
0.8
0.2 0.5
0.8
0.2
0.5
0.8
0.2
0.5
0.8
0.2 0.5
0.8
9
Figure 8: Lasso estimates for problem with binary feature, quasi-normal feature, and interaction feature. We have varied the true signal 3 and use two different normalization strategies for the interaction, where Stratogy represents standardization and Strategy 2 is mean-centering together with scaling by $182
14

---

Published in Transactions on Machine Learning Research (11/2025)
4.2 The Weighted Elastic Net
The weighted elastic net can be used as an alternative to normalization to correct for class balance bias when À > and X2 > 0. To simplify the presentation, we parameterize the elastic net as λ = ad and A = (1 - a, so that a controls the balance between the ridge and lasso. We conduct an experiment with the same setup as Section 4.1.3, but here we use the weighted clastic net instead with a = 0.5. We use 1000 and vary w, using the weights Uj 0j = (gj - o7)00 as we suggested in Section 3.4. Our results (Figure 9) show that = 1 leads to seemingly unbiased estimates.
0.0
0.5
1.0
1.00
(o 0.255)
0.75
0.50
0.25
0.00
1.00
2 (a 0.5)
0.75
Bermoulligg)
0.50
Normal 0.5)
0.25
0.00
1.00
0.75)
0.75
0.50
0.25
0.00
0.5
0.7
0.9
0.5
0.7
0.9
0.5
0.7
0.9
Figure 9: Weighted elastic net estimates for = 0.5 for a problem with a binary feature with class balance g (Bernoull(g)) and quasi-normal with standard deviation 1/2 (Normal(O, 0.5)). indicates the scaling of the penalty weights.
4.3 Additional Results
In Section we present extended results of our experiments as well as several additional experiments. For instance, we also consider an experiment on the Boston housing dataset in Section E.5, where we try estimate the effect of normalization on dichotomized versions the features and find that ō leads feature ranks that best correspond to the linear regression solution.
5 Practical Recommendations
The results presented in this paper directly concern practitioners working with regularized regression on data with binary features. In general, we want to stress that the choice of normalization affects not only the computational aspects of fitting the model, but also the model itself. The bias introduced by ridge, lasso, and clastic net regression depends on the normalization scheme used, and does so in a penalty-specific manner, which means there exists no one-size- mie-siz-fits-all solution. In the following sections, we provide some concrete guidelines for practitioners on how to normalize.
15

---

Published in Transactions on Machine Learning Research (11/2025)
5.1 Feature Selection
We have shown that, lo avoid bias against class-imbalanced binary features, need to scale them with their standard deviations for ridge regression and with their variances for the lasso. Therefore, if the goal is feature selection and the cost of missing a relevant feature is high, then scaling with variance may be preferable. Scaling with variance, however, will increase the variability of the estimates, which means that the choice of normalization is a bias variance trade-off. Naturally, this also means that the size of the dataset matters: with more data, users may be able to afford the increased variance that comes with scaling with variance.
5.2 Prediction
If the goal prodiction, then scaling with standard deviation or even not scaling at all will typically prove the better choice. If the practitioner can afford the computational cost, we also recommend treating normalization hyper-paramettr optimization problem, which will allow users look for normalization scheme that is optimal for their specific dataset and prediction task. The user can, for instance, cross-validate over few common normalization schemes or alternatively use the parameterization through that we have proposed in Section 3.1.
5.3 The Elastic Net
For ridge and lasso regression, the class imbalance can be controlled through the choice of normalization This is not, however, the case for the elastic net. In this case, we therefore recommeed users scale the penalty weights instead of the features themselves, as described Section 3.4. We believe this is a more intuitive approach, given that the elastic net mix of two different norms, which interact with the features' scales separate ways.
5.4
Mixed Data
For mixed data, the choice of normalization is tricky since implicitly affects the relative penalization of binary and normal features. Practitioners need decide how they want to treat binary features relative normal feat. We discuss this topie in Section 3.2 and introduce the parameter that allows practitioners to control this effect through the normalization strategy. While we provide concrete recommendations for setting this parameter, we believe that awareness of this implicit relationship is important.
5.5 Interactions
Interactions introduce another layer of complexity because normalization will affect the correlation between the original features and their interaction terms. We recommend always centering the features before creating the interaction terms, since the means of the features otherwise affect the estimates. Second, we suggest a new type of normalization for interaction features, which is to scale them based on the product of the scales of the normalized original features. We have shown that this can mitigate the flass-balance bias effect.
6 Discussion
This is the first paper to study the effects of normalization in lasso, ridge, and elastic net regression with binary data. We have discovered that the class balance (proportion of ones) of these binary features has a pronounced effect on both lasso and ridge estimates and that this effect depends on the lype of normalization used. For the lasso, for instance, features with large class imbalances stand little chance of being selected if the binary features are standardized or, to an greater extent, if they are not scaled at all even if their relationships with the response are strong. Not scaling binary features is a common approach in practice, for instance recommended in the scikit-learn documentation ion when the data is sparse (scikit-learn developers, 2025). As is standardization, which the default in many software packages for the lasso, such as glmnet (Fricdman al., 2010), as well as the practice taken by countless applications in rescarch. All
8rr equivalently, using max-abs max normalization.
16

---

Published in Transactions on Machine Learning Research (11/2025)
in all, this means that the class-balance bias effect is likely pervasive in practice and risks having already influenced the conclusions drawn in analyses of many datasets.
The driver of this bias is the relationship between the variance of the feature and type of normalization. This works as expected for normally distributed features. But for binary features it means that a one-unit change is treated differently depending on the corresponding feature's class balance, which we believe may surprise some. We have, however, shown that scaling binary features with standard deviation in the case of ridge regression and variance in the case of the lasso mitigates this effect, but that doing so comes at the price of increased variance. This effectively means that the choice of normalization constitutes a baasinariance trade-off.
We have also studied the case of mixed data: designs that include both binary and normally distributed features (Section 3.2). In this setting, our first finding is that there an implicit relationship between the choice of normalization and the manner in which regularization affects binary vis-à-vis normally distributed features, even when the binary feature is perfectly balanced. The choice of max abs normalization, for instance, leads to a specific weighting of the effects of binary features relative to those of normal features.
For intcractions between binary and normal features (Section 4.1.4), our conclusion is that the interaction feature- feature-contrary to what recent literature on interactions in the lasso recommends-should be scaled with the product of the standard deviation of the normal feature and variance of the binary feature to avoid this effect of class imbalance. We have not seen this recommendation in the literature before, but it is a natural extension of our other results.
We note that our theoretical results are limited by a few assumptions: 1) fixed feature matrix X, 2) normal and independent errors, and 3) orthogonal features. The first and second of these assumptions are standard in the literature. The third assumption on orthogonality, however, is strong and rarely satisfied in practice. Yet, as we show in Section A and our experiments, the assumption does not in fact appear to be restrictive for our results, which, at least empirically, hold under much more general settings. We have also focused on the case of binary and continuous features here, but are convinced that categorical features are also of interest and might raise additional challenges with respect to normalization. Finally, most of our results are restricted to least-squares loss, but since all generalized lincar models (GLMs) are paramcterized by the linear predictor, which we have shown to depend directly class balance, we believe that our results are also relevant for other loss functions. Our initial results in Section E.4 seem to support this claim, but we defer further investigation of this to future work.
Regularized regression models are widely used in practice, and are staples of popular machine learning and statistical software packages such as glmnet (Friedman et al., 2010), scikit-learn (Pedregosa et al., 2011), mlpack (Curtin et al., 2023), skglm (Bertrand et al., 2022), LIBLINEAR (Fan al., 2008), and MATLAB (The MathWorks Inc., 2022). Our results suggest that the choice of normalization is important aspect of using these models that, in spite of the popularity of these methods, has so far been overlooked. We hope that our results will motivate rescarchers and practitioners to consider the choice of normalization more carcfully in the future.
Acknowledgments
We are grateful to Mathurin Massias and Malgorzata Bogdan for valuable feedback on carlier drafts of this manuscript.
References
Barry Becker and Ronny Kohavi. Adult, 1996. URL httt::://dii.or/11..2220
Quentin Bertrand, Quentin Klopfenstein, Perre-Antoing Bannier, Gauthier Gidel, and Mathurin Massias, Beyond L1: Faster and better sparse models with skglm. In Advances in Neural Information Processing Systems 85, volume 35, pp. 88950-38965, New Orleans, USA, 2022. Curran Associates, Inc. ISBN 978-1-7138--108-8.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B. Shah. Julia: A fresh approach to numerical computing. SIAM Review, 5:65) )::5)):55.. 98, February 2017. ISSN 0036-1445. doi: 10. 10.1137/141000677
Jacob Bien, Jonathan Taylor, and Robert Tibshirani. A lasso for hierarchical interactions. The Armals of Statistics, 41(3):1111-111 June 2013. ISSN 0090-5364, 2168-8966. doi: 10. 10.1214/13 A0-1009
Bo-Juen Chen, Ming-Was Wei Chang, and Chih-Jen lin Load forecasting using support vector machines: A study on EUNITE competition 2001. IEEE Transactions on Power Systems, 19(4):1821 1830, November 2004.
ISSN 1558-0679. doi: S0.09//TWSS.20000
Ryan R. Curtin, Marcus Edel, Omar Shrit, Shubham Agrawal, Suryoday Basak, James J. Balamuta, Ryan Birmingham, Kartik Dutt, Dirk Eddelbuettel, Rishabh Garg, Shikhar Jaiswal, Aakash Kaushik, Sangyeon Kim, Anjishnu Mukherjee, Nanubala Gnana Sai, Nippun Sharma, Yashwant Singh Parihar, Roshan Swain, and Conrad Sanderson. Mlpack 4: A fast, header-only C++ machine learning library. Journal of Open Source Software, 8582),5026, 2023. doi: 10.21105/joss.000222
David L. Donoho and Iain M. Johnstone. Ideal spatial adaptation by wavelet shrinkage. Biometrika, 81(3): 425-455, August 1994. ISSN 0006-3444. doi: 10.2307/2337118.
Rong-Ea En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang- Rui Wang, and Chil-Jen Lin. LIBLINEAR: library for large linear classification. Journal of Machine Learning Research, 9(61):1871 1874, August 2008, ISSN 1533-7928.
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1):1-22 22, Jamuary 2010. doi: 10.18637/jss.v033..
Andrew Gelman. Scaling regression inputs by dividing by two standard deviations. Stotistics in Medicine, 27 (15):2865 2873, July 2008. ISSN 02776715, 10970258. doi: 10.1002/s0 111002/im..307.
T. R. Golub, D. K. Slonim, P. Tamayo, C. Huard, M. Gaasenbeek, J. P. Mesirov, H. Coller, M. L. Loh, J. R. Downing, M. A. Caligiuri, C. D. Bloomfield, and E. S. Lander. Molccular classification of cancer: Class discovery and class prediction by gene expression monitoring. Science, 286(5439):531 537, October 1999. ISSN 0036-8075. doi: 2.11126/scienne.. 286.5439.531
David Harrison and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. Journal of Environmental Economics and Management, 5(1):81- 102, March 1978. ISSN 0095-0696. doi: 10.1016/0095,
0696(78)90006-2.
Trevor Hastic, Robert Tibshirani, and Ryan Tibshirani. Best subset, forward stepwise lasso? Analysis and recommendations based on extensive comparisons. Statistical Science, 35(4):579 592, November 2020. ISSN 0883-4237. doi: 10.1214/19-STS733
R. J. Henery and C. C. Taylor. StatLog: An evaluation of machine learning and statistical algorithms. In Yadolah Dodge and Joe Whittaker (eds.), Computational Statistics, pp. 157-162, Heidelberg, 1992. Physica- Verlag HD. ISBN 978-3-662-268111 doi: 10.1077/978-3-622228811
J. D. Hirst, R. D. King, and M. Sternberg. Quantitative structure-activity relationships by neural networks and inductive logie programming. II. The inhibition of dihydrofolate reductase by triazines. Journal of Computer-Aide Aided Molecular Design, 8(4):421-45) -422, August 1994. ISSN 0920-654X. doi: 10.1007/BF001253776
Ross D. King, Jonathan D. Hirst, and Michael J. Sternberg. Comparison of artificial intelligence methods for modeling pharmaceutical QSARs. Applied Artificial Intelligence, 9(2):213 233, March 1995. ISSN 0883-9514. doi: 00.1000008839515550
Simon Kornblith. Lasso.jl, March 2024.
Michael Lim and Trevor Hastic. Learning interactions via hicrarchical group-lasso regularization. Journal of Computational and Graphical Statistics, 24(3):627-65) September 2015. ISSN 1061-8600. doi: 10.1080/
18

---

Published in Transactions on Machine Learning Research (11/2025)
Haikady N. Nagaraja and Herbert A. David. Order Statistics. Wiley Series in Probability and Statisties. John Wiley & Sons Inc, Hoboken, N.J, 3 edition, July 2003. ISBN 978-0-171-38926-2.
Fabian Pedregosa, Gaäl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, ((885):2825- 28300, 2011. ISSN 1533-7928.
John C. Platt. Fast training of support vector machines using sequential minimal optimization. In Bernhard Schölkopf, Christopher J. C. Burges, and Alexander J. Smola (eds.), Advonces in Kernel Methods: Support Vector Learning, pp. 185 208. MIT Press, Boston, MA, USA, edition, January 1998. ISBN 978-0-262- 1130.003.0016.
28319-9. doi:
Ross Quinlan, Australian credit approval, 1987.
Soo-Yon Rhee, Jonathan Taylor, Gauhar Wadhera, Asa Ben-Hur, Douglas L. Brutlag, and Robert W. Shafer. Genotypic predictors of human immunodeficieney virus type drug resistance. Proceedings of the National Academy of Sciences, 103( (03(4)):1335) 17755-135-77,, November 2006. doi:
Fadil Santosa and William W. Symes. Linear inversion of band-limited reflection seismograms. SIAM Journal on Scientific and Stalistical Computing, 7(4):1307-1334) October 1986. ISSN 0196-5204. doi:
10.1137/0907087.
scikit-learn developers. 6.3.1.2. Scaling sparse data. http:://athhlaeemmssal////hhdeet May 2025.
The MathWorks Inc. MATLAB. The MathWorks Inc., 2022.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B, 58(1):2667265 2888 1996. ISSN 0035-9246. doi: 00.111/22577611111996..b8
Natalia Zemlianskaia, W. James Gauderman, and Juan Pablo Lewinger. A scalable hierarchical lasso for gene-environment interactions. Journal of computational and graphical statistics joint publication of American Statistical Association, Institute of Mathematical Statistics, Interface Foundation of North America, 14): 2022. ISSN 1061-8600. doi: 10.1050/1061660 0000066000022223333
Hui Zou. The adaptive lasso and its oracle properties. Journal of the American Statistical Association, 101 (476):1418-144) 1429,, December 2006. ISSN 0162-1459. doi: 10.1198/0166 001624450600000000
Hui Zou and Trevor Hastie. Regularization and variable selection via the Elastic Net. Journal of the Royal Statistical Society, Series B (Statistical Methodology), 67(2))301 320, 2005. ISSN 1369-7412.
A Orthogonality Assumption
A key assumption in our theoretical results is that the features are orthogonal to one another. In general, this is a strong assumption that is rarely satisfied in practice. In the high-dimensional setting with binary data, it is even impossible to achieve. In spite this, it turns out that the assumption holds little bearing on our results.
The primary reason for this is well-known behavior of regularized estimators when features are correlated. attain lower
Since information about the effect is shared between the correlated features, the objective value by favoring one of the features over the others. The effect is particularly strong in the lasso, which is known to select only one of the correlated features and ignore the others given a sufficiently large correlation and penalty strength.
A second reason for why the assumption not as restrictive as may seem that the correlation between two features, at least one being binary, tends to zero as class balance increases towards onc. We first demonstrate this empirically in the following experiment. We consider a setting with two binary features: the first, 21
19

---

Published in Transactions on Machine Learning Research (11/2025)
with class balance Q1 0.5, and the second, æ2; with varying balance 92 [0.5, 0.9]. We set the true effect to 07 32 = 1 and the level of correlation, P to three different levels (0, 0.4, and 0.6). The noise level dz is set. to obtain a signal-to-noise ratio of 1. We generate n = 10 000 observations and fit the lasso with A == Amax/2.
The results are shown in Figure 10 where we see that the effect correlation has no impact on the shrinkage imposed from decreasing class balance of the second feature. The results fact suggest that the effect of Q2 is stronger when the features are correlated, which is due the nature of the lasso that previously discussed.
p= 0.0.0
p == 0.4
= 0.6
0.6
A
3
0.5
0.4
0.3
0.2
0.1
0.0
0.9
0.5
0.7
0.9
0.5
0.9
92
Figure 10: Estimates of the regression coefficients from the lasso, B3, for the two features in the experiment in Section A. The first. feature is binary with class balance Fı 0.5 and the second is binary with class balance 42 E [0.5, 0.9]. The features are correlated with correlation p. The plot shows means and 95% normal confidence intervals averaged over 100 iterations. Note that it's impossible to achieve high levels of correlation when the second feature is highly imbalanced, which is why the results for = 0.4 and 0.6 do not cover the whole range of 92 values.
In the following theorems and corollary, we provide further evidence of this behavior by showing that the correlation between a binary feature and a continuous or binary feature tends to zero as the class balance increases.
In conclusion, although the orthogonality assumption is strong in gemeral, unrealistic, and in many other areas of statisties make results hard to generalize to wider setting, it does not seem to have significant impact on results.
Theorem A.1 (Correlation for Dichotomized Normal Variables). Let X and Y be two standard normal random variables with correlation p. Define Z 1[Y a) where -0 (q) is the quantile at which Y is dichotomized. Then
Corr(X, p po(a)
q 9-1
V7(1- -
Proof. Let Z 1[Y > a) with (g). Then, using the law of total expectation, have
E(XZ) = E(E(X Y)) E(pr1[y al) p / yd()) dy pδ(a).
Since Var(X) and Var(Z) = q(1 q), it follows that
E(XZ)
po(a)
Corr(X, 7) Var(X) Var(Z)
Vg(1 -9) 9)
And since ((a) 0 exponentially fast q + it follows that Corr(X,2) 2) 0 q +
7
20

---

Published in Transactions on Machine Learning Research (11/2025)
Theorem A.2 (Correlation with Bernoulli Variable). Let X be continuous rondom variable and Y be Bernoulli random variable defined as:
l with probability p. with probability - p.
Y =
Let p1 E po = E 0) and xx = V (X) by:
Then the correlation between X and Y is given
Px,Y (14 mO)
vp(1 - p).
ox
Proof.
The correlation Px, is defined as:
Cov(X,Y)
PX,
Var(X))Var(Y)
We have Var(Y) p(1 -p)- Using the law of total covariance:
Cov(X,Y) = E(XY) E(X)E(Y)
PP1 (p/4 (1 p)uo)p
p(1 p)(1 -Ho).
Since Var(Y) = p(1 p) the result follows.
Corollary A.3 GGaussian-Bernoull Case). Suppose X and Z are jointly Gaussian random variables with:
x, N Corr(X, 2) p.
and define = 1[2 > a]. Then the correlation between X and Y is:
pΦ(a)
Px,Y (1(a)(1-(()))
where ((a)) and (a)) are the PDF and CDF of the standard normal distribution, respectively. Further for
=aa)), we have:
PX, +
9 1.
Proof. From the theorem above, we identify:
== P=Z a) = 1-1((a).
Since () is jointly normal, X z N(p2,1 -
-p²), 1-
we have:
P1 X Z > al pE|Z Z al = P ②(a)
(
Similarly,
((a)
No = E|x 25 pE
25 =
((a))
Thus:
6(a)
s(a)
pa(a)
140 = (T ((a)
{(a)
{aa +(a))
21

---

Published in Transactions on Machine Learning Research (11/2025)
Substituting these results into the theorem, we obtain:
pá(a)
(1 +()))
Px. Φ(a)(1 - +(a))
pQ((a
(a)(( ((a))
Finally, note that p(a) is bounded hence Px,Y + 0 as = ((a) +
One can also compute bounds for two correlated Bernoulli variables: Let X ~ Bernoullipp) and Bernoulligg) with 0 < p. Their correlation coefficient is given by
p P(X = 1,Y = P9
Vp(1 (11pp)(11-0 -4
where the joint probability P(X 1, 1) satisfics the Fréchct bounds:
max (0, p + q 1} $ P(X 1,Y 1) s min g).
Thus, the extreme values of are:
min min /} pq
Pmax(p.9)
vp(1 pp)((1 9)
and
Pmiu(p.q) maxx00 + 9 13 pa
vp(1 - 11lpp))(11 ②)
B Additional Theory
B.1
Maximum-Absolute and Min-Max Normalization for Normally Distributed Data
In Theorem B.1, we show that the scaling factor in the max abs method converges in distribution a Gumbel distribution.
Theorem B.I. Let X1, x Xn be a sample of normally distributed random variables, each with p and standard deviation Then
lim Pr max (EEm)
[X. Sx (xlx =(x),
where G the currulative distribulion function of a Gumbel distribution with parometers
1
bn = F7 1/n) and on (
where fy and Fy' are the probability distribution function and quantile function, respectively, of a folded normal distribution with mean μ and standard deviation a.
The gist Theorem B.1 is that the limiting distribution of max X,| has expected value b, + yan, where y is the Euler-Mascheroni constant, which shows that the scaling factor depends on the sample size. In Figure 11(a), observe empirically that the limiting distribution agrees well with the empirical distribution in expected value even for small values of n.
In Figure 11(b) we show the effect of increasing the number of observations, in a two-feature lasso model with max-abs normalization applied to both features. The coefficient corresponding to the Normally distributed feature shrinks as the number of obscrvation increascs. Since the expected value of the Gumbel distribution diverges with n, this means that there's always large enough n force the cocfficient in a lasso problem to zero with high probability.
22

---

Published in Transactions on Machine Learning Research (11/2025)
0.25
Bernoulli
45
Normal
max, (0
0.20
40
0.15
35
Empirical
0.10
Theoretical
30
250
500
750
1000
0
100
200
300
100
500
Thcoretical empirical distribution of the maximum absolute value normally distributed random variables.
(b) Estimation of mixed features under maxi-
mum ahsolute value scaling
Figure 11: Effects of maximum absolute value scaling
For min-max normalization, the situation is similar and omit the details here. The main point is that the scaling factor is strongly dependent on the sample size, which makes it unsuitable for normally distributed data in several situations, such as on-line learning (where sample size changes over time) model validation with uneven data splits.
B.2 Solution to the Elastic Net
Let ((m)) be solution to the problem in Equation (2). Expanding the function, we have
こ ( 2 Bo) + (X3 + (x ()) A A
A2
2/ |
Taking the subdifferential with respect to f and 3o, the KKT stationarity condition yields the following system of equations:
XT 30 y) + \ıg + X2 3 0, n30 + (X - y =
(11)
where g is subgradient of the norm that has elements 9: such that
{ignn 3.} if & 00 1-, otherwise.
9i E
B.2.1 Orthogonal Features
If the features of the normalized design matrix are orthogonal, that is, X diag (#f aTāØ), then Equation (11) can be decomposed into a set of p + conditions:
( + 16 y) + A203 + A190 j [ 12,0 (X
= 0.
The inclusion of the intercept ensures that the locations (means) of the features do not affect the solution (except for the intercept itself). We will therefore from now on assume that the features are mean-centered that c Ø, for all and therefore #11 O. A solution to the system of equations is then given by the following set of equations (Donoho & Johnstone, 1994):
3(m)
Sx, (x)y)
((m) y
$JØ;+\2²
where Sx(=) sign(=) max([=) X. 0) is the soft-threshodding operator.
23

---

Published in Transactions on Machine Learning Research (11/2025)
B.3 Bias and Variance of the Elastic Net Estimator
Here, we derive the results in Section 3 in more detail. Let
Zj - x + (2, + e) and dz d X2)
so that 3, Sx, / Since dj is fixed under our assumptions, we focus on Sx (Zz). First observe that. since Cj = J
EJ0; 5 c1)- - -c,) 2J.; nc}
ny
$j
5j
2/2; = - ( - 1c;) = nu 8j
8j
where Uj is the uncorrected sample variance of Z.j. This means that
2₁ = + Binv; - =J= + c,
nvy
and
dj 5-=8 8)
+
+x2
(12)
For the expected value and variance of Zg we then have
87
EZj = μ4j = E ( (ar + + = =
8j
Var Zj 03 = Var (a]e) = =Tā = nuyo?
53
The expected value of the soft-thresholding estimator is
(S, = f Sx(3) = [ (2+ (1+)) /1- (2 (z)2) (2)
And then the bias of Bg with respect to the true coefficient 18 is
E 83 = dg
(Z)) - B9.
Finally, we note that the variance of the soft-thresholding estimator is
(z (2) (<
/ (z - de)
Var Sx(z, - Sx(Z))=
(13)
and that the variance of the elastic net estimator is therefore
1 Var B, d3
Var (2,)
B.3.1 Normally Distributed Noise
We now assume that e is normally distributed. Then
Zg Normal (p, 2]2, =
Let 8 = -Hy A1 and 73 = My \1 Then the expected value soft tottshhresolding of Z, is
ESA (Z,) =
(a.u - 0,) δ(u) du +
(o,u + (g) Φ(u) du
00
8
0
Yj
= =0 +
og o
to o
24

---

Published in Transactions on Machine Learning Research (11/2025)
where p(() and (()) are the probability density and cumulative distribution functions of the standard normal distribution, respectively. Computing Equation (13) gives us
of
0j
00
2
Var Sx(Z))
erf
exp
+
2
T2V2
π
20
0j
02
20, Φ
+
09 $
oj erf
11
2
7
exp
+
2
02V2
0
π
20²
+
74
+
+73 +
03
jj
(2)))
B.4 Derivation of Estimate in the Noiseless Case
If we assume 8j = (gg - 03)², then Equation (7) becomes
Sx, (3)n(g) (7))
B, n(qs - 09) 17) + (g) 97)0
If we are in the lasso case, then X2- and
β, Sx (8)n(0) (gj (3)
n(gj 97) 1
so we need to choose 5 = 1, and hence 8, = 9j T3. to get rid of the dependency on dj- For the ridge case, we instcad have A₂ and hence
39n(9 - 99,) 1-5
8în
85 n(gj 93) 15)1 (g) 0o + (gj 93 2 A22
which shows that we need to choose 1/2 and hence 8j v 93 - gj. to get rid of the dependency on qj in this case.
Note that if \1, A2 0 (the elastic net case), then there is no choice of d that will make the estimator independent of qj-
B.5 Bias and Variance for Ridge Regression
Corollary B.2 (Variance in Ridge Regression). Assume the conditions of Theorem 3.1 hold, except that
A == 0. Then
0
if 1/4,
lim Var 3y
if 1/4,
=
if>> 1/4.
B.6 Centering and Interaction Features
The main motivation for centering is that it removes correlation between the main features and the interaction, which would otherwise affect the estimates due to the regularization. Centering normal features is also important because it ensures that their means do not factor into the estimation of their effects, which is otherwise the case since the variance of Ø3 would then be (o(02+ + 2² qı)) in the case when æ1 is centered
and (q: (o² + 2²) otherwise. Centering binary features is also important because the variance of the interaction term is otherwise q1 (provided 2y is centered), which would mean that the encoding of values of the binary feature (e.g. (0, 1) versus (-1,1}) would affect the interaction term.
25

---

Published in Transactions on Machine Learning Research (11/2025)
B.7 Extended Results Bias and Variance for Ridge, Lasso, and Elastic Net Regression
In Figure 12, we show bias, variance, and mean-squared error for the weighted elastic net. We see that the behavior of bias as qj +1 depends on noise level and that there is bias variance trade-off with respect w. As in Section 3.2, we modify the weighting factor to have comparability under = 2.
0 = 0.25
= 0.5
1.0
2.0
MSE
3
2
1
0.0
0.0
Bias
0.25
0.5
-15
1.0
1.5
0.4
Variance
0.3
0.2
0.1
0.0
0.50
0.75
1.00
0.50
0.75
1.00
0.50
0.75
1.00
0.50
0.75
1.00
0
4
Figure 12: Bias, variance, and mcan-squared error in the case of the one-dimensional weighted clastic net. The measures are shown for different noise levels (σz), class balances (qz), and values of (w), which controls the weights that
set to U = V/ 2 4w-1(q q²) in order for the results to be comparable across different values of w. The dotted lines represent the asymptotic bias of the estimator in the case of w = 1. In the case of w > 1, the limit of the bias is zcro.
C Proofs
D Proof of Theorem 3.1
To avoid excessive notation, we allow ourselves to abuse notation and will drop the subscript j everywhere this proof, allowing 3*, 8, and so on to respectively denote 38,, 8y et cetera.
Since (q ( we have
0
8ngg 72)
aV q 92 b(g 2/121112
0
o, V/lig 1/26/26
7
aVq d² b1g 8)) 2-6-1/2
I'n X1 (2)-11
d= b(q 126
18, + 2zaa - -2)
+ dz (2)
2 I'n bga - 92)
0 = - g6 92)- \1
d
n A2 92226-1
3" bln 9 1-5 A
26

---

Published in Transactions on Machine Learning Research (11/2025)
0. 0.25
de 0.5
de 1.0
d, 2.0
3
MSE
2
1
0.0
8
0.5
0.0
Bias
1.0
0.25
0.5
-15
1.5
-2.
0.4
Variance
0.3
0.2
0.1
0.0
0.75
q
q
q
Figure 13: Bias, variance, and mean-squared error for one-dimensional lasso problem, parameterized by noise level (oz), class balance (q), and scaling (8). Dotted lines represent asymptotic bias of the lasso estimator in the case when 8 = 1/2.
0e 0.25
0e 0.5
0g 1.0
0. 2.0
3
MSE
2
0
0.0
8
0.5
0.0
Bias
0.25
-1.0
0.5
-1.5
1.5
-2.
0.4
Variance
0.3
0.2
0.1
0.75
0.50
1.00
0.75
g
q
q
Figure 14: Bias, variance, and mean-squared error for one-dimensional ridge regression, parameterized by noise level (az), class balance (4), and scaling (8). Dotted lines represent asymptotic bias of the ridge estimator the case of 1/4.
27

---

Published in Transactions on Machine Learning Research (11/2025)
with
3 Vn
\ Oe n
and b=
We are interested in
lim E, = lim q 9-+1-
1
( (-00
(-)
0 (-)
7
?
+ Φ
(2)
(14)
9-+1- d
Before we proceed, note the following limits, which we will make repeated use of throughout the proof.
8 if < 6 << 1
0 lim 2 9-11 9--1
lim
b if 2'
(15)
if > 2'
Starting with the terms involving & inside the limit in Equation (14), for now assuming that they are well-defined and that the limits of the remaining terms also exist seperately, we have
lim (-a
0
+ 7 ② d
(
(1ntbelg
8'n
A n(g - -2) + >2 - q2)6 q22)
+ (d)
= lim g-+l
+
d
- 92)26-1
3'n
A
(22
+
n >2 92)26-1
nl 92) + \2 92)0
3'n
lim
+ +
n >2 - 92)26-1
A1
lim
(16)
g+1 n1g q²) 1 + 22 92)8
Considering the first term Equation (16), we see that
0
if 2
0 n d2(g - Q2)26-1 (0(2) + (2)
8'n
2n n+ 3
((-b)
if
For the second term in Equation (16), we start by observing that if == 1, then (2- = 1, and if > 1, then lim (q 92yr-1 = O. Moreover, the arguments of approach 0 in the limit for 2 1, which means that the entire term vanishes in both cases (8 2 1).
For S d 1, the limit is indeterminite of the form oc O. We define
(44 = (2) + (2) and
g(g) n(q 92)2-5 + d2 22)0
such that we can express the limit as lim f(1)/g(g). The corresponding derivatives are
f(( = (- ? (1- (1-24)(4 -21 - 1/2) - 2g) - 92)6-3/22) 1
(1 - 2q) 12) b(8 - 1/2)(1 -2) -2)0- 3/2)
2
g((g) n01 6) 29) -0) 0z0 120(1-44) - q2)6-1
Note that (() and f(g) are both differentiable and gg( 4 0 everywhere in the interval (1/2, 1). Now note that have
f((
g'(g) n011 6)(g - 6 - 02)6-1/2
(-(6
+ b65 1/2)(q (2)8-1) (
0
b(6 1/2) (2)--1) (2)) (17)
2
28

---

Published in Transactions on Machine Learning Research (11/2025)
For 5 5 < 1/2, - (g)) 0 since the exponential terms of Φ in Equation (17) dominate in the limit.
For d = 1/2, we have
lim f'(2) 4-1 g'(g)
(0())
④
0
(
2a ((bb)
lim
+ Φ
n + X2 q=
n + 22
so that we can use L L'Hôpital's rule to show that the second term in Equation (16) becomes
2 AIv 1 0z (22)
(oevTi)
(18)
For > 1/2, we have
f(()) lim
9 (6 + +2)
lim
4++1
g((g)
9-+1 n11 (0 - (200--2)1/2-6111 + - 2q)(g 0-)6-1/2
b(6 ()) -
+ lim
9-1 n(1 (2)-0 A20(1 2q) 2q) 922)2/2
b(6 - (1 (o 1) (e-abla enb(s-09))
lim
--1
22 (n(1 (g 22)3/2-26 92) + 122(11 h20(1-29)(q 02)2/2)
since the exponential term the numerator dominates. Now we proceed to consider the terms involving in Equation (14). We have
(0 -0 ( (2)) =avv lim g-1
o ( - ( 9-1 nlg - 02)2/2 + A2(g - q2)
Tim
(19)
For 0 & 5 < 1/2, we observe that the exponential terms in & dominate in the limit, and so we can distribute the limit and consider the limits of the respective terms individually, which both vanish.
For 2 1/2, the limit in Equation (19) has an indeterminate form of the type 0. Define
(2) ( Φ ( and v(() = n(g -2)--2))122 + d2lg 22))2-11/2 1/2
( - o
which are both differentiable in the interval (1/2, 1) and (q) * everywhere in this interval. The derivatives are
= ='(4)=-0
7 7 + δ
- (2)-1/2) b(6 1/2)(1 24) 02)0-3/2 2 a(1 24)(2-
0
[a(1 - -2) 1/22 + 6(6 1/2)(1 - 2q)(a -2)0-
0'(4) = m (1 2q)(4 / 22 - 1 (1 - 24) 22)20-3/2
2
And so
u'(g)
1
(a 1)(9- -3)0---) 1 (2) (2)
0(())
+ 140 - () 92)26-1
0
(20)
+ (a+ 1 - 5 x
29

---

Published in Transactions on Machine Learning Research (11/2025)
Taking the limit, rearranging, and assuming that the limits of the separate terms exist, obtain
lim u'(q) lim 9-11 v'(4) 9-11 n + A2 1) 92)20-1
1
(0()) 2 Φ
8 0 σ
1 0
0
b(28
lim 9-1- n A2(40 1)(g 92)26-1
1
④ (2) (()))
(a(q (2) 1/-1/22 - -b(g -22)226-377 -/26)
(
(-a(g 92)0-112 1/2 - bla (333-3333) 3/2) (21)
For 8 1/2, we have
lim ['(g) (-bo(-b) - b)( + 0 2abo(-b) 28 oz(n+ + A2)
(ooVT
1-1 (4(4)
n n+ A2
Using L'Hôpital's rule, Equation (19) must consequently be
2,3 2/
( + 22)
which cancels with Equation (18).
For 5 > 1/2, we first observe that the first term in Equation (21) tends to zero due to Equation (15) and the properties the standard normal distribution, For the second term, we note that this is essentially of the same form as Equation (17) and that the limit is therefore 0 here.
D.1 Proof of Theorem 3.2
The variance of the elastic net estimator is given by
Var 3,
02 10" d² 2
2 + erf
0/22/ )
))
exp
20
8²
+erf
+ erf
(022)
?
-
σ V
exp (-
(-2²²) ))
27
22
+ 2 (2) + 72 (2) (2)) (2 (22)
+ 20a Φ
+ 02 +
We start by noting the following identities:
0² n3*n)² (q 2)2-22 + + + 22,3 72)2-6
d² n? (q 2-22 + 2ndz 92) >g19 92)26
00 = - os (3"m 3/2 3/22(9 - 3/23/2-23 28 + Vndii1 - 02)112-0).
0
σ = (q (2) + l2(g - (2) 22)26 + 2ab(a g2)5.
T n
d n(g (2)$ + X2 - 28-
Expansions involving y instead of 0 have identical expansions up to sign changes of the individual terms. Also recall the definitions provided in the proof of Theorem 3.1.
30

---

Published in Transactions on Machine Learning Research (11/2025)
Starting with the case when 0 8 < 1/2, we write the limit of Equation (22) as
lim Var B
9-+1-
(00)
0
2
82
= lim
9-1 (nlq 92)1/2 + >2(9 - 02)26-11232
1+ erf
(o22)
V π
exp
20²
+ onn lim
g-11 (n(q - q2)1/2 + )2(4 - q2)26-1/2)
1 erf erf
7
7
0
V
π
exp
20
72
200
0
0
27
?
?
+ lim
+ lim
+ lim
0
+
lim 9-1
9-+1-
d?
9-11 d²
9-1
d²
d2
lim
EBg
9-11
assuming, for now, that all limits exist. Next, let.
0²
fi(9) = 1 + erf
exp
20
7
7
2 exp
72
fe(g) + erf
ov2
0
π
20
(()) (n²(q (²)
2n
22)
+ +\3(1
9
12)
And
0
2
0²
f((g) 0²
exp
π
20
22
2
72
f1(0)) o²| exp
V
20
o (w) (1 24) ((g
(2) + 4ndd2(q- - -2,26-1 + X3(40 - 1)(4 -2))48-2)
J1, fi and g are differentiable in (1/2 and g4(4) * 0 everywhere in this interval. fi/9 and f2/g are indeterminate of the form 0/0. And we see that
lim fi(9) lim fá(g) 0 9-1 g(()) - g'(q)
due to the dominance of the exponential terms as 0/a and 7/a both tend to -oc. Thus fi/g and fa/g also tend to 0 by L L'Hôpital's rule. Similar reasoning shows that
20g d2
lim 9-11
lim
$
0.
d2
q
The same result applics to the respective terms involving 7. And since we in Theorem 3.1 showed that lim 1 F. 0, the limit of Equation (22) must be O.
For 8 1/2, we start by establishing that
lim [ (z (2)))
/ 0
y² d(y) dy + 20g / yd(v) dy + 0² / d(y) dy
lim 9-11
02
(23)
is a positive constant since 0/a + -b, o Jzyn 0 - and Ao os Vnd. An identical argument can be made in the case of lim / (z 2) fz(2) dz. We then have
C+
lim
(z + x)2 fz
00,
--1- de -00
lim df
0
31

---

Published in Transactions on Machine Learning Research (11/2025)
where C+ is some positive constant. And because li E 3" (Theorem 3.1), the limit of Equa- tion (22) must be co. Finally, for the case when d > 1/2, we have
: lim 1 o² 3² 6(y) dy + 200 [ yo(u) dy + 0² { d(y) dy 9-+1 d
no²
= lim 9-1
y² Φ(v) dy
(n(1 - /2/ + \2 q2 (2)26-1/2)2
20.v (3"n(9 9²)1- X1)
8
((v) dy
(n(g (2)) + A2 02)36/2-1114 1///2
-8 0 A1 n(g q2)2 >2 92)6
2 / x(v) dry (24)
Inspection of the exponents involving the factor (q - q²) shows that the first term inside the limit will dominate. And since the upper limit of the integrals, 0/a + as q + 17, the limit must be co.
D.2 Proof of Theorem B.2
We have
lim Var By lim 0² (nig- (n(g- 92)2-6 + >2 92)6) =
Te -2 12-1
2 onn
lim (q - 92)1-18 9-11-
g-11 d² In(a-
X
from which the result follows directly.
D.3
Proof of Theorem 3.3
D.3.1 Expected Value
Starting with the expected value, our proof follows a similar structure as in the proof for Theorem 3.1 (Section D). We start by noting the values of some of the important terms. As before we will drop the subscript j everywhere to simplify notation. We have
P = (g 2²)00
-v qp - b(g 2200-1/2
o.V Vulg (2);
u/a q2 blq -22 82)0-112 1/2
-B'n \ı 92 2
d nlq - (2) + >2 - g2)00
n + d249 9²)
I'n \1 w2)=-
0 8'n(g - (2) >ı - 92)0,
7
n p2sa - q
7 = lg - (2) \ı - 22)0.
First note the following limit (which is analogous to that in Equation (15)).
0
if <
< 1
2
lim --1
lim 2 b
if 7
(25)
1-1 σ
if
As in Section D, we are looking to compute the following limit:
1
(
0 - o
+ ΦΦ (2) + Φ (2) (2))
lim F,β = lim
-
(26)
d
9+1 9-1
32

---

Published in Transactions on Machine Learning Research (11/2025)
Starting with the terms involving & and assuming that the limit can be distributed, we have
0 (2)
I'm + X1 (q - g )
0
lim 9-11
(
lim g-11
n 22 q² --)
3en - \1 q )2)
① (2)
+ lim
n A2 n 12)--1
9-1-
if w < 1
A""
il
(27)
8 1f
The derivation of the first case in Equation (27) depends on w. For 0 w S 1/2, it stems from the facts that ((0/o) - 0 and (0//o) + 0 as q- 1 numerator and denominator. For 1/2
- together with the existence of the (q - q²) factor in both
< 1, the terms cancel each other out. In the second case, when
= 1, the result stems from 4(0/o) and (17/o) both tending to 1/2 q 1 And finally for > 1, the terms involving the - q factors vanish and again the values of the cumulative distribution functions tend to 1/2.
Now, we turn to the terms involving the probability density function Φ. Again, we assume the limit is distributive so that
(0)))
(2)
lim á 8-1-
lim 7-11 d
lim
(28)
q d
Starting with the first term on the right-hand side of Equation (28), we have
0. / ② (2)
lim Φ (2) n(q - 1/2 A2(9 12) 1/2'
q 9--1 d
For S w < 1/2, this limit is 0 since the exponential terms in the numerator will dominate as + 17. For
w = 1/2, we have the limit as vevnn((-b) X2- For w > 1/2, the limit is indeterminate of the type 0/0. Let
fi o 2 and ((g) nl q2)a/2 + d2 -2)
and observe that f1 and are differentiable and g'(g) * 0 for q (1/2, 1). The derivatives are
f((4)) = - [ (1 2q /22-1/2 1/2 b(u 1/2)(1 24) - 3/23 2 (2)
a
g' (q) 21 2q (2) + >2(w - 1/2)(1 - 2q) 3/22
n
Next, we find that
f((9) -a b(2w 1) () - q2y0-1
?
(29)
g'(9) n \2w - 1)(g 92) -
Taking the limit of Equation (29) and invoking L L'Hópital's ''s rule yields
lim f((9) = g-+1 g(())
both when 1/2 < w < 1 and w 2 1 since 7/0 tends to 0 as + 1 for w > 1/2 and the Φ term tends to a constant, plus the fact that the remaining factor in the expression also tends to constant since the terms involving (q 2- vanish whon w > 1, are constant when w = 1, and cancol cach other out in the limit when w 1.
Finally, if we now consider the second term on the right-hand side of Equation (28), set f2(g) = (0/o), and perform the same steps as above, we find that the limits are the same in all cases, which means that the limits in Equation (28) cancel in the case when = 1/2 and therefore that
lim ( (0)) (0) =
for 5 w. The limit in Equation (26) is given by Equation (27).
33

---

Published in Transactions on Machine Learning Research (11/2025)
D.3.2 Variance
The proof for the variance result is in many ways equivalent to that in the case of variance of the normalized unweighted elastic net (Section D.1) and we therefore omit many of the details here.
In the case of 0 S < 1/2, the proof is simplified since the de term tends oo whilst the numerator takes the same limit in the normalized case, which means that the limit is 0 in this case. For w 1/2, we consider Equation (23) and observe that it again tends to a positive constant whilst lim d2 = 0, which means that the limit of the expression, and hence variance of the estimator, tends to co. For w > 1/2, an identical argument for the expression in Equation (24) holds and the limit is therefore oc in this case as well.
D.4 Proof of Theorem B.1
If X2 ~ Normal(j,.)), then X ~ FoldedNormal(u.e) a). By the Fisher Tippett Gnedenko theorem, we know that (max, [x,] a,) converges in distribution to either the Gumbel, Fréchet, or Weibull distribution, given a proper choice of an > 0 and b, E R. A sufficient condition for convergence to the Gumbel distribution for absolutely continuous cumulative distribution function (Nagaraja & David, 2003, Theorem 10.5.2) is
(1-F()) f(x) ) = O.
lim 2-000 dr
We have
1 1 Fy(a)
1-
1
2
erf
x-H
V2
2
1 erf
V20
x+
fx(I)
[x-p)²
(1
20
+ V2na2
20
V2x02
2 + 1-
(Ø 2-1
+ 1+
a(1 (xx)) as n
n,
6(x)
where & and & are the probability distribution and cumulative density functions of the standard normal distribution respectively. Next, we follow Nagaraja & David (2003, example 10.5.3) and observe that
d ( - ((x))
αx(1 - ((x)) + as
dx
δ(z)
σ(x)
since
1 1- (xx)
((x) I
In this case, we may take b, = Fy 1/n) and an (nff (bn))
E Additional Experiments
In this section we present additional and extended results from the main section.
E.1 Power and False Discoveries for Multiple Features
Here, we study how the power of correctly detecting = 10 signals under qj linearly spaced in [0.5, 0.99] (Fig- ure 15(a)). We set 83 = 2 for each of the signals, use n 100 000, and let a. = 1. The level of regularization is set to A₁ n4 /10. As we can see, the power is directly related to q and for unbalanced features stronger the higher the choice of 8 is.
We also consider version of the same sctup, but with p lincarly spaced in [20, 100) and compute normalized mean-squared error (NMSE) and false discovery rate (FDR) (Figure 15(b)). As before, we let & 10 and consider three different levels of class imbalance. The remaining k features have class balances spaced
34

---

Published in Transactions on Machine Learning Research (11/2025)
evenly logarithmic scale from 0.5 to 0.99. Unsurprisingly, the increase in power gained from selecting 8 = imposes increased false discovery rates. We also see that the mean-squared error depends class balance. In line with our previous results, å E {0, 1/2] appears to work well for balanced features whilst works better when there are large imbalances. In the case when qj = 0.99, the model under scaling with does not detect any of the true signals.
9
0.5
q 0.9
0.99
1.00
1.00
NMSE
0.75
0.0
0.50
0.25
Power
0.00
0.50
1.00
6
FDR
0.75
0.0
0.50
0.25
1.0
0.00
0.00
0.5
0.6
0.7
0.8
0.9
1.0
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
P
(a) The power (probability of detecting all true signals) of the lasso. our orthogonal setting, power constant over P, which why we have omitted the parameter in the plot.
(b) NMSE and FDR: the rate coefficients incorrectly set non-zero (false discoveries) to the total number estimated coefficients that are nonzero (discoveries).
Figure 15: Normalized mean-squared error (NMSE), false discovery rate (FDR), and power for a lasso problem with k- 10 true signals (nonzero 83), varying p, and dj [0.5, 0.99). The noise level is set al ae = and A1 0.02.
E.2 Support Size and Predictive Performance
In this section analyze the support size of the lasso estimates for the experiment in Section 4.1.2. Tn Figure 16, have, in addition to NMSE on the validation set, also plotted the size of the support of the lasso (cardinality of the set of features that have corresponding nonzero coefficients.. Here we only show results for 8 E (0, 1/2
E.3 Predictive Performance for Simulated Data
In this experiment, we consider predictive performance terms of mean-squared error of the lasso and ridge regression given different levels of class balance (g, E {0.5, 0.9, 0.999), signal-to-noise ratio, and normalization (5). All of the features are binary, but here we have used n 300 and p 1000. The 10 first features correspond to true signals with 87 and all have class balance q. To set signal-to-noise ratio levels, we rely the same choice in Hastie et al. (2020) and use log-spaced sequence of values from 0.05 to 6. We use standard hold-out validation with equal splits for training, validation, and test sets. And fit full lasso path, parameterized by lpg-ep grid of 100 values?, from Amax (the value of À at which the first feature enters the model) to 2 on the training set and pick À based on validation set error. Then we compute the hold-out test set error and aggregate the results across 100 iterations.
The results (Figure 17) show that the optimal normalization type in terms of prediction power depends on the class balance of the true signals. If the imbalance is severe, then gain by using d = 1/2 which gives chance of recovering the true signals. If everything is balanced, however, then do better by not scaling. In general, 1/2 works well for these specific combinations of setlings.
9This is standard choice of grid, used for instance Friedman (2010)
35

---

Published in Transactions on Machine Learning Research (11/2025)
rhee2006
eunite2001
triazines
250
12.5
50
Support Size
200
10.0
150
7.5
100
5.0
50
2.5
10
0.0
1.0
1.8
0.8
Optimal
0.8
1.6
NMSE
0.6
0.5
1.4
0.4
0.4
1.2
1.0
10
10
10
10
10
10-4
10-3
10
10"
10a
10-1
10-3
10
10
10
109
X/ Amax
Figure 16: Support size and normalized mean-squared error (NMSE) for the validation set for the lasso fit to datasets rhee2006, eunite2001, and triazines across combinations and X. The optimal is marked with dashed black lines and the best combination of å (among 0, 1/2, and 1) and shown as a dot.
= 0.5
g = 0.9
q == 0.99
0.8
NMSE
0.6
0.4
0.0
0.5
1.0
10
10"
10
10a
10
10a
Signal-to-Noise Ratio
Figure 17: Normalized mean-squared prediction error in a lasso model for different types of normalization (6), types of class imbalances (Øy), and signal-to-noise ratios (0.05 to 6) a dataset with n 300 observations and p 1000 features. The error is aggregated test-set error from hold-out validation with 100 observations in each of the training, validation, and test sets. The plot shows means and Student's t-based 95% confidence intervals.
36

---

Published in Transactions on Machine Learning Research (11/2025)
E.4 Comparisons of Normalization Methods Real Data
In this section we present an extended and modified version of the experiment in Section 4.1.2, by considering several additional datascts, including datasets with binary responses to which have fit & and lz-regularieed logistic regression. Instead of only considering parameterization over 6, we also extend the benchmarks to cover additional normalization types. For each of the datasets, we have fit the lasso and ridge for 10-times repeated 10-folds cross-validation over grid of À and normalization method. In the case of the method we call "ours", we have extended the grid over as well, and return the results for the with lowest crror.
The results are shown Figure 18 and show that the optimal choice of normalization depends on the dataset and type of model. Using our strategy, which corresponds to standardization for continuous data and hyper-tuning over 5 attains best results for most of the datasets, only ever slightly worse than the best performing method. Among the methods, & normalization seems to perform poorly in general. Given the fact that that it corresponds to variance-scaling for binary data, which we have shown results in considerable variance, this is not particularly surprising.
E.5 Dichotomization and Feature Selection
In this experiment, we attempt to study the effect of normalization choice in feature selection datasets with dichotomized features and the following dichotomization scheme to convert the continuous features to binary.
Each variable was converted to a binary feature by comparing against threshold based on either historical standards, regulatory guidelines, or domain-specific knowledge of urban housing factors in the 1970s (Table 3).
Table 3: Dichotomization procedure and linear regression coefficients for the features of the Boston housing dataset used in the experiment in Section E.5.
Feature
Dichotomization Rule
B
Qj
Crime Rate Zoning
Above/below national U.S. average crime rate (1970 1971)
-1.7
0.90
Presence/ presence/absence of large lot zoning (residential lots over 25,000
1.96
0.26
sq.ft)
Business
Residential vs industrial areas (below/abon) 10% non-retail
-2.41
0.47
business acres)
Charles River NOx Concentration
Original binary feature (borders river or not)
5.12
0.07
A Above/beloy below EPA air quality standard for NOx (53 parts per billion)
-261
0.52
Rooms
Above /below typical family dwelling size (more/less than 3.49 rooms)
0.66
Age
Newer vs historic housing (less/more than 50% built before 1940)
-1.10
0.71
Distance
Close vs far from employment centers (less/more than 5 miles)
4.76
0.27
Highway Access
Limited vs good highway access (accessibility index be-
1.25
0.26
low/abows 20)
Property Tax
Below/abowt Massachusetts average property tax rate (approx- imately $12 per $1000 in 1970s)
7.35
0.97
Pupil-Teacher Ratio
Relow/abows recommended educational value (16 students per teacher)
6.57
0.83
Demographics Lower Status
More/less diverse population (above/below 85% white)
3.25
0.95
Middle-class vs lower-income areas (below/abowy 15% lower status population)
-7.24
0.32
Having dichotomized the data, we then first fit standard linear regression model to the data, and use this proxy for feature importance. Then, for three types of normalization: Ey-normalization (8 =1),
37

---

Published in Transactions on Machine Learning Research (11/2025)
1.00
NMSE
max.abs
min
0.50
purs
0.25
0.00
ala
australian
eunite2001
heart:
housing
rhee2006
triazines
wla
(a) Results for the lasso
1.5
1.0
NMSE
max
min max
ours
std
0.5
0.0
ala
australian
aunite2001
heart
housing
rhee2006
triazines
wla
(b) Results for ridge regressioo
Figure 18: valisavali error from 10-times repeated 10-folds cross validation for the lasso and ridgo and various datasets and normalization strategies. The error is normalized mean-squared error (NMSE). the case of datasets aia, wla, heart, wla, and australian, we have fit regularized logistic regression and otherwise regularized linear regression. The error bars show 95% confidence intervals.
38

---

Published in Transactions on Machine Learning Research (11/2025)
standardization, and min-max normalization, fit lasso model to the data and compute ranks of the features by checking at which point they enter the model.
Finally, we compare the ranks of the features in the lasso model to the ranks of the coefficients from the standard linear regression model, using the latter as the reference. We use four different metrics to compare the ranks: Spearman's 's rank correlation, Kendall's т, mean absolute difference, and normalized discounted cumulative gain (NDCG). The results are presented in Table 4. The results show that the normalization method using & ennorraaizatiin (6 1) best corresponds to the ranks of the linear regression coefficients.
Table 4: Comparison between ranks of ordinary least-squares coefficients and ranks given by the order of model entry along the lasso path for the Boston housing dataset. The metries used are Spearman's 's and Kendall's rank correlations, normalized discounted cumulative gain (NDCG), and mean absolute difference (MAD). Best values are marked in blod face. For all measures except MAD, higher values are better.
Metric
4 N-oomaalization
Standardization
Min Max Maax/MXxxX- Abs
Spearman
0.7308
0.5714
0.5
Kendall
0.5128
0.4359
0.3846
MAD
2.0
2.7692
3.0769
NDCG
0.9515
0.9351
0.9186
F
Categorical Features
This paper has focused on binary features, but since categorical features are typically encoded as set of binary (dummyy features in regression settings, similar principles apply. For instance, categorical features will necossarily result in imbalanced reprosentations with respect to the other features, given that onc-hot encoding results in a set of mutually exclusive features. This means that the same issues with variance and regularization that we have discussed in the main part of the paper will also apply here.
We leave thorough investigation of this to future work, but will here present small experiment to illustrate another aspect of categorical features and class imbalance that is related to which reference category is chosen in one-hot encoding.
Consider a variable Z ~ Categorical(q), where = (41 92, 93). To represent this variable in a regression model, the standard approach is to one-hot encode it into a set of binary variables X₁ = 1(2 = 1}. X2 = 1(2 = 2), and X3 = 1 = 3}, and then drop one of the variables to avoid redundancy. The dropped variable is called the reference category, and the coefficients of the remaining variables are interpreted in relation this category.
Let us assume that we either use the first or the last category as reference, and that 41 = 1/20, = 1/2, and Ø3 = 9/20. In the first case, 3o:A corresponds to the comparison between categories C and A. In the second case, A: corresponds to the comparison between categories A and C. In other words, these coefficients correspond to the same comparison, but the class balances of the features differ greatly. For a standard ordinary least-squares regression, this fact has no implication for the fit and either choice of reference category would yield the exact. same fit. But for the lasso (or ridge), the choice of reference category does matter, since the regularization will affect the coefficients differently depending on the class balances of the features.
We set ВB.A = 3B.C = 1/2 and SC:A= BA.C = 1, and generate data according to the linear model y = XB. Then, we fit a lasso model to each case, using no normalization. The results are shown in Figure 19, where we plot the regularization paths for the two cases. The plot shows that Sc:a enters the path earlier than BAc, even though they converge to the same value as À + 0. This means that the choice of reference category affects the order in which the features enter the model, and therefore potentially also affects feature selection.
G Summary of Data Sets
In Table summarize the datasets we in our paper.
39

---

Published in Transactions on Machine Learning Research (11/2025)
Table 5: Details of the real datasets used in the experiments. The median q value refers to the median of the proportion of ones for the binary features in the data. Note that in the case of housing, there is only a single binary feature.
Dataset
Response
Design
Median
Description
eunite2001
336
16
continuous
mixed
0.856
Mid-term load forecasting competition dataset (EUNITE 2001) using National Tai- University's winning approach. Con- tains 15 features including 7-day histori- cal loads (scaled) with winter-only training data from 1997-1998 to predict January 1999 daily maximum loads (Chen al., 2001).
housing
506
13
continuous
mixed
0.931
Boston housing dataset with information about housing in the Boston area. Re- sponse is median value of owner-occupied homes in $1000s (Harrison & Rubinfeld, 1978).
triazines
186
60
continuous
mixed
0.973
Pharmaceutical dataset relating molecular structures of triazine derivatives to their ability to inhibit dihydrofolate reductase (Hirst et al., 1994; King et al., 1995).
rhee2006
842
361
continuous
binary
0.995
HIV-1 drug resistance data with protease and reverse transcriptase mutations as fea- tures. Response measures in vitro suscep- tibility to antiretroviral drugs (Rhee et al., 2006).
leukemia
38
7129
binary
continuous
Gene expression data for leukemia patients. Classifies between acute myeloid leukemia (AML) and acute lymphoblastic leukemia (ALL) (Golub et al., 1999).
australian
690
14
binary
continuous
0.557
Credit approval dataset, originally from the StaLog database. The task to predict credit approval using number of different features (Quinlan, 1987; Henery & Taylor, 1992).
heart
270
13
binary
mixed
0.678
Heart disease dataset, originally from the StatLog database. The task is to predict the presence of heart disease from a number of features that have already been selected from larger set of features (Henery & Taylor, 1992).
aia
1605
123
binary
binary
0.970
Subset of Adult dataset derived from sus data. Predicts whether income exceeds $50,000/year based census information (Becker & Kohavi, 1996; Platt, 1998).
wia
2477
300
binary
binary
0.976
Derived from web page data, classifying whether pages belong to specific categories (Platt, 1998).
40

---

Published in Transactions on Machine Learning Research (11/2025)
1.00
BB
BAC
BC:A
Ba:c
0.75
000 0.50
0.25
0.00
20
40
20
40
80
Step
Step
Figure 19: The coefficients along the regularization path for an example of fitting the lasso to a dataset with a categorical feature with three levels. The left plot shows the path when using category as reference, and the right plot shows the path when using category as reference. The coefficients 3Cra and BA.C correspond the same comparison, but enter the model at different points along the path.
We also visualize the distribution of class balance among all the binary features in Figure 20. We note that the class imbalance for many of these datasets is quite severe.
ala
wla
rhee2006
30
40
100
125
20
60
10
28
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
triazines
eunite2001
australian
0.6
0.7
0.8
0.9
1.0
0.7
0.9
0.7
0.9
1.0
heart
1.0
0.8
0.6
0.4
0.2
0.0
0.5
0.6
0.7
0.8
0.9
1.0
Figure 20: Histograms over the distribution (class balance, that is, the proportion oncs) for the binary Teatures in each of the datasets used the paper.
41


Reviews:
[{"review_id": "IE8xuxr4Ep", "reviewer": "Reviewer_hdbJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper tackles an overlooked but important problem: how normalization choices affect shrinkage in regularized regression.\n- It combines detailed theoretical analysis (deriving how class balance interacts with normalization) with extensive simulations and experiments on real datasets. \n- The work provides insights for practices, for example, scaling binary features by variance or standard deviation depending on the method, or using weighted penalties for the elastic net.\n\n### Weaknesses\n\n- The study mainly focuses on binary and continuous features under least-squares loss. It would be better to discuss further how the results can be extended to categorical variables with more than two levels or broader classes of models. \n- It would be better to discuss how the studies can be extended to some deep learning scenarios, for example, $\\ell_2$ distance regularization for fine-tuning pretrained deep neural networks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies how the choice of normalization affects shrinkage behavior in regularized regression methods, specifically lasso, ridge, and elastic net. While normalization (centering and scaling features) is standard practice to address feature scale sensitivity, this paper shows that the type of normalization applied has significant and often overlooked effects on model estimates, especially for binary features. They demonstrate theoretically and empirically that class balance in binary predictors directly influences coefficient estimates, with standard approaches such as standardization or max-abs scaling leading to systematic biases. For lasso and ridge, scaling binary features by their variance or standard deviation, respectively, can mitigate these biases, though at the cost of increased estimator variance. For the elastic net, no normalization fully removes this effect; instead, scaling penalty weights offers a solution. The study also extends to mixed binary–continuous data and interaction terms, showing that normalization implicitly governs relative penalization across feature types. Overall, the work highlights normalization as not merely a preprocessing step but a critical modeling choice that shapes bias–variance trade-offs and selection behavior in regularized regression. \n\n- The paper shows that the class balance (proportion of ones) in binary features strongly affects coefficient estimates in lasso, ridge, and elastic net regression. It demonstrates that scaling by variance (lasso) or standard deviation (ridge) can reduce this bias, but only by trading off with increased estimator variance\n\n- The authors prove that, unlike lasso and ridge, elastic net cannot eliminate class-balance bias through normalization alone. Instead, they propose scaling the penalty weights rather than the features themselves as an effective solution. \n\n- The study extends its analysis to mixed binary–continuous designs and interaction terms, revealing that normalization choices implicitly control how regularization penalizes different feature types. They also show that commonly used normalization of interaction terms leads to biased estimates, and they provide an alternative scaling strategy that avoids this problem.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be better to include a discussion of highly sparse binary data (e.g., text or clickstream data), where normalization strategies may interact with sparsity constraints differently.\n\n- The bias–variance trade-off can be quantified more clearly, e.g., error rates or feature selection accuracy in simulated settings with varying SNRs. \n\n- It would be better to add more intuitive examples and visualizations (e.g., small toy datasets) to illustrate how normalization changes coefficient paths for non-technical readers.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wb2w5PuEJq", "reviewer": "Reviewer_NRCr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strength\n\nThis paper combines both empirical results and theoretical guarantees on feature normalisation in lasso, ridge and elastic regression, revealing the potential bias and possible mitigation with variance tradeoff. This paper also discussed the assumptions and limitations of the theoretical results in details. \n\n# Weakness\n\nA slight weakness is that the paper is based on strong assumptions in a very simple setting. Also, results only concerns the expected value of the regression coefficient at the limit of extremely unbalanced binary features.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the effect of different input normalizations on the regression coefficients with Lasso, ridge or elastic regularization. \n\nIn more details, they assume the response $y$ is generated by \n\\begin{equation}\ny = \\beta^*\\_0 + X \\beta^* + \\varepsilon\n\\end{equation} \nwith the target coefficients $\\beta^*\\_0, X \\beta^*$ fixed, $X \\in \\mathbb{R}^{n\\times p}$ the feature matrix, and $\\varepsilon$ independent Gaussian noise. \n\nThe normalized feature matrix  $\\tilde{X}$ is given by $\\tilde{x}\\_{ij}=(x_{ij}-c\\_j)/s\\_j$ for some centering coefficient $c_j$ and scaling coefficient $s_j$. This paper requires that the normalised features to be orthogonal, i.e. $X^\\top X$ is diagonal. \n\nThen the paper shows that the regression coefficient of binary features is biased depending on the normalization and the regularisation. This bias can be mitigated by corresponding normalisations. it also extends the discussions to gaussian features and mixed feature settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In table 1, the $\\hat{\\beta}\\_\\text{max-ab}$ column is entirely zero. Is it correct or a mistake?\n\nAlso, could the authors explain the possibility or the difficulty how to extend the results to non-binary and non-Gaussian features?", "broader_impact_concerns": "No there is no or minor broader impact concerns I am aware of. This is a theoretical paper on linear regression.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "l86XqZ7Cwe", "reviewer": "Reviewer_2vk5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is exceptionally well written, and the figures are all very well presented. I believe the findings are interesting for both linear regression but also machine learning in general.\n\nThe main weakness is the lack of clearly stated lessons for practitioners; some guidance is available but the main lessons appear to be scattered throughout the text. I believe consolidating these into a single discussion section could be valuable.\n\nI believe the main contributions are original but I am not as familiar with the field; I will defer to other reviewers on this point.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this submission, the authors study the relationship between normalization schemes and fit coefficients for regularized linear regression (elastic net). In particular, they delve into the role that class imbalance plays in the effect of normalization. Their main findings include:\n\n* Ridge regression and Lasso admit normalization schemes (feature std deviation and feature variance respectively) which remove class imbalance dependence from the final fit.\n* In general elastic net does not admit a simple normalization scheme which does this.\n* However, elastic net coupled with a specific coordinate-dependent regularization does have this property.\n* In general, the choice of normalization scheme leads to a bias-variance tradeoff.\n\nThey conclude with experiments showing the tradeoffs between regularization strength and normalization scheme in the context of generalization error on real datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Light request: instead of labeling e.g. the design matrix X_{ij}, consider using mixed greek and roman indices like X_{\\alpha j}. This improves the clarity of which quantities are in feature space vs data space.\n\nThe main request is to add a more detailed, separate section on practical lessons from the theory and the experiments, e.g. about how to choose the normalization scheme and/or sanity checks for making sure the normalization scheme is well selected. In particular it seems like choosing nontrivial $\\delta$ was beneficial in the experiments, which may lead to recommendations for practitioners to use non-traditional normalization schemes.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)