How does overparametrization affect performance on minority groups?
Authors: Saptarshi Roy
Subha Maity
Songkai Xue
Mikhail Yurochkin
Yuekai Sun
Venue: Accepted by TMLR ()

Abstract: The benefits of overparameterization for the overall performance of modern machine learning (ML) models are well known. However, the effect of overparameterization at a more granular level of data subgroups is less understood. Recent empirical studies demonstrate encouraging results: (i) when groups are not known, overparameterized models trained with empirical risk minimization (ERM) perform better on minority groups; (ii) when groups are known, ERM on data subsampled to equalize group sizes yields state-of-the-art worst-group accuracy in the overparameterized regime. In this paper, we complement these empirical studies with a theoretical investigation of the risk of overparameterized random feature regression models on minority groups with identical feature distribution as the majority group. In a setting in which the regression functions for the majority and minority groups are different, we show that overparameterization either improves or does not harm the asymptotic minority group performance under the ERM setting when the features are distributed uniformly over the sphere.

Published in Transactions on Machine Learning Research (06/2025)
How does overparametrization affect performance on minor- ity groups?
Saptarshi Roy
saptarshi. wtw.::itioooooouuo.uuuuu..e www.:sssiii.tet..ee..e.e....
Department of Computer Science University of Teras Austin
Subha Maity
smaityQwaterriin edu
Department Statistics & Actuarial Science University of Waterloo
Songkai Xue Department of Statistics University of Michigan, Ann Arbor
srue@umich.. edu
Mikhail Yurochkin
mikhail, www.chuinoooobmmmmmmmmmm
MIT-IBM Watson AI Lab
Yuekai Sun
yuekai@umichh. edu
Department of Statistics
University of Michigan, Ann Arbor
Reviewed on OpenReview: https: openreviee.. net/ forum? id- P0neexxuuuuurrrerrrrrr
Abstract
The benefits of overparameterization for the overall performance of modern machine learning (ML) models are well known. However, the effect of overparametteriaal ion at a more granular level of data subgroups is less understood. Recent empirical studies demonstrate encouraging results: (i) when groups are not known, overparameterized models trained with empirical risk minimization (ERM) perform better on minority groups; (ii) when groups are known, ERM on data subsampled to equalize group sizes yields state-of-the-art worst-group accuracy in the overparameterizeed regime. In this paper, complement these empirical studies with theoretical investigation of the risk of overparameterized random feature regression models on minority groups with identical feature distribution as the majority group. In setting in which the regression functions for the majority and minority groups are different, we show that overparameterization either improves or does not harm the asymptotic minority group performance under the ERM setting when the features are distributed uniformly over the sphere.
1 Introduction
Traditionally, the goal of machine learning (ML) is to optimize the average or overall performance of ML models. The relentless pursuit of this goal event led to the development of deep neural networks, which achieve state-of-the-art performance in many application areas. prominent trend the development of such modern MI. models overparamcterization:: the models are so complex that they are capable of perfectly interpolating the data. There is a large body of work showing overparameterization improves the performance of MI. models in variely of settings, e.g. ridgeless least squares Hastie et. al. 2019) random
Codes available httper///.hhaaa...ee/irtttaaaaar hrap faimess

---

Published in Transactions on Machine Learning Research (06/2025)
feature models Belkin et al. 2219 Mei & Montanari 2019 and deep neural networks Nakkiran et al. 2019
However, as ML models find their way into high-stakes decision-making processes, other aspects of their per- formance (besides average performance) have been under scrutiny. One aspect that is particularly relevant to the fairness and safety of MT. models is their performance on traditionally disadvantaged demographic groups. There is a troubling line of work showing ML models that perform well on average may perform poorly minority groups of training examples. For example, Buolamwini & Gebru (0018) shows that com- mercial gender classification systems, despite achieving low classification error average, tend to misclassify dark-skinned people. In the same spirit, Wilson ct al. (2019) shows that pedestrian detection models, despite performing admirably on average, have trouble recognizing dark-skinned pedestrians.
In lieu of this issue, prior literature examines the effect of model size on the generalization error of the worst group. For example, the simple classification setting with spurious correlation, Sagawa et al. (2020) shows that increasing model size beyond the threshold of zero training error (overparameeterized regime) could harm the test error for minority groups. In nutshell, the authors argue that overparameterized models do not effectively learn structure that generalizes well for both majority and minority groups; instead, they tend to "memorize" the minority points owing to an inductive bias. This is rather surprising as a more extensive set of experiments on benchmark data in Le Pham et al. (2021) reveals that increasing model size either improves or does not harm the worst-group test performance across all settings. In an experiment with California housing prices dataset (see Appendix D), where the target is to predict the median house prices, we find that overparameterization in a random feature regression model (Mei & Montanari] 2019) improves mean squared error for the minority group comprising inland houses. This is in sharp contrast to the findings in Saaawa et al. (020)) Therefore, comprehensive theoretical investigation is necessary in this area to gain deeper understanding of the impact of overparameterizattion on the minority group.
In parallel, Sagawa et al. 2020 An et al. (2021) also shows that subsampling the majority groups is far more successful than upweighting the minority groups in reducing worst-group error. [drissi et al. (202I) recommends using simple methods, í.e. subsampling and reweighting for balanced classes or balanced groups, before venturing into more complicated procedures. They also suggest that newly developed robust optimiza- tion approaches for worst-group error control (Sagawa et al. 2019 Liu et al.. 2021 could be computationally demanding, and that there is no strong (stalistically significant. evidence of advantage over those simple methods.
Main contributions. In light of these existing works, we investigate how overparamellerizaiinn affects the performance of ML models on minority groups in a regression setup when the features are generated uniformly over the sphere. In particular, we study the effect of overparancterization under both the usual linear regression and the two-layer random feature (RF) model (Belkin et al.) 2019 Mei & Montanari, 2019). As we shall see, overparamaterizalii generally improves or slabilizes the generaliaation error of the minority groups under the random feature model. On the contrary, the generalization error under the linear regression setup deteriorates with incrcasing overparameteriaatio. To elaborate more on this, we lay out a more detailed discussion of our contributions below:
1. Complete picture of limiting ERM risk under RF model: We develop a simple two-group model for studying the effects of overparameterization on (sub)groups. Our model has parameters controlling signal strength, majority group fraction, overparameterization ratio, discrepancy between the two groups, and error term variance that display a rich set of possible effects. In a high- dimensional asymptotic setting, we develop a fairly complete picture for the limiting risk of the ménimum norm interpolator under the ERM setup in the two-layer RF regression model.
In particular, we provide clear picture of the bins-variance trade-off under this setup the number of samples n and the number of hidden features N diverge to infinity. For a better understanding of the main result, we present the following informal version of our main theorem:
Theorem 1.1 (Informal version of Theorem 3.9 Let Rminor denote the generalization risk of the minimum norm interpolalor (defined in 2.6
on the minorily group under the RF model. Under
2

---

Published in Transactions on Machine Learning Research (06/2025)
suitable distributional assumptions the input and hidden features, and regularity assumptions the octivation function, we have
lim E[Rminor] N,
RR + T2-
where the RF is the inductive bias (see Section of the RF model and r2 is the variance term due to the noise with veriance 7².
Minority group risk/ Inductive Bias
20.0
Minority group risk (Linear)
Linear
17.5
Bias (Linear)
RF
Minority group risk (RF)
Bias (RF)
Variance
10.0
7.5-
5.0
10
10. Overparameterizatin level
10
Overparameterization level
Figure Comparison of Minority group risk, inductive bias, and variance for linear and RF model un- der 20% minority population. The minority group risk monotonically decreases for the RF model the overparameterization level increases. However, for the linear model, the trend is increasing under the high overparameterization regime. The trend in variance is similar for both linear and RF models.
Section 3 provides a more detailed explanation of all the quantities. As Figure [ shows, a more detailed analysis reveals that the bias term RF does not increase with the level of overparameter- ization (N n)), which is in sharp contrast to with the linear model case. Also, the trend in the variance is decreasing for both linear and RF models. This shows that the trend of the risk in both cases is essentially influenced by the trend in the inductive bias. As result, the empirical risk of the RF model in the minority group stabilizes. This phenomenon can be attributed to the strong generalization property of the random feature model due to its highly non-linear architecture. It follows that under a suitable choice of the activation function, the RF model able to capture or explain all possible directions of the signal that turn does not allow the bias term deteriorate with increasing level of overparameterization.
Complete picture of limiting ERM risk under linear model: We also study the effect of the linear regression model. In contrast to the RF model, we show that the test error for the minority group worsens with the increasing level of overparameterization In particular, in the case of high signal-to-noise ratio, the prediction accuracy for the minority group worsens with the overparameterization of the model (See Figure 1. In particular, we show that the linear model fails to explain part of the signal that is orthogonal to the input features. This results in a higher inductive bias and it aggravates with the increasing level of overparameteeriaation. This highlights the limitation in generalization power of linear model compared to other non-linear models. We defer this discussion along with thcoretical analysis of overparamoterized linear models to Appendix A
overparameterization
Performance of majority group sub-sampling: In Section 3.1 we also analyze the effect of majority group subsampling on the minority group for the overparameterrzze RF model. We lay out concrete theoretical foundation that shows that majority group subsampling improves the minority group performance of the RF model in the overparameterized regime.
3

---

Published in Transactions on Machine Learning Research (06/2025)
Beyond the study of overparameterization in regression tasks with random feature models, we also provide simulation study for the effect of overparameterization on for classifications with random features. While the details of the simulation settings and the observations are deferred to Appendix B as a short description of the observations, we see that both the conclusions Sagawa et al. (2020) and Pham ct al. (2021)
can be replicated, depending on the simulation setting. II the difference between majority and minority
subgroups are small, then model overparamcterization generally helps with the performance on minority group, which is the same conclusion found in Pham et al. (2021). On the contrary, for larger differences, model overparameterizztion harm the performance of minority group, conclusion that is similar to SSaawa et al. 2020). While a detailed theoretical analysis on classification tasks may reveal the intricate reasons for such distinctions, such analysis is beyond the scope of the paper.
1.1 Comparison with prior work
Overparameterized ML models. The benefits of overparamettrization for average or overall perfor- mance of ML models are well-studied. This phenomenon is known "double descent", and it asserts that
the (overall) risk of ML models decreases as model complexity increases past certain point. This behavior
has been studied empirically AAdanii & Saxe (2017); ) Belkin et al. (2018); Nakkiran et al. (2019): Yang et al.
(2020) and theoretically Hastie et al. (2019); Montanari et al. (2020); Mei & Montanari 2019 Deng et al.
(2020g; Liu et al. (2024); Matt & Stöger (2025). All these works focus on the average performance of MI. models, while we focus on the performance of ML models on minority groups. The most closely related paper in this line of work is Pham et al.) (2021). where they empirically study the effect of model overpa- rameterization on minority risk and find that model overparametteriaation generally helps or does not harm minority group performance.
Improving performance on minority groups. There is a long line of work on improving the perfor- mance of models on minority groups. There are methods based on reweighing haaainng//ubaaapingggggg. (Shimodaira
(2000g; Cui et al. (2019) and (group) distributionally robust optimization (DRO) Hashimoto et al. (2018):
Duchi al. (2020); Sagawa et al. (2019). The DRO approach has also been adopted in Hu & Chen (2022)
to improve fairness in survival analysis. Soma al. (2025) proposed generalized DRO approach with faster
convergence rates and tighter bound over the worst-group errors. In parallel, Peet-Pare
Preealaae et al. (2022 adapt
performative predicition (Perdomo et al.] 2020 to DRO approach for more dynamic environments. Some
recent works have also adopted regularization frameworks (An et al.] 2022 Zhao et al. 2022), contrastive
learning (Park et al. 2022 Zhang et al.) 2022). and active learning (Wang et al. 2023 frameworks which uses proxy-sensitive information to induce fairness. In the overparameterized regime, methods based on reweighing are not very effective (Byrd Lipton| (201)), while subsampling has boen empirically shown
improve performance on the minority groups (Sagawa al. 20200 Idrissi al. 2021]. Our theoretical results confirm the efficacy of subsampling for improving the performance for minority groups and demonstrate that it benefits from overparameteriiation..
Group fairness. literature proposes many definitions of fairness (Chouldechova & Roth 2018), some of which require similar accuracy on the minority and majority groups (Hardt et al. 2016)). Methods for achieving group fairness typically perform ERM subject some fairness constraints (Agarwal et al, 2018 Kenfack al. 2023). The interplay between these constraints and overparameetrrization is beyond the scope of this work.
Paper organization. The rest of the paper is organized as follows: In Section 2 we introduce the data- generating model and the random feature model. We also discuss the majority group subsampling scheme under this setup. In Section 3 we present the main theoretical results of the random feature model related to the ERM risk and majority group subsampling along with simulation experiments. Finally, in Section discuss some of the implications of our work.
4

---

Published in Transactions on Machine Learning Research (06/2025)
2 Problem setup
2.1 Data generating process
Let X' C Idd be the feature space and y C R be the output space. To keep things simple, we consider a two-group setup. Let Po and P1 be probability distributions on X x V. We consider Po and P1 as the distribution of samples from the minority and majority groups, respectively. In the minority group, the samples (z, y) X x y are distributed
y x = 3/ +c, c, E(() = 0, var(c) = =², x; e are independent each other
(2.1)
and Px is the marginal distribution of the feature I, Bo Id is vector of regression coefficienss, and y2 E (0) o)) is the noise level.
In the majority group, the marginal distribution of features is identical, but the conditional distribution of the output is different:
y -|== =T= E(c) 0, var(c)=
(2.2)
where 31 E Rd is a vector of regression coefficients for the majority group. We note that this difference between the majority and minority groups is a form of concept drift or posterior drift: the marginal dis- tribution of the feature is identical, but the conditional distribution on this sctting because it not only simplifics our derivations, but also isolates the effects of concept drift between subpopulations through the difference 8 _ 31 Bo- If the covariate distributions between the two groups different, then overparameterized model may be able to distinguish between the two groups, thus effectively modeling the groups separately. In that sense, by assuming that the covariates are equally distributed, we consider the worst-case.
the output is different. We focus
Let 9: E (0, 1} denote the group membership of the i-th training sample. The training data ((. consists of a mixture samples from the majority and minority groups:
Vi. P
9i ~ Ber(M))
(2.3)
(Ia; yi) 9i ~ Pg:
where [ 1] is the (expected) proportion of samples from the majority group in the training data. We denote n as the sample size for majority group in the training data.
2.2 Random feature models
Here, consider random fcature regression model Rahimi & Recht 2007 Montanari et al. 2020
f(r, a, θ) E (0TT2//
(2.4)
where af) is a non-linear activation function and θ the N d weight matrix with 0g the jth row. In vector notation, Equation (2.4] can be written as f(x, a, θ) =a=z, z, where = (0xx Vd). In this vectorized notation, (+)) is applied coordinate wise. The random feature model is similar to a two-layer neural network, but the weights 0, 's of the hidden layer are set some (usually random) initial values. In other words, random feature model fits linear regression model on the response y using the non-linear random features {a(0] x/Vd)) (eng))))snstttteeeeee instead of the original feature vector These types of models have recently been used Montanari et al. 2020)) to provide a theoretical explanation for some of the behaviors observed in large neural network models.
2.3 Training of the random feature model
We consider two ways in which a learner may fit. the random feature model: empirical risk minimization (ERM) that does not require group annotations; subsampling that does require group annotations. The availability of group annotations highly application dependant and there a plethora of prior works
considering either of the scenarios (Hashimoto et al.] 2018 Zhai et al., 2021 Pham al. 2021] Sagawa
et al. 2019 2020
Idrissi et al. 2021
5

---

Published in Transactions on Machine Learning Research (06/2025)
2.3.1 Empirical risk minimization (ERM)
The most common way to fit predictive model is to minimize the empirical risk on the training data:
à E arg min aER
1 fy= f(1,, a, 0)}²
Σ
2
im1
(2.5)
arg min GERN
72 2 (vi (0.000aa./Va))
where we recall that the weights 8 s of the first layer are randomly assigned. The above optimization (.5 has unique solution when the number of random features or neurons less than the sample size (N < n) and we call this regime underparemettertaed. When the number of neurons is greater than the sample size (N > n), which call the oterpanameterized regime, à is not unique in (2.5). as there are multiple that interpolate the training data, i.e., Yi = Eド= ()])).))) i e [n], resulting in zero training error. In such a situation we set at a specific a E IN which (1) interpolates the training data and (2) has minimum h-norm. This particular solution is known as the minimum norm interpolant solution (MNIS) (Hastic et al. 2019 Montanari et al. 2020) and is formally defined as
RN
mniis arg min f1all2: Vi Σi (0) v\/Va), iE [m]] (ZT Z)
(2.6)
where 2 E R with Zr, o00 [00=z./Vā v./Va) and A is the Moore-Penrose inverse of A. The minimum norm
interpolant solution (26) has an alternative interpretation: it is the limiting solution to the ridge regression problem at vanishing regularization strength (Hastie et al. 2019 Section 2.2),
mmiss
lim a(A), ā(A) arg min eER
1 Σ {ur -0000,000
+ NA [al2
(2.7)
A-0+
In fact, the conclusion in 2.7 continues to hold in the underparameterized (N < n) regime. Hence, we combine the two regimes N < and N > n) and obtain the ERM solution as 2..7 In the literature
Hastie et al. (2019) it is known as the ridgeless solution. Finally, having an estimator a the response of individual with feature vector is predicted to be:
(z) f(z, a, θ) 90
(2.8)
2.3.2 Majority group subsampling
It known that a model trained by ERM often exhibits poor performance on minority groups Sagawa et al. (2019). One promising way to circumvent the issue is majority group subsampling: randomly drawn subset of training sample points is discarded from majority groups to match the sample sizes for the majority and minority groups in the remaining training data. This emulates the effect of reweighted-ERM,, where the samples from minority groups are upweighted in the ERM.
In the underparameterrizel case, the reweighting is preferred over the subsampling due to its superior statis- tical efficiency. On the contrary, in the overparameterized case, reweighting may not have the intended effect on the performance of minority groups [yrd Lipton| (2019). but subsampling does (Sagawa Idrissi al. 2021p. Thus, we consider subsampling the overparameterized regime. We note that the subsampling requires the knowledge of the groups/sensitie attributes, so its applicability is limited to problems in which the group identities are observed in the training data.
al.] 2020)
way of improving performance in minority groups in
3 Main results for the MNIS under random feature regression model
In this section we focus on the minority group performance of the MNIS estimator âmnis defined in Equation (2.6). From here on, we drop the subscript for notational brevity and simply use â denote the MNIS estimator, and we write the mean squared prediction error for the minority samples in the test data as
Ro (ā) E.z- px (1 0) - -80 x Ex-rx Px 11 (0) x/ x/Vd) x1
(3.1)
6

---

Published in Transactions on Machine Learning Research (06/2025)
Table
Table of notations
Notations
Descriptions
Total sample size
roo R1
Sample-sizes in minority and majority groups
N
Number of neurons
d
Covariate dimension
1
lim N/d
222
limn,d-rco n/d lim N/n (Oxerparametetzaation
7
Majority group proportion Regression coefficients for minorily and majority groups
So. 31
31 - -0 Noise variance
72
where x ~ Px independent of the training data. One may alternatively study (a (x, à, 0) -)- -) as the mean square prediction error for the minority group, but for fixed noise level it has the same behavior 3.1], suggested by the following: B(za)) (1fa)),,[[ff, ((, e) y} = Ex- [[f +) 8 x) 7
A finite sample study of Ra(a) 3.1 is difficult, in this paper we study its limiting behavior in high- dimensional asymptotic setup. Specifically, recalling the notations from Table I we consider N/d + th > 0, n/d + tb > 0 and n//n - π E [1/2, 1]. The problem underparameterized if y ///122 = (o
and overparameterized if > 1.
To study the minority risk, we first decompose into several parts and study each of them separately. The first decomposition follows:
x and the variance Y (So, 6, T) Lemma 3.1. We define the bias B d, Ex-Px (e, à, 9)]
Px [are Va à, O)}], where = (c1,..., (n) are the errors in the data generating model, i.e. y1 x Pn. and Var denotes the variance with respect to e. The minority risk decomposes as
El- a, 0) x1 (B,, + (Bo,
(3.2)
The proof is realized by decomposing the square in [f(z, 9) 3J x]²²²== = [1(,, à, a, [, 0)]} + [f. a, 0)] - 23 x312. We now provide an interpretation for the variance term.
The variance term. Firstly, we note that the torm (( 8, captures the average effect of variance of the prediction rule f(x a, Q) with respect to the feature distribution Px. In addition, let = = (Ow Vd) as the random feature for new covariates x and note that 7/ y(3,, 6, r) does not depend on the values of 8o or 8, shown in the following display:
Var. (f a, θ)} = Var (2 a} = Vare T (ZTz)
2) zT y} = Var (eeef==(zZAz ZZjZT..
(3.3)
Hence we drop 3n and 5 from the notation and write the variance term as (T). Since r,'s have variance 72 we realize that the exact dependence of the variance term with respect to r is the following: V(T) 2/ (r = 1). This also implies that the variance term (T) leases oul the contribution of dala noise level 7 in the minorily risk.
The inductive bias term. We notice that the term BB0, 8, in decomposition average bias in prediction for the minority group. We refer to this term as the inductive bias of the RF
3.2 represents the
7

---

Published in Transactions on Machine Learning Research (06/2025)
model induced by the MNIS estimator. Before we elaborate on this, first note that the bias term does not depend on the noise level т (both [e à, 0) and 8 x do not depend on 7) and hence, we denote it as ( 6) by dropping т. Next, we separate out the contributions of ßo and in the bias term via a decomposition. For the purpose we recall, Z = (2a.g (es.))iem))jE( E xR" is the matrix of random features and define the following: (i) X = [.] is the covariate matrix in training data, (ii) X1 is the covariate matrix consisting only of the samples from the majority group, and (iii) Z1 is the random feature matrix for the majority group. The decomposition for the bias term follows:
B(B 8) Eu Px 11(2 (Z' (Z Z) x ))0} + (2 (Z 2) )8,0 +2(2 x='80 x
(3.4)
(1 2)
The first terms on the right-hand side of 3.4) essentially capture the inductive bias due the misspecification in the random features model, i.e., the part of the signal x: (o that cannot be explained by the best fitted RF model. On the other hand, the remaining terms capture the part of the bias arising due to the model misspecification in the two- woo-gropp model. These two bias terms together contribute to the overall inductive bias in the RF' model. Next, we make some assumptions on So and ô vectors: Definition 3.2. The quantities Fg, Fs > O and F3.8 E R are defined as follows: F9 := 1150112, Fs == ||012 and F3,6 == (So. 5).
Here, F3 and Fs are the & signal strengths of 8o and 6. The decomposition in the next lemma separates out the contributions of 3o and 8 in (Bo, 8).
Lemma 3.3. Define the following:
Bg Ex~Px || (ZTZ)'z²= X - x 12/d|
B8 Ex ||| (z x/16/d]
(3.5)
6B. = 2EExxx
1( T (z 2)'z X - - (1 2)
Then we have B(o. 8) = F3 + FB + F3.666.6.
In Lemma 3.3 we see that F3 + 8,6 quantifies the contribution of the model misspecification in the two-group model, and we call this misspecification ervor term. For the other terms in the decomposition we utilize the results of Montanari et al.] (2020) who studied the effect of overparameterization on the overall performance (i.e., in a single group model), thus the misspecification error term does not appear in their analysis. Our theoretical contribution in this paper is studying the eract asymptolics of the misspeciffeatiin error terms. Before we present the asymptotic results we introduce some assumptions and definitions. First we make distributional assumption on the covariates is's and the random weights 0y's which allow for easier analysis of the minority risk.
Assumption 3.4. We assume that {a,)i= and (0,1,² are i.ddd ((d))) i.e., uniformly over the surface of a d-dimensional Euclidean ball of radius vd and centered at the origin.
Note that when Ie follows an isotropic standard Gaussian distribution, for large d one can approximate the distribution by Unif (Vā)), because d'1/2 | R Also, in practice, one can scale the features so that they reside on some scaled version of the unit sphere. The main reason to consider this simplifying assumption is that it lends us certain technical advantages while studying asymplotic convergence results.
Next we assume that activation function a has some properties, which are satisfied by any commonly used activation functions, e.g. ReLU and sigmoid activations.
Assumption 3.5. The activation function IR + R is weakly differentiable with weak derivative o' and for some conslands co > 0 il holds (a(u),, lo (w)) & clll
Both assumptions 3.4 and 3.5 also appear in Montanari et al. 2020 Note that the weak differentiability condition in Assumption 3.5 is fairly general and enjoyed by standard activation functions such as RELU or soft-max activation function. Below we define some quantities which we require to describe the asymptotic results.
8

---

Published in Transactions on Machine Learning Research (06/2025)
Definition 3.6. We define the following quantities:
1. For the constants
pa E{o(G)), A1 -=
= -B= -a mr.
(3.6)
where the expectation is taken with respect to G N(O,1) we assume that po: H1.Mi. oo and define { p1/
2. Recall that N/d + define
O, n/d 1/2 and ni [1/2, 1]. We set v min and x= Kiee --- 11 2)
100 -812-
26
We furthermore define the following:
60 xx + +3x (1i0)2 11 10 -2x - -x
+ (1i + $2 3-- + 1)x25*+2X* +x² 333 -0/8/2
& = v2x25* - - + 11 - 11
E 55 -3x45" + (th 1)x²(" +2x + +(- -
2
Equipped with the assumptions and the definitions we are now ready to state the asymptotics for each of the terms in the minority group prediction errors. The following lemma, which states the asymptotic results for Bg and y (r), has been proven in Montanari al. (2020).
Lemma 3.7 (Theorem 5.7, Montanari et al. (2020)) ). Let the assumptions 3.2
3.4 and 3.5 hold. Define
0 /80 and y 8g/ / where 60 & and 62 are defined in Definition 3. Then the following hold:
lima- B[Bp] B* sima [
where the expectation E taken over {- {
Trend in the variance term. We again recall that the variance term ((r) teases out the contributions of noise in the minority group prediction error. Here, the trend that we're most interested in is the effect of overparameterization. how does the asymptotic 7* behave as a function of t 2//2/22 = lim,, +o (N/n) when e/2 lim,,d+< (n//d) is held fixed, and 1. Although it is difficult to understand such trends from the mathematical definitions of y** we notice in Figure 2 (loft) that the variance docreases in overparamcterized
regime (7 > 1) with increasing model size (7))
0.6 n
45
Variance
Bias
135
Bias
n= 0.9
Figure 2: Trends for bias and variance in the minority ERM risk for varying overparameterization with d = 200 and 400. Left: variances for varying Middle: biases for different and where a is the angle between S and 3.. Here the majority group proportion π is set at 0.8. Right: biases for different a's and 7's when a is set at 180" The solid lines are the theoretical predictions and the points with error-bars represent the empirical predictions over 10 random runs.
Next, we study the asymptotics of Bg and Ez,s which, as discussed in lemma 3.3 quantify the part of the minority group prediction risk that comes from the model misspecification in the two group model (233

---

Published in Transactions on Machine Learning Research (06/2025)
Lemma 3.8 (Misspecifeattion error terms). Let the assumptions 3.2 3.4 and 3.5 hold. Following the
definitions of 8 and Y in lemma 3.7 we further define V/2 = B
11 2(x bb), where x and sb are defined
in Definition 3.6 Then
lim B[AB] = M11 _ t(1 x) + x 22 d
lim 3[6,,,] = Mar & x 7 - -1- -144) d=000
where in lemma 3.7 the expectation E is is taken over {r.) {0, }
Below we describe the observed trends in the bias term in the minority risk.
Trends in the bias term. Here, we study the asymptotic trend for the bias term (Bp. 8). We see from the bias decomposition in Lemma 3.3 and the term by term asymptotics in Lemmas 3.7 and 3.8 that the bias term asymplotically converges to F T* + +FG -Fg-Mi + F.. We mainly study the trend of this bias term in overparameteriee regime with respect to the three parameters: (1) the overparametrization parameter 7, (2) the majority group proportion т and (3) the angle between the vectors 8 and 31, which is denoted by The middle and right panels in Figure give overall understanding of the trend of the bias term with respect to the aforementioned parameters. In both of these figures we set || |1 = 1. In the middle panel, we see that for a fixed angle a, the bias generally decreases with overparameterriaatoon. In contrast, for a fixed level of overparameterization 7, the bias increases as the angle a increases. This is not surprising as intuitively, as the angle a grows, the separation between 30 and 81 becomes more prominent. As a consequence, the accuracy of the model worsens and that is being reflected in the plot of the bias term.
Similar trend can also be observed when the majority group proportion स varies for fixed angle a (right 180 As expected, the bias increases
most panel of Figure 2 Here we set 81 -50 i.e., the angle with parameter π for fixed because of higher imbalance in the population. Whercas, for fixed æ, the bias decreases with increasing 7. Thus, in summary, we can conclude that overparameterization generally improves the worst group performance the model for fixed instances of a and π.
In all the plots in Figure we also show the empirical variances and biases along with their error bars computed over 10 repetitions. Within each repetition, a random features model was trained with a dataset generated according to the model described in Section 2.1 The sample sizes and remaining data generating configuratoons, such as d, IT, a, and 7, of the training datasets are described within Figure 2 We then calculated their mean variances using Equation (3.3) and biases using Equation (3.4] over a test dataset with sample size 500 generated according to the same distribution. The same strategy is also used for Figure
The following theorem summarizes the asymptotic results in Lemmas 3.7 and 3.8 and states the asymptotic for minority group prediction error.
Theorem 3.9. Let the assumptions 3.4 and 3.5 hold. Following the definitions of H y Mi and Mi in Lemmas 3.7 and 3.8 have the term by term asymptotics:
lim E[ 88,
lim E (-))
d=
lim E| Mi, and lim E MGs d
d
and the ménority group prediction error has the following asymptotic
E[Ro(a)] = F3E + F + Fo.5 E (T ===10 1)]
F3 F3 M + F3 2
A complete proof of the theorem is provided in Appendix B
Trend in minority group prediction error for ERM. To understand the trends in minority group prediction error we combine the trends in the bias and the variance terms. We again recall that we're interested in the effect of overparameterization, i.e. growing y when & > In the discussions after Lemmas 3.7and andd 3.8 (and in Figure 2 we notice that both the bias and variance terms decrease or do not increase with
10

---

Published in Transactions on Machine Learning Research (06/2025)
0
0.6
45
0.7
Subsampling ERM
90
Subsampling ERM
#= 0.8
a 135"
77 0.9
a 180
-2-
10-1
10P
10
10a2
10*
10a
Y
Figure 3: Difference in minority risk between subsampling and ERM with d 200, 400. Left: the differences in minority risks for several values of a (the angle between 80 and (1) and overparameterzation 7 where majority proportion ı is set at 0.8. Right: the minority risk differences for several values of æ and when a is set at 180"
growing overparameeerrizaiion for any choices of the majority group proportion क and the angle a between Bo and $1. Since the minority risk decomposes to variance terms (3.2) notice that similar trends hold
for the overall minority risk. In other words, overparameterization improves or does not harm the minorily
risk. These trends agree with the empirical findings in Pham et al. 2021
3.1 Limiting risk for majority group subsampling
Though overparameterization of ML models does not harm the minority risk for ERM, they generally produce large minority risk Pham ct al. (0021) To improve the risk over minority group Sagawa et al. (2020); Idrissi et al. (2021) recommend using majority group subsampling before fitting ERM and show that they achieve ofate-of-the-rr minority risks. In addition, An et al. (2021) also theoretically showed that resampling produces superior results while training ML models under simple optimization setup. However, the authors do not investigate the asymptotic test error under more complicated model. Here, we complement these prior works with theoretical study on the minority risk for random feature models in the two group regression setup (2.3). We recall from subsection 2.3.2 that subsampling discards a randomly chosen subsample of size no from the majority group (of size n;) to match the sample size (na) of the minority group, and then fits an ERM over the remaining sample points. Here, we repurpose the asymptotic results for ERM (Theorem 3.9 to describe the asymptotic behavior for subsampling by carefully reviewing the changes in the parameters 1 th2 and æ. First we notice that the total sample size after subsampling is ns = 2no (both the majority and minority groups have sample size no,s 11,55 = no) which means the new majority sample proportion is gg no,5/ng 1/2. We further notice that 14 remains unchanged, i.e., $1.s 1/1 whereas d lima-+00 n/d changes to 4/2,5 = lima-x ns/dd = lim 2no/d = lim 2(no/n) (n/d) 2(1 2) The subsampling setup is underparameterizeel when nış = 2no > N or 7s = v1..//v2. = 7/(2 and overparameterized when 7s > 1. Rest of the setup in subsampling remains exactly the same as in the ERM sctup, and the results in Lemmas 3.7 3.8 and Theorem 3.9 contimue to hold with the new parameters: 11.s, V2.3 and Ts.
m)) <
ERM vs subsampling. We compare the asymptotic risks of ERM and majority group subsampling in overparameterized setting. Figure 3 shows the trend of the error difference between subsampling and ERM. Similar to the previous discussion, we denote by a the angle between B1 and Åo, and set || = ||ol||2 = 1. In the left panel, we fix the majority group parameter ग = 0.8 and vary a. The empirical quantities are also evaluated in the similar fashion as in Figure [] It is generally observed that subsampling improves the worst group performance over ERM, which is consistent with the empirical findings in Sagawa et al. (2020) and [drissi al.] (2021). Morcover, the improvement is prominent when the group regression coefficients are well separated, i.e., a is large. Intuitively, when is very large, the distinction between majority and
11

---

Published in Transactions on Machine Learning Research (06/2025)
minority groups is more noticeable. As consequence, the effect of under representation on minority group becomes more relevant, due to which the worst group performance of ERM is affected. Whereas, subsampling alleviates this effect of under representation by homogenizing the group sizes, On the other hand, the effect of under representation becomes less severe when 3n and 31 are close by, i.c., a is small. In this case, we see subsampling does not improve the worst group performance significantly In fact, for larger values of y, subsampling performs slightly worse compared to ERM. This again not surprising, as for smaller separation, the population structure of the two groups becomes more homogeneous. Thus, full sample ERM should deliver better performance compared to subsampling.
Similar trend is also observed in the right panel where we fix a 180 and vary я. Again, improvement due to subsampling is most prominent for larger values of which corresponds to greater imbalance in the data. Unlike the previous case, the subsampling always helps in terms of worst group performance but the improvement becomes less noticeable with decreasing ι as overparameterization grows.
4 Summary and discussion
In this paper, we studied the performance of overparamcterized ML models on minority groups. We set up a two-group model and derived the limiting risk of random feature models in the minority group (see Theorem 3.9 In our theoretical finding, we generally see that overparameterriation improves does not harm the minority risk of ERM, which complements the findings in Pham et al. (2021). We also show theoretically that majority group subsampling is an effective way of improving the performance of overparameterized models in minority groups. This confirms the empirical results on subsampling overparameterized neural networks given in Sagawa et al. (2020) and Idrissi et al. 20211.
Assumption implications. In our study, an important assumption is that the feature distributions be- tween the two groups are the same (Concept drift, see Section 2.1). This is the worst case for the minority group because an overparameterized model can not loverage any difference in the feature distributions be- tween the groups to improve minority group performance. It is encouraging that overparanedderiaation does not hurt the minority group even under this worst-case setting. Extending our results to the two-group setup with a shift in feature distribution is an interesting future work direction.
Practical implications One of the main conclusions of this paper is that overparameterization generally helps or does not harm the worst group performance of ERM. In other words, using overparameterized models is unlikely to magnify disparities across groups. However, we warn the practitioners that overparam- eterization should not be confused with method for improving the minority group performance. Dedicated methods such group distributionally robust optimization (Sagawa et al. 2019 and subsampling Sagawa ct al. (020): Idrissi et al. (2021) are far more effective. In particular, subsampling improves worst-group performance and benefits from overparameterivation as demonstrated in prior empirical studies Sagawa et al.
2020 ; Idrissi et al. 2021 and in this paper.
References
Madhu S. Advani and Andrew M. Saxe. High-dimensional dynamics of generalization error in neural net- works. arxiv:1710 0.06666777777 (physics, q-bio, statj, October 2017.
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A Reductions Approach to Fair Classification. urXiv:1803.02. [es], July 2018
Bang An, Zora Che, Mucong Ding, and Furong Huang, Transferring fairness under distribution shifts via fair consistency regularization. Advances in Neural Information Processing Systems, 35:32582-3222 -22597, 2022.
Jing An, Lexing Ying, and Yuhua Zhu. Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients. International Conference on Leurning Represenlations, 2021.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning practice and the bias-variance trade-off. arXiu: 1812. 11118 fes, stat], December 2018.
12

---

Published in Transactions on Machine Learning Research (06/2025)
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. ar: (es, stati, March 2019.
Joy Buolamwini and Timnit Gebru. Gender Shades: Intersectional Accuracy Disparities Commercial Gender Classification. In Proceedings of Machine Learning Research, volume 87, pp. 77 7-91, 2018
Jonathon Byrd and Zachary Lipton. What is the Effect of Importance Weighting in Deep Learning? In International Conference on Machine Learning, pp. 872-881. PMLR, May 2019.
Alexandra Chouldechova and Aaron Roth. The Frontiers of Fairness in Machine Learning. arXi::18000888 (es, statj, October 2018
Y. Cui, M. Jia, T. Lin, Y. Song, and S. Belongic. Basss-Balnnced Loss Based on Effective Number of Samples. In 2019 IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CVPR), pp. 9260 9269, June 2019, doi: 10.1109/ M0.109/CVVP.2011 CVPR.2019.00944
Zeyu Deng, Abla Kammoun, and Christos Thrampoulidis. A Model of Double Descent for High-dimensional Binary Linear Classification. arXiv: 1rxii=191110 191919911122 [cs, cess, stati, May 2020.
John Duchi, Tatsunori Hashimoto, and Hongscok Namkoong. Distributionally Robust Losses for Latent Covariate Mixtures. arXic:2007.138: 13982 [cs, statj, July 2020.
Moritz Hardt, Eric Price, and Nathan Srebro. Equality of Opportunity in Supervised Learning. arXiv:1610.084 0.3113 [es}, October 2016.
Tatsunori B. Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness Without Demographics in Repeated Loss Minimization. arXic:1806. 08010 fes, stat], June 2018.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in Bigh-Dimensional Ridgeless Least Squares Interpolation. 1XXi::1033088.0 fes, math, statj, November 2019.
Shu Hu and George H Chen. Distributionally robust survival analysis: A novel fairness loss without demo- graphics. In Machine Learning for Health, pp. 62 87. PMLR, 2022.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive wonst-group-accuracyy arXiv preprint arXim:2100. 14503, 2021.
Patrik Joslin Kenfack, Samira Ebrahimi Kahou, and Ulrich Aivodji. Fairness under demographic scarce
regime. arXiv preprint orXiv:2307.1381, 2023.
Alan Le Pham, Eunice Chan, Vikranth Srivatsa, Dhruba Ghosh, Yaoqing Yang, Yaodong Yu, Ruiqi Zhong, Joseph E Gonzalez, and Jacob Steinhardt. The effect of model size on worst-group generalization. Tn NewrIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Fanghui Liu, Leello Dadi, and Volkan Cevher. Learning with norm constrained, over- pee-parramtterized two- layer neural networks. Journal of Machine Learning Research, 25(138):1 42, 2024.
Hannes Matt and Dominik Stöger. Linear regression with overparameterived linear neural networks: Tight upper and lower bounds for implicit II-regularization. arXiv preprint #xXiw:2566001443 2025.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 2019.
Andrea Montanari, Feng Ruan, Younttak Sohn, and Jun Yan. The generalization error of max-margin lincar classifiers: High-dimensional asymptotics in the overparametrized regime. arXiw:1911.0151/ [math, stat), July 2020.
13

---

Published in Transactions on Machine Learning Research (06/2025)
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep Double Descent: Where Bigger Models and More Data Hurt. orXiu: 112.0222992. (es, statf, December 2019.
Sungho Park, Jewook Lee, Pilhyeon Lee, Sunhee Hwang, Dohyung Kim, and Hyeran Byun. Fair contrastive learning for facial attribute classification. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10389-10398, 10398, 2022.
Liam Peet-Pare, Nidhi Hegde, and Alona Fyshe. Long term fairness for minority groups via performative distributionally robust optimization, 2022. URL
Juan Perdomo, Tijana Zrmic, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. In PMLR, 2020.
International Conference on Machine Learning, pp.
Alan Pham, Eunice Chan, Vikranth Srivatsa, Dhruba Ghosh, Yaoqing Yang, Yaodong Yu, Ruiqi Zhong, Joseph E Gonzalez, and Jacob Steinhardt. The effect of model size worst-group generalization. arXiv preprint ftfe:2111.0409/ 2021.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Proceedings of the 20th International Conference Neural Information Processing Systems, NIPS'07, pp. 1177-1184, Red Hook, NY, USA, December 2007. Curran Associales Inc. ISBN 978-1-60560-352-0.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally Robust Neu- ral Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. arXi::191100778 [cs, stat/, November 2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An Investigation of Why Overparam- eterization Exacerbates Spurious Correlations. arXiv:2005.043 04345 [cs, stat), August 2020.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90(2):227 244, October 2000. ISSN 0378-3758.
doi: 10.1016/S0330 (0777883555
Tasuku Soma, Khashayar Gatmiry, Sharut Gupta, and Stefanie Jegelka. Near-optimal algorithms for group distributionally robust optimization and beyond, 2025. URL http::////aariv.o/aaa/////// 22tpg22..//22/......./ 13669
Guanchu Wang, Mengnan Du, Ninghao Liu, Na Zou, and Xia Hu. Mitigating algorithmic bias with limited annotations. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 241-258. Springer, 2023.
Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern. Predictive arXiv:1902.1 10097 fes, stat], February 2019.
Inequity in Object Detection.
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking Bias Variance Trado-off for Generalization of Neural Networks. In International Conference on Machine Learning, pp. 00767-10777. PMLR, November 2020.
Runtian Zhai, Chen Dan, Zico Kolter, and Pradeep Ravikumar. Doro: Distributional and outlier robust optimization. In International Conference on Machine Learning, pp. 12345 -22355. PMLR, 2021.
Fengda Zhang, Kun Kuang, Long Chen, Yuxuan Liu, Chao Wu, and Jun Xiao. Fairness-aware contrastive learning with partially annotated sensitive attributes. In The Eleventh International Conference Learn- ing Representations, 2022.
Tianxiang Zhao, Euyan Dai, Kai Shu, and Suhang Wang. Towards fair classifiers without sensitive attributes: Exploring biases in related features. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 4433-1442 2022.
14

---

Published in Transactions on Machine Learning Research (06/2025)
A Ridgeless regression
Similar to Section 3, here we describe analyzing the lincar regression model in the overparameterized regime. The learner fits a linear model
f(x) - 3Fx
the training data, where 3 E Idd is a vector regression coofficients. We note that although the lincar model is well-specified for each group, it is misspecified for the mixture of two groups. The learner fits linear model to the training data in one of two ways: (1) empirical risk minimization (ERM) and (2) subsampling.
A.1 Model fitting
Empirical risk minimization (ERM) The most common way of fitting linear model is ordinary least squares (OLS):
DOtS E arg min Ei= - 3F-x, arg min 2n lly X
where the rows of X E Rxd are the xt's and the entries of R= are the 's. We note that ERM does not use the sensitive attribute (even during training), so it is suitable for problems in which the training data does not include the sensitive attribute.
In the overparameterized case (n < d), BOlS is not well-defined because there are many linear models that can interpolate the training data. In this case, we consider the minimum & norm (min-norm) linear model with zero training error:
Buin E arg (1l
x = (XT (XTx)xX'
where xx denotes the Monro-Penrose pseudoinverse of x X. The min-norm least squares estimator is also known as the ridgeless least squares estimator because Amin lim Bx where
3x arg min minl fly 3 + B1
We note that if X has full column rank (i.e. X is non-singular), then 8min is equivalent to Bots.
Majority group subsampling It is known that models trained by ERM may perform poorly minority groups Sagawa et al. (2019). One promising way of improving performance minority groups is majority group randomll discarding training samples from the majority group until the two groups are equally represented in the remaining training data. This achieves an effect similar to that of upweighing the loss on minority groups in the underparameterizee regime.
In the underparameterized case (n > d), reweighing more statistically efficient (it does not discard training samples), subsampling is rarely used. On the other hand, in the overparameterized case, reweighing may not have the intended effect the performance of overparameterized models in minority groups [yrdd & Lipton (2019), but Sagawa el al. (2020) show that subsampling does. Thus also consider subsampling here as way of improving performance in minority groups. We note that reweighing requires knowledge of the sensitive attribute, so its applicability is limited to problems in which the sensitive attribute is observed during training.
A.2 Limiting risk of ERM and subsampling
We are concerned with the performance of the fitted models on the minority group. In this paper, we measure performance on the minority group with the mean squared prediction error on a test sample from the minority group:
RB( A E[((3 x2)2
x E[(B -
E(B 30) x].
(A.1)
We note that the definition of A.1 is conditional on X; i.e. the expectation is with respect to the error terms in the training data and the features of the test sample. Although it is hard to evaluate Rg exactly for
15

---

Published in Transactions on Machine Learning Research (06/2025)
finite and d, it is possible to approximate it with its limit in high-dimensional asymptotic setup. In this section, consider asymptolic setup in which n, - oo in a way such that d - (0, oo). If y 1 the problem is underparameterizet; if y > 1, it is overparameterized.
To keep things simple, we first present our results in the special case of isotropic features. We start by formally stating the assumptions on the distribution of the feature vector Px.
Assumption A.1. The feature vector Px has independent entries with zero mean, unit variance, and finite 8 te moment for some 0.
A.3 Empirical risk minimization
We start by decomposing the minority risk (A.1) of the OLS estimator BOLS and the ridgeless least squares estimator 8min Let no and ny be the number of training examples from the minority and the majority groups respectively. Without loss of generality, we arrange the training samples so that the first no examples are those from the minority group:
Xo Xi
yo y1
X
y=
Lemma A.2 (ERM minority risk decomposition). Let & be either Bots Pmin. Ro(3) [[ (o) (g 00 (1a - 110 + =² Th(Ett +
We have
T [0)]
Onoxd Xi
(5t)(x+
Onoxd
/noj
Xi
(A.2)
+ +5 { [nnxxd
X/n} {/n)(51)355) 2
where 11x & X is the projector onto ran(XT), ), 8 a 80 B1, and E e 1 X is the sample coveriance matrix of the features in the training data.
Inductive bias We recognize the first term on the right side of A.2 as a squared bias term. This term reflects the inductive bias of ridgeless least squares: it is orthogonal to ker(X), so cannot capture the part of 3o in ker(X). We note that this term is only non-zero in the overparameterized regime: BOLS has no inductive bias
Lemma A.3 (astii et al. (2019), Lemma 2). In addition to Assumption A.1 assume |100 = 8o for all
n, d. We have
0 (Ta xx) 3 s0 (1 })} V as n. d + 00, 7-
Variance
The second term on the right side of A.2 is a variance term The limit of this term in the hige--imensional asymptotic setting is known.
Lemma A.4 (Hastic et al. 2019). Theorem 1, Lemma 3). Under Assumption A.1 we have
<< 1
d
P
>>1 n, Xi, 7-
n
Approximation error The third and fourth terms in A.2 reflect the approximation error of BOLS and Amin because the linear model is misspecified for the mixture of two groups. Unlike the inductive bias and variance terms, this term does not appear in prior studies of the average/overall risk of the ridgeless least squares estimator Hastic al. (2019).
Lemma A.5. In addition to Assumption A. assume || = r and 8 Bo c for all n, d. As d we have
( O0 x
(5))(
T
Onoxd X1
/m]6 Ps
r
ri-
xy
+ 22 (1)2)
<<>
X1
( + x (-
<>1
-(--1)
16

---

Published in Transactions on Machine Learning Research (06/2025)
7
7-
π= 0.6
a 0
6-
6-
n=
ar = 45
n= 0.8
a = 90
ERM
n= 0.9
ERM
a 135
4-
180
10-1
10
10-4
10a
10A
V
Y
Figure 4: Minority group prediction error for the ridgeless least square ERM in the isotropic features
The left plot considers the setup with varying when the angle between 8 and B| is set at 180% The right plot sets π = 0.8 and considers different values of a. Here the SNR is set at || 1 = 10.
and
8T {
Onuxd X
T
(5 & (1((//))
Before moving on, note that (the limit of) the approximation error term is the only term that depends on the majority fraction æ. Though the inductive bias may increase at overparameterized regime (7 > 1) with growing when the SNR (= |/001//²)) is large, we notice that the approximation error terms always decrease with growing overparameterization. In fact they tend to zero as y + 00.
We plot the (the limit of) the minority risk prediction error (MSPE) Figure 4 In both overparameterized and underparameterizedd regimes, the MSPE increases as æ increases. This is expected: as the fraction of training samples from the minority group decreases, expect ERM to train model that aligns more closely with the regression function of the majority group.
We also notice that in the overparameterized regime (y > 1) when the SNR (118212/7²) is high the MSPE increases with growing 7 if the two groups are aligned (the angle between Bn and 31 is a 0). This trend is also observed in Hastie et al. (2019)..
A.4 Inadequacies of ridgeless least squares
There is one notable disngreement between the asymptotic risk of the ridgeless least squares estimator and the empirical practice Pham et al. (2021) in modern ML: the risk of the ridgeless least squares estimator increases as the overparameteriiation ratio y increases, while the accuracy of modern ML models generally improves with overparameterization. This has led ML practitioners train overparameterized models whose risks exhibit double-dcscent phenomenon Belkin et al. (2018). Inspecting the asymptotic risk of ridgeless least squares reveals the increase in risk (as y increases) is due to the inductive bias term (see A.3) For high SNR problems (3o > 7²), the increase of the inductive bias term dominates the decrease of the variance term (see A.4), which leads the overall risk to increase. This is a consequence of the fact that problem dimension (the dimension of the inputs) and the degree of overparameterization are tied to ridgeless least squares. In order to elucidate behavior (in the risk) that more closely matches empirical observations, we study the random features models, which allow us keep the problem dimension fixed while increasing the overparameterization by increasing the number of random features.
A.5 Majority group subsampling
We note that the (limiting) minority risk curve of majority group subsampling is the (limiting) minority risk curve of ERM after change of variables. Indeed, it is not. hard to check that discarding training samples
17

---

Published in Transactions on Machine Learning Research (06/2025)
Subsampling ERM
Subsampling ERM
= 0.6
a= 0
= 0.7
0 45
= 0.8 == 0.9
a 90
a= 135
a= 180
10
10"
10'
10
10
Y
Y
Figure 5: Minority error differnce between subsampling and ERM for the ridgeless least square in the isotropic features case. The left plot considers the setup with varying a when the angle between Bo and Bi is set at 180 The right plot sets ı 0.8 and consider different values of a. Here the SNR is set |/ = || 112/ 5.
from the majority group until the groups are balanced in the training data leads to reduction in total sample size by factor of 2(1 - m) (recall π [ 1]). In other words, if the (group imbalanced) training data consists of n samples, then the (group balanced) training data will have 2(1 - т) training samples. In the n, + oc, 1 - y limit, this is equivalent to increasing Y by a factor πパ-жx The limiting results in ERM can be reused with the following changes: under subsampling (1) changes to 1/2, (2) y changes to 7/(2- - 2m), and (3) the rest of the parameters for the limit remain the same.
In Figure 5 where we plot the differences between subsampling and ERM minority risk, we observe that subsampling gencrally improves minority group performance over ERM. This aligns with the empirical findings in Sagawa et al. (2020); Idrissi al. (2021). The only instance when subsampling has worse minority risk than ERM is when the angle between 30 and 81 is zero, i.e. 8o = 31. This perfectly agrees with intuition; when there is no difference between the distributions of the two groups then subsampling discards some samples for the majority group which are valuable in learning the predictor for the minority group, resulting in an inferior predictor for the minority group.
A.6 Proofs of Lemmas A.2 and A.5
A.6.1 Proof of Lemma A.2
Denoting U = (XTX/n) X/n we note that the estimation error - Bo is
Onoxd] X1
Onoxd
i 8-3- U(X +
8 + - :o = =ITxx 12). + U
5 ++
X1
18

---

Published in Transactions on Machine Learning Research (06/2025)
have the following decomposition of ERM minority risk
Ro(
(B [3- - ( - [00]
{
Ongxx x
Ono X1
=R (1Ix - To)30 +
8+ Ue
( - I2) +
+ Uc
X1
Ongxa X1
Ongxd X1
30 (Ilx 3.0 + B[ (U)
Uq +oT
(U)
8+
I1
1II
Onoxd x X1
Ong xd Xi
2
(U) (U) (Ixx Ia) + 2
(TTx
Ia) + U
8
Uc
IV
The term is equal to S (IIx - Lo because 11x - La is projection matrix. By the linearity of expectation and trace operator, the term II is equal to
B[Tr(eT Ue)] = Tr St 7 - S -
(TE))
n
By properties of Moore Penrose pseudoinverse, the term III and IV are equal to
8T{ { Onoxd] X1
X/n) X
Ong X1
/n and 20 {
Ong X1
X/n x/n](E1) ($1)3
respectively.
The term V is 0 since E(e] = 0. Hence we complete the proof.
A.6.2 Proof of Lemma A.5
We first prove for the cross term that
B xT
(A.3)
To realize the above we notice that for any orthogonal matrix O it holds
E E E{}} x xT.0 E oΣ (E)) x0
where X is replaced by Xo = XO and the replacement in X changes the covariance matrix to Eo and it's Moore-Penrose inverse to St 0. Noticing that X Xo, E a Eo and St d Et we conclude
B13 oΣ ( (0, X0, To = B O
1 X XT (o.x7)) n
We now let O to be uniformly distributed over the set of all d x orthogonal matrices (distributed according to Haar measure) which results in
Ei8
1 72
BMT E(5 X (x,x 580 7 E|T Σ X (8T8)))))
To calculate the term
T E(}1}} (T(0, (T
19

---

Published in Transactions on Machine Learning Research (06/2025)
notice that
R[TH E(ET)(( 1 x X1)
1
Σ B ((;,)(-);;;))
d
1=10+1
and each of the terms to conclude
Tr( Σ(E)) (.a!) are identically distributed. We appeal to exchangability argument
Σ T (1 nz [ -,,,))= i=
(
T 2(E)) C3. )
d
d
= B[T Σイン)
where we notice that ΣΣ is a projection matrix and conchude
1 ETT
ブインッ = 1 F(TT ΣΕ
If d < i.c. 7 < then Σ is invertible and and it holds ΣΣ Ia. Tf d > n then there are exactly n many non-zcro cigen-values in ΣΣ and it holds Tr ΣΣ n. Combining the results we obtain
B = d
Now, to calculate the term
E {10,
we first use similar argument to
A.3 and obtain
BiO X, ) (x (0,x = x X1
X X1 H
Σ
n
n
We rewrite
XTX,
XIT X1
Tr{
= n2 Fi, + mn - F1.2
where we define
{ - H
Fi.1 4
B[r
F1, A
Tr
x
E1 H
n
n
To calculate nF notice that
nF1.1
7
n
E
T1 (St)
22 }]
1 a a xT (E
7
Denoting 44 as the L1 convergence we notice that Iıs 4 d.e. [[[[|]] -- 0. We now calculate
the convergence limit for (St)am Noticing that
z 2T(1))221 lim
2
E 2Ia) -2
n
2+0 n
we write E + 2lla = (xισ) /n) + Ag, where A₂ = = n + z1g. Using the Woodbery decomposition
- /n)
((11p]/)) = A -1
1 + 2 1
20

---

Published in Transactions on Machine Learning Research (06/2025)
obtain that
2 A72r1 2x1
- -x =Ta)
2x
n
(1 + 2 A x1
In Lemma A.6
we notice that
E 1 A
7 <1
- - x1
(1-7)a
<
as d 00.
71
> 1
1リ-
ア>
- T A var
A,
var -
1 T
A,3x1)
+
which implics Fin converges to
7 9<1
nF1,1 4
7(-1) 7>1
Noticing that
d EEIT ΣETH
1
E|TT Σ(Σ))
nF + n(n 1,
d
we obtain the convergence of n(n - 1),2 as
1-27
n(n - n(n-1)F112 4
T-y
9<1
7-2
7>1
7(7-1)
which finally yields
x X1 (Et 2 X1
B
= n1Fi,1 + 11 - 1)F1.
x xnFi,1 + + 1)F2
2(1-11) <<1
(7-22
1>1
7-1 7(1-1)
Lemma A.6. As + 0+ and d have
<<
E
x1
T
Ag
21
n
>>1
7<1
E
31
T
A
11
(1-
n1
17-153
7>1 1
and
1 T A
var
æ1).
varl
A 3x1 +0
n
n
Proof of Lemma A.6 To prove the results about variances first establish that for any symmetric matrix B ((by)) Baxd it holds
var(e] Bay) S cTTr(B2).
(A.4)
for some c > 0. Writing Iı = (u....., ua)T we see that
var(x] Bxi) = var( Eis biubbiz)
=E bybbrcor(uu,, ux))
=
Noticing that
cov(u,u u1) = Eluuuunul E|,,,,]EB [[ugan
that
21

---

Published in Transactions on Machine Learning Research (06/2025)
1. If one of i,j, k is distinct from the others then cov(u,uy, u.4u = 0.
2. If i = and & = and i then
Bluuy ui - 1 E[ufu]] E Blu E [ 0.
3. If j= h: = then co,(,, u = var(uf).
4. If (i- = and = 1 and i + j} or (i = 1 and = and i i j} then we have
E Elu,uyugun - Muuuu,BBluuee var(u:u;) = var(uru2).
Gathering the terms we notice that
var(z; BrT)
8 Σ
var var(n;)))3 + Σ by var(usuz)
by cTr(B3).
(,
where c 2(var(uf) V var(u1u2)).
Using
A.4
we notice that
lim var(x] A-5x1/n) ////)= 2-0
1
TT( 101
x {(()))) - 0.
n²
We notice that
E
x- A,
m1
(4,1)
TT
n
n
n
which appears in the variance term in Lemma A.4
and we conclude that
1
A
as d + co.
>>1 1
To calculate
IF
T
2
- T
consider case by case.
For < we see that
1 Ir((St))) = - -2
n
+7 1 / (s)
The Stieltjes transformation of May is
s7(2)= =
T d11((x) (x)
? 2
(1 7 - 2) 472
V7.1 -EC11--7.1++ V
27/2
For |2| < 1 - VT
(2( =
(x)(
1
1
duy
2/20
80
I 0=
22

---

Published in Transactions on Machine Learning Research (06/2025)
Hence, we have
d (8y())) E[1/x]].
lim 2- 2
dz
Now,
1-7 1---22 2 - 472
=1 =1-1- (1+ + 27 22 + 222 472)) 1/2
=1-7- ((1 + - 27z) 1/2
x1 x1-9- - (1 - 7) + 2- 22 - 272
(2² 22 - 272)2
as 2 - 0
2(1 - 71)
8(1 - -4
2- 22 2/22
2² + 7)2
x1-7-2 (1 7) +
as 2 - 0
2(1 7)2
2(1 7)
22 1 7)
2 47z
2(1 7))3
2(1 7)
22
27z
+
(1-7)3
1-7
which implies
+ 1
8y(2)= x (1 - 7)3 1-7
This establish that
TT((5)))) = ? lim
d
(s,(2))
7
n
--0 dz
(1 - 7)3
For 7 > we notice that
TT((2)) = Σ
where sy's are non-zero eigen-values of X x X/n. This is also equal to
Σ
PP i=1 2
where 4 the eigen-values of the invertible matrix XX /d. Hence we have
TT((E)))) 1 Tr((XXT/a)) /d) + 72 + dz (51/s(2)) = (7 1)3
d
7
lim
722 n
We combine the limits for y and y to write to write
E 1 A 2x1 +
[-- 7
as - x0.
n
17-1)3
? 1
m
B Proof of Theorem 3.9
Before diving into the main proof give rough sketch of the whole proof. Essentially, the proof has three major components.
In the first part, we establish the limiting risk where we treat S and 8 uncorrelated random variables. (Section B.1
23

---

Published in Transactions on Machine Learning Research (06/2025)
2. In the second part of the proof establish the necessary bias-variance decomposition under or- thogonality of the fixed parameters Bo and 8. (Section B.2
3. In the final part, we use the results of previous two parts to establish the limiting risk result for Bo and ò in general position. (Section B.3 and B.4
B.1 Limiting risk for uncorrelated parameters
We start this section with some definitions of some important quantities. Definition B.1. Let the functions vn; v C+ C+ be uniquely defined by the following conditions: (i)
Vi, V2 analytic on C+. (i) For Im({) 0, vn (S) and uz(C) satisfy the equations
-1
V1 1/1
- - 12
62
8²v1
V2 1/1
-
(2
(ii) P2(S)) is the unique solution of these equations with
(n((0) s w//Tm(()) (2((O) S w2/ [m(()) for Tm(() > C2
with C sufficiently large constant.
Let
x= u1
and
x5 + + (01 1i v/2 1)
-2x -x3 - 3ei + 1 2x²
+x² 33++
GE, 12, = 22 - +
+1212,
12, A) 26 -xx (t) 36 + 22x + +32
+ (- - 1 - 2x² - x2².
We then define
Ei 1/,, 1/2,
Bridge (E, 1h1, 1/2, A)
(B.1)
6o(E, th, 1/22 A)
E2 1/1, 1h2 A)
Veidge (E, 1/1 1b2, A)
(B.2)
Eo 1/1, 1/2, A)
The quantities derived above be casily derived numerically. But for our interest, need to focus the limiting case when X + 0. It can be shown that when X + 0, the expressions of Bridge and Vridge reduces to the form of 3* and y defined in Lemma 3.7 respectively. For details we refer to Section 5.2 in Mei & Montanari 2019).
Now we are ready to present the proof the main result.. Recall that
a(x) = arg m
Σ Vi Σ
N 00 x,
wv/Va))
NA + d
|all2
By standard lincar algebra it immediately follows that
a(A) Vā
(Z Z + X 1
TZ
(B.3)
24

---

Published in Transactions on Machine Learning Research (06/2025)
vi where Z = oxx /vd) d1,
d and d22 a- Also, define the square error loss (x, = (x Bu (A); where r is a new feature point.
For brevity, we denote by the tuple (X, 0, Er [ 6, A)], where x is a new feature point from minority group. The main recipe of our proof is the following:
20,00, 6). Recall that we are interested the out-of-sample risk
We first do the varianar decomposition the expected sut-of-sample-rikk.
We analyze the bias and variance terms separately using Lechniques from random matrix theory.
Finally, we obtain asymptotic limit of the expected risk by using the asymptotic limits of bias and variance terms.
Bias variance decomposition
The expected risk new point coming from the minority group can be decomposed into following way:
Er Rar (2,, X, G, Er (xr Bo f(z; (();
E. Bo E. [x, a(A);
(B.4)
Bias term Var Var.Eraraaae
Er Variance term
a()
The above fact follows from Lemin 3.3, which basically uses the uncorrelatedness Bo and 6. Now we define the matrix
(Z T 7 + Tx)
(B.5)
In the next couple of sections we will study the asymptotic behavior of the bias and variance terms by analyzing several terms involving V. In order obtain asymptotic limits of those terms, will be borrowing results of random matrix theory in Mei & Montanari 2019
Decomposition of bias term
We focus the bias
B(6,A) = Er [ [TTT 30 - [=(((=((A)))))
(B.6)
= EE
xx TTTT--o(Oxi) ((er/aa)
Vd T
X + V2 X
It turns out that it rather difficult to directly analyze the bias term B(5 A). The main difficulty lies the fact that the terms involving 3o and are coupled the bias term and poses main difficulty in applying well known random matrix theory results directly. Hence, in order to decouple the terms we look at the following decomposition:
B(6, A) = B(8, A) - B)0, B(0,
(1)
(1T)
It is fairly simple to that term (II) is only function of Sp. Next, we will show that term (I) does not contain 8g. To this end, We define the matrices Ü,U U NxN as follows:
F===(00 [0] x/Va)a(0 s/Va), Ug = B(OG) V(. E [N] (N].
(B.7)
25

---

Published in Transactions on Machine Learning Research (06/2025)
Next, due to Assumption 3.2, it easily follows that
B X) - B(0, X)
8o [= ( V + =2
= Er
Vd
- (T70-o(00//)) vd (
( x860]²
Ey
(B.8)
Er
LE
[Tr(X] 2,
F E (Tr (x] /,00 xi)}
d²
Above display shows that term (I) does not contain Sg. In the subsequent discussion, we will now focus on obtaining the limits of term (T) and (IT) separately.
Next, by the property of trace, we have the following:
{
Tr(x] ZUU X1) = Tr
Σ
YUM
Σ 2
Σ
1=n++1
i=
Σ
Tr (VUV2,3,
1= -=
Ij Tr (V
(B.9)
n +=
a Tr (1
Σ
JJ
Thus, by exchangeability of the terms (Tr (VU ) and [x] Ir (WUV2y2) Iiatisisgnn we have
[T B
(T Z xi) mIJ((11) +m(m 1))(1,2),
(B.10)
where í(i,j) E [aJa, Tr (VUVzz=])) / and < i, < 2. Hence, together with Equation B.8 and
Equation B.10
it follows that
B(6 - B(0, A) = (ni (1, 1))(1)
(B.11)
Furthermore, from Equation (8.10), Equation (8.25), Lemma 9.3 and Lemma 9.4 of Mci & Montanari (009 we get
((0, A) = Dride (5, v1, 12 2/2 + oa((1,
(B.12)
where Bridgo (E, 11, 22,, A/u²) is defined as in Equation B.1]. Thus, it only remains to caleulate the limit of B(6, A) - B(0, in order obtain the limit of the bias term B(5, A).
Analysis of variance term
Now we briefly deffer the calculation of term (I) in Equation B.11 and shift our focus to variance term. The reason behind this is that is hard to directly calculate the limits of the quantities appearing in the right hand side of Equation B..1p. We obtain these limiting quantities with the help of the limiting variance.
26

---

Published in Transactions on Machine Learning Research (06/2025)
Note that for the variance term, we have
((A) Er [(x;)(()))
1 12T EryEaa o(Oxx/Va)
Vd
= (TCUUZ 2/}}²
(B.13)
-²
(B(TI(UUVE 2)) d
Wgr?
where Wg:= := ==:=E(T [Tr (WUV2 2)} /d. Again by exchangeability argument, we have the following:
1
Tr (QUVaR))
1) E
dP
1=1
Tr (QUVins))
E
(as x. 2:/d 1)
(B.14)
E Tr TTUUU 2)
V 3-
Thus, in the light of Equation [B.11 and Equation B.14 we reemphasize that it is essential to understand the limiting behavior of ((i,j) in order to obtain the limit of expected out-of-sample risk.
Obtaining limit of variance term:
To this end, define the matrix A MRM M N + with parameters q (81, 82,1,,2,,, E IR. in the following way:
A = A(q) = SiIS + s
Z +W
7 p W 4I + 1
where
1 L0e
P1 X H. XXT W
Q
For & E C+, define the log-determinant of A
G(E,q) = IM (A,(A) - {IM)
Here Log is the branch cut on the negative real axis and (X{(A))E[M) denotes the eigenvalues A in non-increasing order.
Define the quantity
1 Tr TT (UUZZZ 2).
13 = d
From Equation [B14], trivially follows that E(w²) = nf(1, 1). The key trick lies replacing the kernel matrix U by the matrix A Qf pLIN. By (Mei Montanari] 2019 Lemma 9.4), know that the asymptotic error incurred in the expected value of Vi by replacing U in place of A is oa(1). To elaborate, first define
1/3 == 1 IIr 2).
Then by Mei & Montanari 2019 Lemma 9.4), we have
E B Vzl oa()).
(B.15)
Thus, instead of V3, will focus V3. By Mei & Montanari
2019
Proposition 8.2) know
27

---

Published in Transactions on Machine Learning Research (06/2025)
33 = -, 4 0) ng0xg, G2(((1182211/220)
(B.16)
Also by Mei & Montanari 2019
Proposition 8,5), for any fixed E R4, have the following:
lim E d000
Viu (0l:;;))]00 1) =0
(B.17)
As Vy a bi-linear form of V
A) (See Equation B.16 ) together with Mei & Montanari
2019. Equation (8.26)), follows that
lim E 1 VeigglEE 1/1, 1/2, A/p2) - 0, d
(B.18)
where Yndee (5, 1/, 2/2, A/p2) is as defined in Equation B.18
B.2 Thus, both Equation B.15 and Equation
yields that
lim EM - Vndno 11, 12, A/1 +
(B.19)
d-000
Hence, by Equation B.13 we get the following:
V(A) = Vndge 1/1, 2/2 A/p²) ol(
(B.20)
Obtaining limit of bias term:
Now we revisit the term B(6, (--))-B(0,)) B(0, A). We will study the terms f1, and nı n1(nn 1 -1)f(1,2) in Equation
B.11 separately.
First, focus on the term (1, Equation B.14 Equation [B115 and Equation B.19 show that
nf(1,) 1) Vrddge (5, 11, 2/2, A/r²) + oa()
(B.21)
Next, in order to understand the limiting behavior of f(1,2), 2). we define
Tr (DUDZ HZ),
2 == _ Tr (VAVZ HZ).
Again due (Mei & Montanari 2019 Lemma 9.4), we have
V/3 V/l od())
Hence, we only foeus on V2 A calculation similar to B.9 shows that
E(V2) nf(11 + n(n 1)f(122) + oa(1).
(B.22)
Let g(E;4) ④ be the analytic function defined in MMei & Montanari
2019
Equation (8.19)).
Using
Meei &
Montanari 2019 Proposition 8.5), we get
E(V2) =- -150 .. (i(1 (1 4/2 0) 1&0 1. (1( o1 (E 11, 22 A, M1) + ozl.
This along with B.21 and
B.22 show that
n(n 1),(1.2) 2)
WE 11, 12, A,H,, V (5, U2, 12 A/p2) + os(1).
(B.23)
Thus, finally using Equation B.11 Equation B.12 and the fact ni ni/n + n, we conclude that
B(5, 2) = F2 B (Si Vh, 22, 2 + [ - T) Trd (S, th,, 12 A/12)
(B.24)
+ 3 VE v/, 12, X, Mr. p1)] oa(1).
Lastly, using Equation B.20 Equation
B.24 and plugging the values in Equation
B.4
we gel
lim Ex.,, BRF (z, X. θ A)]
d0
B3ge (E, v1, 222 A/ a/22) F2 [T (011-T)-0 Vesdgge v/,, 1/2, A/ 212)
(B.25)
+ x V V1, 2/2, X, uc, v1) + 7 Vridge (E, 1/,, 122, 2/12).
28

---

Published in Transactions on Machine Learning Research (06/2025)
Ridgeless limit
Finally, for ridgeless limit we need to take + 0+ on both sides of Equation (B.25) Following similar calculations as in the proof of (Mei & Montanari] 2019 Theorem 5.7), or more specifically using (Mei
Montanari 2019 Lemma 12.1), we ultimately get
lim lim Ex,0, (2, x,), = =F3 + FF + py.
X- d000
This completes the proof.
B.2 Bias vaas-variance decomposition under orthogonality of parameters
In this section we will demonstrate that the same bias-variance decomposition in Lemma 3.3 contimues to hold rather weaker assumption than Assumption 3.2. Specifically, in this scction we only assume the paramcter vectors 30 and are orthogonal, iee, 8 0. The following Lemma shows that under this orthogonality condition, the desired bias variance decomposition still holds.
Lemma B.2. Define the followings: B3
T (Z Z x113/dd and Bs Exllxlll>*
Ex [||=] [ Z) X1 x11//d]. Also assume that 3/ 8 = 0. Then, have A (Bo: 5) F3 FB.Bo.
Proof. Similar to Equation
B.4 her also we have
(So. 8) = Ex Px [1(22 (Z" ZZT=ztzTxx x 2 + (Z Z 2) X18}²
(B.26)
+2(= (
2)²2"x x ))08 x 71 (7 2)²=].
The only difference in Equation
B.4 compare to Equation (B.26) is that Bu and 5 are now fixed vectors in Rd instead of being random. To this end we define the space of d x orthogonal matrices as follows:
( = (0 E Rddd o- = = =T]]
Now we consider the following change of variables for a matrix 0 E O(d):
X XO X
00000 2
(B.27)
6o + 0 - 3o. 8 8808 I O2
The key thing here is to note the following
7 a oX Na
Thus, the expression of á(X) in Equation B.3 remains unchanged. Finally, due to distributional assumption
and Lemma C.1 have F Unif ((v)) for every O E Old.. Finally, note that 5 3 x
which entails that o B(3, 8) = (30, 5) for all O E Odd, where
(3. 8) Eg 2 1 T ))
(Z 2 X1882
+ 2 22) x ) XT 2)
This motivates us to consider the change of variables in Equation B.27). when 0 is sampled from Haar measure Ha on (d) and independent of X, 0, Also, note that due to Lemma C.1 and Lemma C.2 the random vectors &y and satisfy the setup of Section B. Thus the result follows immediately by noting the fact that
0 ( 8)
EO 18 8)] F3 + Fg
where the last incquality follows from taking A + in Equation [..1 and Equation B.12
29

---

Published in Transactions on Machine Learning Research (06/2025)
B.3
Limiting risk for general Bu and 8
Unlike the previous section, this section studies the general property of the asymptotic risk Ro(â) when the parameters in and ô are in general position. Specifically, in this section we relax the assumption that Sg 8 + 0. Thus, the only difficulty arise in analysing the cross-covariance term
2B [( (2 7)² 7 X/vdd - x ")00 x (z(Z' /)/=/vd] in Equation (3.4) as it does not vanish under the absence of orthogonality assumption. Thus, essentially it boils down to showing the results in Lemma
3.3 and Lemma 3.8
B.3.1 Proof of Lemma 3.3
We begin with proof of Lemma 3.3
We recall the right Equation 3.4 i.e,
( 8) = Ex 11(
2 - x =T) )00 + (2 (z 2) X16}2 +2(z (z 2) 2 x x1 3808 x (1 2)²2].
Apart from the coossccorreltton term, The first two-terms of the equation does not depend on the interaction between So and 6, To be precise, the first two terms are only the functions of 30 and individually and does not depend on the orthogonality of Bo and 6. Thus, an analysis similar to Section B.2 yiclds that
BBo, = F3 + Ff 2(z (Z z)z - 1000 x 21 2)12
Thus, it only remains to analyze the cross-covariance term
E,s a 2x (Z x 3066 x, (1 2)²2),
To get the desired form as Lemma 3.3, we again introducce that random variables Bo. 6, X1, 2, Z1 and as in Section B.2, Also, define the quantity
63,8 a 2x [(=T(2TT) (Z" 3300 x, Z1(2" 2)²2].
Following, a similar argument in Section
B.2 have E3,5 EOwH. Ha (, Next, note that,
6,,6 TT
2
2) 2(2 (Z 2 300
B0o
Thus condition on and and applying Lemma C.2 the result follows, i.e.,
6g5 = F9.5
(B.28)
B.3.2 Proof of 3.8
it can be shown that the limiting values of [] [[])) and (T)( remains the same, i.e. the results of Lemma 3.7 and first part of Lemma 3.8 remains truc. Thus, it only remains to analyze the cross-covariance term
Following the same arguments Section B.1 and Section B.2
Eg,6 - 2E, ((2 (Z Z)!ZTx x x J000 x, Z1Z 2)12),
to fully characterize the asymptotic risk asymptotically. To begin with, the vector 5 can be written direct sum of two components ó and 82, where 82 is the orthogonal projection of on span((Bo)). In particular
we have
8 81 52. and 82 (Ta 30.30 / [186113)..
30

---

Published in Transactions on Machine Learning Research (06/2025)
Thus, there exists μ E R such that ó1 1Bs- Thus, 69,5 be decomposed in the following way.
6356 = 2ulBar Pxx 11:00 x z(Z 2) 2 X180
P 11:00 x (z Z) 30)
2112
0,a
2I Px [1= (Z (Z Z) Z X x D062 X Z1 2)
As 8y 82 = 0, following the arguments in Section B.2 we have 6 = 0. To analyze the terms (1.1))
and
(1.1) 6,6 we will gain analyze their corresponding ridge equivalents
6,,5 (A) (1).)()) = 2/1 Ex ExTZvūZZZ x, X,8)
d
(0 (A) 24 B2 (g0 x Z1 Vux T
Vd
where v and 0 are defined in Equation (B.5) and Equation B.7 respectively and vdz. We focus these terms separately. To start with, by an application of exchangeability argument we note that
2uF3
3,5 (A)) =
d2 [r
= = + nn (m - 1)f(1, non non ((,,)).
Using Equation B.21 and Equation B.23 it be deduced that
B(GL)) (A)) 2 thi, via A, Hi) oa(1).
(B.29)
Let us denote by V the matrix B(ur") This shows that
B(
2/
2p
E (BT
Vd
Again by exchangeability argument we get
E(G 3,5
(A)) =
11
2μ
E (gT x ZVVB))
Vd
Let use define Ti va (B) X B.). By the arguments of Scction 9.1 in Mei & Montanari (0119 can also conclude that
F2
T 0,0( +)((1)))
the function is defined in (Mci & Montanari
2019
Equation (8.19)). Thus we have E( (A))
3,8
nF (F3 + oa(1). This along with Equation
B.29 yields the following:
E((g,)) = lim (
(A)) + E (1.2 9,6
(A)) + oa(1)
X=0
8.8
lim uo F2 [24T 14 12 >, - 0,0(((110))))) (1 x=0
+aa(1)
F - + v33 + (1()
31

---

Published in Transactions on Machine Learning Research (06/2025)
Finally, by simple algebra it follows that
Fs cos Ø3,6, where Fp
(3,5)
Φ arccos
FgFo
This shows that pF FaFa cos(03,6) limg-so [[ - -1+
Fg.
Now recalling Equation B.28
have
- v.). This finished the proof of Lemma 3.8.
B.4
Final form of limiting risk
Finally, Gathering all the results from Section B.1
B.2 and
B.3
have
lim
= F3 +
Mi + F3,.
+
d
with E0 = T 1 W2). This concludes the proof of Theorem 3.9
C Uniform distribution on sphere and Haar measure
In this section we will discuss some useful results related to Uniform distribution on sphere and the Haar measure Ha on ((d).
Lemma C.1. Let U ~ Unif(s+ and O E O(d). Then OU Unif(S* (Vd)). Also, i ~ His and is independent of U, then also OU ~ Unif(S*-I(Va))
Proof. Note that U L VJG/ [|G||2, where G N(O,Ia) Next define ===0. By property of Gaussian
random vector we have G G. Now the result follows from the following:
OU 1 Vd OG = Vd & & Vd
G U.
|G
|Gl|
Next, let ~ Hy be an independent random matrix from U. Thus conditional on O, we have
OU |0 ~ Unif(s"-1((e)))
Thus, unconditionally we have OU Unit(g(-t(Vi)))
Lemma C.2. Let B, 8 E Rd such that T = F3, 8. Also, define the random vectors 3 033 and = 05, where O Ha- Then the followings are true:
Fg.6 T,
B(3ōT)
d
B(33 )/ 118112 - E(55 1|112 d
Proof. Let O 00100000
(2,,00,.... oa. ie, of is the ith row of O. Let T (3(8TT 1E(O88TOT) Also, by
property of Haar measure, have πO d O for any permutation matrix II. This shows that jo exchangeable. As consequence we have
B(o,0,)) Ed B(oLoT)
E(OTO) 1 d
d for all i e [d].
d
Now we are equipped compute matrix T. Note that.
Ta To,,, B8 ox) Eς 010 8} 87 3/d Fa.s/d.
32

---

Published in Transactions on Machine Learning Research (06/2025)
Also, for + similarly get
Tiş = B 050, 33.
Now again by property of Haar measure we know - (Ia 2 where e, denotes the ith canonical basis of I This implies that 0,0o _ -ojoT Using, this property of Hd, we also get B(oro,) 0. Thus, get Ti 0 and this concludes the proof for uncorrelatedness.
Next, we define VB == E(B3T), By similar argument, it also follows that
113112
(V))k.
d A.j.
where Dij is the Kronecker delta function. The result for can be shown following exactly the same recipe.
D Random feature regression model on California housing price dataset
We fit random feature regression model on the "California Housing Price" dataset that contains information from the 1990 California census from given district for predicting the median house price. The dataset has 10 columns containing summary statistics of approximately 18,000 houses. For example, it contains information like median household income, median house price (Larget), and ocean proximity. More details about the dataset can be found in https:: :/sp:://ww/ //www. https/////////.lee.. kaggle. wwtsgle..co//// www.cmm/aaaaao homm//aaasess//aaaa wttpa///tcmmnnnn/caeeeeeeeeeeeee cttet//aafffoiiaaooonoeo phtooiiaahousinn-Pa poouusnng-piiess /
In this section, we only consider randomly chosen subset of size 2000 which is further partitioned into two groups based on the ocean proximity. The data has originally 5 groups based on the ocean proximity: (1) within 1 hour from ocean, (2) Near ocean, (3) Island, (4) Near bay, and (5) Inland. In this experiment, merge the groups 1 and 2 to form the majority group Near Ocean, and set the group Inland as the minority group. We drop the rest of the groups to limit our study in the 2-groups setting. Furthermore, we split the data into training (80%) and test (20%) datasets. The group proportions along with the median house value are shown in Figure G which clearly shows that there is a significant difference in the house price across the two groups.
Nedian house Valua by Ocean Prosimily
Deemn Neaiedy
Tex Drres Roairelty
SLAND
MaAt COCAN
Piirrnty
Figure 6: (left) Box plot for the median house value across two groups. (right) Histogram of ocean-proximity for training and test datasets.
We consider two cases:
Case 1: We coder the training dataset without any down sampling of the minority group. In this case, the majority group proportion is π x 0.62.
2. Case 2: We down sample the minority group in the training dataset to make the majority group proportion 0.91.
In both the above cases, we fit the RF regression model and evaluate the mean squared error (MSE) performance on the test minority group (Inland) across 20 independent runs. F the performance of the fitted RF model for varying choices of the overparameterization parameter y which clearly shows that the overparameterization (7 1) improves the MSE for the test. minority group.
33

---

Published in Transactions on Machine Learning Research (06/2025)
0.6 Test Loss vs Y. majority group proportion
031
Test loss inland data
Figure 7: MSE of RF model for test minority group under varying choices of T.
E Random feature classification model
In this section, consider an overparametterized classification problem. We consider two group setup: E 1]. Without loss of generality, we assume > (so the 1 group is the majority group). The data generating process the training examples {(x; y1) ys)
0: Ber(I))
~ N)
(E.1)
+1, w.p. f(x Bo) ( =
v 9:
11 w.p. 1 - [xx Bo) - 0) (1)
where S0, 31 E Rd are vectors of coefficients of the minority and majority groups respectively, and f(t) = (1 (1+e-t)-1 is the sigmoid function.
Consider a random feature classification model, that is, for newly generated sample (In41,9n41). the classifier which predicts a label in-1 for the new sample
7 = sign Σ ago (0 (ffann/vaa)
where a(-) is non-linear activation function and N is the number of random features considered in the model. In the overparameterized setting (n S N), we train the random feature classification model by solving the following hard-margin SVM problem:
arg minagRN
||l||
}
ae
subject to no a 1, (m))
where = (s.,1...., =T with 2. o00 x.//v/// for i i [m],] e (N].
We wish to study disparity between the asymptotic risks (i.e., test-time classification errors) of on the minority and majority groups, that is, comparing
Ro(a) = ({())=[((0.... # yo 1 = 0) and Rí (a) E[I # yatt ) Hga+ = 1]
asymptotically. Moreover, we consider high-dimensional asymplotic regime such that d d= 41 > 0, n/dd vi > 0 and //////2-
lim (N/n} 2 1. d
Therefore, y encodes the level overparameterization.
In the simulation for Figure g we let o(-) be the ReLU activation function and 0,, be IID standard normal distributed. Moreover, let 0.95, 30 10e,, 8 10cos(0)er 10sin(0) 400, d 200, N m
34

---

Published in Transactions on Machine Learning Research (06/2025)
where e1 and 62 the first two standard basis of R We tune hyperparameters 0 E {0", 45", 90", 135", 180%} and 7 E (0.5, ......3,,, 3), then report test errors averaged over 20 replicates.
As seen in Figure S the misclassification error either decreases or doesn't increase with overparameeeeriation when the angle a I 90%. This the regime that is most related to Pham et al. (2021). On the contrary, an opposite behavior is observed for a 90%; when the differences across subgroups are large. The mis- classification error for the minority group increases with overparamettriiation,, which is more aligned with
the observations in Sagawaa al. (2020). Interestingly, in > 90" regime, undersampling improves the misclassification error for minority class compared to ERM.
0.7
Misclassfiicatin error
Misclassiiictionn error
0.50
0.6
45
0.5
0.45
90a
0"
0.4
135
45
180
0.40
90
0.3
180
0.35
0.5
1.0
2.0
2.5
3.0
0.5
1.0
1.5
2.0
2.5
3.0
7 N/n.
7
N/u
Figure Left: misclassification errors at different levels of overparameteriiation under ERM. Right: the sample with undersampling. In each case, the errorbars are calculated over 20 repetitions.
35


Reviews:
[{"review_id": "cJ3Ybu5EJp", "reviewer": "Reviewer_9wer", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The research problem of interest is stated clearly, and the broad area of machine learning under non-standard evaluation metrics (for example group-balanced risks under group imbalance, etc.) is definitely of importance in practice.\n\nOn the other hand, while at first glance the writing may seem fine, I feel that the presentation of the technical content in this paper is unsatisfactory at present, and below the standard of clarity that is expected for a TMLR submission (noting that TMLR emphasizes clarity above impact and significance). I will try my best to give some concrete examples to back up my criticism, breaking this up into two major points. First is poor technical presentation, and second is a lack of clarity regarding what the authors have done on top of previous literature.\n\n__On technical presentation issues__\n\nThroughout the technical exposition in this paper, there are countless points which I tripped up on; in my opinion this is sloppy writing. Here is a list of some examples, not in any particular order, which is not exhaustive, but I think sufficiently illustrative.\n\n- Figure 1: it says \"variance\" is being compared; where is this?\n\n- Eqn (2.1): $\\\\epsilon$ should be $\\\\varepsilon$.\n\n- Section 2.2 and onward: $\\\\Theta$ is undefined.\n\n- Section 2.2: $x\\_{j}$ is used to represent individual features whereas $x\\_{i}$ is used to represent vectors.\n\n- For (2.6), writing $\\\\hat{a} \\\\in \\\\text{arg min}\\\\{\\\\ldots\\\\}$ is misleading, since the solution is unique, as shown by the equality in the same equation.\n\n- The notation $\\\\hat{a}$ is used in several contexts so as to make the first equality in (2.7) quite unclear. It really should be that the right-most quantity shown in (2.6) is equal to $\\\\lim\\_{\\\\lambda \\\\to 0\\_{+}} \\\\hat{a}\\_{\\\\lambda}$, right? On top of this, this \"fact\" is given as common knowledge, but really a precise reference would be nice.\n\n- Before (2.8), they say \"having an estimator $\\\\hat{a}$ of the parameter $a$...\", but in reality $a$ is just a placeholder variable, not a pre-fixed unknown parameter to be estimated.\n\n- Section 2.3.2, second paragraph: \"due to its superior statistical efficiency\"; is this a formal statement? A reference would be desirable.\n\n- Table 1: $\\\\psi\\_{1}$ and $\\\\psi\\_{2}$ are just ratios, whereas in the main text they are the *limits* of the given ratios. This mismatch is troublesome.\n\n- Page 6, last paragraph: \"the variance ... captures the average variance of...\" poor vocabulary.\n\n- Page 6, last paragraph: what is $\\\\hat{f}$? A typo perhaps.\n\n- Page 6, last paragraph: $z = \\\\sigma(\\\\Theta x/\\\\sqrt{d})$ appears, but $\\\\Theta$ is undefined, and implicitly assumed to be a sequence of vectors, making this difficult to interpret. I know what the authors have in mind, but this is sloppy.\n\n- Lemma 3.1: why is $\\\\epsilon$ used here instead of $\\\\varepsilon$ as in the data generation section?\n\n- Assumption 3.2: I have no idea what this assumption means, or why it is even called an assumption. How is this any different from the *definition* of $F\\_{\\\\beta}$, $F\\_{\\\\delta}$, and $F\\_{\\\\beta,\\\\delta}$? What is being *assumed*?\n\n- Assumption 3.5: Maybe some readers are quite familiar with this assumption, but it stood out to me. $\\\\sigma$ is assumed to be \"weakly differentiable\". This is a rather technical notion; is this degree of generality really needed? The bound on the (weak) derivative also is not something I'm familiar with. Some additional commentary would be natural considering the audience of TMLR.\n\n- Lemmas 3.7 and 3.8: all of a sudden it seems that $\\\\beta\\_{0}$ and $\\\\beta\\_{1}$ (and thus $\\\\delta$) are random. Does this appear anywhere in the preceding assumptions or exposition of the data generating process? So $F\\_{\\\\beta}$, $F\\_{\\\\delta}$, and $F\\_{\\\\beta,\\\\delta}$ are thus also random, correct? Or are these \"lengths\" and \"angles\" assumed to be constant in Assumption 3.2? None of this is clear.\n\n- Section 3.1: $\\\\theta$ is used to denote the angle between $\\\\beta\\_{1}$ and $\\\\beta\\_{0}$; this clashes with previous usage of $\\\\theta$.\n\n- I would say that the most critical assumption underlying the analysis in section 3 is the restriction to the minimum norm interpolator, but this is given as a nonchalant statement buried in section 2. This must be made explicit, I think (yes, I know it is stated in section 1).\n\n\n__What is new here?__\n\nClearly much of the technical argument is rooted in previous work such as Montanari et al. (2020). Is their problem setting identical to this one? Is the main difference just that they did not analyze the same error terms as the authors have? I'm not complaining about a lack of novelty, it's a lack of clarity regarding what has been done (and learned) in the current manuscript versus what was already done and known in the closely related literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors are interested in how the use of overparameterized models impacts performance (assuming an ERM learning algorithm) on \"minority groups\" in the context of regression problems.\n\nHere the notion of \"group\" is formulated in terms of a difference in conditional mean of the response. That is, for each sampled point, with probability $0 < \\\\pi < 1/2$, the mean of $y \\\\,\\\\vert\\\\, x$ is $\\\\beta\\_{0}^{\\\\top}x$ (the \"minority\" case); otherwise it has mean $\\\\beta\\_{1}^{\\\\top}x$ (the \"majority\" case). Everything else (variance, marginal distribution of $x$ etc.) is identical.\n\nAs mentioned above, the focus of this paper is on ERM learning algorithms, with a potentially overparameterized model, essentially a feed-forward neural network with one hidden layer, non-linear activation function, and $N$ units. The weights used in taking the linear combination of the raw inputs (denoted by vector $\\\\theta\\_{j}$ for each $j=1,\\\\ldots,N$) are assumed to be random, and the ERM training just sets $N$ weights used to combine the outputs of the hidden layer (see their equation (2.4)).\n\nTheir main theoretical results look at the \"mean squared prediction error for the minority samples\", defined in their (3.1), which is the mean squared prediction error of the conditional mean (of $y$), conditioned on the group being the minority group. Their theoretical analysis is for the asymptotic case, where the number of features $d$, samples $n$, and units $N$ grow together without bound. The \"overparameterized\" setting is that in which $N/n$ converges to a number greater than $1$ in this limit. Their results are not for any arbitrary ERM solution, but rather the unique \"minimum norm interpolator\".\n\nThe main results are somewhat difficult to parse, but by breaking down the risk of interest into bias and variance terms, the authors look at how the impact of the ratio $N/n$ in the limit impacts these terms, and come to the conclusion that overparameterization should not be expected to harm the minority risk, all else equal.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the comments raised above. I think the paper represents a genuine effort, but it still needs a fair bit of polish.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pGGy68i5xg", "reviewer": "Reviewer_FusR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The paper studies an interesting problem: theoretically explain the empirically observed impact of overparametrization in the minority group. \n\n2. The paper presents interesting theoretical results. \n\n\n## Weaknesses\n1. The notation is not easy to follow. In fact, Table 1 is tricky to parse. For example, some of the variables defined are fixed, while others are the limit of the expression in the description. However, the text makes it easier to understand this distinction. It is also hard to parse the notation after Definition 3.4. It could be easier to understand if the authors provide some interpretation of the variables they define in Def. 3.4.\n\n2. On the assumptions: I know that the theoretical results from Montanari that the paper relied on, for example, a very restrictive distribution on $x_i's$ -- uniform in the sphere. However, in reality, is this assumption reasonable? How much would your results change if you don't have uniformity in the sphere? Can you discuss more of the limitations of this (and the rest of the) assumption?\n\n3. Intuitively, it makes sense that the assumption that minority and majority feature distributions are equal implies an easier-to-learn distribution for the minority. However, there might be some cases where this assumption does not hold. For example, imagine that the minority is distributed as $x_0$ with probability one and zero probability mass on the rest. The model can memorize the output to $x_0$. Therefore, the mode generalizes because the input always has only one fixed point $x_0$ and its respective memorized output. I make this point to note that maybe equal distribution is not the worst case. Hence, we need some math to prove the argument you make in Section 4. \n\n4. All theoretical results are asymptotic. The paper argues that overparametrization helps the minority group in RFM. However, this is only shown in the limit. Should we expect this result to hold in $d <<  \\infty $? If yes, why?\n\n## Questions:\n1. What do you know what to subsample? We only have access to the data $(x, y)$. You are assuming that the features are indistinguishable across the groups; therefore, we can't tell the group of $x$ by looking at it. Also, depending on the values of $\\beta_0$ and $\\beta_1$ it might be challenging to decide what is the minority group. How can we perform oversampling in this case?\n\n2. In fairness, we are usually interested in *decisions* of consequences. However, the paper focuses on regression. How could you generalize your results for classification?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to theoretically investigate the effect of overparametrization on the risk of underrepresented groups in the dataset. The paper develops its theoretical results using the ERM of linear regression and random features models. The authors then prove that: \n\n1. Asymptotically, the expected risk of the minority group can be fully characterized when using strong assumptions on data distribution, random feature distribution, and the activation function. Using their characterization, we conclude that, in the limit, the risk in the minority group decreases with overparametrization.\n\n2. Similarly, the paper analyzes the risk in the minority group using the linear regression ERM and concludes that overparametrization increases the risk in the minority group. \n\n3. Finally, the authors consider using subsampling with the aim of improving the risk of minority groups. Their asymptotic characterization indicates that subsampling improves the risk for the minority group compared to just using the ERM.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Can the authors improve the clarity of Table 1, as discussed in the weakness? \n\n2. Could you discuss the interpretation of variables defined in Def. 3.6.\n\n3. Is it possible to write additional limitations discussing the assumptions necessary in your main results? Namely: Data uniformly distributed in the sphere. \n\n4. The last paragraph in the conclusion seems speculative \"using overparameterized models is unlikely to magnify disparities across groups.\" You only know this for RFM, and even your own results in linear regression go against this claim. \n\n5. Is it possible to tone down the abstract as well? You argue that \"we show that overparameterization always improves minority group performance.\" However, from my understanding, this finding is (i) limited to RFM and (ii) only asymptotic. [4 & 5 make the results not support the claims].\n\n6. In the paragraph after Eq. 3.1, could you not use $a$ in the expectation? It is a little confusing when you have the coefficients being $a$.\n\n7. Can you add a section describing how to compute the empirical results for Figures 2 and 3?", "broader_impact_concerns": "1. The conclusion in the paper seems very limited and specific to the random features model. \n\n2. The theoretical results are interesting, but they seem like applications of the results from Montanari alongside additional algebra. It is unclear to me if there is any novel theoretical technique in the proofs that the authors would like to highlight. \n\n3. The results hold on the limit of dimensionality, which is not the case for regressors. Therefore, it is hard to know if the results are applicable in practice. \n\nConclusion: the message of the paper is not surprising from a practical point of view, the theoretical results only hold in a limited scenario, and the authors didn't highlight any novel technique used in their proof.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XKIFdveeBv", "reviewer": "Reviewer_ZvA5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- nice (and technically sound) extension of results in Hastie et al etc to the new setting considered in the theory part of this paper. In particular, Lemma 3.8 seems to be the main theoretical contribution necessary to relate prior results to the data generating process introduced in this paper.\n- Theoretical results are presented in a mathematicaly rigorous manner.\n\nWeaknesses\n- there is a stark discrepancy between the data distribution chosen to prove theoretical results in this paper and the distribution assumed in prior experimental analyses that this paper claims to attempt to explain (e.g. Pham et al; Sagawa et al; etc). In particular, two major differences stand out:\n    1. this paper analyzes a regression problem, while all experimental works cited as motivation focus on classification.\n    2. in the prior empirical analyses that motivate this paper (e.g. Sagawa et al), the ground truth $\\beta^\\star$ is indeed the minimizer of the 0-1 loss in the low noise regime, while for large noise levels a trade-off between average and worst-group performance emerges. In contrast, in this paper, neither $\\beta_0$ nor $\\beta_1$ are optimal for prediction, regardless of the noise regime.\n    - When studying the impact of learning algorithms on worst-group error, the choice of the data distribution is crucial. For instance, depending on the distribution, improving worst-group performance may or may not come at a cost for average-case performance. Therefore, these discrepancies raise questions about how suitable the data distribution considered in this paper is for explaining prior empirical observations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the effect of overparameterization on the worst group accuracy of machine learning models. The authors analyzed this problem with a random feature regression model and derived precise asymptotic risk for the minority group. The theoretical results demonstrate a double descent curve for the minority group risk as a function of overparameterization, which suggests that increasing the model size would be beneficial to learn the minority group.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major changes:\n- The paper should better motivate the relevance of the theoretical results. For instance, do there exist empirical results for regression that highlight the poor performance of ERM where the data distribution is akin to the one considered in this paper? The literature on fairness for regression (e.g. https://proceedings.mlr.press/v97/mary19a.html, https://arxiv.org/abs/1905.12843, etc) might help to guide the process of designing a data generating process that resembles more closely the one in experimental analyses.\n- Alternatively, the theoretical analysis could perhaps consider classification instead of regression, and can attempt to extend works like https://arxiv.org/abs/1803.06964 and https://arxiv.org/abs/2005.08054. In addition, the work of Sagawa et al (https://arxiv.org/abs/2005.04345) considers a simple high-dimensional setting for their theoretical results that could potentially be adapted to show how increasing overparameterization affects worst-group error for classification.\n- The main takeaways of the theoretical analysis should be discussed only in the context of the problem setting. For instance, the title, abstract and introduction should reflect more clearly the key characteristics of the data distribution (i.e. regression, groups with identical support). Alternatively, more evidence should be provided to demonstrate the extent to which the assumptions hold for the experimental settings considered in Pham et al, Sagawa et al, etc. \n\nMinor changes:\n- At the end of page 1 it is stated that the works of Le Pham et al and Sagawa et al are “in sharp contrast”. My understanding  is that these papers are not contradicting each other, but rather focusing on different aspects: Sagawa et al discusses the average vs worst-group error gap (without discussing trends that emerge when varying model size), while Le Pham et al focus on how worst-group error changes with model size once we’re already in the overparameterized regime.\n- in Figure 1 and Theorem 1.1, do the authors mean “Inductive bias” or “(Statistical) Bias”? If the former is true, then the paper should include a formal definition of what is understood as “Inductive bias” and how one can quantify it.", "broader_impact_concerns": "This work raises no outstanding ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was reviewed by three expert reviewers on the topic. All three reviewers initially expressed concerns about the disconnect between the theoretical findings and the claims of the paper. Subsequently, the authors have revised the paper to better mirror the findings.  However, I think the paper still suffers from this disconnect as mentioned by Reviewer ZvA5 in their decision letter.

"my primary concern about this paper regards the disconnect between the setting analyzed theoretically (regression instead of classification, and an inherently inconsistent problem where the ground truth is not the minimizer of the 0-1 loss) and the setting of the empirical works that the current paper claims to try to explain. In my view, the changes to the manuscript do not address these concerns. As suggested in my review, I think it is necessary that the paper either analyze a classification problem or, instead, demonstrate empirically that this phenomenon is indeed prevalent in regression with a similarly misspecified setting (i.e. with ground truth not minimizing risk)." 

Hence, I think the paper still needs another round of revision to reach the publishable state. In particular,
- Please empirically verify that a similar phenomenon to classification also appears in a real-world regression task. For example, you may want to look into the empirical setup of https://arxiv.org/abs/1905.12843 or https://arxiv.org/abs/2405.04034.
- Please update the literature survey. The paper's citations on fairness literature is out of date and sparse especially beyond 2021. It is important to make a thorough effort to contextualize the work correctly. For example, fairness regularization is not discussed at all. Please explore the last 4 years of literature in TMLR, ICML, etc and discuss their relevance. You can start with https://arxiv.org/abs/2207.07068 and https://openreview.net/forum?id=3HE4vPNIfX and the references therein.

The paper is accepted subject to the verification of these remaining issues by the AE. Congratulations!
Decision: Accept (poster)