An Effective Theory of Bias Amplification
Authors: Arjun Subramonian
Samuel Bell
Levent Sagun
Elvis Dohmatob
Venue: ICLR 2025 Poster (2025)
Keywords: fairness
algorithmic bias
machine learning theory
random matrix theory
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We present a precise, effective theory of machine learning bias that explains empirical observations found in the literature, such as bias amplification and minority-group bias.

Abstract: Machine learning models can capture and amplify biases present in data, leading to disparate test performance across social groups. To better understand, evaluate, and mitigate these biases, a deeper theoretical understanding of how model design choices and data distribution properties contribute to bias is needed. In this work, we contribute a precise analytical theory in the context of ridge regression, both with and without random projections, where the former models feedforward neural networks in a simplified regime. Our theory offers a unified and rigorous explanation of machine learning bias, providing insights into phenomena such as bias amplification and minority-group bias in various feature and parameter regimes. For example, we observe that there may be an optimal regularization penalty or training time to avoid bias amplification, and there can be differences in test error between groups that are not alleviated with increased parameterization. Importantly, our theoretical predictions align with  empirical observations reported in the literature on machine learning bias. We extensively empirically validate our theory on synthetic and semi-synthetic datasets.

Published as a conference paper at ICLR 2025
AN EFFECTIVE THEORY OF BIAS AMPLIRICATION
Arjun Sabbaamnnin' Samuel J. Bell?, Levent Sagun', Elvis Dohmatob 2,3,1 UCLA
MMeta FAIR
*Concordia University *Mila
ABSTRACT
Machine learning models can capture and amplify biases present in data, leading to disparate test performance across social groups To better understand, evalu- ate, and mitigate these biases, a deeper theoretical understanding of how model design choices and data distribution properties contribute to bias is needed. In this work, we contribute precise analytical theory in the context of ridge regression, both with and without random projections, where the former models feedforward neural networks in a simplified regime. Our theory offers a unified and rigorous explanation of machine learning bias, providing insights into phenomena such as bias amplification and minority-grou, bias in various feature and parameter regimes. For example, we observe that there may be an optimal regularization penalty or training time to avoid bias amplification, and there can be differences test error between groups that are not alleviated with increased parameterization. Importantly, our theoretical predictions align with empirical observations reported in the literature on machine learning bias. We extensively empirically validate our theory on synthetic and semi-synthetic datasets.
INTRODUCTION
Machine learning (ML) datasets can encode a plethora of biases which, when said data is used to train models, can result in systems that can cause practical harm. Datasets that encode correlations that only hold for subset of the data may cause disparate performance when models are used more broadly, such as an X-ray pneumonia classifier that only functions on images from certain hospitals (Zech et al., 2018). This issue is magnified when coupled with redeerrepresnntation, whereby a dataset fails adequately reflect parts of the underlying data distribution, often further marginalizing certain groups. Lack of representation results in systems that might work well on average, but fail for minoritized groups, including facial recognition systems that fail for darker-skinned women (Buolamwini & Gebru, 201 8), large language models that consistently misgender transgender and nonbinary people (Ovalle al., 2023), image classification technology that only works in Western contexts (de Vries et al., 2019; Richards et al., 2024).
Unfortunately, contemporary models may exhibit bias amplification, whereby dataset biases only replicated, but exacerbated (Zhao et al., 2017; Hendricks et al.. 2018; Wang & Russakovsky, 2021). While previous research has shown that amplification is a function of both dataset properties and how we choose to construct our models (Hall et al., 2022; Sagawa et al., 2020; Bell & Sagun, 2023), it not fully clear how bias amplification occurs mechanistically, nor do we precisely understand which settings lead to its emergence. Thus, in this work, we propose novel theoretical framework that explains how model design choices (e.g., number of parameters, regularization penalty) and data distributional properties (e.g., number of features, group imbalance, label noises) interact to amplify bias. Moreover, our framework provides an account of prior work on bias amplification (Bell & Sagun, 2023) and minority-groon group bias (Sagawa et al., 2020).
theory bias amplification is important for several reasons. First, as empirical research necessarily yields only sparse data mainss-ofter foeused only the most common regimes- theory allows us to interpolate between past findings, and reason about how bias emerges in under-explored settings. Second, a precise theory gives us the depth of understanding needed in order intervene, potentially supporting the development of both novel evaluations and mitigations. Finally, beyond explaining
Work done while interning Meta. Corresponding author: Arjun Subramonian (urjunsu@@csuullas

---

Published as a conference paper at ICLR 2025
Atready-known phenomena, our theory makes new predictions, suggesting new avenues for future research.
1.1 MAIN CONTRIBUTIONS
In this work, we develop a unifying and rigorous theory of ML bias the settings of ridge regression with and without random projections. In particular, we precisely analyze test error disparities between groups (e.g., demographic groups or protected categories) with different data distributions when training mixture of data from these groups. We characterize these disparities in high dimensions using operator-valued free probability theory (OVEPT), thereby avoiding possibly loose bounds on critical quantities. Our theory encompasses different parameterization regimes, group sizes, label noises, and data covariance structures. Moreover, our theory has applications to important problems in ML bias that have recently been empirically investigated:
Bias amplification. Even in the absence of group imbalance and spurious correlations, a single model that is trained on a combination of data from different groups can amplify bias beyond separate models that are trained data from each group (Bell & Sagun, 2023). With our theory, we reproduce and analyze the bias amplification findings of Bell & Sagun (2023) in controlled settings. We further observe how stopping model training early or tuning the regularization hyperparameter can alleviate bias amplification.
Minority-grout bias. Overparamaterization can hurt test performance on minority groups due to spurious features (Sagawa al., 2020; Khani & Liang, 2021). We theoretically analyze how model size and extraneous features affect sanority-group bias.
We extensively empirically validate our theory in controlled and semi-synthetic settings. Specifically, we show that our theory aligns with practice in the cases of: (1) bias amplification with synthetic data generated from isotropic covariance matrices and the semi-synthetic dataset Colored MNIST (Arjovsky al., 2019), and (2) minority- grrou bias under different model sizes with synthetic data generated from diatomic covariance matrices. In these applications. we expose new, interesting phenomena in various regimes. For example, a larger number of features than samples can amplify bias under overparameterization, there may be an optimal regularization penalty or training time to avoid bias amplification, and there can be differences in test error between groups that are not alleviated with increased parameterization. Our observations of phenomena in Sections 4 and 5 are largely empirical but are supported by their agreement with our theory. Our theory of ML bias can inform strategies to evaluate and mitigate unfairness in ML, or be used to caution against the usage of ML in certain applications.
1.2 RELATED WORK
Bias amplification. A long line of research has explored how ML exacerbates biases data. For example, single model that is trained on a combination of data from different groups amplify bias (Zhao et al., 2017; Wang& & Russakovsky. 2021), even beyond what would be expected when separate models trained on data from each group (Bell & Sagun, 2023). Hall al. (2022) conduct a systematic empirical study of bias amplification in the context of image classification, finding that amplification can vary greatly as a function of model size, training set size, and training time. Further- more, overparameterization despite reducing a model's overall test error, can disproportionately hurt test performance for minority groups (Sagawa et al., 2020; Khani & Liang, 2021). Models can also overestimate the importance of poorly-predictive hurting performance on these groups (Leino et al., 2019). In this paper, we distill a holistic theory of how model design choices and data distributional properties affect disparate test performance across groups, which can encompass seemingly disparate bias phenomena.
low-signal features for minority groups, thereby
High-dimensional analysis of MI. A suite of works have analyzed the expected dynamics of ML in appropriate asymptotic scaling limits, e.g., the rate of features d to samples n converges to a finite values as d and respectively scale towards infinity (Adlam & Pennington, 2020b; Tripuraneni et al., 2021; Lee et al., 2023). Notably, Bach (2023) theoretically analyzes the double descent phenomenon (Spigler et al., 2019; Belkin et al., 2019) in ridge regression with random projections by computing deterministic equivalents for relevant random matrix quantities in a proportionate scaling limit. Like Adlam & Pennington (2020b); Tripuraneni et al. (2021); Lee et al. (2023), we leverage the tools of
2

---

Published as a conference paper at ICLR 2025
OVFPT (Mingo & Speicher, 2017), which is at the intersection of random matrix theory (RMT) and functional analysis. However, Adlam & Pennington (2020b) focus on training and testing a random features model on data from the same Gaussian distribution. Furthermore, Tripuraneni et al. (202 Lee et al. (2023) focus on training a random features model on data from one Gaussian distribution and testing the model on different Gaussian. In contrast, study the random features model in the setting of training on a mixture of Gaussian distributions and testing on each component. Because mixture is more expressive than a single Gaussian, our theoretical results cannot be derived as special case of these other works. Furthermore, our theory von-trivially generalizes Bach (2023), which we recover Corollary J.1 as a special case, and requires more powerful analytical techniques.
Certain prior theoretical work precisely analyzes the bias of models trained on mixture of data from different groups in high-dimensional setting (Mannelli et al., 2024; Jain et al., 2024). Like Mannelli et al. (2024); Jain et al. (2024), we study linear models and consider bias as the disparity in test performance of a model between groups. We further consider some similar factors that give rise to bias amplification (e.g., group imbalance, group data variance, inter-group similarity, dataset size). We also share some theoretical conclusions, such as bias can occur even when the groups have the same ground-truth weights (see Section 5) and are balanced (Section 4.1). Additionally, we both discuss the paradigms of training a single model for both groups vs. separate models for each group. However, the main distinction between our work and Mannelli al. (2024); Jain al. (2024) is that we precisely characterize how models amplify bias in different parameterization regimes, that is, we examine the impact of model size on bias. This enables us to expose new, richer insights into the impact of over- and underparametterzzaton on bias amplification (see Figure 1, Section 4, and Section 5). See Appendix C for further comparison of our work to Mannelli al. (2024); Jain et al. (2024).
PRELIMINARIES
2.1 DATA DISTRIBUTIONS
We consider ridge regression problem on dataset from the following multivariate Gaussian mixture with two groups and 2. These groups could represent different demographic groups or protected categories.
(Group ID) Law(s) Bernoullip), (Features) Law(x 5) (N Σ.), (Ground-truth weights) (aw(wf) = NN 0/d), Law wi) - (0,0 (2)
(Labels) Law(y) s, = NN) with (x) = x
w. T...
The scalar p (0. controls for the relative size of the two groups (e.g., p = 1/2 in the balanced setting). For simplicity of notation, we define matrices En and Σg are the covariance matrices for the different groups. The d-dimensional vectors wI and w are the ground-truth weights vectors for each group. wi and wi ww are independently sampled from zero-mean Gaussian distributions with covariances O/d and A/d, respectively. In particular, setting A= 0 corresponds to the case that both groups have identical ground-truth weights. We define O = θ, O2 = + A. Finally, o" corresponds to the label noise for each group s. While we consider the case of two groups only for conciseness, our theoretical methods readily extend to any finite number of groups
p and p2 = 1 - p. The d x d positive-definite
2.2 MODELS AND METRICS
Learning. A learner is given an IID sample D = [xx1 (iii,,)).... (In,) Wr)) = (X E Raxd Y R") of data from the above distribution and it learns a model for predicting the label y from the feature vector z. Thus, X is the total design matrix with ith row x; and y the total response vector with äth component y,. Let D = (X E R", xd s.dYY E IR*)) be the data pertaining only to group s, so that D Dd D2 a partitioning of the entire dataset. Two choices are available to the learner: (1) learn a model a f. E F on each dataset D*, or (2) learn a single model f E F on the entire dataset D. In practice, choice is made based on scaling vs. personalization considerations.
We consider two solvable settings for linear models: classical ridge regression in the ambient input space, and ridge regression in feature space given by random projections. The latter allows to
3

---

Published as a conference paper at ICLR 2025
study the role of model size in ML bias, by varying the output dimension of the random projection mapping. This output dimension m controls the size of a feedforward neural network in a simplified regime (Maloney et al., 2022; Bach, 2023).
Classical Ridge Regression. We will first consider the function class F C {I< + R} of linear ridge regression models without random projections. For any vector R the model with parameters w is defined by f(x)) = x w, for all x E RR, and is learned with Ez-regularization. We define the generalization error or risk of any model with respect to group s as:
(5) We consider ridge regression because in addition to its analytical tractability, can be viewed as the asymptotic limit of many learning problems (Dobriban & Wager, 2018; Richards et al., 2021; Hastie et al., 2022). We now formally define some metrics related to bias amplification.
(f( = E [(f(=) ft (z)) s]
Definition 2.1 (Bias Amplification). We isolate the contribution of the model to bias when learning from data with different groups. This intuitive conceptualization of bias amplification allows us to quantify the phenomenon. Grounded in the literature (Bell & Sagun, 2023), we define the Expected Difficulty Disparity (EDD) as:
EDD = IE
ERI(f)) -
(6)
where the expectations are w.r.f. randomness in the training data and any other sources of randomness in the models. The EDD captures the difference test risk between models trained and evaluated on each group separately. In contrast, we define the Observed Difficulty Disparity (ODD) as:
ODD R - E ER((f))
(7)
The ODD captures the bias (i.e., difference in test risk between groups) of a model trained on both ODD
groups. Finally, we define the Amplification of Difficulty Disparity (ADD) as ADD EDD. We say
that bias amplification occurs when ADD > 1. See Appendix for further motivation of ADD.
Ridge Regression with Random Projections. We consider feedforward neural networks in a simplified regime which can be approximated via random projections, i.e., a one-hidden-layer neural network fxx)= = T Sx with a linear activation function. In particular, we extend classical ridge regression by transforming our learned weights as random projection with entries that are IID sampled from Wo 1/d). Ridge regression with random projections offers analytical tractability while exposing bias amplification phenomena related model size; such phenomena are not exposed by classical ridge regression (see Figure 1). Moreover, it has been shown that in high dimensions, training a one-hidden-layer neural network with gradient descent effectively learns a linear predictor over random features (Yehudai & Shamir, 2019). Furthermore, Adlam & Pennington (2020a); Bach (2023); inter alia are able to reproduce interesting phenomena like double descent using the random features model. Nevertheless, Yehudai & Shamir (2019) have shown that the model often cannot learn even a ReLU neuron, suggesting that some mechanisms of bias amplification could be different in nonlinear networks.
SS E R where S E Rdxm is
3 THEORETICAL ANALYSIS
Assumptions. Some of our theorems will require standard technical assumptions that we detail here and in Appendix B. Assumption 3.1 describes the proportionate scaling limits, standard in RMT, in which we will work. These limits enable us to derive deterministic analytical formulae for the expected test risk of models. Our experiments (see Sections 4 and 5) validate our theory.
Assumption 3.1. In the case of ridge regression with random projections, we will work in the following proportionate scaling limit:
n, n1 122 d + 0o, ni/n + pl, n2/n - p2, d/n + à, m/n + 1h, m/d + 7, d/m + Φ1, m/n1 + 11. d/nzz - Φ2, m/n1 + 1/2,
(8)
(9)
for some constants 01, o2 0, Ur, the, vb E (0). The scalar & captures the rate of features to samples. Observe that o = p1 and d captures the rate of parameters to samples. The setting vp > 1 (resp. zb < 1) corresponds to the overparametericee (resp. underparameterized) regime.
o- p262. We note that dy sb and d. vh.. The scalar
4

---

Published as a conference paper at ICLR 2025
3.1 MAIN RESULT: RIDGE REGRESSION WITH RANDOM PROJECTIONS
To provide a mechanistic understanding of how ML models may amplify bias, our theory elucidates differences in the test error between groups when a single model is trained on combination of data from both groups vs. when separate models are trained on data from each group. We first consider the classical ridge regression model in Appendix before studying ridge regression with random projections below, which is a more realistic but still analytically solvable setup.
Single Random Projections Model Learned for Both Groups. We first consider the ridge regression model I with random projections, which is learned using empirical risk minimization and 62- regularization with penalty A. The parameter T of the linear model f is given by the following optimization problem:
き Sij R with w- arg min L(n) EERn
Σ -1, - + Almll
(10)
==1
Explicitly, one can write T S(Z Z + - result for the random projections model, we provide some relevant definitions.
where XS. Before presenting our
Definition 3.1. Let ir A (1/d) ir A be the normalized trace operator and (el, e2, T, U1, U2 p) be the unique positive solution to the following fixed-point equations:
1/T = 1+ LK-
1/e. = 1 1+ vbrtE,KK for 8 E {1,2),
(11)
r' x
Ug === th E, (y pla) for {1,2),
(12)
where: L= - Pi + D2 K K= L + Ala, D = EI +
+ P282 + B.
(13)
For deterministic d x d PSD matrices A and B, we define the following auxiliary quantities:
Ttr 7fr A n,g (A): :=
(14)
h (A, := := py AE b - + - 2
- (p uz Ey +
(15)
hy ( (A, :=
AΣ, (Te p-
p
+ 2 B + pla)
+uj(DynegTTz + >la) >
(16)
(A, B) ==
- Py - E3)
- -XT(eguy + +
(17)
In Appendix H. we intuitively interpret the scalars e,, T, Ua, P in the setting where separate model is learned for each group. In essence, our theory extends the scalars to the more general setting where a single model is trained on mixture of data from different groups. Furthermore, each of the terms h/50 ..... h2 capture the limiting values of different sources of covariance between the sample covariance matrices for the groups, the resolvent matrix, and the random projections matrix S. These sources of covariance are written explicitly in Appendix G, and naturally arise from expanding the solution to the ridge regression problem with random projections.
We now present Theorem 3.1, which is our main contribution. Theorem 3.1 presents a novel bias- variance decomposition for the test eиror Rf( for each group E {1, in the context of ridge regression with random projections. It is a non-trivial generalization of theories in high-dimensional ML which requires the powerful machinery of OVFPT (see proof in Appendix G).
Theorem 3.1. Under Assumptions B.2 and 3.1, it holds that ( ~ B, V.(f), with
v.( Σ 0 20 (Ta. Ed), 1=1
(2)
(18)
B.( TO,E + 1| (0,, th E.) + +2h (0.,, (1) Ex)
1
(3) (0.,
(19)
(0,E, - (0,E.) + 1/ (A, E.y.)
(20)
-2 (0.
s= 1,
-2 (1(0) (
(21)
(A, (2) (0) (A. nnح 1 (AV%),
= 2.
5

---

Published as a conference paper at ICLR 2025
0DD
EDD
ADD
4
4
4
10
10
10
10
10
10
10
10
Figure 1: ODD, F.D, and ADD phase diagrams for ridge regression with random projections. We plot the bias amplification phase diagrams with respect to & (rate of features to samples) and & (rate of parameters samples), as predicted by our theory for ridge regression with random projections (Theorems 3.1, 3.2, 3.2). Red regions indicate theoretical predictions greater than 1 (i.e., bias amplification in the rightmost plot), while blue regions indicate theoretical predictions less than 1 (i.e., bias deamplification in the rightmost plot). Darkness indicates intensity. We consider isotropic covariance 21a, A = la. Additionally, π = 1 x 10a, of = a = We further choose x= A1 X2 1 10 to approximate the minimum-norm interpolator. We show that bias amplification can occur even in the balanced data setting, ie., when p1 = P2 1/2.
matrices: ΣS = 21a, E₂ Is.
The unregularized limit corresponds to the minimum-norm interpolator, and alternatively may be viewed training neural network until convergence (Ali al., 2019). We discuss methods for, and the complexity of, solving the above fixed-point equations in Appendix I. Furthermore, in Appendix J, we directly express the bias and variance of the test risk of unregularized model trained just group in terms of the second and first-order degrees of freedom of Σ, and the parameterization rate v).. Moreover, in Appendix M, we derive the approximate bias amplification profile of an unregularized model with respect to the ratio eigenspectra of the covariance matrices have power-law decay.
o of label noises, in the setting where the
Separate Random Projections Model Learned for Each Group. We now consider the ridge regres- sion models fi and fe with random projections, which learned using empirical risk minimization and fz-regularization with penalties Å and 22, respectively. particular, we have the following optimization problem for each group arg min L(w) n, || Y,112 + >>1lvi12 Alter- natively, the reader can think of each fa as the limit of when pa - In this setting, we deduce Theorem 3.2, which follows from Theorem 3.1.
Theorem 3.2. Under Assumptions B.2 and 3.1, it holds that R, Bo(f.) + VV.(s), where V.(f.) = limp,-.1 Vx(F) and B, B.( =limp,--1 Bx( (see Appendix H for explicit formulae).
Phase Diagram. The phase diagram for the random projections model (Figure 1) offers rich insights into how the rate of parameters to samples (1²), in interaction with the rate of features to samples (o), affects bias amplification. In the ODD and EDD profiles, we observe phase transitions at á = (when & 0.5) and & 0.5 (i.e., 1 र/2 1), where these metrics begin decreasing significantly. tb, = 1 is a known interpolation threshold for random features models (Adlam & Pennington, 2020a; D' Ascoli et al., 2020). In contrast, at &o 1 and ø the ODD drastically increases. Furthermore, at ø = 26 (when vo < 0.5) and 6 = 0.5 (for và > 0.5), the EDD greatly increases. Accordingly, the ADD profile, we observe phase transitions at Φ = va (when vi < 0.5), s 0.5, v = and é 1, where bias amplification begins occurring (i.e., ADD > 1). However, bias seems to be deamplified (i.e., ADD < 1) at & = v (when v < 0.5) and = 0.5 (when > 0.5). Some observations are less visible due the granularity of the color thresholding in Figure 1.
BIAS AMPLIFICATION
We empirically show how ridge regression models with random projections may amplify bias when a single model is trained combination of data from different groups vs. when separate models are trained on data from each group (Bell & Sagun, 2023). We further show how our theory: (1) predicts bias amplification, and (2) exposes new, interesting bias amplification phenomena in various regimes.

---

Published as a conference paper at ICLR 2025
4.1 ISOTROPIC COVARIANCE
Setup. To mirror the setting of Bell & Sagun (2023), we consider balanced data (p1 = P2 1/2)
without spurious correlations (E1 = djld, D2 = agla. a. for 01, a2 > 0). The groups have different ground-truth weights (G = 2Ta, A Ta). Refer to App. K.1 for full details due space limitations.
Validation of Theory. Figure 2 and the figures in Appendix L reveal that Theorems 3.1 and 3.2 closely predict the ODD, EDD, and ADD of ridge regression models with random projections under diverse settings. Note that, as indicated by the error bars, some of our empirical estimates (especially those with larger magnitude) have higher variance and their variance is influenced by the choice of vi, o, d1, 02 oz. Notably, our theory predicts the observation of Bell & Sagun (2023) that models can amplify bias even with balanced groups and without spurious correlations. We present new phenomena predicted by our theory below.
Effect of Label Noise. In the ODD profile, when the label noise ratio c - / oo is larger, the right tail is higher for & (rate of features to samples) closer to than other é. This suggests that under overparameterizztion, larger noise ratio and similar number of features and samples can increase disparities in test risk between groups when a single model is learned for both groups. We aim to explain this phenomenon analytically in Section M. Moreover, the EDD curve is generally higher for larger c, suggesting that larger noise ratio increases disparities in test risk when a separate model is learned for each group. This finding is supported by our experiment with real data (see Figure I1).
Effect of Model Size. We observe interesting divergent behavior & (rate of parameters to samples) increases for different & (rate of features to samples). When Φ > 1, as vi increases, the ODD increases and then decreases, peaking at the interpolation threshold at rb
1. Similarly, when 6 > 0.5 (i.e., 6 = Φ2 > 1), as vo increases, the EDD increases and then decreases, peaking at the interpolation threshold at d/ = 0.5 (i.e., 1/1 = 1h2 = 1). Accordingly, when ø > 0.5, bias is effectively deamplified (ADD < 1) at vo = 0.5 and when á > 1, bias amplification peaks (ADD > 1) at & = In contrast, when Φ 1, the ODD decreases as & increases, plateauing at different finite values. Similarly, when & < 0.5, the EDD generally decreases and plateaus as & increases; in some cases, the EDD dips and/or increases and plateaus. A notable exception to these trends occurs when @ ~ 1, with the corresponding ODD and ADD curves increasing as 16 increases, plateauing at significantly larger value (i.e.. ADD > 1) than the curves corresponding to other values of Φ. We observe similar phenomenon for the EDD curves when Φi = 22 R 1. Hence, overparameterization can greatly amplify bias when the number of features is close to the number of samples. Regardless of the regime of o, the left tail of the ADD profile appears plateau at 1. The right tail plateaus at different finite values, with the curves corresponding to o > 1 consistently platcauing above 1. This suggests that when there are more features than samples, overparameterization amplifies bias.
Some of the peaks and valleys in Figure 2 can be attributed to double descent. However, double descent in high dimensions has primarily been studied in the setting where data are drawn from a single Gaussian distribution; this corresponds to the EDD setting, where a separate model is learned for each group. In Figure 1, we observe a double descent peak in the EDD vh = 1/2 = 1 (Adlam & Pennington, 2020a; D' Ascoli et al., 2020). Our work extends the theoretical treatment of double descent to the setting of training a model on a mixture of Gaussians. However, our theory of bias amplification cannot be reduced to double descent. For example, we note other interpolation thresholds in Figure 1; our use of a linear activation does not have confounding effect here, as interpolation thresholds have also been observed in random features models with nonlinear activations (Adlam & Pennington, 2020b). In addition, much of Sections and 5, and Appendix M, are devoted to studying the tails or limiting behavior of bias amplification with respect to 1 and d.
Effect of Number of Features. In the ODD and ADD profiles, when the rate of features to samples
> 1. the right tail generally plateaus higher values (i.c., greater than 1) when & is closer to 1. This suggests that with a similar number of features and samples, under overparameterization bias amplification increases. In contrast, when ø < 1, the right tail of the ODD and EDD D curves seems to plateau at higher values when & is larger. Regardless of the regime of the rate of features to samples o, the left tails of the ODD and EDD curves are generally higher for larger c.
4.2 REGULARIZATION AND TRAINING DYNAMICS
We now explore how regularization and training dynamics affect bias amplification.
7

---

Published as a conference paper at ICLR 2025
10
8-
6-
8
A20
2-
2-2
2 22
23
20
Figure 2: Our theory predicts that models can amplify bias even with balanced groups and without spurious correlations. We empirically validate our theory (Theorems 3.1 and 3.2) for ODD, EDD, and ADD under the setup described in Section 4.1, with
0.5, o. 1, and o3 =1 10-5, The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We plot ODD and EDD on the same scale for easy comparison, and include a black dashed line at ADD = 1 to contrast bias amplification vs. deamplification. We include all the plots with error bars in Appendix L.
Setup. We revisit the experimental setup for Section 4.1. We modulate a1 02 (rate of parameters to samples), as well as Ả (regularization penalty) to understand the effects of regularization and early stopping on bias amplification. We fix of = o² 1, and the rate of features to samples Φ = 0.75.
Effect of Regularization and Training Time. In simplistic settings, we can simulate model learning over training time t by setting À 1/t (Ali et al., 2019). In the figures in Appendix O, we observe that regardless of the regime of 15, ADD se 1 (i.e., there is neither bias amplification nor deamplification) with high regularization or a short training time. When & > 1 (i.e., in the overparameterized regime), the ADD is generally greater than across values of À (i.e., bias is amplified), while when v < 1 (i.c., in the underparameterized regime), the ADD is generally less than (i.c., bias is dcamplified), Moreover, when v > 1, regularization decreases (or training time increases), bias amplification increases and plateaus. In contrast, when t < 1, as regularization decreases (or training time increases), bias deamplification increases and plateaus. A notable exception to this trend occurs when vi is close to 1, where bias is initially deamplified and then amplified as À decreases (or t increases). This suggests that there may be an optimal regularization penalty or training time to avoid bias amplification and increase bias deamplification. Intuitively, as training progresses, overparameterized models may discover "shortcut" associations (Geirhos et al., 2020) that do not generalize equally well across groups, yielding bias amplification. In practice, an optimal À or can be selected by searching for values that strike a desired balance hetween overall validation error and empirical bias amplification. The search space can be reduced by using the above ADD trends w.r.t. and & that our theory predicts for over- vs. underparameterized models (see Appendix R for more details). It is important for ML practitioners the consider the interplay between high vs. low feature- to-sample regimes and overparameterization in inducing bias amplification vs. deamplification when selecting optimal hyperparameters (see Figure 1).
In general, the calibration À 1ltt may not yield a theoretically tight picture of how bias evolves with f. The use of discrete gradient descent in practice rather than continuous-tin might yield further discrepancies. However, the calibration À = 1/t yields a ratio of gradient flow to most 1.69, with no assumptions on the features X (Ali et al., 2019). Moreover, in the controlled settings considered by Ali et al. (2019), this ratio empirically appears to be quite close 1, and thus may be sufficient for extrapolating our results. Like us, Jain et al. (2024) and Hall et al. (2022) find that bias and bias amplification can vary substantially during training: future work can establish stronger connections between our observations and the results of Jain et al. (2024), who analytically identify phases in the evolution of bias and a crossing phenomenon in the test ervor curves of groups during training. However, Jain et al. (2024) do not consider the effect of over- and imderparameterization
gradient flows
ridge risk that
bias evolution. While analysis relies on the simplistic calibration = 1/t, reveals divergent behavior in how bias evolves depending on model size.
Corroboration on Real Data. We further investigate the effect of training time bias amplification on a more realistic dataset. We train a convolutional neural network (CNN) on Colored MNIST (see Appendix K.2 for more details). Colored MNIST is semi-synthetic dataset derived from MNIST where digits are randomly re-colored to be red or green (Arjovsky et al., 2019). We treat the color of each digit as its group, and we manipulate the groups have different levels of label noise. In
8

---

Published as a conference paper at ICLR 2025
experimental protocol: (1) the color of each digit (in both train and test) is chosen uniformly at random (i.e., with probability 0.5) and independently of the label; (2) by default, in the training set, the labels of red digits are flipped with probability 0.05 while the labels of green digits are flipped with probability 0.25; (3) labels are binarized (i.e., digits 0-4 correspond to O while digits 5-9 correspond batch of 250 instances. Although Colored MNIST is classification task and we use complex CNN architecture, our theory correctly predicts that as the training time increases, the ODD of the CNN is relatively low while the EDD is much larger, producing bias deamplification.
1); and (4) each training step constitutes a step of gradient descent based
Taking & - 30 corresoonds to the setting of X - 0+ in our theory (Theorems 3.1, 3.2). Because we as- sign the colors at random, the only difference in im- age features between groups is color; therefore, we expect the covariance matrices Σ₁ and Eg to roughly coincide and A 0 (i.e., wi w.). Note that we
DDD
EDD
0.4
oimcuity disgans
0.3
do not make any assumptions about the structure of E1, E2. Furthermore, P1 P2 1/2, and thus, Φ1 = Φ2 and t/ = th2- Additionally, we analo- gize the probability of label flipping to label noise in ridge regression. Hence, e1 = 62, 11 W2. Accordingly, lim (f) = lim (f))
0.2
00.1
0.0
20
40 60 80
Training time t
limx-o+ B2( lim B2(f2). Simultane- ously, lim Vi(f) s lim V2(l). However,
Figure 3: Our theory predicts that dis- parate label noise between groups deam- plifies bias on Colored MNIST. We plot the ODD and EDD of a CNN over train- ing time t for Colored MNIST. As in- creases, the ODD is relatively low while the EDD is noticeably higher. The error bars capture the standard deviation com- puted over 10 random seeds.
limx-101 (fi) = o./2.V 1022V==0. =0.05/2-1 V = 0..055V
(where V = o1h1² (La, E)). while limx V2((2) = 02/2- V = 0.25/2- V = 0.125V. This results in ODD 0 while EDD
V., which explains the divergence of ODD and EDD in Figure? 3. Intu- itively, the high label noise for the green digits prohibits the separate model fa from achieving a low test risk compared to f1: the single model f achieves comparable test risk on both groups, effectively deamplifying bias, because of the better learning signal from the red digits. This phenomenon is similar to positive transfer, wherein the EDD of a model generally tends to be higher than the ODD when the labeling rules of imbalanced groups are sufficiently similar (Mannelli et al., 2024). However, Mannelli et al. (2024) do not explore the impact of model size on positive transfer. We show that the ODD can be less than the EDD depending on & in Figure 2, where A = Ia (i.c., the groups have different labeling rules). Future work can study the ADD ODD EDD profile when A 0. Refer to
Appendix P for additional Colored MNIST experiments.
5 MINORITY-GROUP BIAS
Recent work has revealed that overparamettrrizatio may hurt test performance on minority groups due to spurious features (Sagawa et al., 2020; Khani & Liang, 2021). Our theory provides new insights into how model size and extraneous features affect minority-group bias.
Setup. To mirror the settings of Sagawa et (2020); Khani & Liang (2021), we consider diatomic
(A
0
covariance matrices of core and extraneous features. We define A a B
and choose
0
B)
Σ, a1 Ind e (1 Eg u2/nd @ b2 (1-x)d for x E (0, 1), 01, b2 > 0, a 022 Refer to App.
K.1 for full details and a discussion of extraneous vs. spurious features (due to space limitations).
Interpolation Thresholds. The together R2 (i.e., the test risk for the minority group in the single model setting) has different interpolation thresholds as 1b (rate of parameters to samples) increases, depending on & (rate of features to samples) and и (fraction of core features). Notably, as ø increases, the interpolation thresholds occur at larger model sizes, culminating at e = 1. This suggests that for higher rate of features to samples, larger model size can greatly increase the together test risk of the minority group. Furthermore, the interpolation thresholds all occur closer to y = for larger n, collapsing single threshold
when π - (as in Appendix L... Therefore, lower fraction
9

---

Published as a conference paper at ICLR 2025
15
Togeter Rt.
Toguttar Ry
Saparate R.
Saparoate Rs
10-
0
2
Figure 4: Minority-group test risk can peak with different model sizes depending on the rate of features to samples. We empirically demonstrate that minority-groot bias is affected by extraneous features. We validate our theory (Theorems 3.1 and 3.2) for together R1, R2 (i.e., single model learned for both groups) and separate R1 R2 (i.e., separate model learned per group) under the setup described in Section 4.2, with = 2, b2 0.2, and = 0.5. The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We include a black dashed line at ADD 1 to contrast bias amplification vs. deanplification, All y-axes are on the same scale for easy comparison. All the plots with error bars are in Appendix Q.
of core features can yield more possible model sizes that increase the test risk of the minority group. In addition, the together R2 exhibits steeper rate of growth around the interpolation thresholds for larger by, suggesting that higher variance in the extraneous features can also increase the test risk of the minority group in the single model setting. The phenomenon of different interpolation thresholds is not visible for R2 when separate model is trained per group; however, we do observe the expected double descent peaks in the separate Ri and R2 curves at r²1 and x/2 1, respectively.
Overparameterization. The right tails of the together R2 curves plateau at different finite values depending on 6. In partieular, for closer to I, the together R2 curves generally plateau at higher value, suggesting that a similar number of features and samples can exacerbate minority-group bias under overparameterization. Furthermore, for smaller π and certain values of < 1, the right tail of the together R1 curve plateaus at lower value than the together R2 curve. This suggests that there can be differences in test error between groups that are not alleviated even with increased model size. This phenomenon diminishes in magnitude as the fraction of core features increases. This phenomenon supports the finding of Sagawa et al. (2020) that overparameterization with spurious features can increase test risk disparities between groups. We identify that the magnitude of this phenomenon may depend on both the rate of features samples and fraction of core features.
CONCLUSION
We present a unifying, rigorous, and effective theory of ML bias in the settings of ridge regression with and without random projections. Our theory predicts interesting insights into bias amplification and minority-grone -gropp bias in different feature and parameter regimes. These findings can inform strategies to evaluate and mitigate unfairness in ML (see Appendix R for more details). However, there remain practical challenges to assessing whether model is prone to bias amplification. These include robustly estimating the feature covariance matrices (Bickel & Levina, 2008) and label noises (Frénay & Kabán, 2014) for groups from sample data, especially for minority groups which have limited data. Even so, practitioners can use our theory and empirical observations to form intuition about when disparities in the variability of features and labels across groups can amplify bias.
Our theoretical methods are easily extendable to the case of more than two groups and can accom- modate label noise sampled from other distributions. However, our theory is not directly extendable to different proportionate scaling limits (e.g., d /n has finite limit instead of d/n). Additionally, our theory requires approximately normally-distriuute data and thus does not currently account for missing features, which are common in the real world (Feng et al., 2024). Furthermore, our theory implicitly assumes that group information is known, which is not always true (Coston et al., 2019); however, because we work in an asymptotic scaling limit, having access to group information with o(min (n1, n2)) noise is sufficient. As future work, we can leverage "Gaussian equivalents" (Goldt et al., 2022) to extend our theory to wide, fully-trained networks in the NTK (Jacot et al., 2018) and lazy (Chizat et al., 2019) regimes; this will enable understand how, apart from model size, other design choices like nonlinear activation functions and learning rate may affect bias amplification.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Ben Adlam and Jeffrey Pennington. Understanding double descent requires a fine-grained bias- variance decomposition. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 11022-11032. Curran Associates, Inc., 2020a.
Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning, pp- 74-84. PMLR, 2020b.
Alnur Ali, J Zico Kolter, and Ryan Tibshirani. A continuous-ting time view of early stopping for least squares regression. In The 22nd international conference artificial intelligence and statistics, pp. 1370-1378. PMLR, 2019.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprini arXiv: 19XXx09907.228000 02893, 2019.
Francis Bach. High-dimensional analysis of double descent for linear regression with random projections, 2023.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias variance trade-off. Proceedings of the National Academy of Sciences, 116632):15849-1588 15854, 2019.
Samuel J. Bell and Skyler Wang. The multiple dimensions of spuriousness in machine learning, 2024.
Samuel James Bell and Levent Sagun. Simplicity bias leads to amplified performance disparities. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, FAceT '23, pp. 355-369, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701924 doi: 10. 145/3593013.359499
Peter J. Bickel and Elizaveta Levina. Regularized estimation of large covariance matrices. Annals of Statistics, 36:1 199-227,, 2008
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities commercial gender classification. In Sorelle A. Friedler and Christo Wilson (eds.), Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 of Proceedings of Machine Learning Research, pp. 77-91. PMLR, 23-24 Feb 2018.
Andrea Caponnetto and Emesto de Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7:331-368, 2007.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
Amanda Coston, Karthikeyan Natesan Ramamurthy, Dennis Wei, Kush R. Varshney, Skyler Speak- man, Zairah Mustahsan, and Supriyo Chakraborty. Fair transfer learning with missing protected attributes. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES '19, pp. 91-98, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450363242. doi: 10. 30.1144/3306111.3334422
Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: the crossover from the noiscless to noisy regime. Journal of Statistical Mechanics: Theory and Experiment, 2022(1 1):1 nov 2022.
Stéphane Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble double descent: Bias and variance(s) in the lazy regime. In Hal Daumé III and Aarti Singh (eds.), Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2280-2290. PMLR, 13-18 Jul 2020.
Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. Does object recogni- tion work for everyone? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 52-59, 2019.

---

Published as a conference paper at ICLR 2025
Edgar Dobriban and Stefan Wager. High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics, 46(1):247-279, 2018.
Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The of regression. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomczak, and C. Zhang in Neural Information Processing Systems, volume 37, pp. (1979-47013. Curran Associates, Inc., 2024a.
Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. Strong model collapse. arXiv preprint arXiv:2410.048.0 2024b.
Reza Rashidi Far, Tamer Oraby, Wlodzimierz Bryc, and Roland Speicher. Spectra of large block matrices. arXiv preprint cs/06 10045, 2006.
Raymond Feng, Flavio Calmon, and Hao Wang. Adapting fairness interventions to missing values. Advances in Neural Information Processing Systems, 36, 2024.
Benoît Frénay and Ata Kabán. A comprehensive introduction label noise. In The European Symposium on Artificial Neural Networks, 2014.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1):665-673, 2020.
Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mézard, and Lenka 7de- borová. The gaussian equivalence of generative models for learning with shallow neural networks. In Mathematical and Scientific Machine Learning, pp. 426-471. PMLR, 2022.
Melissa Hall, Laurens van der Maaten, Laura Gustafson, Maxwell Jones, and Aaron Adcock. A systematic study of bias amplification, oct 2022.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949, 2022.
Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 771-787, 2018.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization neural networks. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
Anchit Jain, Rozhin Nobahari, Aristide Baratin, and Stefano Sarao Mannelli. Bias in motion: Theoretical insights into the dynamics of bias in sgd training. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 24435-24471. Curran Associates, Inc., 2024.
V. Kargin. Subordination for the sum of two random matrices. The Annals of Probability, 43(4):2119 2150, 2015.
Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups disproportionaaelly In Proceedings of the 2021 ACM Conference on Fairness, Accountabiliiy, and Transparency, FAccT '21, pp. 196-205, New York, NY, USA, 2021. Association for Computing Machinery. iSBN 9781450383097. doi: 41444/3344188834455333
Donghwan Lee, Behrad Moniri, Xinmeng Huang, Edgar Dobriban, and Hamed Hassani. Demys- tifying disagreeemenn-on-ttee
high dimensions. In International Conference on Machine Learning, pp. 19053-19093. PMLR, 2023.
Klas Leino, Matt Fredrikson, Emily Black, Shayak Sen, and Anupam Datta. Feature- wise bias amplification. In International Conference on Learning Representations, 2019.
12

---

Published as a conference paper at ICLR 2025
Alexander Maloney, Daniel A Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprint arXiv:2210.1655 2022.
Stefano Sarao Mannelli, Federica Gerace, Negar Rostamzadeh, and Luca Saglietti. Bias-inducing geometries: exactly solvable data model with fairness implications. In ICML 2024 Workshop on Geometry-grounded Representation Learning and Generative Modeling, 2024.
V.A. Marčenko and Leonid Pastur. Distribution of eigenvalues for some sets of random matrices. Math USSR Sb, 1:457-483, 1967.
James A. Mingo and Roland Speicher. Free Probability and Random Matrices, volume 35 of Fields Institute Monographs. Springer, 2017.
Anaelia Ovalle, Palash Goyal, Jwala Dhamala, Zachary Jaggers, Kai-Wei Chang, Aram Galstyan, Richard Zemel, and Rahul Gupta. "I'm fully who am": Towards centering transgender and non-binary voices measure biases in open language generation. In Proceedings of the 2023 ACM Conference on Fairness, Accountabiliry, and Transparency, FAccT '23, pp. 1246-1266. New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701924. doi: 10.11445/59330133359900
Dominic Richards, Jaouad Mourtada, and Lorenzo Rosasco. Asymptotics of ridge (less) regression under general source condition. In International Conference Artificial Intelligence and Statistics, pp. 3889-3897. PMLR, 2021.
Megan Richards, Polina Kirichenko, Diane Bouchacourt, and Mark Ibrahim. Does progress on object recognition benchmarks improve generalization on crowdsourced, global data? In The Twelfth International Conference on Learning Representations, 2024.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameteriizatoon exacerbates spurious correlations. In International Conference on Machine Learning, pp. 8346-8355 PMLR, 2020.
Stefano Spigler, Mario Geiger, Stéphane d' Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under-to over-paramettizzatoon affects generalization in deep learning. Journal of Physics A: Mathematical and Theoretical, 52(47):474001, 2019.
Nilesh Tripuraneni, Ben Adlam, and Jeffrey Pennington. Overparameterization improves robustness to covariate shift in high dimensions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.
13883-13897. Curran Associates, Inc., 2021.
Angelina Wang and Olga Russakovsky, Directional bias amplification. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10882-10893. PMLR, 18-24 Jul 2021.
Sierra Wyllie, Ili Shumailov, and Nicolas Papernot. Fairness feedback loops: training on synthetic data amplifies bias. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, FAceT '24, pp. 2113-2147, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704505. doi: 10. 0444///63116.365900
Gilad Yehudai and Ohad Shamir. On the power and limitations random features for understanding neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bos Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
John R. Zech, Mareus A. Badgeley, Manway Liu, Anthony B. Costa, Joseph J. Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLOS Medicine, 15(1 l)xe nov 2018. ISSN 1002683.
1649-1676. doi: 10. 10.13a//j
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. Association for Computational Linguistics. doi: 10. 1/136////DDD1-1333.
Copenhagen, Denmark, sep 2017.
13

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
A Warm-up:
Deriving Marchenko-Pastiee Law via Operator- Valued Free Probability Theory
Step Constructing Linear Pencil
A.2 Step 2: Constructing the Fundamental Equation via Freeness
16
A.3 Step 3: The Final Calculation
17
B Technical Assumptions
18
C Related Work (Continued)
19
High-dimmnssionll analysis of bias.
19
Bias amplification metrics.
19
D Warm-Upe Classical Linear Model
20
D.1. Single Model Learned for Both Groups
20
D.2 Separate Model Learned Per Group
20
D.3 Phase Diagram..
21
E Proof of Theorem D.2
22
E.1 Variance Term
22
Bias Term
24
F Proof Theorem D.1
F.1 Variance Terms
26
Bias Terms
Proof Theorem 3.1
36
G.I Computing Etr /
37
G.2 Computing Etr
38
G.3 Computing Etr 3
40
G.4 Computing Etr r
Thcorem 3.2
Solving Fixed-Point Equations for Theorem D.1 Proportional Covariance Matrices
45
L.
45
I.2 The General Regularized Case Unregularized Limit
45
1.3
47
J
Corollary J.1
49
J.1 Case 1: 0o 0
49
J.2 Case 2: 0, > 0
50
K Experimental Details
52
K.1 Synthetic Experiments
52
Setup for Section 4.1.
52
Setup for Section 5.
52
Extraneous vs. Spurious Features.
52
14

---

Published as a conference paper at ICLR 2025
K.2 Colored MNIST Experiments
53
L Bias Amplification Plots
54
Power-Law Covariance
57
N Proof Corollary M.1
0 Bias Amplification During Training
60
P Colored MNIST Plots
61
Minority-Group Bias Plots
62
Actionable Insights from Theory
65
Searching optimal hyperparameters.
65
Informing evaluation and miligation strategies.
65
15

---

Published as a conference paper at ICLR 2025
A WARM-UP: DERIVING MARCHENKO-PASTUR LAW VIA OPERATOR- VALUED FREE PROBABILITY THEORY
We provide a detailed example of how apply linear pencils and operator-valued free probability theory (OVFPT) to derive the classical Marchenko-Pase -asttrr (MP) law (Marčenko & Pastur, 1967). Let = (1/n)x X Rdd be the empirical covariance matrix for an n d random matrix with IID entries from (00, 1). If n tends to infinity while d is held fixed, then S converges to the population covariance matrix, here Σ I.. If d also tends infinity, then the limit seizes to exist. It turns out that one can still make sense of the limiting distribution of eigenvalues of S in the case d/n stays constant, i.e.,
n, - -d xc, d/n - E (0,
(22)
In partieular, seek to understand the behavior of the random histogram:
A
(23)
where "I,..., Ad are the eigenvalues of S. In the aforementioned limit, 7n converges to a determin- istic law μMP on R. called the MP law. This is central to the field of random matrix theory (RMT), primary tool in probability theory, statistical analysis of neural networks, finance, etc. We are interested in an even more powerful tool free probability theory (FPT) which is powerful enough to give precise picture of deep learning in certain linearized regimes (e.g.. random features, NTK) and interesting phenomena (e.g., triple descent) via analytic calculation.
A.1 STEP 1: CONSTRUCTING A LINEAR PENCIL
For any positive A, consider the 2 block matrix Q defined by:
I
X
Q
(24)
X
La
Vnd
Let fr be the normalized trace operator on square matrices and set y = E fr. This gives random
(n d) x (n d) matrices the structure of a Neumann algebra A. Define 2 x 2 matrix
G = G(Q) by:
G = (I2 2 p0 i.e 9i.i <((Q~))..) = [), for all i,j E (1,2).
(25)
Thus, the operator (I2 Ø p)Q-1 extracts the expectation of the normalized trace of the blocks of the inverse of the a 2 x 2 block matrix Q
Observe that:
E (S ALa)- 92,2
(26)
This is a direct consequence of inverting a x 2 block matrix (namely Schur's complement). The mechanical advantage of Equation 26 is that the resolvent (S + Ala) depends quadraticalyy on X while 92.2 is defined via Q, which linear in X. For this reason, Q is called linear pencil for (S + ALa) -1 The construction of appropriate linear pencils for rational functions of random matrices is a crucial step in leveraging FPT.
A.2 STEP 2: CONSTRUCTING THE FUNDAMENTAL EQUATION VIA FREENESS
For any B E (C)+ define a block matrix B & 1A by:
(bijaa
if dj
[ [B 14l0
(27)
00 clse
16

---

91.1 = X + 792,2 Published as a conference paper at ICLR 2025
Here, b b is the number of blocks in the linear pencil Qx, that is, 2. Now, observe that
write Q F Qx. where:
(_
la
F
0 1, 12 60 1 and Qx =
(28)
x
0
Vnd
One can then express = = (I, & -0))=====------0-0 (I, p) Qx)=
-! From operator-valued FPT, we
know that in the proportionate scaling limit given Equation 22, the following fixed-point equation (due to the asymptotic freeness of Qx and F) is satisfied by G:
G (4s @ (F R @ LA)
(29)
where RQxx (G), and Rox the R-transform of Qxx which maps (C itself like so:
Rox (B)is
Σ
(30)
k.C
Here, a(i, k; E,j) is the covariance between the entries of block (i, k) and block (E,j) of Qx, while ag is the dimension of the block (k, E).
A.3 STEP 3: THE FINAL CALCULATION
By the structure of Qx, compute from Equation 30:
71,1 = d nd 92.2 = 2
A 92.2.
(31)
71,2 = 0,
(32)
72, O,
(33)
222 = n
91,1 = 91.1-
(34)
nd
Combining this with Equation 29, one has:
1 + (7/2)g222
G - (12 RT-
1 12.21
(35)
\/((A + 192.2)
x/(A+ (1,))
Comparing the matrix entries, this translates the following scalar equations:
A
(36)
X
92,2 = À + 91.1
(37)
92.1 = 91.2 = 0.
(38)
Plugging the second equation into the first (to eliminate 91.1) gives:
92,2 X A/(2 792.2)
Setting m 92.2/ / then gives m (A + 1/(1 + 7mm)
i.e.
1 A
(39)
m
1+ami
which is precisely the functional equation characterizing the Stieltjes transform (evaluated at X = -) of the MP law with shape parameter 7. By treating as a complex number and applying the Cauchy-inversin formula, we recover μMP.
17

---

Published as a conference paper at ICLR 2025
B TECHNICAL ASSUMPTIONS
Assumption B.1. In the case of classical ridge regression, we will work in the following proportionate scaling limit:
n. n1. n2, - oc, n1/n P1, R2/n P2, d/n1 + 01, d/n2 + 62, d/n Φ. (40)
for some constants 01 02 @ E (0, 0o). The scalar & captures the rate of features to samples. Observe that & p1 and p202-
Assumption B.2. The per-group covariance matrices Σn and Ea and ground-truth weight covariance matrices & and are all simultaneously diagonalizable; hence, all these matrices commute.
While Assumption B.2 may appear reductive, our goal to analyze the bias amplification phenomenon in sufficient setting that does not introduce complexities due to non-commutativity. Notably, our main theoretical result does not assume isotropic covariance. For example, our theory accommodates diatomic covariance (see Section 5) and power-law covariance (see Appendix M).
Assumption B.3. In Corollary M.I, we assume the following spectral densities exist when d + oc:
12g of the eigen- v E P(R+) is the limiting spectral density of E of the ratios 1 values of the respective covariance matrices.
11 E PP(,,, [R4 is the joint limiting density of the spectra of E and Ex,
ग E EP(44) is the limiting density of the spectrum of A.
18

---

Published as a conference paper at ICLR 2025
C RELATED WORK (CONTINUED)
High-dimennional analysis of bias. Mannelli et al. (2024) employ the replica method, which is non-rigorous, while we OVFPT, which is entirely rigorous. Moreover, Mannelli al. (2024); Jain et al. (2024) study the application of linear classification to Gaussian data with isotropic covariance; in contrast, we study the application of regression with random projections (a simplified model of feedforward neural networks) to Gaussian data with more general covariance structure (i.e., covariance matrices that are simultaneously diagonalizable) and noisy labels. This allows us to analyze the effects of these additional factors on bias. We make additional connections between our work and Mannelli et al. (2024); Jain et (2024) Section 4.2.
Bias amplification metrics. Our definition of ADD is consistent with the conceptualization of bias of Bell & Sagun (2023). At a high level, our definition quantifies how many times worse model bias would be if a ML. practitioner opted train single model mixture of data from two groups (i.e., the setting in which bias is observed in practice) vs. separate models for the data from each group (i.e.. the setting which corresponds
the bias in the data alone, and thus the priori amount of bias we would expect in the case of single model). In sum, we seek to isolate the contribution of the model bias when learning from data with different groups.
19

---

Published as a conference paper at ICLR 2025
D WARM-UP: CLASSICAL LINEAR MODEL
Technical Difficulty. The analysis of the test errors (e.g., R,(f) amounts to the analysis the trace of rational functions of sums of random matrices. Although the limiting spectral density of sums of random matrices is a classical computation using subordination techniques (Marčenko & Pastur, 1967; Kargin, 2015), more involved analysis required in our case. This difficulty is even greater in the setting of random projections (see Section 3.1). Thus, we employ OVFPT to compute the exact nigh-dimmnssonal limits of such quantities. We derive Theorems D.2 and D.1 using OVFPT (in Appendices F and E). Theorem D.I is non-trivial generalization of Proposition from (Bach, 2023), which can be recovered by taking Ps + (i.e., Ps 0).
D.1 SINGLE MODEL LEARNED FOR BOTH GROUPS
We first consider the classical ridge regression model f. which is learned using empirical risk minimization and fg-regularization with penalty X. The parameter vector E IR of the linear model f is given by the following problem:
W arg min L(w) = Σ En'ww Yall2 + Al|w||i. weRd
(41)
The unregularized limit 0+ corresponds to ordinary least-squares (OLS). We provide in Theorem D.1 novel bias-variance decomposition for the test error R, (f) for each group s E {1,2). We first present some relevant definitions.
Definition D.1. For any group index s E (1,2), we define (ei, E2, (()) (l))) to be the unique positive solution to the following system of fixed-point equations:
1/e, 1 otr otr E.
(8) uk = Φetr (pru) (e) E p2u2 (8) E2 E.)K Ke (1,2), (42)
where = P1c1 + P262L2 + AIa and fr = (1/d) tr is the normalized trace operator.
The fixed-point equations for e, are non-linear and often not analytically solvable for general Σi, Ez. This is typical in RMT.
Theorem D.1. Under Assumptions B.2 and B.1, it holds that: Rs B.(f) + vo(f), with
Vv( ( (f)
(43)
v( = PROPØt En (e+E, Au
(5) Ta + pk Ex e K-2
(44)
B.) = B) (33 ( + f0.
8=1,
(45)
(2b! (f), 5=
B(" = p. AE, (px (1 + p.u ee² Eg E, + u (pecoE + X(a)
(46)
B/21 (f) = PIA E, (1 + 22 e E2 (2) +N))
(47)
B(3) (f) A (p (5) 31 p2u (3) Eg E.KK?
(48)
where 1' -/=22 and 2' 1.
D.2 SEPARATE MODEL. LEARNED PER GROUP
We now treat the case of fitting separate model f. per group. Suppose that the classical ridge regression models Fi and f2 are learned using empirical risk minimization and Ez-regularization with penalties ar and de, respectively. In particular, we have the following optimization problem for each group s:
| Y.Il2 + A Nellw |
arg min L(w)= wERd
(o,T v.)2 + Aallwali (D
Σ
(49)
The
11
We first present some relevant definitions.
20

---

Published as a conference paper at ICLR 2025
Definition D.2. Let (r (1) ir sig (Exx + Ila) and Ki be the unique positive solution to the equation K₂ A₂ df (K.).
In this setting, we deduce Theorem D.2. Theorem D.2. Under Assumptions B.2 and B.I, 1, it holds that:
Ro(fa) Bf(fa) + V.(fs), with
(50)
0. 02 (5) (Nis) (f. 12tro,2 (E, kold)
Vo(fa)
(51)
oad -1 (Kis)
2 (Ma))
D.3 PHASE DIAGRAM
We present the bias amplification phase diagram (Figure 5) predicted by Theorems D.1 and D.2 for the classical ridge regression model. The phase diagram offers insights into how (rate of features to samples) affects bias amplification. To obtain the precise phase diagram, we solve the scalar equations numerically. the ODD profile, we observe an interpolation threshold at Φ = To the right of the threshold, observe tail that descends towards 1. To the left of the threshold, the ODD descends below with local minimum at á &s 0.25 before increasing. In contrast, we observe that the EDD continually grows & increases, ascending from small value, exhibiting inflection point at = 0.5, and plateauing after & = Accordingly, the ADD increases significantly as e decreases (with an intermediate inflection point at ø 0.5), peaks at ø 1, and descends towards 1 as increases (i.e., bias remains amplified in this phase). In sum, bias is most amplified when the rate of features samples Φ and Φ 1. Interestingly, bias amplification consistently occurs (i.e.,
ADD > 1) across all observed values of ø.
101
107
101
10-2.
10-2
000
EDO
ADD
10*
20-4
10-9
10-4
10-4
101
10-3
Figure 5: ODD, EDD, and ADD phase diagrams for classical ridge regression. We plot the bias amplification phase diagrams with respect to ó (rate of features to samples), as predicted by our theory for ridge regression without random projections (Theorems D.1, D.2). Dashed black lines indicate theoretical predictions. We consider isotropic covariance matrices: Σ; 21a, Ee la. θ 2/a- A = X2
= Ja. Additionally, n = 10a of of = We further choose approximate the minimum-norm interpolator. We observe that bias amplification can occur even in the balanced data setting, i.c., when P1 = P2 = 1/2, without spurious correlations.
10 to
21

---

Published as a conference paper at ICLR 2025
E PROOF OF THEOREM D.2
Proof. We define M, = X, X, and Eg = Y, - X Note that Wg = (X, ngdgla) (ada)=-x (X,w, + E,) (M, + - M,w, + (M1 +nddgla) Xx Es We deduce
that (f.) B.(f.) + V.s)... where:
(f().) = BII(M, +ndddla) 1a
M - - v.(f.) = E (I,,, + n2A,Ta) -1x E.
(52)
(53)
E. VARIANCE TERM
Note that the variance term Vf( of the test error of f, evaluated on group is given by:
Vo(Fa) a. tr (M, + m.A.la) (M, + -, X
(54)
= ir (M, + m... nad.la) -E
(55)
We can re-express this as:
n2 V...).
of (H, \ X.1 + -
(56)
of Er Etr (Ha/>. 1a) (H.,/ / Ia) -E
(57)
A
where H, X Xs/n,, and X, 7, _,/ wwtt Z1 E Rh xd and Zz R being independent random matrices with IID entries from NN0 1). Thus, the variance term is proportional to:
tr (Hl + Xalaa= H, + \=
(58)
WLOG, we consider the case where 1. The matrix of interest has linear pencil representation given by (with zero-based indexing):
(H//a1 +aa +a) (H.A)) + I Q
(59)
where the linear pencil Q is defined follows:
la
0
0
0
0
JAn 19 In
0
0
VAL
0
Ia
0
0
0
Q
0
Ia
VALyAI ZJ
0
(60)
0
Z1
VALVIE
0
0
Ia
-X
Σ
0
Ja
0
0
0
Ia
We compute Q using the NCMi ni ma Descri resssriittRReeee iizttii inn function of the NCAlgebra library'. We further symmetrize Q by constructing the self-adjoint matrix Q:
Q= Q
0
Q
(61)
0
This enables us to apply known formulae for the R-transform of Gaussian block matrices (Far et al., 2006). We note that Q0.17 = Q Taking similar steps as Lee et al. (2023), we use OVFPT on Q. Let G = (Jis @ E E R 188118 be the matrix whose entries are normalized traces of blocks of Q We provide detailed example of how to apply OVFPT derive the MP law in Appendix A. One can arrive at that, in the asymptotic limit given by Equation 40, the following holds:
Go.77
Etr (1I + HA + (a
X1
(62)
with Go.17 = (Gs,14 G2 (E, + Ta) En (E1G5,14 + AT) -1 E1.
d
'ttpps: :/ //t.thttbb.tt////////////t.... com NCA WCCAgeebra/Ne /NC By convention, the trace of non-square black zero.
22

---

Published as a conference paper at ICLR 2025
We will now obtain the fxee-point equations satisfied by G2 and G5,14. We observe that:
A
22,11 ->1
G3,10 = - (2, + =
(63)
G3,10
7 G2,11 1
(64)
+ A1Ta)-1>
A1
G5,14 = --===----- - + Φ) 13,
G6 = - 1 (2) + Atla)
(65)
Gs,14
(66)
1 1 (2) G5,14 + +1da) -1
We recognize that we must have the identification é1 = G2 = Gs.14, where e1 2 0. Therefore:
61
(67)
c + बी ( (N1/21)
ie, = e + Φ df (A1/e1) = \1/11 + df
(68)
K1 \ + k Φ | (mi)
(69)
where (1) = fr Eg (E, + tTa) and x₁ = >//e1. Additionally:
11113
G3.13
G2,11
= QiET
(70)
(->1 + (-x
A
G3,13
(T + (1 +) ( -
(71)
G2,14 dr (k1) + A (Σ + Anla) E1,
=
et
(72)
G3
tr (Eie) + -E.
(73)
Then:
2 (1-0h
G3.10 + G3.13)
G5,14- G2,11 = -2,11
(74)
d1
G3,10 + +3,13
G2,14
di (k1) + A1Tr (Eie) + AT -2
(75)
A
oî
(r(ie
+ lala)
(Eier + E
(76)
G2 (mI)
El fT (kI)
of
(77)
Gs.11 G2.11 15.11-221100 (K))
(78)
of
We define:
G5,14 -G2,14 1 + dIc (1)
(m1) (xi))
q 2 1,c
(79)
i.e.,
(80)
1- 01
(1) (m1)
Hence:
Go.17 cıdr (k1) (k1)
dr. (k1) 1 cT (151)
(81)
In conclusion:
K1 = A1 + of
(82)
oî (L) (K1)
vi(hi)
(83)
1 d- (K1) (11)
23

---

Published as a conference paper at ICLR 2025
Following similar steps for (2(f)), get:
Kig = 22 + k202dl1 (Niz):
(84)
V2((2) o (122)
(85)
1 - 22d12 (2) (K2)
To further substantiate our result, let consider the unregularized case where A 0 and Φ2 1:
0.0.
Kg = 0. =
(86)
From an alternate angle, we know that:
R2 (fa) = E 10 = 1B X,) -1 x F.
(87)
= of Xx(X, xx) Ex X,)-1x; X
(88)
d
0
of (X. x.) EE ng d tr Ia 0 n2 1-11
(89)
1 0z
where we have used Lemma E. below.
Lemma F.1. Let n. and d be positive integers with n 2 + 2. If Z is an n. x random matrix with IID rows from ((, E). then:
B(Z 1)
n d-1 E-1
(90)
E.2 BIAS TERM
We can compute the bias term Bz(fs) of the test error of evaluated on group s as:
B.. E |I n.AaTa) M, u w. 1
(91)
[M + naA.Ta) - u - (M, + nn,nn,,,,h n,
+ n. maAoTa)m:313.
(92)
E (M,, + n.A,daa n,,,
(93)
=nggBB tr (M, + n.A,Ta) (w" (M, -nd
(94)
We can re-express this as:
(Z. EGr (H, -1) OH, A, Ta) -'S,
(95)
X
B.) = (H,/A, + 1 + Ia) -E.,
(96)
(0, -
==1 WLOG, we consider the case where 1. The matrix of interest
where θ
10+ ++ A.
2 2"
has linear pencil representation given by (with sero-based indexing):
(Hı/A1 + Ta) (H,, /A1 Ta)- = Q
(97)
where the linear pencil Q is defined as follows:
la
la
ZT
%
VAyA
Is
0
0
Q
(98)
0
La
0
Z7
VAVA
0
I...
ViVR
0
0
0
1a
24

---

Published as a conference paper at ICLR 2025
We note that 2o,17 Q0.s Using OVFPT, we deduce that, the limit given by Equation 40, the following holds;
Er 1 + 1a) (H1 + - Go.17.
(99)
with Go.17 A (2, + (PTي) (A1 12,) G6,15
A1 E1.
(100)
We will now obtain the fixed-point equations satisfied by G2,11 and G6,15. We observe that:
A
G2 = ->1 + $1 G3,10
G3,10 = - 2i C2, + A
(101)
-2,11 1 Φnli En (E1 A1aa)-1'
(102)
A1
G7 = - (6, + Aela) G6,15 = -Y1 + 61G7,1
(103)
= Go,15
(104)
1 Φn E₁ (E,Go,15 + Aala) -I
We recognize that we must have the identification
G2,11 = Go,15. where ei 2 0. Therefore:
(105)
1 0i En (E1e) + Aa
i.e. K1 A1 + 1 (M1).
(106)
Additionally:
\1σI G3,14
Φιei G3,14
G2,15 (->1 G1G3s (-x1 G7,14)
(107)
A1
G3.14 = fr (E, + Ta) G2,12,15 + 110)E1
(108)
A1
G2,15 di (1) (k1)
\1 ti (i) kala) -021,
(109)
G2,15
+1 G2 (m1)
+ A (Σ) + ka ?0E,,
(110)
\1
i.., G2,15
tr (E1 + k2la) A0
(111)
1 - 21
(1) (k1)
Hence:
Gu = -in=R,th N²tEr (E1 + 12 -2 0Σ, dr 2allz(10 (R1)
(1)
G2,15
(112)
\1
x d (m1
= kfft (21 + K5 42) -2 +ズ; + ト
tr (2) + kn
(113)
1 1 و1 df (MI)
)
-11 1 - (1+1-
2IdI (K5)) (k1)
1rtrr (E, Tx) -2 OΣ
(114)
(kx)) (kI))
In conclusion:
tr (Σ) + Ki I1) 2
Bi(s)
(115)
1 1 و1 dı (M1)
(1)
Following similar steps for (2)))), we get:
Bx) natrr (E2 + k2)a) "(0 A}E₂
(116)
1 - -2ddf? (2) (122)
We observe that in the unregularized case (i.e., X = 0), Ks = 0. In this setting, (f. = 0 as expected.
25

---

Published as a conference paper at ICLR 2025
F PROOF OF THEOREM D.
Proof. We define M X nAla. Note that one has:
④ M (M +X Ei + M2 + X22 E2).
(117)
We deduce that R&(f) = B.(f) + Vo(f). where:
BA( E || M M M w
(118)
vo(f) = 1I Ei + X2 E2) 12.
(119)
= -EIIM-XX E, |3.5 + M- X2 E2 E213..
(120)
with 5' 22 11 2
s=1 1
F.1 VARIANCE TERMS
Note that V2(f) of the test error of f evaluated on group is given by:
v.( oi tr X xx + + tr X2M 2.M-LMM--L
(121)
= of tr M E M
M
(122)
We can re-express this as:
mvvi = of (H Ala) HH + Ala) -aa) -E, + + tr (H + - + Ala) -1a) -E,(31)3))
where H - H1 H2, H, = X] and X, 2 with Zı I and Zz Raaxd being independent random matrices with IID entries from (o0 1).
WLOG, we focus on tr (H Ala) H2 + AI,) The matrix interest has linear pencil representation given by (with zero-based indexing):
HH /X H2/A -
1) = QT -
(124)
26

---

Published as a conference paper at ICLR 2025
where the linear pencil Q is defined follows:
(125)
21
0
0
0
0
12
0
0
Using OVFPT, we deduce that, the limit given by Equation 40, the following holds:
EG (H1 + Ho -aa H2 + H2 + Ala) - G1,23
(126)
with:
G1,23 = PA + XGo - pr GoGGSGGG + E, Go,27
(127)
X
E5 + E2 + A2a)
(128)
27

---

Published as a conference paper at ICLR 2025
By identifying identical entries of Q Q- we must have that Ga.20
G6
G10.25 A
Go.15
G2.1r
G18.28 For G6,21 and 12, we observe that:
G6,21 = -X + $07,20
G7 = -tr Ci,21 + +22227217 + Ma))
(129)
1
= G6,21 1 1tr I1 (6,2 + P2 12 Ala)
(130)
X
G2,17 = - -X + GG
G3,15 = - Eg (p1 + P2 + Ald)
(131)
1
- -2,17 1 Øtr In G6 + 12 + \
(132)
We define 7/1
G6.21 A
Ga.r 1/2
- X
with 1/1 2 0, 1/2 2 0. Therefore:
7 - øtr E
(133)
where K = +
+ la. Additionally, by identifying identical entries of Q have that G5,24 Go,25, Go.27 22228. We observe that:
+ m2p2i2
we must
1 = -x + @G
(134)
XpG7.24
G7.24
G6,25
(135)
(-> - + 60 20 ( - (G11.21
A
G7,21 x 2 Kー
2 (p) E G6,25 22 \11)E111
(136)
À
= G6,25 = or - Go,25 + 22 - EI,
(137)
-
G13,28 -X + @G11.27
(138)
X6G3.27
2 G3,27
G2,28 (-X (G3,15) (-X (G14,77)
222113 1/2 X
(139)
G3.27 xx - K 22 1i + 2222222222 >E,)
(140)
X
G2,28 on K 2 (p) 2 Go,25 25 + 22 \E,)222
(141)
We now define 0 G6,25
u2
(+)
= G2.28, with tj 2 0, 0, 2 0. Therefore, u U2 obey
the following system of equations:
of
tn (0 + 22 22 + AL,)EE.
(142)
We further define (.)
Putting all the pieces together:
2 x - ti 222 - u La + p1 En (722) (3) -1 ))
(143)
By symmetry, conclusion:
= =v.vvv + V(2 (s)
(144)
((()())= = -1 11 Pk- Ex (n=E. u/ Ia Pk Ev (n ))) KK=
(145)
[2, k 1
with k'
1, & 2
28

---

Published as a conference paper at ICLR 2025
We now corroborate our result in the limit P2 - 1 (i.e., P1 + 0) and = 2. We observe that:
Φ A + A2,
(146)
(f) 0,
(147)
V2 (221
= tr 12(12 u/2 la)K-²
(148)
A 2202
2 (2)
S2 -22 2122 EO + >2
(149)
02(21) (2) + X2) (2) (1:2);
(150)
Φzd12 (122)
(2)
2
(2)
1 - 22dl2 2 (k2)
(2)
(151)
V220( (f)
k2d11 (N2) (N22) 02 ( E2 + Ia)-
(152)
x 6203
k2dt2
(2) (N2) kz
(2)
tr 2g (22 k2ld) -2
(153)
xxdl1 (2) (k2) K2
(2)
(k2) d1 (K²))
(154)
=x2(1 u2
(2)
Jdl2
(2)
(K2) K2tl
(2) df (2(2) (m2)
(155)
K2 k2 (122)
di (K2)
(2)
(156)
- 222012 (122)
Adr1 (1i2)
(157)
@odl? (Niz)
of oz (2)
(Ka2
v (
(158)
or (122)
which exactly recovers the result for V2(fe) as expected.
F.2
BIAS TERMS
Recall that:
BB( EIIM -EIII--MMMMUSS
M + M M,w; w
(159)
Now, observe that M M w = M-1M - M-1M -M -M2
nAM-1 ww.
Let = wi-- wf. Then:
B. = EIIM (-10- - 1
(160)
= Eir 8T MM.M
(161)
- 22(-1)--'ABB1 5- 1_,M- 1033
(162)
+ +n² (1:) M. Iw
(163)
= - 2(-1) (()) + (
(164)
where:
B (f) = Etr 8T ///////AAA////////////////////// H2 Ia) 'E,,
(F) Etr (H1/A+
(165)
H2 Ia) -w,
(166)
B(O) ( EEr (H1/> H2/X+ 1,) H²/
(167)
Because 8 and wi are independent and sampled from zero-centered distributions:
B ( 0. () (f) Eir (HI/A+ - H*/A+
(168)
'
(169)
29

---

Published as a conference paper at ICLR 2025
WLOG, for B we focus the 1. The matrix of interest has linear pencil representation given by (with zero-based indexing):
BAA-+AAAAAAAA-E///AAAAAAEAAAAAA H//A+ Ia) - = Qi,16.
(170)
where the linear pencil Q is defined follows:
(171)
0000
0000
0. 2 0000
Using OVFPT, we deduce that, the limit given by Equation 40, the following holds:
(/aaa///aaaa//AAAa//////A/////// = G1,33,
(172)
with:
G1,33
Ex -
D1 E, + ATa)
(173)
Gp,22 + +2 -
30

---

Published as a conference paper at ICLR 2025
By identifying identical entries of Q' we must have that 7
Ga.23
Gr,24
G11.88
A
722
G2.19
Ga.20
G14.31
For G7,24 and 23,20, observe that:
A
G7 = -X + "G8,23
G8,23 = -tr En (7,2 P2 +a)
(174)
1
= GT,21
(175)
Otr 21 (p1 p2 Ala)
G3,20 = -- + "G4,19
G4,19 = - Mn (pr E1, + 22 + Ma)
(176)
1
= G3,20 1 øtr Ie +2 Aaa)
(177)
By again identifying identical entries of Q
we further have that o()) G6,27 = -G7,28, (1)
G2 30
-3,31- We observe that:
G7,28 = 2/ Gs,27
(178)
X
G8,27 = - K K
(179)
X
Vi
(1)
= pe K K- 11
(6)
E, + )
(180)
G4.30
G3,31
2 1/2
(181)
G4.30
2 K 2 (p) E G + C2 \E
(182)
)
U22
(1)
= o K (2) (8)
E + (4)
D2 + 1E1
(183)
Putting all the pieces together:
= = -1p===i2(AApp022211 + (1 + uy ( + 1a2
(184)
In conclusion:
Bi
+ D ul (...
(185)
Now, switching our focus to (f) the matrix of interest has linear pencil representation given by (with zero-based indexing):
(Hi/+ H ////////////// H -a) 1
(186)
31

---

Published as a conference paper at ICLR 2025
where the linear pencil Q is defined follows:
(187)
0.000
0 000
Like before, the following holds:
Bir (W1/^+ H2/A+ /////// - G1.25.
(188)
with:
G1,25
=(A,, + AG2 -p2 + 22222222666663 Gp,1E + Go.22 + Ala)
(189)
32

---

Published as a conference paper at ICLR 2025
By identifying identical entries of Q we must have that 7
G2.s8
Ga.19
G11.77
A
722
Ga,22
Gr.23
G14.30
A For G3,19 19 and G7,23, observe that:
A
G3,19 = -X + "Ga,18
= = -tt E1 Po
+ P2.207,25 + Na)
(190)
1
= G3,19
(191)
1 1 + 1 (p: + Ala)
p2
G7,23
Gs.22 = - Ca,1 + p2 P2 + Aa)
(192)
-X + $G8,22
G7,23
(193)
1 1 øtr I2 (pI + 22 Xla)
By again identifying identical entries of Q
we further have that 0i 22,26
(2)
-G3,27, 02
(2)
-G G7,30 We observe that:
G3,27 = 2x²1)/ G4.26
(194)
X
G4,26 - K - 2i G
122)2111
(195)
X
11
(2)
क: K ( pi E
(2)
+
02 12E2 + EE,
(2)
(196)
Gg,29
G7
2 1/2
(197)
G8,29 X 2r K 2 (p: Eg G3.27 G7 1Eg)EEs
(198)
0/2
(2) = or K
(1) (2)
pt
E,
+ 1/2 P2D, + 1E,
(2)
(199)
Putting all the pieces together:
(f = =0, ( Ei (²) la+ 12+ 1241 (22 (2))
(200)
(201)
Finally, switching our focus to 3 by (with zero-based indexing):
(f). the matrix of interest has linear pencil representation given
(H//+ H2/A - H2/A+ -a) =Q
(202)
33

---

Published as a conference paper at ICLR 2025
where the linear pencil Q is defined follows:
(203)
6
0.
22
0
0
Moo
0000
The following holds:
1// H -1);;; G1,23,
(204)
with G1,23 Atr E - P2 E252255
+ XE,, (m) E, G2,17 + P2 E2 + Ala)
By identifying identical entries of Q and following similar steps before, must have the identification 7/1 G2.17
1/2 Gs as well as u (1)
-G2,44 U2 -G5 Therefore, in
(1)
conclusion:
B fr (prup( En + p2ub (")E, -
(205)
34

---

Published as a conference paper at ICLR 2025
In the limit ps - 1 (i.e., Ps - 0), observe that:
Φ + Øz A + As.
(206)
( 0,
(207)
( +0,
(208)
B ( fr +1jE,K
(209)
4 (o) = o. K (v( + >a)22²
(210)
= (v.0 + Xo 20l2 (Mi))
(211)
3 = @sdr (1ix)
(212)
1 ball (Kis)
a(a) 2, + Kala)
(213)
1 c, (Kas
B,( 1
(214)
which matches up exactly with (fa) expected.
35

---

Published as a conference paper at ICLR 2025
G PROOF OF THEOREM 3.1
Proof. The gradient the loss L is given by:
VL(n) Σ S x (X,Sn Y.)/n An Σ ST M,Sn -s Xx, Y,/n+ Y,/nn+ X
= -n Σ S x, Y./n
where H = MS + Alm E R= with M = Mi + M2 and M, = X2 X,Xh/nn X./n. Thus, setting
R = H-1, we may write:
= - SRS (X]Y, Y2)/n
= SSS (Mww + M2w3) + SRS X1 Bijn+ SRS X2 E2/n.
We deduce the following variance decomposition:
1 " = B.( + v.(f), where
vf(f) (f) (f) with ( = ofoBtr M,SRST
E.SRS
B.( IEI (Miwj + M2w3) - w
We can further decompose B,( first considering the case s 1. We define = ==w w
SRST (M + M2w3) will 12
E|[(SRS (M1 M2) Ia) + SRS 1
=E][SSSS' M Ia) E SRS M₂ 15
= =RGrO(MSRS' - (SRS M - Ia) + EIr AM,SRS A E,SRS M₂ -Eir OΣ, + Bir BMrSSs'' SRS M
M + Eir AM M2.
We can similarly decompose B2:
SISRS (Miwi + M2w3) - w 13 E|
(ASRST(MWW (Mw + M2w3) w 1
B][(SRS (M1 + M2) - Ta)ws SRS M
= EEI(SRST M - Ia)w5 1/555133 13. + +EISRS M1 2E (wE) (MSRS" - Ia)E,SRS' M1
Ef - (SRS M - Ia) + Eir EWSSS Mi
2EE A(MSRS" 12)S Mi
Eir 12 ELr R. E2SRS M 2EL 2EL O5S2RRS M
+ Err AM, ERSRS M1 2 AMSRST
AMSRSTE,SSS EySRS M1 + 2Er AE M1.
Furthermore, observe that:
Eir AMSRS B M
(215)
Rir AM, BSRS M1 + Bū AM,SRS BSRST M₂ + 2R AM,SRS BSRS M2.
(216)
Etr ASRS M = Etr ASRST Mi + Etr ASRS M2.
(217)
Hence, we desire deterministic equivalents for the following expressions:
(A) ASRS M,,
(218)
(A, B) AM,SRS BSRST
(219)
(3) (A, B) AM,SRS BSRS M,,
(220)
(4) (A, B) = AM, AMS BSRS M,
(221)
36

---

Published as a conference paper at ICLR 2025
where:
M, = x.y 2y R MS 1 M1 M2,
(222)
M, M, R= XR, - M//.
(223)
In summary:
(f) = 0302
- EEir " (Id, E2),
(2)
(224)
BJ) O,E,
(225)
Bir (3) (6,, -,,) + Bir (3) (0.. E,) + 2Etr (4) (0,, Ex)
(226)
2EET (1 (0,E.) - 2Etr (1) (9,E,
(227)
E (A,Es)
(228)
0, Etr r1 (A, E2) + Etr Etrr (2) (A, Mg) Efrr
s 1,
(229)
(3)
==2
G.1 COMPUTING E (1)
WIOG, we focus on r1 The matrix of interest has a linear pencil representation given by (with zero-based indexing):
11)
Qi,
(230)
where the linear pencil Q is defined as follows:
la
0
S
0
0
00
0
A
la
0
0
0
0
Im
S
0
0
0
D
0
la
>:7
->
ld
VA
Q
0
0
In.
VX
1/1
0
(231)
0
0
Ia
0
0
0
0
12
0
VI
0
1/2
VI
00
0
Ia
0
Ia
Using the tools of OVFPT, the following holds:
Eir (1) G1,21
(232)
with:
G1 =
P2GG.119 + Ala)
(233)
For G5,16 and G8,19, we observe that:
- Go.15 1 + P2 + Ala)
G5,16 -X + @Go,15
(234)
- G5,10 1 22211t "GGasstTB(CGGasaaAAGGGee.e P2E +1a)
(235)
-1
G3 = -X + +GGg,18
G Eo + p2 D2 + Ala)
(236)
Gs
(237)
1 G2,113tr Eel Gr P22 Ala)
37

---

Published as a conference paper at ICLR 2025
We define e1 G5,16. 62 18,19, with e1 2 0,e2 e2 2 0. We further observe that:
1
22,13 1+ G3,11
(238)
G3111 ti (p, 121oG,.10/1021111i2 + P2I2G8,19) + A
(239)
We define 7 = G2,13 2 O. We further define L - p1e1 + P262 K YTL + Ma. Therefore,
we have the following system of equations:
6,
+ Prt E K-1
1 fr L. -K
(240)
In conclusion:
(1) D =P;yeTTrr AE
(241)
G.2 COMPUTING Etr r
WLOG, we focus on ri (2) zero-based indexing):
The matrix of interest has a linear pencil representation given by (with
ri
(2)
Q -1
1,13
(242)
38

---

Published as a conference paper at ICLR 2025
where the linear pencil Q is defined follows:
(243)
areg
00000000000000000000000000000000
The following holds:
Etrr (2) -G1,33
(244)
with:
G1.33 = -E
(245)
P1
(246)
+ Tr.
AG + XGs,15G123ela
(247)
Pa (7G6,26 (pr G3,2s + Tp2229 Ala)
(248)
(7G12,32 (p) G15,35 + E3 + Ala).
(249)
Following similar steps as before and recognizing identifications, we arrive at that:
E1 G3,23 G15,35
(250)
E2 = Gg G18,38
(251)
G6 G12,32
(252)
39

---

Published as a conference paper at ICLR 2025
We now focus on the remaining terms. We observe that:
G4.14
G3,35 = 1
(253)
G4,14
E1 (pT
(n1 D2 XB) AG6,324)
(254)
110,37
G9,38 062
(255)
G10,37
ytr Σ2(
(pi (pi + P2 C3,,35
AB) AGo,32la)
(256)
A
We define 1
Gs.38
X
U2
Go.ss x
with I 0, U2 S 0. We further define D = Pi +
P2u222 E2 B. We now observe that:
G7,31
G6,32
G7,31
(257)
(G7.25 + () +1)
G7 = tr (7Go,32 + K²
(258)
Defining p G6,32, must have the following system of equations:
v, ve Ea(rT2 + pla)K p = r't (xpL² + + >2D) K-
(259)
(260)
In conclusion:
P1 K2,
(261)
P 2yer7 B X 2y - ezu1) + Aeplla NuniTa,
(262)
Eú (2) >p,yfr
eyu;) + eepla Au,TTa)KK²
(263)
G.3 COMPUTING r
WLOG, we focus on ri (3) zero-based indexing):
The matrix of interest has a linear pencil representation given by (with
r]
(3)
QT
(264)
40

---

Published as a conference paper at ICLR 2025
where the linear pencil Q is defined follows:
(265)
8 8
ong 0000
9 9
2000
900
00 2o0 00000000000000000000000000000000 0000
op
holds that B = G1,41. We immediately observe that:
e1 G3,21. G15
(266)
Gy,30, G18,39,
(267)
T G6,27; G12,33
(268)
U1 G3,36
(269)
G9,39
U2 X
(270)
P Ga,33
(271)
In conclusion:
Etr (0) = + + pla) + u, + 2
(272)
41

---

Published as a conference paper at ICLR 2025
G.4 COMPUTING E
(4)
WLOG, we focus on ri (4) The matrix of interest has a linear pencil representation given by (with zero-based indexing):
= Q1
(273)
where the linear pencil Q is defined as follows:
(274)
0000
900 0000
1000 2009
00000000000000000000000000..0..0
000000
holds that Būr (4) = GI,41. We immediately observe that:
é G3,24 G15,361
(275)
e2 Gg,30, G18,39,
(276)
T Go,27 G12,33+
(277)
U1 G3,36
(278)
G9.39
U2 X
(279)
0 Ga,33-
(280)
42

---

Published as a conference paper at ICLR 2025
In conclusion:
Etr : -
(281)
-Arlejuy +
(282)
43

---

Published as a conference paper at ICLR 2025
H THEOREM 3.2
Definition H.1. Let (e1, C22 T1, T2, u1, U2 PI, P2) is be unique positive solution to the following
system of fixed-point equations:
6 1hsTst En( + A²Td) for E {1,2)
(283)
To ( \ala) for {1,2}
(284)
U, = v.e E, (7T?
+ (us (us + pe + A,Id) " for 8 E {1,2]
(285)
Ps = 7/ (7pa
Xus + 1)T + Aala)
for {1,2}.
(286)
For deterministic d d PSD matrices A and B, we define the following auxiliary quantities:
(A) == A2,T3)) -yll)) (A): == ytr
(287)
(288)
h (A) := ir (76 + -
(289)
Under Assumptions B.2 and 3.1, it holds that:
(a(fa) = + Va(f)), with v.(fs) lim v.(f), Balda) = lim Bs(
(290)
p.-1
More explicitly:
va(fa) =
B.( = G + +1. - 20,
(291)
Proof. Theorem 3.2 follows from Theorem 3.1 in the limit p. + (i.e., Ps' - 0).
The scalars c. Ts Mgs Ps can be intuitively interpreted in the setting where a separate model learned for each group. For ridge regression with random projections and À -> 0+,, we show in Equations 337 and 338 that c, Ta are related the normalized first-order degrees of freedom I of the population covariance matrix E.. e, captures the effect of the feature rate d, while captures the effect of the parameterization rate y,. Similarly, for classical ridge regression, we show in Equation
that e, is related the normalized first-order degrees of freedom (r($) On the other hand, u, and Ps be understood as pseudo-variances Indeed, for ridge regression with random projections and À -> 0+, Equations 341 and 349 show that Us Pss V, are all related to the normalized second-order degrees of freedom /2,2 of Σ.,
44

---

Published as a conference paper at ICLR 2025
1 SOLVING FIXED-POINT EQUATIONS FOR THEOREM D.
I.1 PROPORTIONAL COVARIANCE MATRICES
When À 0+, it is not possible analytically solve the fixed-point equations for the constants in Definition 3.1 for general En, Dg. As such, we consider a more tractable case where the covariance matrices are proportional, i.e., Σ1 aj and Σg azΣ, for some E Rdxd
We define 0 pr(appeertappac and 17 = tr E + 0Ia) -1. Then, we have that:
60,77
1/e, - - E,K
(292)
01P161 + 02P2"2
1/T= t 1+ (n/7)/' n/>
(293)
0. O, then T/0 = Therefore, c/ + a1piettaappee which a quadratic fixed-point equation. Accounting for the constraint that 6, > 0, the fixed-point equation requires that ø < 1. Moreover, 7-> 1/7, which requires that 7y
ea.
We further observe that p (I which implies that p - 0. We can then see that, for c E {ai, a2):
Us 2 (a a2p2u2 + c)tr E2K-
(294)
pa + 02p2u2 + c)
(295)
(a1Ple1 a2p2e2)
which is linear fixed-point equation in ug. In contrast, if 0g > 1, we have e, =1 vra.nd and the equation:
70
(296)
(1 (1- m/7)
RYPI OEET (1110/230j Od
GgPa 1110n
which is quartic equation in 7). This highlights the difficulties of rigorously isolating the effects of different components on bias amplification. We empirically investigate how different components (e.g.. covariance structures, group sizes) affect bias amplification and minority-grout bias in Sections 4 and 5, and extensively validate that our theory predicts these implications.
I.2 THE GENERAL REGULARIZED CASE
We now consider the case where the covariance structure is the same for both groups, i.e., Σ₁ Σg Σ. The calculations presented is this subsection are shared with Appendix H.2 of Dohmatob et al. (2024b), of which two authors of this work are also authors.
In this setting, it is clear that 61 = e2 and U2 where (T, u, p) now satisfy:
1/e = 1 17 = v= + p 7"r (7pK3 + XPL'KK L' :- u) (298)
1/T= 1 ir KoK-1 -1 where Ko == == K = 7T Ko + Alas (297)
We first introduce some notation related to the degrees of freedom of Σ, before proceeding with our theoretical result.
Definition I.1. Let df) ir Th (Σ+ + (Ia) for any positive integer m. Furthermore, define ((, = Gr EE - 1bb) for any positive integers a, b.
Lemma I.1. The scalars and p¹ p/(r²2) solve the following pair of linear equations:
u = 22..2 + u) + (1)p)
(299)
TP' = (22) - ( u).
Furthermore, the solutions can be explicitly represented as:
Φ=
02 12
11
/ = 7 0z I (8)
(300)
0z - (2)
45

---

Published as a conference paper at ICLR 2025
where (7) - (0)) + 6 1,,( In particular, in the limit ry - oc, holds that:
(k2
df2(x)/n
0 ~ R. 00 1 2122(())
(301)
df2(x)//n
where K. > O is uniquely satisfies the fixed-poine equation K -xt E(E + *Ia))
Proof. The equations defining these scalars
u == ti + pla) p = ~ + X2T'KK
(302)
(303)
where Ko = =C,, K = Kr + Ala, and == u. + B. Further, since = E, we have u.). Now, rewrite the previous equations like
I' = (1
wh = - 077 ptr ΣK
p 7'tr (xpe + x1) = yr ptr M p + E)
This can be equivalently written as:
u = == 7P' p
(304)
(305)
Now, observe that:
3ee ( 122(0)
(306)
-e= -1/0+ -,,
(307)
AT == =000-20012000 (.2)))
(308)
- -===0(((++))))//m ;
(309)
= = =1,2(0)((77))
(310)
where we have used the definition a/(Tre). Thus, v and p have limiting values which solve the system of linear equations:
u . . 2 22 + u) + , - 2 =220= + + TP' 122( + 8² (1) + u) = (2) 1, (1
ΦJ 2T,2(0))'
where we have used the identity Qy = sb. These correspond exactly to the equations given in the lemma. This proves the first part.
For the second part, indeed, T = 1 - 70/7 + 1 in the limit 7 + co, and so A/(re) which verifies the equation:
A F A - 1 -12)= 0
0 ~ A Aaber ++)==+))-= A A
qe
qe
i.e. 0 A 0 dd, (0)/n and 0 > By comparing with the equation x: - X (x)/nn satisfied by > 0 in Definition D.2, we conclude 0 = k.
Now, Equation 299 becomes p' = 0, and 222 (k) + u), ie..
(12,(KK) df2(m)/n
1 212,2(1))
1- df2(k)))
claimed.
46

---

Published as a conference paper at ICLR 2025
1.3 UNREGULARIZED LIMIT
The calculations presented in this subsection are shared with Appendix H.2 of Dohmatob et al. (2024b), of which two authors of this work also authors.
Define the following auxiliary quantities:
X
A
θ
X
R:=
(311)
"TE
where 7, and are as previously defined.
Lemma 1.2. In the limit À + 0+, we have the following analytic formulae:
X + Xo = - (1-b))+ 70:,
(312)
Kg (e) 1)+ 00/0,
(313)
T + To = 1 - -70071
(314)
ÔTT
(315)
where 0g is the unique positive solution of the fixed-point equation 110 = (00)
Proof. Observe that Kg = eΣ and Ko Al = yTe (E +E+01.) 01a).. Defining == [1,(0),
one can then rewrite the equations defining and follows:
= x UTA EK A
1 + +1a) = A+ one,,
(316)
YTC
Ae
x AEr Atr KoK
A
E + 01a) X (1/7)7/
(317)
YTE
We deduce that:
=
1 -n/²' 7'E = X70.
(318)
In particular, the above means that 17 S min(1 1/6). The last part of equations Equation 318 can be rewritten as follows:
61) - n/-) =0, i.e., 6n)² - (Ø7 + 1)17 +
(319)
This is quadratic equation for 7 as function of and 0, with roots
Φ7 + + V + 1)2 - 4(0: - (0/0)A) +11 ((b + 1)2 4(bb - 6/0)
(320)
2x
2Φ
Now, for small and
* 1, we can do Taylor expansion get:
+1 + 14 -
n
1 X 0(23).
2Φ
0|
More explicitly:
11/0 1/((1~1)0)) 1))),
if <1
=0
( 1/((0-1)0
if
フ = 0(2) + (7- 101-A1(11000))) +)0),
i
11/0-\((11) 1)00)
i
Because 7) $ 1 7), we must have the expansion:
7 1/1a((1110) (((1-+)00) +)0),
if < if
~ 0(A)) +
110 A/((1) - 1)0),
(321)
(1-1))0 A + O(13),
170
47

---

Published as a conference paper at ICLR 2025
provided 8g 0, i. nu # 1. in this regime, we obtain:
7' A
[/((1 - 1+ (11((10-0000))) (1 -4-00 [1/(1 - 1/1 + o(1)) 0,
if 1 1 if >>>
1 7/7
d' = A
[1/(1- & + o(1)) +0,
if <
67
1/(1-11 1+ 1-001+(00((0 1)86) - (v6 1)0b/0,
of > 1
== 7/7 70/7 (1 1//))+
= On ~ 1 ono = (1 - +)+
On the other hand, if 0 = 0 (which only happens if < 1 and 7 > 1, and ø < 1), is easy to see from Equation 318 that must have T' - 0, e' -0, 1- 0 2 0.
L
48

---

Published as a conference paper at ICLR 2025
J COROLLARY J.1
As special case of Theorem 3.1, we recover Corollary J.1, which aligns with Proposition 4 from (Bach, 2023). Theorem 3.1 is non-trivial generalization of Proposition 4.
Corollary J.1 captures how the covariance matrix affects the test risk of a model through the nor- malized second and first-order degrees of freedom of Σ.. Corollary J. also reveals that in the underparameterized regime (4/, < 1), the bias and variance of the test risk of the model strictly increase as a function of qi, (rate of parameters to samples); the test risk of the model explodes (i.e.. there is catastrophic overfitting (Bach, 2023)) when V gets close to regime (v,, > 1), the bias and variance the test risk decrease as 1/' increases.
1. In the overparameterized
Corollary J.1. Under Assumptions B.2 and 3.1, it holds in the unregularized setting X - 0 that
e, (S. 1-
7.th/ <
(fa.
1, 1 S.. S7,
8 e, (0, 1-0.22,(P1))
1,1,1 27
4.-1
(322)
7.4% <
I-
va(fa)
0.
1/ 1, 2 or S 3/ S
(323)
1-0,
(.00) (0)
+ 11
v/ 2 1, v/ スソ
where (t) ur (E + tIa) for any positive integers a, and 0g is the unique solution the
following non non-linear equation:
%
7:1h <
(,0))
1,
1,,,,
1or S....
(324)
1/6s:
1, 27
Proof. Define e' 1/e, 2 0,,00,'1111
= 1/7, 20,0 -0 2 and = (0) E [0, 1). One can then
#=1+ + o.me' express e' and 7'
(325)
X,1a)= - (1///)77 (n/7)7'
(326)
We deduce that:
è' = 1 62l)"
(327)
7' 1 n//
(328)
Ar'e' 70
(329)
We define the following limiting values:
lim
- 8o: lim 17 - 170
(330)
X -
-
lim
6. - co:
lim Ts - To.
(331)
A, -
x, --
lim
u, + uo
lim Ps + Po-
(332)
X,
x. -
There are now two cases to consider.
J.1 CASE 1: 0g 0
This implies To = 1. Therefore, by simple computation, 6n = 1/c6 = 1 2.70 = 0, and
To 1/70 1 1/7. This requires di and y 1.
49

---

Published as a conference paper at ICLR 2025
J.2
CASE 2: 0u > 0
Equation 329 can be re-written as:
X
A
(1 (l - n/7) = -0, i.e. @.1 (1hx 1)n + 7
0.
(333)
0
We solve this quadratic equation for 1/, arriving at the solutions:
17t # (t's + 1)2 4(0's (0,/0)2a) 4 + + (14 + 1)2
4(1', (o./0)1a)
(334)
2Φ,
20h
Taking the limit of 11 as Aa + 0+ gives:
1/ + 1 +
| -
w////s= =7
if 2 1,
17
20.
11/6ar
if < 1,
(335)
1/
114
(1/6a)
if 2 1, if < 1.
2Φ.
1///0s =Y,
Recall that we have the following constraints:
e' 120 2072
n E 10.
We can show that 170 = 1/0, is incompatible with t/. < 1. Indeed, otherwise we would have TO 1/(11 70/7) 1/(1 1/1bs) < 0. Similarly, if c/ 1, would have eo 1 1 Φ,7 1- th < 0. Therefore, 1/o = 17_ Furthermore, if 4,, < 1, it must be that On > 0 and ?/o = 7 Instead, if 4/. < 1, 2 1, we must have that Φ. S 1, and therefore, O and 1/o 1. Similarly, if
v. 2 1 2 1, and 0. I (i.e., S v S 7), we must have that 9, = and foo = 1. In all other cases where v. 2 1, it must be that 7/0 1/é, (which additionally requires o. > th 2 7). Succinctly:
7. < 1
110
I,
v. < 1, 2 o 1 S 3. ST
(336)
1/6..
V. 1, b. 27
Plugging this into Equation 327 and Equation 328 gives:
60 = 6.70 = 1 (00))
(337)
To - = 1 (,00)//
(338)
We will now solve for un and Po/Tă We can re-write u, and Ps/ T as:
P=/T3- - - (u, + 1) 1),,,
(339)
T: = (u. + () + 1 p.. (.)1.
(340)
Solving for and pa/ră yields:
ΦC
7.8
N4g 7 7-06 Φ5 I2,)(0)) (f))
Po/ 2
(341)
7- Φc T,,)(00)
where (2,((0)) (()) 2
(342)
50

---

Published as a conference paper at ICLR 2025
We then see for the variance term that:
= -at.
(343)
= +,2 0Ta)
(344)
21 +
(345)
(346)
- u00 0.0.0010 T0
(347)
1 1 (60)
711 - (.0))
(348)
0.5 + 1,, (0o) -
where & = (0n) 2T,, (222 + (2,)(00) and we have used the fact that T,((0) =
(I,1(0) ( Plugging in (0,) = 170 we have that:
7, v <
vo(fa)
0
1, or v/ ST.
(349)
1
v. 2 1,
2.2
0.-1
where have used that (2,) 12,((0) in the second case.
Likewise, for the bias term, we obtain:
B. B.(F)) = - htt.::t..........eeeeeeetr2rt-.. pola) + & 2
(350)
2yer:.ifo..3((TT..3...11111.. ATTa
(351)
+ 20,, () -
(352)
(353)
( BQ
(354)
(00,2,(00u04a) (Z. +ola)) -2
(355)
+ 00,E,(-2E, 20 E,, (E, 0QIa) -2
(356)
020
+ 0ola) + 11110
(357)
Again, plugging in [,, = Too we have that:
6.6 e, E, (E, 1 -0,
7. <1
B.(fa) +
=0,
14 4 2 or S 14 27,
1 (2,
O, 2. 1.2(6))
Boti (E, 4961a)
42 27
1.
(358)
where we have used that tr (l = and in the second case, 0, = and 22,2 (4))
2
51

---

Published as a conference paper at ICLR 2025
K EXPERIMENTAL DETAILS
K.1 SYNTHETIC EXPERIMENTS
Across all experiments on synthetic data, we choose n 400. We further use 5 runs estimate test risks (e.g.. ER, ()) (f.)), and 5 runs to capture the variance the estimators, for a total of 25 runs. We use 10,000 samples to estimate test risks.
Our experiments validate that bias amplification occurs even in low-dimensional regimes. In Sections 4 and 5, and Appendices I., O, and Q, we show that our theory predicts bias amplification for models trained on only n = 400 samples. The high-dimensional regime is commonly studied in ML theory and statistical physics (as we mention in Section 1.2), as it makes precise analysis more tractable.
Setup for Section 4.1. We further choose À = 1 x 10 6 to approximate the minimum norm interpolator; we henceforth set À = A1 X2 for simplicity. We modulate d1, ,1 of oz. well as to (rate of parameters to samples) and Φ (rate of features to samples) to understand the effects of model size, number of features, and sample size on bias amplification. We consider diverse and dense values of these variables to obtain a clear picture of when and how models amplify bias.
Setup for Section 5. The first xd features represent common core features of groups I and while the latter (1 - π Jdd features capture unshared extraneous features for group 2 (c.g.. spurious features). Intuitively, this setting can model: (1) learning from data from two groups where one group suffers from spurious features (Sagawa et al., 2020), or (2) learning from mixture of raw data (i.e., with spurious features) and clean data (i.e., without spurious features) for a single population (Khani & Liang, 2021). We ask: Does our theory predict how the inclusion of different amounts of extraneous features affect the test risk of minority group (compared to the majority group) when a single model is trained on data from both groups
separate model is trained per group?
Although Sagawa et al. (2020) consider classification instead of regression, to mirror their experi- mental setting, we pick P1 = 0.9 (i.e., group 1 is much larger than group 2) and θ = L2, A 0 (i.e., wi w.). We additionally choose À = x 10-6 and of = o3 = 1. We modulate 01 b2, as well as vi (rate of parameters samples) and é (rate of features samples). Notably, this setting also captures learning problems with o(d) overlapping core and extraneous features in our asymptotic scaling limit. An extremization of this setting is choosing Σ₁ = and + (((1-x)d, Σ₂ 0l /2/ (1-)dd where groups 1 and 2 have no overlapping features.
The experiments in Section validate that our analysis does not rely on conditional dependence heterogeneity. That is, we empirically verify that our theory still holds and predicts bias amplification occurs even when - = w (see Figure 4). In essence, the structure and eigenspectra of the covariance matrices of the two groups still contribute to bias amplification even when the ground- -ruth weights for the groups are the same. In our theory, we only allow the possibility of w + w/ to be as general as possible. In practice, labeling rule heterogeneity may be leveraged, for example, to train a mixture of experts that is regularized to deamplify bias.
Extraneous vs. Spurious Features. Our usage of extraneous features (i.c.. features that are different across groups and correlated with labels) differs from classical definitions of spuriousness (i.e., non- causal correlations between features and labels) (Bell & Wang, 2024); indeed, the extraneous features are used to generate the labels of the minority group. For example, Khani & Liang (2021) model both the labels and spurious features in linear regression as being separately generated by the core features, such that the labels and spurious features are associated but not causally related. However, this setup not encompassed by our modeling assumptions, as it entails that the ground-truth parameter and feature covariance matrices are not jointly diagonalizable. In contrast, Sagawa et al. (2020) study spurious correlations in classification. At high level, Sagawa et al. (2020) create four subgroups of population with different combinations of class labels y E (-1, 1} and group labels a E { (-1, 1,11 1). The core and spurious features are then sampled from normal distributions parameterized by y,, a (respectively) and different variance levels. By setting the spurious features to have a significantly lower variance than the core features and making y and a highly associated (i.e., imbalanced groups), the authors coerce models to perform classification as function of primarily the spurious features of the majority group. which does not generalize to the minority group. To capture this spirit, our setup uses imbalanced groups, and the data for the majority group provides no learning signal for the
52

---

Published as a conference paper at ICLR 2025
extraneous features; this coerces models perform regression function of primarily the features, without learning appropriate parameters for the extraneous features, and thus generalize poorly to the minority group.
K.2 COLORED MNIST EXPERIMENTS
Train-test split. Colored MNIST has total of 60k instances. Each image is 28 x 28 x 3 pixels. We use the prescribed 0.67-0.33 train-test split. We do not perform validation of hyperparameters, which we mostly adopt'.
Model architecture. By default, our CNN architecture consists of: (1) convolutional layer (3 in-channels, 20 out-channels, kernel size of 5, stride of 1); (2) max pooling layer (kernel size of 2. stride of 2); (3) second convolutional layer (20 in-channels, 50 out-channels, kernel size of stride of 1); (4) second max-pooling layer (kernel size of 2, stride of 2); (5) fully-connected layer (R.000 + R 500); and (6) a second fully-connected layer (R 500 + )
Model training. We train each model with batch size of 250 for single epoch with respect to groups (i.c., 80 training steps given there are two groups). We use a cross-entropy loss and the Adam optimizer with learning rate 0.01. We run all experiments single NVIDIA L40S. We report results over 10 random seeds.
'nttps://colab. research reogerch.oooolee.e..oooo.......e http://....o..tt/////iaaaaaaaaa htt.a//aaoooccccooooooooooooooo/
zation/biiii colored_mnist. ipynb
mnvariant-risk-minn / htt.a/////.n/naeeeeeaaaaaaaaaaaa zat on_
53

---

Published as a conference paper at ICLR 2025
L
BIAS AMPLIFICATION PLOTS
0.5, 0. of =f000ff==100. 1.0e 0.5
log
10
00D
ADD
2
20
2
2
2"
2
2
2
0,5,
1.0, of
1.0, o3
1.0e0
log
20
20
10
6-
000
EDD
10
2
2
2
2
29
2
2
2
0.5, 1.0,
1.0, of 1.0e1.0
log
200
200
100
80
60
EDO
ADD
100
100
40
50
20
2
2
2.
2
2 2
27
Figure We empirically demonstrate that bias amplification occurs and validate our theory (Theorems 3.1 and 3.2) for ODD, EDD, and ADD under the setup described in Section 4.1. The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We plot ODD and EDD on the same scale for casy comparison, and include black dashed line at ADD = 1 to contrast bias amplification vs. dcamplification. The error bars capture the range of the estimators over 25 random seeds.
54

---

Published as a conference paper at ICLR 2025
= o= 1.0e 0.5
oo 0
40
20
30
30
15
88
10
10
2
2
29
2.0, 1.0, 1.0, of 1.0e0
log 0
30
30
15
000
E00
ADO
10-
10:
2.a
2
2
2
2-2
2.0, 8, 1.0, of 1.0, o3 1.0e1.0
log 6
200
200
100
150
150
60
100
EDO
100
ADD
40
1
20.
-2
0
2-
Figure 7: We empirically demonstrate that bias amplification occurs and validate our theory (Theorems 3.1 and 3.2) for ODD, FDD, and ADD under the setup described Section 4.1. The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what theory predicts. We plot ODD and EDD on the same scale for easy comparison, and include black dashed line ADD = 1 to contrast bias amplification vs. deamplification. The error bars capture the range of the estimators over 25 random seeds.
55

---

Published as a conference paper at ICLR 2025
= 0 =.0,, =.0, =f 1.00-00e000000 0.5
lo 0
150-
80
100
100
75
75
40
-1
2
2
2'
2
31 8.0, a2 1.0, of 1.0, of 1.0e0
log 0
60
100
100
75
EDD
40
0-
2-
2
8.0, 8, 1.0, of 1.0, of 1.0e1.0
log 6
200
200
100
150
150
60
1.00
EDD
100
ADD
40
1
20.
-2
0
Figure 8: We empirically demonstrate that bias amplification occurs and validate theory (Theorems 3.1 and 3.2) for ODD, FDD, and ADD under the setup described Section 4.1. The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what theory predicts. We plot ODD and EDD on the same scale for easy comparison, and include black dashed line ADD = 1 to contrast bias amplification vs. deamplification. The error bars capture the range of the estimators over 25 random seeds.
56

---

Published as a conference paper at ICLR 2025
M POWER-LAW COVARIANCE
To better understand how & (rate of features to samples) and the label noise ratio с affect bias amplification. we derive explicit phase transitions in the bias amplification profile of unregularized ridge regression with random projections in terms of these quantities. We consider the setting of power-law covariance, as analytically tractable and can be translated the case of wide neural networks (Caponnetto & de Vito, 2007; Cui et al., 2022; Maloney et al., 2022), where the exponents
can be empirically gauged. Let the eigenvalues x/=) 23, have power-law decay, i.c., X k- for all k and some positive constants 31 and 32. WLOG, we will assume 31 > 82. Note that 3, controls the effective dimension and ultimately the difficulty fitting the noiseless part of the signal from group If 33 is large, then all the information is concentrated in few features, and so the learning problem is easier. We similarly assume that the eigenvalues μą Á have power-law decay H4 = k- for all & and constant >> O. Finally, we consider balanced groups (i.e., p1 p2 = 1/2). Under this setup, we have the following corollary.
Corollary M.1. Suppose that in the single model setting, as X 0t, (21, T, u1 U2 p) is the unique positive solution to the following fixed-point equations:
1/T 1 1/(rr), 1/e, = ofr for s E (1,2),
(359)
p= Us - E,DL for E {1,2],
(360)
where: L pieiii + p2e2222, D = poug2in B.
(361)
Furthermore, suppose 1. < <,,,21111 or S t/. S. Under the assumplions of Theorem 3.1 and Assumption B.3, as X - +0+ we have the following approximate analytical phase transitions in the bias amplification profile of ridge regression with random projections:
lim d
ADD + c 1|
lim lim d,n. R22000
ADD - 0,
(362)
0 +2
2122-26
lim
lim d,m 221102200
ADD - 1,
lim d.n..1g--0 21,2-26
lim
ADD - X.
(363)
-1,2-24
We relegate the proof to Appendix N and empirically assess the validity of this result in Figure 9, The phase transitions reveal that bias amplification peaks near 1, bias deamplification peaks when + 0+,, and bias is roughly neither amplified or deamplified when c - oc. Furthermore, the right tail of the ODD profile (which Corollary M. predicts be proportional to e) is higher than the left tail (i.e., 0) for larger However, the left tail of the EDD profile (which Corollary M. predicts to be proportional to Jc - 1|) does not increase steeply as c -> 0+. Interestingly, in the proof of Corollary
M.I, we observe that the bias term depends on (r ΛΣ,; therefore, the setting Yk, A > 1/px (e.g.. common in learning from synthetic data (Dohmatob et al., 2024a)) can prevent the bias term from vanishing or even cause it to explode. This may explain why training models on synthetic data (i.c., data previously generated by the model) may amplify unfairness (Wyllie et al., 2024).
57

---

Published as a conference paper at ICLR 2025
81 0
20
20
600
EDO
ADD
logc
Figure 9: Our theory predicts that bias amplification larger for higher noise ratios than lower noise ratios. We observe that Corollary M.1 generally predicts the ADD profile with respect to the noise ratio e. The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what Theorem 3.1 predicts. We plot ODD and EDD the same scale for easy comparison, and include a black dashed line at ADD = to contrast bias amplification deamplification. The error bars capture the range of the estimators over 25 random seeds. We consider the setup described in Appendix M with e = 0.3, & = 0.2, and À = 1 x 10-6.
58

---

Published as a conference paper at ICLR 2025
N PROOF OF COROLLARY M.1
Proof. We begin by computing the ODD in the limit À - 0+. We define ul() = u, for B g. By Assumption B.3, we can re-express the constants in Definition 3.1 in terms of the limiting spectral densities of the covariance matrices:
1
1
e1 1 1 Jo pier-p2e2r du(r)
62
1 1 So pieltizeer du(r)'
(364)
-I=1111770 =0, 0,
(365)
I
77
(1) (1)
(1) Ui Pir + 02 P2r + dd(r),
(1)
(1)
P1 + U2
P2r +
-du(r), uz (1) = =
"1
Dej
(f161 + P262r)2
(pi6) + (2222))
(366)
"i
(2)
dej
"1 P1 + u2 P2r + du(r), (2) 0e2 J
(2) (2)
"1 P1" + U2 (2)
(2)
p2r + y2
du(r).
00
(Dic1 + p2622)2
(pre) + p2621)2
(367)
Since 81 > - (-3,) > 0. As such, for d + oc, the ratios Th = /1 have the approximate limiting distribution // = ôy=co, i.c., Dirac atom at infinity, Thus:
(2)
e1 = e, = =1
1- -2 1 1/7,p=0
(368)
P2
(1) "j = 0, U2 0, uj
(1)
(2)
0, u12 P2(D2 - -6)
(2)
Φ
(369)
Now, we can re-express the variance terms as:
Vi(f) = o (pi + p2e22)
Pi dv(r) + Ø00 -002 S /
D2e2r
dv(r) = =0,
(370)
(pi + p2e2r)²
/= pir + p1p2ul 22 du(r) + 2 /00 (piel + p262r)² du(r)
22e21
Vi(f) oo /
(371)
(p1 + p2c27)²
P2 6
Likewise, we can re-express the bias terms as:
/ 0
00
ade2p3r2
1000 000 1o ad (0) = 0, (372)
Bi
(EIP) + e2p2r)2 du(r, a))
0
jo
B2(
(373)
In this calculation, we observe that the adversarial setting Ak: (1 > 1/ /bg can prevent the bias term from vanishing. Putting these pieces together and recalling that P2 = 1/2:
2007
ODD + v1(h - v=( = 1 1- 1-26 c.
(374)
We now compute the EDD. We can once again re-express the constants in Definition H.1 in terms of the limiting spectral densities of the covariance matrices:
1+ /./e = 1- 6, To =1-1/7.
(375)
By Corollary J.1, because v/.
< 11,7 2 or S v. S 54 B. = 0 and V.(f.)
of..
1-
Therefore, because o = Psds:
EDD + [i(Pi) - -Va(Fa) = 1- 2Φ
26
0 02
2
200
|c - 1/,
(376)
1 2Φ
ADD
(377)
c
59

---

Published as a conference paper at ICLR 2025
O
BIAS AMPLIFICATION DURING TRAINING
= =05, -2 =1.0
log
6-
4-
ADD
2-
0-
10-3
10
10
10
10-7
10
10-
16-1
10
= 2.0, d2
1.0
log
000
0 15
2
0-
-2
10
10-1
10
10-1
10
10
10-3
10
8: = 8.0, 02 = 1.0
log
08
ADD
20
10
10-
10
10
10
10 10 10
10
10 10
Figure 10: Our theory reveals that there may be an optimal regularization penalty to deamplify bias. We empirically demonstrate that bias amplification can be heavily affected by À and validate our theory (Theorems
and 3.2) for ODD, EDD, and ADD under setup described in Section 4.2. The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We include black dashed line at ADD to contrast bias amplification vs. deamplification. The error bars capture the range of the estimators over 25 random seeds.
60

---

Published as a conference paper at ICLR 2025
P COLORED MNIST PLOTS
We further assess the applicability of our conclusions about the effects of label noise (Figures 3, 11) and model size (Figure 12) on bias amplification for Colored MNIST. Please see Section 4.2 for discussion of Figure 3. We observe in Figure 11 that as we increase the label noise ratio, the EDD generally increases, while the ODD remains relatively low, which suggested by our theoretical reasoning in Section 4.2. Furthermore, in Figure 12, as the hidden dimension m of the penultimate layer of the CNN increases, the ODD appears to decrease and plateau, which is predicted by our theoretical results (see Section 4.1) in the Colored MNIST regime where Φ < 1. However, the EDD does not appear decrease; while this plausibly predicted by our theory, it requires going beyond simplistic assumption that 21 roughly coincides with Eg and studying the interplay between He, sha, Di for each group s (as suggested by Appendix L).
Dificulty disparity
0.4
0.3
0.2
0.1
0.0
EDD
-0.1
2
aj/o
Figure 11: Our theory predicts that more disparate label noise between groups deamplifies bias Colored MNIST. We plot the ODD and EDD of CNN for different label noise ratios c = o for Colored MNIST. As c increases, the EDD generally increases while the ODD remains relatively low, which is predicted by our theory (see reasoning in Section 4.2). In our experiments, of = 0.05 stays fixed while of varies, For each value of c, the model is evaluated after t training steps and has penultimate layer with dimension m 500. The error bars capture the standard deviation computed over 10 random seeds.
ODD
Difficulty disparity
0.4
0.3
0.2
0.1
0.0
0.1
150
360
450
600
Hidden dimensian
Figure 12: Our theory predicts that larger model size reduces bias on Colored MNIST in the single model setting. We plot the ODD and EDD of a CNN for different model sizes m (where is the dimension of the penultimate CNN layer) for Colored MNIST. As m increases, the ODD appears to decrease and plateau, which is in line with what our theory predicts in the regime where Φ 1 (see analysis in Section 4.1). The EDD does not tend towards 0. In our experiments, of = of = 0.05. For each value of m. the model is evaluated after t = 80 training steps. The error bars capture the standard deviation computed over 10 random seeds.
61

---

Published as a conference paper at ICLR 2025
Q
MINORITY-GROUT BIAS PLOTS
2, =.0, 0,000, n.
Togather
Togatnar
Separate
20
-1
2
22000,0000 D2 2.0.b, m
log
30-
Toget R
Saparate R.
Rg
Together
10-
2-2
2
2
91 2.0, 0.0, 82 2.0, b2 0.2, n 0.9
log
Togathee fiy
Separate Rq
Together
20-
10-
2.2
2
2.2
2
2
2
2
2.
2
Figure 13: We empirically demonstrate that minority group bias is affected by extrancous features. We validate our theory (Theorems 3.1 and 3.2) for together R1, Rg (i.e., single model learned for both groups) and separate R1, R2 (i.e., separate model learned per group) under the setup described in Section 4.2. The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We include a black dashed line ADD 1 contrast bias amplification vs. deamplification. All y-axes are on the same scale for easy comparison. The error bars capture the range of the estimators over 25 random seeds.
62

---

Published as a conference paper at ICLR 2025
21 -,.2.0,b, a0-00aa-2..0 2.0, 2.0,
30-
Togatnar R
Topether
Suparate
20-
0
-1
a102.0, b1 0.0.a: 2.0, 1. 0.75
log
20-
Togethar R.
Saparata R.
As.
Topether
10
2.2
222
2
2
23
24
31 2.0, b2 O.D. 32
2.0. 2.0, 0.9
log
40
Together R
fq
Together
20-
10-
2-2
2
2.2
2
2
2
2
2
Figure 14: We empirically demonstrate that minority- group bias is affected by extraneous features. We validate our theory (Theorems 3.1 and 3.2) for together R1, R2 (i.e., single model learned for both groups) and separate R1, R2 (i.e., separate model learned per group) under the setup described in Section 4.2. The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We include a black dashed line ADD 1 contrast bias amplification vs. deamplification. All y-axes are on the same scale for easy comparison. The error bars capture the range of the estimators over 25 random seeds.
63

---

Published as a conference paper at ICLR 2025
21 -,-2.0,b, 22 2.0. 4.0,
30-
Togatnar Rs
Topether
20-
Supara
0
81 -.0, b1 0.0.a: 2.0, 4.0, n 0.75
log
20-
Togethar R,
Saparata R.
As.
Topether
10
2.2
222
2
2
2
22
31 2.0, D2 O.D. 32 2.0. 4.0, 0.9
log
40
Together Rd
Separate fq
Together
20-
10-
2-2
2
2.2
2
2
2
2
2
2
2
Figure 15: We empirically demonstrate that minority- group bias is affected by extraneous features. We validate our theory (Theorems 3.1 and 3.2) for together Ri, R2 (i.e., single model learned for both groups) and separate R1, R2 (i.e., separate model learned per group) under the setup described in Section 4.2. The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We include a black dashed line ADD I contrast bias amplification vs. deamplification. All y-axes are on the same scale for easy comparison. The error bars capture the range of the estimators over 25 random seeds.
64

---

Published as a conference paper at ICLR 2025
R ACTIONABLE INSIGHTS FROM THEORY
Searching for optimal hyperparameters. In practice, an optimal regularization penalty or training time can be selected by searching for values that strike desired balance between overall validation error (that is not too high) and bias amplification (that is not too high). As we would estimate the test error using the empirical validation error, we can estimate bias amplification using the validation set. Moreover, we would need to train: (1) the main model on a mixture of data from groups, and (2) auxiliary separate models on the data for each group.
However, it may be expensive to train auxiliary models for each candidate value of À and f. The search space can be reduced by using insights from our theory. For instance, with overparamaterization, as decreases (or increases), bias amplification increases and plateaus, and with underparamettrization as À decreases (or increases), bias deamplification increases and plateaus. When the curves are monotone with respect to X, the optimal À is either at the left tail of the curve (e.g.. À 0) or the right tail (i.e., the largest among the reasonable options). In contrast, Figure 10 shows that when ei close to the interpolation threshold of I, bias amplification is often not monotone with respect to X.
Informing evaluation and mitigation strategies. Our theory offers avenues to assess whether ML model trained on certain real-world data is prone bias amplification and mitigate this amplification, even though we may lack direct access to population parameters like E. We can estimate such unable to robustly estimate
parameters using samples (c.g.. Σ = X X). However, even if these parameters, our theory still provides valuable insights. For example, we observe that the ratios of parameters for the groups is often what matters, e.g., label noise ratio o /of (see Section 4.1), ratio of covariance eigenvalues (see Appendix M). Thus, practitioners can use our theory to get intuition about when disparities in the variability of labels and features across groups amplify bias.
Morcover, our findings warn against the conventional wisdom that increased model overparameteriza- tion data balancing can alleviate bias issues. In addition, our theory informs criteria for feature selection (e.g., discarding features with disparate variance across groups) and warns MI. practitioners about the interplay between high vs. low naature-to-sample regimes and overparameterization in inducing bias amplification. Nevertheless, additional work is required to make rigorous connections between our theoretical findings and better strategies for evaluating and mitigating the bias of models.
65


Reviews:
[{"review_id": "hCI5U8DNiV", "reviewer": "Reviewer_X48V", "summary": "This paper considers the theoretical foundation of the bias amplification phenomenon. On a Gaussian mixture data generating distribution and ridge regression model, the authors derive the exact form of the bias and variance formula under a high dimensional limit. Using the theory, the authors show the relationship between bias amplification and the model size, label noise, number of features, regularization, and training dynamics.", "questions": "1. Is the bias amplification phenomena even in a low dimensional model, such as classical linear regression, or it is a special phenomenon on a high dimensional limit?\n\n2. In Figure 1, the EDD plot has an extreme phase transition in the lower left corner, can the authors provide more discussion about this behavior? Moreover, the extreme case seems to mismatch with the ADD plot on the right-hand side. \n\n3. According to Figure 11, most of the metrics are monotone with respect to lambda, how do you conclude the optimal lambda in this case?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed theory is solid with the exact formulation of the limiting risk, and it provides a rigorous foundation on the bias amplification phenomenon and its relationship with various components in machine learning models. The discovery is novel and the contribution is important to deepen our understanding of the bias amplification in ML.", "weaknesses": "1. Although the authors present the exact formulation of the risk in the main text, it is complicated to understand the implications of those formulas. It would be helpful to include more discussion to explain each term to better understand the results. \n\n2. The paper's main contribution is to examine the bias amplification phenomenon using the formula. However, a formal statement about how different components affect the bias amplification is lacking. I would suggest the authors write them in formal theorems. \n\n3. It is unclear how these theoretical findings relate to real-world deep learning models, I would suggest the authors verify the conclusion about the label noise and model size on MNIST and CNN as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y7MOkdVXLV", "reviewer": "Reviewer_HU3q", "summary": "Machine learning models can capture and amplify biases present in data. This paper develops an analytical theory within the context of ridge regression to better understand and address these biases. The theory provides insights into phenomena like bias amplification and minority-group bias, revealing that optimal regularization penalties or training times may help mitigate bias. Their theoretical predictions are supported by empirical validation on diverse synthetic and semi-synthetic datasets.", "questions": "- I have mixed feelings about the analysis of bias amplification. On one hand, the results are somewhat intuitive if we assume two groups, one with $y = x$ and the other with $y = -x$. On the other hand, the theoretical results are based on isotropic covariance, which is a simplified scenario. What would happen if the covariance structure were non-isotropic, for instance, if the covariance between $X_i$ and $X_j$ fixes or exponentially decays? \n\n- In real-world datasets, we often lack information about the detailed properties of each group such as $\\Sigma_1$. How can we assess whether a machine learning model trained on such data is prone to bias amplification?\n\n- Even if we identify bias amplification and attribute it to heterogeneous conditional dependencies, how can we design a practical criterion to select the optimal $\\lambda$ or runtime? Additionally, could the heterogeneity itself be leveraged to develop a better machine learning model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper examines bias amplification in ridge regression with random projections. Theoretical findings not only predict known bias amplification effects but also reveal new amplification phenomena under isotropic covariance. These findings are further validated through empirical analysis on a complex semi-synthetic image dataset.", "weaknesses": "- It appears that the analysis of bias amplification heavily relies on an implicit assumption about the conditional dependence heterogeneity (e.g., $w_2\\neq w_1$) between the two groups. What would happen if this assumption were relaxed or removed?\n\n- The manuscript does not sufficiently justify the definition of “bias amplification.” I think more theoretical backing is needed for this definition. \n\n- More theoretical justification is needed to demonstrate that ridge regression with random projections serves as an effective approximation for neural networks (NN). Additionally, in Line 172, the one-hidden-layer NN model seems lacks a nonlinear activation function?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ZGrdaI6QZ", "reviewer": "Reviewer_m7Lv", "summary": "The authors introduce a simple modeling framework for understanding the phenomenon of bias amplification, i.e. the observation of abnormal disparate model performance on communities unequally represented in the dataset. The setup encompasses a regression with a linear model (both a single-layer network and a linear two-layer network are considered) on a dataset with multiple modalities and different associated rules. This framework allows them to capture some key aspects of bias amplification and to make qualitative predictions on the possible regimes where over-parametrization can help or hinder the model's unbiasedness. The analytical results are derived with the tools of \"operator-valued free probability theory\", and corroborated with some empirical observations on standard benchmarks.", "questions": "MAJOR\n1) As mentioned above, the authors should acknowledge and discuss the connection of their work to arxiv:2205.15935. Ignoring such similarity to a previous work is unacceptable for an ICLR paper. \n2) It seems plausible that much of the observed bias phenomenology in this paper is a direct effect of the double-descent / interpolation peak phenomenology, which is known to shift with the degree of over-parametrization of the network and to cause a critical sensitivity to input noise (hindering the generalization). Could the authors discuss this connection more in detail, and clarify if a different cause is at play? Is the linear activation assumption a confounding effect regarding this?\n3) The imputation of the effect of early stopping in the training dynamics from the effect of a properly scaled regularization (1/t) might lead to simplistic conclusions. Can the authors discuss, for example, the connection to this previous work arXiv:2405.18296? In general, I believe mentioning optimal training time, and training dynamics both in the abstract and in the paragraph titles without specifying that this extrapolation hinges on a very simplistic approximation is misleading. \n4) Can the authors mention how, for example, their results can be used to \"inform strategies to evaluate and mitigate unfairness in machine learning\"?\n\nMINORS\n1) Not clear if in Fig. 1 the setting is with balanced groups, or if there is another figure for this.\n2) The explanations e.g. in lines 311-317, 372-375, 400-403, 443-453, are hardly understandable if you are not extremely familiar with the model parametrization. Maybe a more effective way of presenting the results would be to state in words what setup is being considered, before looking at the parameter values.\n3) I don't think it is fair practice to hide the error bars of Figure 3 in the supplementary. Moreover, the associated theoretical prediction seems very qualitative.\n\n\nAFTER REVISION\nGiven the implemented changes, I will raise my score to a 6.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I believe the topic analyzed in this paper is very relevant, and theoretical contributions in this setting are crucial to understanding how to mitigate the pressing issue of bias amplification. In this context, it is interesting to understand and discuss whether the usual recipe of overparametrization could be more harmful than expected. \nThe authors' analysis seems technically sound, although the methods required the introduction of some modeling assumptions that might be simplistic in some cases.", "weaknesses": "My main concern about this work is its extreme similarity, in goals, modeling setup, and some of the conclusions, with previous work (\"Bias-inducing geometries: an exactly solvable data model with fairness implications\" https://arxiv.org/abs/2205.15935), which is ignored in the present paper. Notice that this same reference is for example cited in ref. (Bell & Sagun, 2023) within this paper. While I can recognize some differences, reasonably motivating the present extension, there should be a clear discussion of the connections and differences.\n\nMoreover, the authors do not discuss at length the limitations of some of their assumptions. For example, using a linear network as a proxy of a deep network, or using regularization intensity to approximate early stopping and to understand the training dynamics (without looking at the training dynamics). Although the assumptions might be \"standard\", the limitations should be discussed also in the present work.\n\nFinally, some of the paragraphs on the results in figures are hardly readable because constantly refer to different values of the model parameters, often without providing a clear explanation of the \"meaning\" of the different regimes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hUcv53C9sB", "reviewer": "Reviewer_KjTp", "summary": "The paper develops a rigorous statistical framework with theory to explain and analyze model bias, particularly focusing on one of the most popular statistical methods, ridge regression, both with and without random projections (which model neural networks in a simplified regime). Its key contributions include:\n\nUnified Theory of Bias: The paper provides a unifying theory to understand how model design choices and data distribution properties lead to bias, including phenomena like bias amplification and minority-group error. This framework explains how bias can emerge and be amplified in different feature and parameter regimes.\n\nBias Amplification: It analyzes how training a single model on combined data from different groups can amplify bias more than training separate models on each group’s data, even without group imbalance or spurious correlations. The theory also suggests that early stopping and tuning regularization can reduce this amplification.\n\nMinority Group Error: The paper investigates how overparameterization and extraneous features negatively affect test performance for minority groups. It explicitly connects model size to error amplification for these groups.\nEmpirical Validation: The theoretical predictions are extensively validated using synthetic and semi-synthetic datasets, and the results align with empirical findings reported in prior literature.\n\nOverall, the paper offers novel statistical theory and new insights into model bias and proposes avenues for mitigating it through informed model design and training choices.", "questions": "Please see the **Weaknesses** section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: This paper has originality. There are many works on model bias, but most of them are empirical work. A unified statistical theory of model bias is underexplored. Compared to previous work on model bias theory, this paper focuses on disparities in test error across majority and minority groups with different data distributions when models are trained on data from both groups. This paper does use existing tools based on random matrix theory from previous work, but the theory is not a special case of the existing ones. Therefore, I think the paper has its originality. \n\n\n**Quality**: This paper has good quality. The analysis using operator-valued free probability theory (OVFPT) and high-dimensional scaling limits look technically sound to me. Fixed-point equations and a detailed exploration of phase transitions (e.g., bias amplification at specific proportions of features to samples) are mathematically rigorous. Further, besides the theoretical analysis, this empirical section demonstrates how ridge regression models with random projections can amplify bias, even in balanced data without spurious correlations. This is an interesting finding. It offers a clear, empirical validation of theoretical predictions, demonstrating the impact of model size, label noise, and training dynamics on bias amplification. \n\n**Clarity**: The structure is clear. From preliminaries in section 2 to a detailed model analysis in section 3, and empirical study in section 4, they all manifest a well-organized and thoughtful presentation. \n\n**Significance**: This paper studies an important problem and provides a good contribution to the field. To be specific, this article tackles the critical issue of bias in machine learning, which can lead to unequal performance across demographic groups. It presents a new theoretical framework using operator-valued free probability theory to explain and predict bias amplification, unifying previous findings and offering a clearer understanding of bias mechanisms. The theory is validated in diverse empirical settings, showing practical relevance. Overall, this work makes a significant contribution by providing a structured approach to understanding and mitigating bias, with implications for fairer machine learning practices and future research directions.", "weaknesses": "There are no significant weaknesses in this paper. However, the following minor weaknesses can be improved. \n\n1. The connection between the theoretical analysis in this paper and the prior work in OVFPT is not discussed clearly enough. This paper mentions that the theory cannot be derived as a special case in Adlame et al., Tripuraneni et al., Lee et al., but does not provide a more involved explanation on why this is the case. It would thus be great if a paragraph discussing this point can be added.\n\n2. Section 4.2 nicely addresses how regularization and training time influence bias amplification, but it lacks a bit of practical guidance on applying these insights in real scenario. After discussing the effects of regularization and training dynamics on bias amplification, the authors could add a bit more remarks or comments. This would be an ideal place to provide concrete guidance on setting regularization parameters or selecting optimal training durations for different dataset sizes and feature-sample ratios. For instance, they could suggest regularization ranges or early stopping criteria when $\\psi$ is close to or above 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)