An Effective Theory of Bias Amplification

## Abstract

Abstract Machine learning models may capture and amplify biases present in data, leading to disparate test performance across social groups. To better understand, evaluate, and mitigate these possible biases, a deeper theoretical understanding of how model design choices and data distribution properties could contribute to bias is needed. In this work, we contribute a precise analytical theory in the context of ridge regression, both with and without random projections, where the former models neural networks in a simplified regime. Our theory offers a unified and rigorous explanation of machine learning bias, providing insights into phenomena such as bias amplification and minority-group bias in various feature and parameter regimes. For example, we demonstrate that there may be an optimal regularization penalty or training time to avoid bias amplification, and there can be fundamental differences in test error between groups that do not vanish with increased parameterization. Importantly, our theoretical predictions align with several empirical observations reported in the literature. We extensively empirically validate our theory on diverse synthetic and semi-synthetic datasets.

### 1 Introduction

Machine learning datasets encode a plethora of biases which, when used to train models, result in systems that can cause practical harm.
Datasets that encode correlations that only hold for a subset of the data cause disparate performance when models are used more broadly, such as an X-ray pneumonia classifier that only functions on images from certain hospitals (Zech et al., ).
This issue is magnified when coupled with under-representation, whereby a dataset fails to adequately reflect parts of the underlying data distribution, often further marginalizing certain groups.
Lack of representation results in systems that might work well on average, but fail for minoritized groups, including facial recognition systems that fail for darker-skinned women (Buolamwini & Gebru, ), large language models (LLMs) that consistently misgender transgender and nonbinary people (Ovalle et al., ), or image classification technology that only works in Western contexts (de Vries et al., ; Richards et al., ).

Unfortunately, many contemporary models may exhibit *bias amplification*, whereby dataset biases are not only replicated, but exacerbated (Zhao et al., ; Hendricks et al., ; Wang & Russakovsky, ).
While previous research has shown that amplification is a function of both dataset properties and how we choose to construct our models (Hall et al., ; Sagawa et al., ; Bell & Sagun, ), it is not fully clear how bias amplification occurs mechanistically, nor do we precisely understand which settings lead to its emergence. Thus, in this work, we propose a novel theoretical framework that explains how model design choices and data distributional properties interact to amplify bias, and provides an account of diverse prior work on bias amplification (Bell & Sagun, ) and minority-group error (Sagawa et al., ).

A theory of bias amplification is important for several reasons.
First, as empirical research necessarily yields only sparse data points—often focused on only the most common regimes (Bommasani et al., )—theory allows us to interpolate between past findings, and reason about how bias emerges in under-explored settings.
Second, a precise theory gives us the depth of understanding needed in order to intervene, potentially supporting the development of both novel evaluations and novel mitigations.
Finally, beyond explaining already-known phenomena, our theory makes novel predictions, suggesting new avenues for future research.

#### 1.1 Main Contributions

In this work, we develop a unifying and rigorous theory of machine learning bias in the settings of ridge regression with and without random projections. In particular, we precisely analyze test error disparities between groups (e.g., different demographic or protected categories) with different data distributions when training on a mixture of data from these groups. We characterize these disparities in high dimensions using operator-valued free probability theory (OVFPT), thereby avoiding possibly loose bounds on critical quantities. Our theory encompasses different parameterization regimes, group sizes, label noise levels, and data covariance structures. Moreover, our theory has applications to important problems in machine learning bias that have recently been empirically investigated:

- •
Bias amplification. Even in the absence of group imbalance and spurious correlations, a single model that is trained on a combination of data from different groups can amplify bias beyond separate models that are trained on data from each group (Bell & Sagun, ). We reproduce and analyze the bias amplification results of Bell & Sagun () in controlled settings, and additionally provide an in-depth theoretical treatment of these results. We further observe how stopping model training early or tuning the regularization hyperparameter can alleviate bias amplification.
- •
Minority-group error. Overparamaterization can hurt test performance on minority groups due to spurious features (Sagawa et al., ; Khani & Liang, ). We theoretically analyze how model size and extraneous features affect minority-group error.

We extensively empirically validate our theory in diverse controlled and semi-synthetic settings. Specifically, we show that our theory aligns with practice in the cases of: (1) bias amplification with synthetic data generated from isotropic covariance matrices and the semi-synthetic dataset Colored MNIST (Arjovsky et al., ), and (2) minority-group error under different model sizes with synthetic data generated from diatomic covariance matrices. In these applications, we expose new, interesting phenomena in various regimes. For example, a larger number of features than samples can amplify bias under overparameterization, there may be an optimal regularization penalty or training time to avoid bias amplification, and there can be fundamental differences in test error between groups that do not vanish with increased parameterization. Ultimately, our theory of machine learning bias can inform strategies to evaluate and mitigate possible unfairness in machine learning, or be used to caution against the usage of machine learning in certain applications.

#### 1.2 Related Work

Bias amplification. A long line of research has explored how machine learning exacerbates biases in data. For example, a single model that is trained on a combination of data from different groups can amplify bias (Zhao et al., ; Wang & Russakovsky, ), even beyond what would be expected when separate models are trained on data from each group (Bell & Sagun, ). Hall et al. () conduct a systematic empirical study of bias amplification in the context of image classification, finding that amplification can vary greatly as a function of model size, training set size, and training time. Furthermore, overparameterization, despite reducing a model’s overall test error, can disproportionately hurt test performance on minority groups (Sagawa et al., ; Khani & Liang, ). Models can also overestimate the importance of poorly-predictive, low-signal features for minority groups, thereby hurting performance on these groups (Leino et al., ). In this paper, we distill a holistic theory of how model design choices and data distributional properties affect disparate test performance across groups, which encompasses seemingly disparate bias phenomena.

High-dimensional analysis of machine learning.
A suite of works have analyzed the expected dynamics of machine learning in appropriate asymptotic scaling limits e.g., the rate of features $d$ to samples $n$ converges to a finite values as $d$ and $n$ respectively scale towards infinity (Adlam & Pennington, ; Tripuraneni et al., ; Lee et al., ). Notably, Bach () theoretically analyzes the double descent phenomenon (Spigler et al., ; Belkin et al., ) in ridge regression with random projections by computing deterministic equivalents for relevant random matrix quantities in a proportionate scaling limit.

Unlike much prior theoretical work, we focus on disparities in test error across groups with different data distributions when models are trained on data from both groups.
Jain et al. () similarly consider the bias of models trained on a mixture of data from multiple groups in a high-dimensional setting, and in particular, how bias evolves during training. However, focus on the linear classification of Gaussian data with isotropic covariance in the setting $d\to\infty$; in contrast, we consider the application of ridge regression with random projections to Gaussian data with more general covariance structure in the setting $d\to\infty$ and the number of parameters $m\to\infty$, which enables richer insights into the impact of (over/under-)parameterization on bias. Furthermore, analytically characterize the evolution of bias by exactly solving a set of ODEs in their setting.

On the other hand, like , , and , we leverage the tools of OVFPT (Mingo & Speicher, ), which is at the intersection of random matrix theory (RMT) and functional analysis. Our theory, however, cannot be recovered as a special case of the theories presented in these papers. Furthermore, our theory non-trivially generalizes (Bach, ), which we recover in Corollary [1](#A14.EGx147), and requires more powerful analytical techniques.

### 2 Preliminaries

#### 2.1 Data Distributions

We consider a ridge regression problem on a dataset from the following multivariate Gaussian mixture with two groups $s=1$ and $s=2$. These groups could represent different demographic or protected categories, for example.

$$ $\displaystyle\textbf{(Group ID) }\mathrm{Law}(s)=\mathrm{Bernoulli}(p),$ (1) $\displaystyle\textbf{(Features) }\mathrm{Law}(x\mid s)=\mathcal{N}(0,\Sigma_{s}),$ (2) $\displaystyle\textbf{(Ground-truth weights) }\mathrm{Law}(w_{1}^{*})=\mathcal{N}(0,\Theta/d),\quad\mathrm{Law}(w_{2}^{*}-w_{1}^{*})=\mathcal{N}(0,\Delta/d),$ (3) $\displaystyle\textbf{(Labels) }\mathrm{Law}(y\mid s,x)=\mathcal{N}(f^{\star}_{s}(x),\sigma_{s}^{2}),\text{ with }f^{\star}_{s}(x)=x^{\top}w_{s}^{*}.$ (4) $$

The scalar $p\in(0,1)$ controls for the relative size of the two groups (e.g., $p=1/2$ in the balanced setting). For simplicity of notation, we define $p_{1}=p$ and $p_{2}=1-p$. The $d\times d$ positive-definite matrices $\Sigma_{1}$ and $\Sigma_{2}$ are the covariance matrices for the different groups. The $d$-dimensional vectors $w_{1}^{*}$ and $w_{2}^{*}$ are the ground-truth weights vectors for each group. $w_{1}^{*}$ and $w_{2}^{*}-w_{1}^{*}$ are independently sampled from zero-mean Gaussian distributions with covariances $\Theta$ and $\Delta$, respectively. In particular, setting $\Delta=0$ corresponds to the case that both groups have identical ground-truth weights. Finally, $\sigma_{s}^{2}$ corresponds to the label-noise level for each group $s$. While we consider the case of two groups only for conciseness, our theoretical analysis readily extends to any finite number of groups.

#### 2.2 Models and Metrics

Learning. A learner is given an IID sample $\mathcal{D}=\{(x_{1},y_{1}),\ldots,(x_{n},y_{n})\}=(X\in\mathbb{R}^{n\times d},Y\in\mathbb{R}^{n})$ of data from the above distribution and it learns a model for predicting the label $y$ from the feature vector $x$. Thus, $X$ is the total design matrix with $i$th row $x_{i}$, and $y$ the total response vector with $i$th component $y_{i}$.
Let $\mathcal{D}^{s}=(X\in\mathbb{R}^{n_{s}\times d},Y\in\mathbb{R}^{n_{s}})$ be the data pertaining only to group $s$, so that $\mathcal{D}=\mathcal{D}^{1}\cup\mathcal{D}^{2}$ is a partitioning of the entire dataset. Two choices are available to the learner: (1) learn a model a $\widehat{f}_{s}\in\mathcal{F}$ on each dataset $\mathcal{D}^{s}$, or (2) learn a single model $\widehat{f}\in\mathcal{F}$ on the entire dataset $\mathcal{D}$. In practice, a choice is made based on scaling vs. personalization considerations.

We consider two solvable settings for linear models: classical ridge regression in the ambient input space, and ridge regression in a feature space given by random projections. The latter allows us to the study the role of model size in machine learning bias, by varying the output dimension of the random projection mapping. This output dimension $m$ controls the size of a neural network in a simplified regime (Maloney et al., ; Bach, ).

Classical Ridge Regression.
We will first consider the function class $\mathcal{F}\subseteq\{\mathbb{R}^{d}\to\mathbb{R}\}$ of linear ridge regression models without random projections.
For any $w\in\mathbb{R}^{d}$, the model $f$ is defined by

$$ $\displaystyle f(x)=x^{\top}w,\text{ for all }x\in\mathbb{R}^{d},$ (5) $$

and is learned with $\ell_{2}$-regularization. We define the generalization error (a.k.a. risk) of any model $f$ w.r.t. to group $s$ as

$$ $R_{s}(f)=\mathbb{E}\,[(f(x)-f_{s}^{\star}(x))^{2}\mid s].$ (6) $$

We consider ridge regression because in addition to its analytical tractability, it can be viewed as the asymptotic limit of many learning problems (Dobriban & Wager, ; Richards et al., ; Hastie et al., ). We now formally define some metrics related to bias amplification.

###### Definition 1 (Bias Amplification) .

We isolate the contribution of the model to bias when learning from data with different groups. This intuitive conceptualization of bias amplification allows us measurements. Further grounding it in the literature (Bell & Sagun, ), we define the Expected Difficulty Disparity (EDD) as:

$$ $\displaystyle EDD=|\mathbb{E}\,R_{2}(\widehat{f}_{2})-\mathbb{E}\,R_{1}(\widehat{f}_{1})|,$ (7) $$

where the expectations are w.r.t. randomness in the training data and any other sources of randomness in the models. The $EDD$ captures the difference in test risk between models trained and evaluated on each group separately. In contrast, we define the Observed Difficulty Disparity (ODD) as:

$$ $\displaystyle ODD=|\mathbb{E}\,R_{2}(\widehat{f})-\mathbb{E}\,R_{1}(\widehat{f})|.$ (8) $$

The $ODD$ captures the bias (i.e., difference in test risk between groups) of a model trained on both groups. Finally, we define the Amplification of Difficulty Disparity (ADD) as $ADD=\frac{ODD}{EDD}$. We say that bias amplification occurs when $ADD>1$.

Ridge Regression with Random Projections.
We consider neural networks in a simplified regime which can be approximated via random projections, i.e., a one-hidden-layer neural network $f(x)=v^{\top}Sx$ with a linear activation function. In particular, we extend classical ridge regression by transforming our learned weights as $\widehat{w}=S\widehat{\eta}\in\mathbb{R}^{d}$, where $S\in\mathbb{R}^{d\times m}$ is a random projection with entries that are IID sampled from ${\cal N}(0,1/d)$.

###### Definition 1 (Bias Amplification) .

### 3 Theoretical Analysis

Assumptions. Some of our theorems will require standard technical assumptions that we detail here and in Appendix [B](#A2). Assumptions [1](#Thmassumption1) and [2](#Thmassumption2) describe the proportionate scaling limits, standard in RMT, in which we will work. These limits enable us to derive deterministic formulas for the expected test risk of models. Our experiments (see Sections [4](#S4) and [5](#S5)) validate our theory.

###### Assumption 1 .

In the case of classical ridge regression, we will work in the following proportionate scaling limit:

$$ $\displaystyle n,n_{1},n_{2},d\to\infty,\quad n_{1}/n\to p_{1},\,n_{2}/n\to p_{2},\quad d/n_{1}\to\phi_{1},\,d/n_{2}\to\phi_{2},\,d/n\to\phi,$ (9) $$

for some constants $\phi_{1},\phi_{2},\phi\in(0,\infty)$. The scalar $\phi$ captures the rate of features to samples. Observe that $\phi=p_{1}\phi_{1}$ and $\phi=p_{2}\phi_{2}$.

###### Assumption 2 .

In the case of ridge regression with random projections, we will work in the following proportionate scaling limit:

$$ $\displaystyle n,n_{1},n_{2},d\to\infty,\quad n_{1}/n\to p_{1},\,n_{2}/n\to p_{2},\,d/n\to\phi,\,m/n\to\psi,\,m/d\to\gamma,$ (10) $\displaystyle d/n_{1}\to\phi_{1},\,m/n_{1}\to\psi_{1},\quad d/n_{2}\to\phi_{2},\,m/n_{2}\to\psi_{2},$ (11) $$

for some constants $\phi_{1},\phi_{2},\phi,\psi_{1},\psi_{2},\psi\in(0,\infty)$. We note that $\phi\gamma=\psi$ and $\phi_{s}\gamma=\psi_{s}$. The scalar $\psi$ capture the rate of parameters to samples, and thus quantifies model capacity. The setting $\psi>1$ (resp. $\psi<1$) corresponds to the overparameterized (resp. underparameterized) regime.

###### Assumption 1 .

###### Assumption 2 .

#### 3.1 Warm-Up: Classical Linear Model

To provide a mechanistic understanding of how machine learning models may amplify bias, our theory elucidates differences in the test error between groups when a single model is trained on a combination of data from both groups vs. when separate models are trained on data from each group.

Single Classical Linear Model Learned for Both Groups. We first consider the classical ridge regression model $\widehat{f}$, which is learned using empirical risk minimization and $\ell_{2}$-regularization with penalty $\lambda$. The parameter vector $\widehat{w}\in\mathbb{R}^{d}$ of the linear model $\widehat{f}$ is given by the following problem:

$$ $\displaystyle\widehat{w}=\arg\min_{w\in\mathbb{R}^{d}}L(w)=\sum_{s=1}^{2}n^{-1}\|X_{s}w-Y_{s}\|^{2}_{2}+\lambda\|w\|_{2}^{2}.$ (12) $$

The unregularized limit $\lambda\to 0^{+}$ corresponds to ordinary least-squares (OLS).We provide in Theorem [1](#Thmtheorem1) a novel bias-variance decomposition for the test error $R_{s}(\widehat{f})$ for each group $s=1,2$. We derive this result using linear pencils and operator-valued free probability theory (in Appendix [D](#A4)). We first present some relevant definitions.

###### Definition 2 .

For any group index $s\in\{1,2\}$, we define $(e_{1},e_{2},u^{(s)}_{1},u^{(s)}_{2})$ to be the unique positive solution to the following system of fixed-point equations:

$$ $\displaystyle 1/e_{s}$ $\displaystyle=1+\phi\bar{\operatorname{tr}}\,\Sigma_{s}K^{-1},\quad u^{(s)}_{k}=\phi e_{k}^{2}\bar{\operatorname{tr}}\,\Sigma_{k}(p_{1}u^{(s)}_{1}\Sigma_{1}+p_{2}u^{(s)}_{2}\Sigma_{2}+\Sigma_{s})K^{-2},\,k\in{1,2},$ (13) $$

where $K=p_{1}e_{1}\Sigma_{1}+p_{2}e_{2}\Sigma_{2}+\lambda I_{d}$ and $\bar{\operatorname{tr}}\,A:=(1/d)\operatorname{tr}\,A$ is the normalized trace operator.

The fixed-point equations for $e_{s}$ are non-linear and often not analytically solvable for general $\Sigma_{1},\Sigma_{2}$. This is typical in RMT.

###### Theorem 1 .

Under Assumptions [3](#Thmassumption3) and [1](#Thmassumption1), it holds that: $R_{s}(\widehat{f})\simeq B_{s}(\widehat{f})+V_{s}(\widehat{f})$, with

$$ $\displaystyle V_{s}(\widehat{f})$ $\displaystyle=V_{s}^{(1)}(\widehat{f})+V_{s}^{(2)}(\widehat{f}),$ (14) $\displaystyle V_{s}^{(k)}(\widehat{f})$ $\displaystyle=p_{k}\sigma_{k}^{2}\phi\bar{\operatorname{tr}}\,\Sigma_{k}\big{(}e_{k}\Sigma_{s}-\lambda u^{(s)}_{k}I_{d}+p_{k^{\prime}}\Sigma_{k^{\prime}}(e_{k}u^{(s)}_{k^{\prime}}-e_{k^{\prime}}u^{(s)}_{k})\big{)}K^{-2},$ (15) $\displaystyle B_{s}(\widehat{f})$ $\displaystyle=B_{s}^{(1)}(\widehat{f})+B_{s}^{(3)}(\widehat{f})+\begin{cases}0,&s=1,\\ 2B_{2}^{(2)}(\widehat{f}),&s=2,\end{cases}$ (16) $\displaystyle B_{s}^{(1)}(\widehat{f})$ $\displaystyle=p_{s^{\prime}}\bar{\operatorname{tr}}\,\Delta\Sigma_{s^{\prime}}(p_{s^{\prime}}(1+p_{s}u^{(s)}_{s})e_{s^{\prime}}^{2}\Sigma_{s^{\prime}}\Sigma_{s}+u^{(s)}_{s^{\prime}}(p_{s}e_{s}\Sigma_{s}+\lambda I_{d})^{2})K^{-2},$ (17) $\displaystyle B_{2}^{(2)}(\widehat{f})$ $\displaystyle=p_{1}\lambda\bar{\operatorname{tr}}\,\Sigma_{1}((1+p_{2}u_{2}^{(2)})e_{1}\Sigma_{2}-u_{1}^{(2)}(p_{2}e_{2}\Sigma_{2}+\lambda I_{d}))K^{-2},$ (18) $\displaystyle B_{s}^{(3)}(\widehat{f})$ $\displaystyle=\lambda^{2}\bar{\operatorname{tr}}\,\Theta_{s}(p_{1}u_{1}^{(s)}\Sigma_{1}+p_{2}u_{2}^{(s)}\Sigma_{2}+\Sigma_{s})K^{-2},$ (19) $$

where $1^{\prime}=2$ and $2^{\prime}=1$. For completeness, we treat the case of fitting a separate model $\widehat{f}_{s}$ per group in Appendix [C](#A3).

Phase Diagram. We present the bias amplification phase diagram predicted by Theorems [1](#Thmtheorem1) and [4](#A14.EGx25) in Figure [5](#A3.F5) (in the appendix). To obtain the precise phase diagram, we solve the scalar equations numerically. In the $ODD$ profile, we observe an interpolation threshold at $\phi=1$. To the right of the threshold, we observe a tail that descends towards 1. To the left of the threshold, the $ODD$ descends below 1 with a local minimum at $\phi\approx 0.2$ before increasing. In contrast, we observe that the $EDD$ continually grows as $\phi$ increases, ascending from a small value towards 1 and plateauing after $\phi=0.5$ (i.e., $\phi_{1}=\phi_{2}=1$). Accordingly, the $ADD$ increases significantly as $\phi$ decreases, peaks at $\phi=1$, and descends towards 1 as $\phi$ increases (i.e., bias remains amplified in this phase). That is, bias is most amplified when the rate of features to samples $\phi\ll 1$ and $\phi=1$. Interestingly, bias amplification consistently occurs (i.e., $ADD>1$) across all observed values of $\phi$.

Technical Difficulty.
The analysis of the test errors (e.g., $R_{s}(\widehat{f})$) amounts to the analysis of the trace of rational functions of sums of random matrices. Although the limiting spectral density of sums of random matrices is a classical computation using subordination techniques (Marčenko & Pastur, ; Kargin, ), a more involved analysis is required in our case. This difficulty is even greater in the setting of random projections (see Section [3.2](#S3.SS2)). Thus, we employ operator-valued free probability theory (OVFPT) to compute the exact high-dimensional limits of such quantities.

###### Definition 2 .

###### Theorem 1 .

#### 3.2 Main Result: Ridge Regression with Random Projections

Single Random Projections Model Learned for Both Groups. For a more realistic but still analytically solvable setup, we now consider the ridge regression model $\widehat{f}$ with random projections, which is learned using empirical risk minimization and $\ell_{2}$-regularization with penalty $\lambda$. The parameter $\widehat{w}$ of the linear model $\widehat{f}$ is given by the following optimization problem:

$$ $\displaystyle\widehat{w}=S\widehat{\eta}\in\mathbb{R}^{d},\text{ with }\widehat{w}=\arg\min_{\eta\in\mathbb{R}^{m}}L(\eta)=\sum_{s=1}^{2}n^{-1}\|X_{s}S\eta-Y_{s}\|^{2}_{2}+\lambda\|\eta\|_{2}^{2}.$ (20) $$

Explicitly, one can write $\widehat{w}=S(Z^{\top}Z+n\lambda I_{m})^{-1}Z^{\top}Y$, where $Z:=XS$. As previously mentioned, ridge regression with random projections can be viewed as a simplification of the high-dimensional dynamics of neural networks that still captures the effect of model size on machine learning bias. Before presenting our result for the random projections model, we provide some relevant definitions.

###### Definition 3 .

Let $(e_{1},e_{2},\tau,u_{1},u_{2},\rho)$ be the unique positive solution to the following fixed-point equations:

$$ $\displaystyle 1/\tau$ $\displaystyle=1+\bar{\operatorname{tr}}\,LK^{-1},\quad 1/e_{s}=1+\psi\tau\bar{\operatorname{tr}}\,\Sigma_{s}K^{-1},\text{ for }s=1,2,$ (21) $\displaystyle\rho$ $\displaystyle=\tau^{2}\bar{\operatorname{tr}}\,(\gamma\rho L^{2}+\lambda^{2}D)K^{-2},\quad u_{s}=\psi e_{s}^{2}\bar{\operatorname{tr}}\,\Sigma_{s}(\gamma\tau^{2}D+\rho I_{d})K^{-2},\text{ for }s=1,2,$ (22) $\displaystyle\text{ where: }L$ $\displaystyle=p_{1}e_{1}\Sigma_{1}+p_{2}e_{2}\Sigma_{2},\,K=\gamma\tau L+\lambda I_{d},\,D=p_{1}u_{1}\Sigma_{1}+p_{2}u_{2}\Sigma_{2}+B.$ (23) $$

For deterministic $d\times d$ PSD matrices $A$ and $B$, we define the following auxiliary quantities:

$$ $\displaystyle h_{j}^{(1)}(A)$ $\displaystyle:=p_{j}\gamma e_{j}\tau\bar{\operatorname{tr}}\,A\Sigma_{j}K^{-1},$ (24) $\displaystyle h_{j}^{(2)}(A,B)$ $\displaystyle:=p_{j}\gamma\bar{\operatorname{tr}}\,A\Sigma_{j}(\gamma e_{j}\tau^{2}B+p_{j^{\prime}}\gamma\tau^{2}\Sigma_{j^{\prime}}(e_{j}u_{j^{\prime}}-e_{j^{\prime}}u_{j})+e_{j}\rho I_{d}-\lambda u_{j}\tau I_{d})K^{-2},$ (25) $\displaystyle h_{j}^{(3)}(A,B)$ $\displaystyle:=p_{j}\bar{\operatorname{tr}}\,A\Sigma_{j}(\gamma e_{j}^{2}p_{j}\Sigma_{j}(p_{j^{\prime}}\gamma\tau^{2}u_{j^{\prime}}\Sigma_{j^{\prime}}+\gamma\tau^{2}B+\rho I_{d})$ $\displaystyle+u_{j}(p_{j^{\prime}}\gamma e_{j^{\prime}}\tau\Sigma_{j^{\prime}}+\lambda I_{d})^{2})K^{-2},$ (26) $\displaystyle h_{j}^{(4)}(A,B)$ $\displaystyle:=p_{j}\gamma p_{j^{\prime}}\bar{\operatorname{tr}}\,\Sigma_{j}\Sigma_{j^{\prime}}A(\gamma\tau^{2}(e_{j}e_{j^{\prime}}B-p_{j}e_{j}^{2}u_{j^{\prime}}\Sigma_{j}-p_{j^{\prime}}\Sigma_{j^{\prime}}e_{j^{\prime}}^{2}u_{j})$ $\displaystyle-\lambda\tau(e_{j}u_{j^{\prime}}+e_{j^{\prime}}u_{j})I_{d}+e_{j}e_{j^{\prime}}\rho I_{d})K^{-2}.$ (27) $$

We now present Theorem [2](#A14.EGx13), which is our main contribution. Theorem [2](#A14.EGx13) presents a a novel bias-variance decomposition for the test error $R_{s}(\widehat{f})$ for each group $s=1,2$ in the context of ridge regression with random projections. It is a non-trivial generalization of theories in high-dimensional machine learning, which requires the powerful machinery of OVFPT (see proof in Appendix [E](#A5)).

###### Theorem 2 .

Under Assumptions [3](#Thmassumption3) and [2](#Thmassumption2), it holds that $R_{s}(\widehat{f})\simeq B_{s}(\widehat{f})+V_{s}(\widehat{f})$, with

$$ $\displaystyle V_{s}(\widehat{f})$ $\displaystyle=\sum_{j=1}^{2}\sigma_{j}^{2}\phi h_{j}^{(2)}(I_{d},\Sigma_{s}),$ (28) $\displaystyle B_{s}(\widehat{f})$ $\displaystyle=\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}+h_{1}^{(3)}(\Theta_{s},\Sigma_{s})+h_{2}^{(3)}(\Theta_{s},\Sigma_{s})+2h_{1}^{(4)}(\Theta_{s},\Sigma_{s})$ (29) $\displaystyle-2h_{1}^{(1)}(\Theta_{s}\Sigma_{s})-2h_{2}^{(1)}(\Theta_{s}\Sigma_{s})+h_{s^{\prime}}^{(3)}(\Delta,\Sigma_{s})$ (30) $\displaystyle-2\begin{cases}0,&s=1,\\ h_{1}^{(3)}(\Delta,\Sigma_{2})+h_{2}^{(4)}(\Delta,\Sigma_{2})-h_{1}^{(1)}(\Delta\Sigma_{2}),&s=2.\end{cases}$ (31) $$

We discuss methods for and the complexity of solving the above fixed-point equations in Appendix [G](#A7). The unregularized limit corresponds to the minimum-norm interpolator, and alternatively may be viewed as training a neural network till convergence (Ali et al., ).

Separate Random Projections Model Learned for Each Group. We now consider the ridge regression models $\widehat{f}_{1}$ and $\widehat{f}_{2}$ with random projections, which are learned using empirical risk minimization and $\ell_{2}$-regularization with penalties $\lambda_{1}$ and $\lambda_{2}$, respectively. In particular, we have the following optimization problem for each group $s$:
$\arg\min_{\eta\in\mathbb{R}^{m}}L(w)=n_{s}^{-1}\|X_{s}S\eta-Y_{s}\|^{2}_{2}+\lambda_{s}\|\eta\|_{2}^{2}$.
Alternatively, the reader should think of each $\widehat{f}_{s}$ as the limit of $\widehat{f}$ when $p_{s}\to 1$. In this setting, we deduce Theorem [3](#Thmtheorem3), which follows from Theorem [2](#A14.EGx13).

###### Theorem 3 .

Under Assumptions [3](#Thmassumption3) and [2](#Thmassumption2), it holds that $R_{s}(\widehat{f}_{s})\simeq B_{s}(\widehat{f}_{s})+V_{s}(\widehat{f}_{s})$, where $V_{s}(\widehat{f}_{s})=\lim_{p_{s}\to 1}V_{s}(\widehat{f})$ and $B_{s}(\widehat{f}_{s})=\lim_{p_{s}\to 1}B_{s}(\widehat{f})$. We relegate the explicit formulae for $B_{s}(\widehat{f}_{s})$ and $V_{s}(\widehat{f}_{s})$ to Appendix [F](#A6).

Phase Diagram.
The phase diagram for the random projections model (Figure [1](#S3.F1)) offers richer insights into how model capacity, in interaction with the number of features and samples, affects bias amplification. In the $ODD$ and $EDD$ profiles, we observe apparent phase transitions at $\phi=\psi$ (when $\psi<0.5$) and $\psi=0.5$ (i.e., $\psi_{1}=\psi_{2}=1$), where these metrics begin decreasing significantly. In contrast, at $\psi=1$ and $\phi=1$, the $ODD$ seems to drastically increase. Furthermore, at $\phi=\psi$ (when $\psi<0.5$) and $\phi=0.5$ (when $\psi>0.5$), the $EDD$ greatly increases. Accordingly, in the $ADD$ profile, we observe apparent phase transitions at $\phi=\psi$ (when $\psi<0.5$), $\psi=0.5$, $\psi=1$, and $\phi=1$, where bias amplification begins occurring (i.e., $ADD>1$). However, bias seems to be consistently deamplified (i.e., $ADD<1$) at $\phi=\psi$ (when $\psi<0.5$) and $\phi=0.5$ (when $\psi>0.5$).

Figure: Figure 1: $ODD$, $EDD$, and $ADD$ phase diagrams for ridge regression with random projections. We plot the bias amplification phase diagrams with respect to $\phi$ (rate of features to samples) and $\psi$ (rate of parameters to samples), as predicted by our theory for ridge regression with random projections (Theorems [2](#A14.EGx13), [3](#Thmtheorem3)). Red regions indicate theoretical predictions greater than 1 (i.e., bias amplification in the rightmost plot), while blue regions indicate theoretical predictions less than 1 (i.e., bias deamplification in the rightmost plot). Darkness indicates intensity. We consider isotropic covariance matrices: $\Sigma_{1}=2I_{d},\Sigma_{2}=I_{d}$, $\Theta=2I_{d}$, $\Delta=I_{d}$. Additionally, $n=1\times 10^{4},\sigma_{1}^{2}=\sigma_{2}^{2}=1$. We further choose $\lambda=\lambda_{1}=\lambda_{2}=1\times 10^{-6}$ to approximate the minimum-norm interpolator. We show that bias amplification can occur even in the balanced data setting, i.e., when $p_{1}=p_{2}=1/2$.
Refer to caption: /html/2410.17263/assets/x1.png

###### Definition 3 .

###### Theorem 2 .

###### Theorem 3 .

### 4 Bias Amplification

We empirically show how ridge regression models with random projections may amplify bias when a single model is trained on a combination of data from different groups vs. when separate models are trained on data from each group (Bell & Sagun, ). We further show how our theory: (1) predicts bias amplification, and (2) exposes new, interesting bias amplification phenomena in various regimes.

#### 4.1 Isotropic Covariance

Setup.
To be consistent with the setting of Bell & Sagun (), we set different ground-truth weights for the groups ($\Theta=2I_{d},\Delta=I_{d}$). We additionally consider balanced data ($p_{1}=p_{2}=1/2$) without spurious correlations ($\Sigma_{1}=a_{1}I_{d},\Sigma_{2}=a_{2}I_{d}$, for $a_{1},a_{2}>0$). We further choose $\lambda=1\times 10^{-6}$ to approximate the minimum-norm interpolator; we henceforth set $\lambda=\lambda_{1}=\lambda_{2}$ for simplicity. We present other experimental details in Appendix [I.1](#A9.SS1). We modulate $a_{1},a_{2},\sigma_{1}^{2},\sigma_{2}^{2}$, as well as $\psi$ (rate of parameters to samples) and $\phi$ (rate of features to samples) to understand the effects of model capacity and sample size on bias amplification. We consider diverse and dense values of these variables to obtain a clear picture of when and how models amplify bias.

Validation of Theory. Figure [2](#S4.F2) and the figures in Appendix [J](#A10) reveal that Theorems [2](#A14.EGx13) and [3](#Thmtheorem3) closely predict the $ODD$, $EDD$, and $ADD$ of ridge regression models with random projections under diverse settings. As indicated by the error bars, some of our empirical estimates (especially those with larger magnitude) have higher variance and their variance is influenced by the choice of $\psi,\phi,a_{1},a_{2},\sigma_{1}^{2},\sigma_{2}^{2}$. Notably, our theory predicts the observation of Bell & Sagun () that models can amplify bias even with balanced groups and without spurious correlations. We present new phenomena predicted by our theory below.

Effect of Label Noise. In the $ODD$ profile, the left tail is higher when the noise ratio $c=\sigma_{2}^{2}/\sigma_{1}^{2}$ is larger (compared to when it is lower), which suggests that under overparameterization, a larger noise ratio can increase disparities in test risk between groups when a single model is learned for both groups. We analytically explain this phenomenon in Appendix [K](#A11). In contrast, the $EDD$ curve is generally higher for larger $c$, suggesting that a larger noise ratio increases disparities in test risk when a separate model is learned for each group. We replicate this finding on real data in Figure [3](#S4.F3). Additionally, we observe that the $ADD$ grows faster close to the interpolation threshold for larger $c$, which suggests that a larger noise ratio can increase the maximum possible bias amplification.

Effect of Model Size. We observe interesting divergent behavior as $\psi$ (rate of parameters to samples) increases for different $\phi$ (rate of features to samples). When $\phi>1$, as $\psi$ increases, the $ODD$ increases and then decreases, peaking at the interpolation threshold at $\psi=1$. Similarly, when $\phi>0.5$ (i.e., $\phi_{1}=\phi_{2}>1$), as $\psi$ increases, the $EDD$ increases and then decreases, peaking at the interpolation threshold at $\psi=0.5$ (i.e., $\psi_{1}=\psi_{2}=1$). Accordingly, when $\phi>0.5$, bias is effectively deamplified (i.e., $ADD\ll 1$) at $\psi=0.5$ and when $\phi>1$, bias amplification peaks (i.e., $ADD\gg 1$) at $\psi=1$. In contrast, when $\phi<1$, the $ODD$ decreases as $\psi$ increases, plateauing at different finite values. Similarly, when $\phi<0.5$, the $EDD$ decreases and plateaus as $\psi$ increases. A notable exception to these trends occurs when $\phi\approx 1$, with the corresponding $ODD$ and $ADD$ curves consistently increasing as $\psi$ increases, plateauing at a significantly larger value (i.e., $ADD\gg 1$) than the curves corresponding to other values of $\psi$. Hence, overparameterization can greatly amplify bias when $\phi=1$. Regardless of the regime of $\phi$, the left tail of the $ADD$ profile appears to plateau at 1. The right tail plateaus at different finite values, with the curves corresponding to $\phi>1$ consistently plateauing above 1. This suggests that when $\phi>1$, overparameterization amplifies bias.

Effect of Number of Features. In the $ODD$ and $ADD$ profiles, when $\phi>1$, the right tail plateaus at higher values ($>1$) when $\phi$ is closer to 1. This suggests that with a similar number of features and samples, under overparameterization, bias amplification increases and may even be inevitable. In the $EDD$ profile, when $\phi>1$, the right tail plateaus at higher values when $\phi$ is larger. In contrast, when $\phi<1$, the right tail of the $ODD$ and $EDD$ curves plateaus at higher values when $\phi$ is larger. Regardless of the regime of $\phi$, the left tails of the $ODD$ and $EDD$ curves are higher for larger $\phi$.

Figure: Figure 2: Our theory predicts that models can amplify bias even with balanced groups and without spurious correlations. We empirically validate our theory (Theorems [2](#A14.EGx13) and [3](#Thmtheorem3)) for $ODD$, $EDD$, and $ADD$ under the setup described in Section [4.1](#S4.SS1), with $a_{1}=0.5,a_{2}=1,\sigma_{1}^{2}=1$, and $\sigma_{2}^{2}=1\times 10^{-5}$. The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We plot $ODD$ and $EDD$ on the same scale for easy comparison, and include a black dashed line at $ADD=1$ to contrast bias amplification vs. deamplification. We include the remaining plots with error bars in Appendix [J](#A10).
Refer to caption: /html/2410.17263/assets/x2.png

#### 4.2 Regularization and Training Dynamics

We now explore how regularization and training dynamics affect bias amplification.

Setup.
We revisit the setting described in Section [4.1](#S4.SS1). We modulate $a_{1},a_{2},\psi$ (rate of parameters to samples), as well as $\lambda$ (regularization penalty) to understand the effects of regularization and early stopping on bias amplification. We fix $\sigma_{1}^{2}=\sigma_{2}^{2}=1$, and the rate of features to samples $\phi=0.75$.

Effect of Regularization and Training Time. We can simulate model learning over training time $t$ by setting $\lambda=1/t$ (Ali et al., ). In Figure [11](#A13.F11) (in the appendix), we observe that regardless of the regime of $\psi$, $ADD\approx 1$ (i.e., there is neither bias amplification nor deamplification) when there is high regularization or a short training time. When $\psi>1$ (i.e., in the overparameterized regime), the $ADD$ is generally greater than 1 across values of $\lambda$ (i.e., bias is amplified), while when $\psi<1$ (i.e., in the underparameterized regime), the $ADD$ is less than 1 (i.e., bias is deamplified). Moreover, when $\psi>1$, as regularization decreases (or training time increases), bias amplification increases and plateaus. In contrast, when $\psi<1$, as regularization decreases (or training time increases), bias deamplification increases and plateaus. A notable exception to this trend occurs when $\psi$ is close to 1, where bias is initially deamplified and then amplified as $\lambda$ decreases (or $t$ increases). This suggests that there may be an optimal regularization penalty or training time to avoid bias amplification and increase bias deamplification. This aligns with the findings of Hall et al. () and Jain et al. () that bias amplification can vary substantially during training; future work can establish stronger connections between our theoretical results and the results of , who identify phases in the evolution of bias and a crossing phenomenon in the test error curves of groups during training. Intuitively, as training progresses, overparameterized models may discover “shortcut” associations that do not generalize equally well across groups, yielding bias amplification (Geirhos et al., ).

Corroboration on Real Data. We further investigate the effect of training time on bias amplification on a more realistic dataset. We train a convolutional neural network (CNN) on Colored MNIST (see Appendix [I.2](#A9.SS2) for more details). Colored MNIST is a semi-synthetic dataset derived from MNIST where digits are randomly re-colored to be red or green (Arjovsky et al., ). We treat the color of each digit as its group, and we manipulate the groups to have different levels of label noise. In our experimental protocol: (1) the color of each digit (in both train and test) is chosen uniformly at random (i.e., with probability 0.5) and independently of the label; (2) the labels of red digits are flipped with probability 0.05 while the labels of green digits are flipped with probability 0.25; (3) labels are binarized (i.e., digits 0-4 correspond to 0 while digits 5-9 correspond to 1); and (4) each training step constitutes a step of gradient descent based on a batch of 250 instances. Although Colored MNIST is a classification task and we use a complex CNN architecture, our theory correctly predicts that as the training time $t$ increases, the $ODD$ of the CNN is relatively low while the $EDD$ is much larger, producing bias deamplification (Figure [3](#S4.F3)).

Figure: Figure 3: Our theory predicts that disparate label noise between groups deamplifies bias on Colored MNIST. We plot the $ODD$ and $EDD$ of a CNN over training time $t$ for Colored MNIST. As $t$ increases, the $ODD$ is relatively low while the $EDD$ is noticeably higher. Please refer to Figure [6](#A9.F6) (in the appendix) for error bars.
Refer to caption: /html/2410.17263/assets/x3.png

Taking $t\to\infty$ corresponds to the setting of $\lambda\to 0^{+}$ in our theory (Theorems [2](#A14.EGx13), [3](#Thmtheorem3)). Because we assign the colors at random, the only difference in image features between groups would be color; therefore, we expect the covariance matrices $\Sigma_{1}$ and $\Sigma_{2}$ to coincide and $\Delta=0$ (i.e., $w_{1}^{*}=w_{2}^{*}$). Note that we do not make any assumptions about the structure of $\Sigma_{1},\Sigma_{2}$. Furthermore, $p_{1}=p_{2}=1/2$, and thus, $\phi_{1}=\phi_{2}$ and $\psi_{1}=\psi_{2}$. Additionally, we analogize the probability of label flipping to label noise in ridge regression. Hence, $e_{1}=e_{2},u_{1}=u_{2}$. Accordingly, $\lim_{\lambda\to 0^{+}}B_{1}(\widehat{f})=\lim_{\lambda\to 0^{+}}B_{1}(\widehat{f}_{1})\approx\lim_{\lambda\to 0^{+}}B_{2}(\widehat{f})=\lim_{\lambda\to 0^{+}}B_{2}(\widehat{f}_{2})$. Simultaneously, $\lim_{\lambda\to 0^{+}}V_{1}(\widehat{f})\approx\lim_{\lambda\to 0^{+}}V_{2}(\widehat{f})$. However, $\lim_{\lambda\to 0^{+}}V_{1}(\widehat{f}_{1})\approx\sigma_{1}^{2}/2\cdot V=0.05/2\cdot V=0.025V$ (where $V=\phi_{1}h_{1}^{(2)}(I_{d},\Sigma)$), while $\lim_{\lambda\to 0^{+}}V_{2}(\widehat{f}_{2})\approx\sigma_{2}^{2}/2\cdot V=0.25/2\cdot V=0.125V$. This results in $ODD\approx 0$ while $EDD\approx 0.1|V|$, which explains the divergence of $ODD$ and $EDD$ in Figure [3](#S4.F3). Intuitively, the high label noise for group 2 prohibits the separate model $\widehat{f}_{2}$ from achieving a low test risk compared to $\widehat{f}_{1}$; the single model $\widehat{f}$ achieves a comparable test risk on both groups, effectively deamplifying bias, because of learning signals from both groups.

### 5 Minority-Group Error

Recent work has revealed that overparameterization may hurt test performance on minority groups due to spurious features (Sagawa et al., ; Khani & Liang, ). Our theory provides new insights into how model size and extraneous features affect minority-group error.

Setup.
To be consistent with the settings of (Sagawa et al., ; Khani & Liang, ), we consider diatomic covariance matrices consisting of core and extraneous features. In particular, we define $A\oplus B=\begin{pmatrix}A&0\\
0&B\end{pmatrix}$, and choose $\Sigma_{1}=a_{1}I_{\pi d}\oplus 0I_{(1-\pi)d},\Sigma_{2}=a_{2}I_{\pi d}\oplus b_{2}I_{(1-\pi)d}$, for $\pi\in(0,1)$ and $a_{1},b_{2}>0$ and $a_{1}=a_{2}$. Here, the first $\pi d$ features represent common core features of groups 1 and 2 while the latter $(1-\pi)d$ features capture unshared extraneous features for group 2 (e.g., spurious features). Intuitively, this setting can model: (1) learning from data from two groups where one group suffers from spurious features (Sagawa et al., ), or (2) learning from a mixture of raw data (i.e., with spurious features) and clean data (i.e., without spurious features) for a single population (Khani & Liang, ). We ask: Does our theory predict how the inclusion of different amounts of extraneous features affect the test risk of the minority group when a single model is trained on data from both groups vs. a separate model is trained per group?

Although Sagawa et al. () consider classification instead of regression, to closely mirror their experimental setting, we pick $p_{1}=0.9$ (i.e., group 1 is much larger than group 2) and $\Theta=I_{d},\Delta=0$ (i.e., $w_{1}^{*}=w_{2}^{*}$). We additionally choose $\lambda=1\times 10^{-6}$ and $\sigma_{1}^{2}=\sigma_{2}^{2}=1$. We modulate $a_{1},b_{2}$, as well as $\psi$ (rate of parameters to samples) and $\phi$ (rate of features to samples). Notably, this setting also captures learning problems with $o(d)$ overlapping core and extraneous features in our asymptotic scaling limit. An extremization of this setting is choosing $\Sigma_{1}=a_{1}I_{\pi d}\oplus 0I_{(1-\pi)d},\Sigma_{2}=0I_{\pi d}\oplus b_{2}I_{(1-\pi)d}$, where groups 1 and 2 have entirely different sets of important features.

Figure: Figure 4: Minority-group test risk can peak with different model sizes depending on the rate of features to samples. We empirically demonstrate that minority-group bias is affected by extraneous features. We validate our theory (Theorems [2](#A14.EGx13) and [3](#Thmtheorem3)) for together $R_{1},R_{2}$ (i.e., single model learned for both groups) and separate $R_{1},R_{2}$ (i.e., separate model learned per group) under the setup described in Section [4.2](#S4.SS2), with $a_{1}=2,b_{2}=0.2$, and $\pi=0.5$. The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We include a black dashed line at $ADD=1$ to contrast bias amplification vs. deamplification. All y-axes are on the same scale for easy comparison. The remaining plots with error bars are in Appendix [N](#A14).
Refer to caption: /html/2410.17263/assets/x4.png

Interpolation Thresholds. The together test risk $R_{2}$ for the minority group has different interpolation thresholds as $\psi$ increases depending on $\phi$ and $\pi$ (fraction of core features). Notably, as $\phi$ increases, the interpolation thresholds occur at larger model sizes, culminating at $\psi=1$. This suggests that for a higher rate of features to samples, a larger model size can greatly increase the together test risk of group 2. Furthermore, the interpolation thresholds all occur closer to $\psi=1$ for larger $\pi$, collapsing to a single threshold at $\psi=1$ when $\pi\to 1$ as in Appendix [J](#A10). Therefore, a lower fraction of core features can yield more possible model sizes that increase the test risk of group 2. In addition, the together $R_{2}$ exhibits a steeper rate of growth around the interpolation thresholds for larger $b_{2}$, suggesting that a higher variance in the extraneous features can also increase the together test risk of group 2. The phenomenon of different interpolation thresholds is not visible for $R_{2}$ when a separate model is trained per group, nor for $R_{1}$.

Overparameterization. The right tails of the together $R_{2}$ curves plateau at different horizontal values depending on $\phi$. In particular, for $\phi$ closer to 1, the $R_{2}$ curves plateau at a higher value, suggesting that a similar number of features and samples can exacerbate minority-group error under overparameterization. Furthermore, when $\phi$ is close to 1, the together $R_{1}$ curves plateau at lower values than their corresponding together $R_{2}$ curves; this suggests that there can be fundamental differences in test error between groups, and thus fundamental model biases, that do not disappear even with increased model capacity. This phenomenon diminishes as the fraction of core features increases, and is not visible in the separate $R_{2}$ curves. Our experiments support the finding of Sagawa et al. () that overparameterization with spurious features increases test risk disparities between groups, and we nuance this finding by identifying that this phenomenon is most prominent in the regime where the number of features tends towards the number of samples.

### 6 Conclusion

In this paper, we present a unifying, rigorous, and effective theory of machine learning bias in the settings of ridge regression with and without random projections. We extensively empirically validate our theory in diverse settings. We further demonstrate that our theory provides interesting insights into bias amplification and minority-group error in various regimes. For example, there may be an optimal regularization penalty or training time to avoid bias amplification, and there can be fundamental differences in test error between groups that do not vanish with increased parameterization. These findings can inform strategies to evaluate and mitigate unfairness in machine learning.

Our methods and theory are easily extendable to analyze the case of more than two groups and can accommodate label noise sampled from various distributions. However, our theory is not directly extendable to different proportionate scaling limits, e.g., $d^{2}/n$ has a finite limit instead of $d/n$. Additionally, our theory requires approximately normally-distributed data and thus does not currently account for missing features, which are common in the real world (Feng et al., ). Furthermore, our theory implicitly assumes that group information is known, which is not always true (Coston et al., ); however, because we work in an asymptotic scaling limit, having access to group information with $o(\min(n_{1}),\min(n_{2}))$ noise is sufficient. As future work, we can leverage “Gaussian equivalents” (Goldt et al., ) to extend our theory to wide, fully-trained networks in the NTK (Jacot et al., ) and lazy (Chizat et al., ) regimes; this will enable us to understand how, apart from model size, other design choices like activation functions and learning rate may affect bias amplification.

## Appendix

### Appendix A Warm-up: Deriving Marchenko-Pastur Law via Operator-Valued Free Probability Theory

We provide a detailed example of how to apply linear pencils and operator-valued free probability theory to derive the Marchenko-Pastur law. Let $S=(1/n)X^{\top}X\in\mathbb{R}^{d\times d}$, the empirical covariance matrix for an $n\times d$ random matrix $X$ with IID entries from ${\cal N}(0,1)$. If $n$ tends to infinity while $d$ is held fixed, then $S$ converges to the population covariance matrix, here $\Sigma=I_{d}$. If $d$ also tends to infinity, then the limit seizes to exist. It turns out that one can still make sense of the limiting distribution of eigenvalues of $S$ in case $d/n$ stays constant, namely the behavior of the random histogram:

$$ $\displaystyle\widehat{\mu}_{n}=\frac{1}{d}\sum_{i=1}^{n}\delta_{\widehat{\lambda}_{i}},$ (32) $$

where $\widehat{\lambda}_{1},\ldots,\widehat{\lambda}_{d}$ are the eigenvalues of $S$. Let us state without any delay that in the aforementioned limit, i.e.,

$$ $\displaystyle n,d\to\infty,\,d/n\to\gamma\in(0,\infty),$ (33) $$

$\widehat{\mu}_{n}$ converges to a deterministic law $\mu_{\text{MP}}$ on $\mathbb{R}$ called the Marchenko-Pastur law. This is central to the field of random matrix theory (RMT), a central tool in probability theory, statistical analysis of neural networks, finance, etc. We are interested in an even more powerful tool – free probability theory (FPT) – is powerful enough to give a precise picture of deep learning in certain linearized regimes (e.g., random features, NTK, etc.) and interesting phenomena (e.g., triple descent, etc.) via analytic calculation.

#### A.1 Step 1: Constructing a Linear Pencil

For any positive $\lambda$, consider the $2\times 2$ block matrix $Q$ defined by:

$$ $Q=\begin{pmatrix}I_{n}&-\frac{X}{\sqrt{n\lambda}}\\ \frac{X^{\top}}{\sqrt{n\lambda}}&I_{d}\end{pmatrix}.$ (34) $$

Let $\bar{\operatorname{tr}}\,$ be the normalized trace operator on square matrices and set $\varphi=\mathbb{E}\circ\bar{\operatorname{tr}}\,$. This gives random $(n+d)\times(n+d)$ matrices the structure of a von Neumann algebra $\mathcal{A}$. Define a $2\times 2$ matrix $G=G(Q)$ by:

$$ $G=\mathbb{(}I_{2}\otimes\varphi)Q^{-1},\text{ i.e }g_{i,j}=\varphi([Q^{-1}]_{i,j})=[\varphi(Q^{-1})]_{i,j}\text{ for all }i,j\in\{1,2\}.$ (35) $$

Observe that:

$$ $\displaystyle\mathbb{E}\,\bar{\operatorname{tr}}\,(S+\lambda I_{d})^{-1}=\frac{g_{2,2}}{\lambda}.$ (36) $$

This is a direct consequence of inverting a $2\times 2$ block matrix (Schur’s complement).
The mechanical advantage of equation [36](#A1.E36) is that the *resolvent* $(S+\lambda I_{d})^{-1}$ depends quadratically on $X$ while $g_{2,2}$ is defined via $Q$, which is linear in $X$. For this reason, $Q$ is called a *linear pencil* for $(S+\lambda I_{d})^{-1}$. The construction of appropriate linear pencils rational functions of random matrices is a crucial step in leveraging FPT.

#### A.2 Step 2: Constructing the Fundamental Equation via Freeness

For any $B\in M_{b}(\mathbb{C})^{+}$ (here $b\times b$ is the number of blocks in the linear pencil $Q_{X}$, and so $b=2$), define a block matrix $B\otimes 1_{\mathcal{A}}$ by:

$$ $\displaystyle[B\otimes 1_{\mathcal{A}}]_{ij}=\begin{cases}b_{ij}I_{d_{i}},&\mbox{ if }d_{i}=d_{j},\\ 0,&\mbox{ else.}\end{cases}$ (37) $$

Now, observe that we can write $Q=F-Q_{X}$, where:

$$ $\displaystyle F=\begin{pmatrix}I_{d}&0\\ 0&I_{n}\end{pmatrix}=I_{2}\otimes 1_{\mathcal{A}}\text{ and }Q_{X}=\begin{pmatrix}0&\frac{X}{\sqrt{n\lambda}}\\ -\frac{X^{\top}}{\sqrt{n\lambda}}&0\end{pmatrix}.$ (38) $$

One can then express $G=(I_{b}\otimes\varphi)Q^{-1}=(I_{b}\otimes\varphi)(F-Q_{X})^{-1}$.
From operator-valued FPT, we know that in the proportionate scaling limit equation [33](#A1.E33), the following fixed-point equation (due to the asymptotic freeness of $Q_{X}$ and $F$) is satisfied by $G$:

$$ $G=(I_{b}\otimes\varphi)(F-R\otimes 1_{\mathcal{A}})^{-1},$ (39) $$

where $R=\mathcal{R}_{Q_{X}}(G)$, and $R_{Q_{X}}$ is the R-transform of $Q_{X}$ which maps $M_{b}(\mathbb{C})^{+}$ to itself like so:

$$ $\mathcal{R}_{Q_{X}}(B)_{ij}=\sum_{k,\ell}\sigma(i,k;\ell,j)\alpha_{k}b_{k\ell}.$ (40) $$

Here, $\sigma(i,k;\ell,j)$ is the covariance between the entries of block $(i,k)$ and block $(\ell,j)$ of $Q_{X}$, while $\alpha_{k}$ is the dimension of the block $(k,\ell)$.

#### A.3 Step 3: The Final Calculation

Due to the structure of $Q_{X}$, one computes from equation [40](#A1.E40):

$$ $\displaystyle r_{1,1}$ $\displaystyle=d\cdot\frac{-1}{n\lambda}g_{2,2}=-\frac{\gamma}{\lambda}g_{2,2},$ (41) $\displaystyle r_{1,2}$ $\displaystyle=0,$ (42) $\displaystyle r_{2,1}$ $\displaystyle=0,$ (43) $\displaystyle r_{2,2}$ $\displaystyle=n\cdot\frac{-1}{n\lambda}g_{1,1}=-\frac{1}{\lambda}g_{1,1}.$ (44) $$

Combining this with equation [39](#A1.E39), one has:

$$ $\displaystyle\begin{split}G=(I_{2}\otimes\varphi)(Z-R\otimes 1_{\mathcal{A}})^{-1}=(I_{2}-R)^{-1}&=\begin{pmatrix}1+(\gamma/\lambda)g_{2,2}&0\\ 0&1+g_{2,2}/\lambda\end{pmatrix}^{-1}\\ &=\begin{pmatrix}\lambda/(\lambda+\gamma g_{2,2})&0\\ 0&\lambda/(\lambda+g_{1,1})\end{pmatrix}.\end{split}$ (45) $$

Comparing the matrix entries, this translates to the following scalar equations:

$$ $\displaystyle g_{1,1}$ $\displaystyle=\frac{\lambda}{\lambda+\gamma g_{2,2}},$ (46) $\displaystyle g_{2,2}$ $\displaystyle=\frac{\lambda}{\lambda+g_{1,1}},$ (47) $\displaystyle\ g_{2,1}$ $\displaystyle=g_{1,2}=0.$ (48) $$

Plugging the second equation into the first (to eliminate $g_{1,1}$) gives:

$$ $g_{2,2}=\dfrac{\lambda}{\lambda+\lambda/(\lambda+\gamma g_{2,2})}.$ $$

Setting $m=g_{2,2}/\lambda$ then gives $m=(\lambda+1/(1+\gamma m))^{-1}$, i.e.:

$$ $\displaystyle\frac{1}{m}=\lambda+\frac{1}{1+\gamma m},$ (49) $$

which is precisely the functional equation characterizing the Stieltjes transform (evaluated at $\lambda=-z$) of the Marchenko-Pastur law with shape parameter $\gamma$. By treating $\lambda$ as a complex number and applying the Cauchy-inversion formula, we can recover $\mu_{\text{MP}}$.

### Appendix B Technical Assumptions

###### Assumption 3 .

The per-group covariance matrices $\Sigma_{1}$ and $\Sigma_{2}$ and ground-truth weight covariance matrices $\Theta$ and $\Delta$ are all simultaneously diagonalizable; hence, all these matrices commute.

While Assumption [3](#Thmassumption3) may appear reductive, our goal is to analyze the bias amplification phenomenon in a sufficient setting that does not introduce complexities due to non-commutativity.

###### Assumption 4 .

We assume the following spectral densities exist when $d\to\infty$:

- •
$\nu\in\mathcal{P}(\mathbb{R}_{+})$ is the limiting spectral density of $\Sigma_{2}\Sigma_{1}^{-1}$, of the ratios $\lambda_{j}^{(2)}/\lambda_{j}^{(1)}$ of the eigenvalues of the respective covariance matrices,
- •
$\mu\in\mathcal{P}(\mathbb{R}_{+})$ is the joint limiting density of the spectra of $\Sigma_{2}\Sigma_{1}^{-1}$ and $\Sigma_{1}$
- •
$\pi\in\mathcal{P}(\mathbb{R}_{+})$ is the limiting density of the spectrum of $\Delta$.

###### Assumption 3 .

###### Assumption 4 .

### Appendix C Theorem 4 : Separate Classical Model Learned Per Group

Suppose that the classical ridge regression models $\widehat{f}_{1}$ and $\widehat{f}_{2}$ are learned using empirical risk minimization and $\ell_{2}$-regularization with penalties $\lambda_{1}$ and $\lambda_{2}$, respectively. In particular, we have the following optimization problem for each group $s$:

$$ $\displaystyle\arg\min_{w\in\mathbb{R}^{d}}L(w)=\frac{1}{n_{s}}\sum_{(x_{i},y_{i})\in\mathcal{D}^{s}}(x_{i}^{\top}w-y_{i})^{2}+\lambda_{s}\|w\|^{2}=\frac{\|X_{s}w-Y_{s}\|^{2}_{2}}{n_{s}}+\lambda_{s}\|w\|^{2}.$ (50) $$

We first present some relevant definitions.

###### Definition 4 .

Let $\bar{\operatorname{df}}_{m}^{(s)}(t)=\bar{\operatorname{tr}}\,\Sigma_{s}^{m}\left(\Sigma_{s}+tI_{d}\right)^{-m}$, and $\kappa_{s}$ be the unique positive solution to the equation
$\kappa_{s}-\lambda_{s}=\kappa_{s}\phi_{s}\bar{\operatorname{df}}^{(s)}_{1}(\kappa_{s}).$

In this setting, we deduce Theorem [4](#A14.EGx25). We derive Theorems [4](#A14.EGx25) and [1](#Thmtheorem1) using OVFPT, which is sufficiently powerful to give the general case in which we are interested (i.e., two groups with general $p_{s},\Sigma_{s}$). Theorems [4](#A14.EGx25) and[1](#Thmtheorem1) are non-trivial generalizations of Proposition 3 from (Bach, ), which can be recovered by taking $p_{s}\to 1$ (i.e., $p_{s^{\prime}}\to 0$).

###### Theorem 4 .

Under Assumptions [3](#Thmassumption3) and [1](#Thmassumption1), it holds that:

$$ $\displaystyle R_{s}(\widehat{f}_{s})\simeq B_{s}(\widehat{f}_{s})+V_{s}(\widehat{f}_{s}),\text{ with }$ (51) $$

$$ $\displaystyle V_{s}(\widehat{f}_{s})=\frac{\sigma_{s}^{2}\phi_{s}\bar{\operatorname{df}}_{2}^{(s)}(\kappa_{s})}{1-\phi_{s}\bar{\operatorname{df}}_{2}^{(s)}(\kappa_{s})},\,B_{s}(\widehat{f}_{s})$ $\displaystyle=\frac{\kappa_{s}^{2}\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}\left(\Sigma_{s}+\kappa_{s}I_{d}\right)^{-2}}{1-\phi_{s}\bar{\operatorname{df}}_{2}^{(s)}(\kappa_{s})},\,\Theta_{1}=\Theta,\,\Theta_{2}=\Theta+\Delta.$ (52) $$

###### Proof.

We define $M_{s}=X_{s}^{\top}X_{s}$. Note that $\widehat{w}_{s}=X_{s}^{\dagger}Y_{s}=(X_{s}^{\top}X_{s}+n_{s}\lambda_{s}I_{d})^{-1}X_{s}^{\top}(X_{s}w^{*}_{s}+E_{s})=(M_{s}+n_{s}\lambda_{s}I_{d})^{-1}M_{s}w^{*}_{s}+(M_{s}+n_{s}\lambda_{s}I_{d})^{-1}X_{s}^{\top}E_{s}$. We deduce that $R_{s}(\widehat{f}_{s})=B_{s}(\widehat{f}_{s})+V_{s}(\widehat{f}_{s})$, where:

$$ $\displaystyle B_{s}(\widehat{f}_{s})$ $\displaystyle=\mathbb{E}\,\|(M_{s}+n_{s}\lambda_{s}I_{d})^{-1}M_{s}w^{*}_{s}-w^{*}_{s}\|_{\Sigma_{s}}^{2},$ (53) $\displaystyle V_{s}(\widehat{f}_{s})$ $\displaystyle=\mathbb{E}\,\|(M_{s}+n_{s}\lambda_{s}I_{d})^{-1}X_{s}^{\top}E_{s}\|_{\Sigma_{s}}^{2}.$ (54) $$

###### Definition 4 .

###### Theorem 4 .

###### Proof.

#### C.1 Variance Term

Note that the variance term $V_{s}(\widehat{f})$ of the test error of $\widehat{f}_{s}$ evaluated on group $s$ is given by:

$$ $\displaystyle V_{s}(\widehat{f}_{s})$ $\displaystyle=\sigma_{s}^{2}\mathbb{E}\,\operatorname{tr}\,X_{s}(M_{s}+n_{s}\lambda_{s}I_{d})^{-1}\Sigma_{s}(M_{s}+n_{s}\lambda_{s}I_{d})^{-1}X_{s}^{\top}$ (55) $\displaystyle=\sigma_{s}^{2}\mathbb{E}\,\operatorname{tr}\,(M_{s}+n_{s}\lambda_{s}I_{d})^{-1}M_{s}(M_{s}+n_{s}\lambda_{s}I_{d})^{-1}\Sigma_{s}.$ (56) $$

We can re-express this as:

$$ $\displaystyle n_{s}V_{s}(\widehat{f}_{s})$ $\displaystyle=\sigma_{s}^{2}\mathbb{E}\operatorname{tr}\,(H_{s}+\lambda_{s}I_{d})^{-1}H_{s}(H_{s}+\lambda_{s}I_{d})^{-1}\Sigma_{s}$ (57) $\displaystyle=\frac{\sigma_{s}^{2}}{\lambda_{s}}\mathbb{E}\operatorname{tr}\,(H_{s}/\lambda_{s}+I_{d})^{-1}(H_{s}/\lambda_{s})(H_{s}/\lambda_{s}+I_{d})^{-1}\Sigma_{s},$ (58) $$

where $H_{s}=X_{s}^{\top}X_{s}/n_{s}$ and $X_{s}=Z_{s}\Sigma_{s}^{1/2}$, with $Z_{1}\in\mathbb{R}^{n_{1}\times d}$ and $Z_{2}\in\mathbb{R}^{n_{2}\times d}$ being independent random matrices with IID entries from ${\cal N}(0,1)$. The typical variance term is proportional to:

$$ $\displaystyle\bar{\operatorname{tr}}\,(H_{s}+\lambda_{s}I_{d})^{-1}H_{s}(H_{s}+\lambda_{s}I_{d})^{-1}\Sigma_{s}.$ (59) $$

WLOG, we consider the case where $s=1$. The matrix of interest has a linear pencil representation given by (with zero-based indexing):

$$ $\displaystyle(H_{1}/\lambda_{1}+I_{d})^{-1}(H_{1}/\lambda_{1})(H_{1}/\lambda_{1}+I_{d})^{-1}\Sigma_{1}=Q^{-1}_{0,8},$ (60) $$

where the linear pencil $Q$ is defined as follows:

$$ $\displaystyle Q=\leavevmode\resizebox{346.89731pt}{}{$\left(\begin{array}[]{ccccccccc}I_{d}&\Sigma_{1}^{\frac{1}{2}}&0&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0\\ 0&I_{d}&-\frac{1}{\sqrt{\lambda_{1}}\sqrt{n_{1}}}Z_{1}^{\top}&0&0&0&0&0&0\\ 0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda_{1}}\sqrt{n_{1}}}Z_{1}&0&0&0&0&0\\ -\Sigma_{1}^{\frac{1}{2}}&0&0&I_{d}&0&0&0&0&0\\ 0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda_{1}}\sqrt{n_{1}}}Z_{1}^{\top}&0&0&0\\ 0&0&0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda_{1}}\sqrt{n_{1}}}Z_{1}&0&0\\ 0&0&0&0&0&0&I_{d}&-\Sigma_{1}^{\frac{1}{2}}&0\\ 0&0&0&0&\Sigma_{1}^{\frac{1}{2}}&0&0&I_{d}&-\Sigma_{1}\\ 0&0&0&0&0&0&0&0&I_{d}\\ \end{array}\right)$}.$ (70) $$

We compute $Q$ using the NCMinimalDescriptorRealization function of the NCAlgebra library(^1^11[https://github.com/NCAlgebra/NC](https://github.com/NCAlgebra/NC)). We further symmetrize $Q$ by constructing the self-adjoint matrix $\overline{Q}$:

$$ $\displaystyle\overline{Q}=\left(\begin{array}[]{cc}0&Q^{\top}\\ Q&0\end{array}\right).$ (73) $$

This enables us to apply known formulas for the $R$-transform of Gaussian block matrices (Far et al., ). We note that $\overline{Q}^{-1}_{0,17}=Q^{-1}_{0,8}$. Taking similar steps as Lee et al. (), we use operator-valued free probability theory (OVFPT) on $\overline{Q}$. Let $G=(I_{18}\otimes\mathbb{E}\,\bar{\operatorname{tr}}\,)\overline{Q}^{-1}\in\mathbb{R}^{18\times 18}$ be the matrix whose entries are normalized traces of blocks(^2^22By convention, the trace of a non-square block is zero.) of $\overline{Q}^{-1}$. We provide a detailed example of how to apply linear pencils and operator-valued free probability theory to derive the Marchenko-Pastur law in Appendix [A](#A1). One can arrive at that, in the asymptotic limit given by equation [9](#S3.E9), the following holds:

$$ $\displaystyle\begin{split}&\mathbb{E}\bar{\operatorname{tr}}\,(H_{1}+\lambda_{1}I_{d})^{-1}H_{1}(H_{1}+\lambda_{1}I_{d})^{-1}\Sigma_{1}=\frac{G_{0,17}}{\lambda_{1}},\\ &\text{ with }\frac{G_{0,17}}{\lambda_{1}}=(G_{5,14}-G_{2,14})\bar{\operatorname{tr}}\,\left(\Sigma_{1}G_{2,11}+\lambda_{1}I_{d}\right)^{-1}\Sigma_{1}\left(\Sigma_{1}G_{5,14}+\lambda_{1}I_{d}\right)^{-1}\Sigma_{1}.\end{split}$ (74) $$

We will now obtain the fixed-point equations satisfied by $G_{2,11}$ and $G_{5,14}$. We observe that:

$$ $\displaystyle G_{2,11}=-\frac{\lambda_{1}}{-\lambda_{1}+\phi_{1}G_{3,10}},\quad G_{3,10}=-\lambda_{1}\bar{\operatorname{tr}}\,\Sigma_{1}(\Sigma_{1}G_{2,11}+\lambda_{1}I_{d})^{-1}$ (75) $\displaystyle\implies G_{2,11}=\frac{1}{1+\phi_{1}\bar{\operatorname{tr}}\,\Sigma_{1}(\Sigma_{1}G_{2,11}+\lambda_{1}I_{d})^{-1}},$ (76) $\displaystyle G_{5,14}=-\frac{\lambda_{1}}{-\lambda_{1}+\phi_{1}G_{6,13}},\quad G_{6,13}=-\lambda_{1}\bar{\operatorname{tr}}\,\Sigma_{1}\left(\Sigma_{1}G_{5,14}+\lambda_{1}I_{d}\right)^{-1}$ (77) $\displaystyle\implies G_{5,14}=\frac{1}{1+\phi_{1}\bar{\operatorname{tr}}\,\Sigma_{1}\left(\Sigma_{1}G_{5,14}+\lambda_{1}I_{d}\right)^{-1}}.$ (78) $$

We recognize that we must have the identification $e_{1}=G_{2,11}=G_{5,14}$, where $e_{1}\geq 0$. Therefore:

$$ $\displaystyle e_{1}$ $\displaystyle=\frac{e_{1}}{e_{1}+\phi_{1}\bar{\operatorname{df}}^{(1)}_{1}(\lambda_{1}/e_{1})}$ (79) $\displaystyle\text{i.e., }1$ $\displaystyle=e_{1}+\phi_{1}\bar{\operatorname{df}}^{(1)}_{1}(\lambda_{1}/e_{1})=\lambda_{1}/\kappa_{1}+\phi_{1}\bar{\operatorname{df}}^{(1)}_{1}(\kappa_{1})$ (80) $\displaystyle\kappa_{1}$ $\displaystyle=\lambda_{1}+\kappa_{1}\phi_{1}\bar{\operatorname{df}}^{(1)}_{1}(\kappa_{1}),$ (81) $$

where $\bar{\operatorname{df}}_{m}^{(s)}(t)=\bar{\operatorname{tr}}\,\Sigma_{s}^{m}\left(\Sigma_{s}+tI_{d}\right)^{-m}$ and $\kappa_{1}=\lambda_{1}/e_{1}$. Additionally:

$$ $\displaystyle G_{2,14}$ $\displaystyle=\frac{\lambda_{1}\phi_{1}G_{3,13}}{(-\lambda_{1}+\phi_{1}G_{3,10})(-\lambda_{1}+\phi_{1}G_{6,13})}=\phi_{1}e_{1}^{2}\frac{G_{3,13}}{\lambda_{1}},$ (82) $\displaystyle\frac{G_{3,13}}{\lambda_{1}}$ $\displaystyle=\bar{\operatorname{tr}}\,(\Sigma_{1}G_{2,11}+\lambda_{1}I_{d})^{-2}(\Sigma_{1}G_{2,14}+\lambda_{1}I_{d})\Sigma_{1}$ (83) $\displaystyle=\frac{G_{2,14}}{e_{1}^{2}}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})+\lambda_{1}\bar{\operatorname{tr}}\,(\Sigma_{1}e_{1}+\lambda_{1}I_{d})^{-2}\Sigma_{1},$ (84) $\displaystyle\frac{G_{3,10}}{\lambda_{1}}$ $\displaystyle=-\bar{\operatorname{tr}}\,(\Sigma_{1}e_{1}+\lambda_{1}I_{d})^{-1}\Sigma_{1}.$ (85) $$

Then:

$$ $\displaystyle G_{5,14}-G_{2,14}$ $\displaystyle=e_{1}^{2}\left(1-\phi_{1}\frac{G_{3,10}+G_{3,13}}{\lambda_{1}}\right),$ (86) $\displaystyle\frac{G_{3,10}+G_{3,13}}{\lambda_{1}}$ $\displaystyle=\frac{G_{2,14}}{e_{1}^{2}}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})+\lambda_{1}\bar{\operatorname{tr}}\,(\Sigma_{1}e_{1}+\lambda_{1}I_{d})^{-2}\Sigma_{1}$ (87) $\displaystyle-\bar{\operatorname{tr}}\,(\Sigma_{1}e_{1}+\lambda_{1}I_{d})^{-2}(\Sigma_{1}e_{1}+\lambda_{1}I_{d})\Sigma_{1}$ (88) $\displaystyle=\frac{G_{2,14}}{e_{1}^{2}}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})-\frac{e_{1}}{e_{1}^{2}}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})$ (89) $\displaystyle=-\frac{G_{5,14}-G_{2,14}}{e_{1}^{2}}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1}).$ (90) $$

We define:

$$ $\displaystyle c_{1}\geq 1,c_{1}=\frac{G_{5,14}-G_{2,14}}{e_{1}^{2}}=1+\phi_{1}c_{1}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1}),$ (91) i.e., $\displaystyle c_{1}=\frac{1}{1-\phi_{1}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})}.$ (92) $$

Hence:

$$ $\displaystyle\frac{G_{0,17}}{\lambda_{1}}=c_{1}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})=\frac{\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})}{1-\phi_{1}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})}.$ (93) $$

In conclusion:

$$ $\displaystyle\kappa_{1}=\lambda_{1}+\kappa_{1}\phi_{1}\bar{\operatorname{df}}^{(1)}_{1}(\kappa_{1}),$ (94) $\displaystyle V_{1}(\widehat{f}_{1})=\frac{\sigma_{1}^{2}\phi_{1}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})}{1-\phi_{1}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})}.$ (95) $$

Following similar steps for $V_{2}(\widehat{f}_{2})$, we get:

$$ $\displaystyle\kappa_{2}=\lambda_{2}+\kappa_{2}\phi_{2}\bar{\operatorname{df}}^{(2)}_{1}(\kappa_{2}),$ (96) $\displaystyle V_{2}(\widehat{f}_{2})=\frac{\sigma_{2}^{2}\phi_{2}\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})}{1-\phi_{2}\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})}.$ (97) $$

To further substantiate our result, let us consider the unregularized case where $\lambda_{s}=0$ and $\phi_{s}<1$:

$$ $\displaystyle\kappa_{s}=0,V_{s}(\widehat{f}_{s})=\frac{\sigma_{s}^{2}\phi_{s}}{1-\phi_{s}}.$ (98) $$

From an alternate angle, we know that:

$$ $\displaystyle R_{s}(\widehat{f}_{s})$ $\displaystyle=\mathbb{E}\,\|\widehat{w}_{s}-w^{*}_{s}\|_{\Sigma_{s}}^{2}=\mathbb{E}\,\|(X_{s}^{\top}X_{s})^{-1}X_{s}^{\top}E_{s}\|_{\Sigma_{s}}^{2}$ (99) $\displaystyle=\sigma_{s}^{2}\mathbb{E}\operatorname{tr}\,X_{s}(X_{s}^{\top}X_{s})^{-1}\Sigma_{s}(X_{s}^{\top}X_{s})^{-1}X_{s}^{\top}$ (100) $\displaystyle=\sigma_{s}^{2}\mathbb{E}\operatorname{tr}\,(X_{s}^{\top}X_{s})^{-1}\Sigma_{s}=\frac{\sigma_{s}^{2}}{n_{s}-d-1}\operatorname{tr}\,I_{d}=\sigma_{s}^{2}\frac{d}{n_{s}-d-1}\simeq\frac{\sigma_{s}^{2}\phi_{s}}{1-\phi_{s}},$ (101) $$

where we have used Lemma [1](#A14.EGx44) below.

###### Lemma 1 .

Let $n$ and $d$ be positive integers with $n\geq d+2$. If $Z$ is an $n\times d$ random matrix with IID rows from ${\cal N}(0,\Sigma)$, then:

$$ $\displaystyle\mathbb{E}(Z^{\top}Z)^{-1}=\frac{1}{n-d-1}\Sigma^{-1}.$ (102) $$

###### Lemma 1 .

#### C.2 Bias Term

We can compute the bias term $B_{s}(\widehat{f}_{s})$ of the test error of $\widehat{f}_{s}$ evaluated on group $s$ as:

$$ $\displaystyle B_{s}(\widehat{f}_{s})$ $\displaystyle=\mathbb{E}\,\|(M_{s}+n_{s}\lambda_{s}I_{d})^{-1}M_{s}w^{*}_{s}-w^{*}_{s}\|_{\Sigma_{s}}^{2}$ (103) $\displaystyle=\mathbb{E}\,\|(M_{s}+n_{s}\lambda_{s}I_{d})^{-1}M_{s}w^{*}_{s}-(M_{s}+n_{s}\lambda_{s}I_{d})^{-1}(M_{s}+n_{s}\lambda_{s}I_{d})w^{*}_{s}\|_{\Sigma_{s}}^{2}$ (104) $\displaystyle=\mathbb{E}\,\|(M_{s}+n_{s}\lambda_{s}I_{d})^{-1}n_{s}\lambda_{s}w^{*}_{s}\|_{\Sigma_{s}}^{2}$ (105) $\displaystyle=n_{s}^{2}\lambda_{s}^{2}\mathbb{E}\,\operatorname{tr}\,(M_{s}+n_{s}\lambda_{s}I_{d})^{-1}w^{*}_{s}(w^{*}_{s})^{\top}(M_{s}+n_{s}\lambda_{s}I_{d})^{-1}\Sigma_{s}.$ (106) $$

We can re-express this as:

$$ $\displaystyle\frac{1}{\lambda_{s}^{2}}B_{s}(\widehat{f}_{s})$ $\displaystyle=\mathbb{E}\,\bar{\operatorname{tr}}\,(H_{s}+\lambda_{s}I_{d})^{-1}\Theta_{s}(H_{s}+\lambda_{s}I_{d})^{-1}\Sigma_{s}$ (107) $\displaystyle B_{s}(\widehat{f}_{s})$ $\displaystyle=\mathbb{E}\,\bar{\operatorname{tr}}\,(H_{s}/\lambda_{s}+I_{d})^{-1}\Theta_{s}(H_{s}/\lambda_{s}+I_{d})^{-1}\Sigma_{s},$ (108) $$

where $\Theta_{s}=\begin{cases}\Theta,&s=1\\
\Theta+\Delta,&s=2\end{cases}$. WLOG, we consider the case where $s=1$. The matrix of interest has a linear pencil representation given by (with zero-based indexing):

$$ $\displaystyle(H_{1}/\lambda_{1}+I_{d})^{-1}\Theta(H_{1}/\lambda_{1}+I_{d})^{-1}\Sigma_{1}=Q^{-1}_{0,8},$ (109) $$

where the linear pencil $Q$ is defined as follows:

$$ $Q=\leavevmode\resizebox{346.89731pt}{}{$\left(\begin{array}[]{ccccccccc}I_{d}&\Sigma_{1}^{\frac{1}{2}}&0&0&-\Theta&0&0&0&0\\ 0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}^{\top}&0&0&0&0&0&0\\ 0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}&0&0&0&0&0\\ -\Sigma_{1}^{\frac{1}{2}}&0&0&I_{d}&0&0&0&0&0\\ 0&0&0&0&I_{d}&\Sigma_{1}^{\frac{1}{2}}&0&0&-\Sigma_{1}\\ 0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}^{\top}&0&0\\ 0&0&0&0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}&0\\ 0&0&0&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&I_{d}&0\\ 0&0&0&0&0&0&0&0&I_{d}\\ \end{array}\right)$}.$ (110) $$

We note that $\overline{Q}^{-1}_{0,17}=Q^{-1}_{0,8}$. Using OVFPT, we deduce that, in the limit given by equation [9](#S3.E9), the following holds:

$$ $\displaystyle\mathbb{E}\,\bar{\operatorname{tr}}\,(H_{1}/\lambda_{1}+I_{d})^{-1}\Theta(H_{1}/\lambda_{1}+I_{d})^{-1}\Sigma_{1}=G_{0,17},$ (111) $\displaystyle\text{with }G_{0,17}=\lambda_{1}\bar{\operatorname{tr}}\,\left(\Sigma_{1}G_{2,11}+\lambda_{1}I_{d}\right)^{-1}(\lambda_{1}\Theta+\Sigma_{1}G_{2,15})\left(\Sigma_{1}G_{6,15}+\lambda_{1}I_{d}\right)^{-1}\Sigma_{1}.$ (112) $$

We will now obtain the fixed-point equations satisfied by $G_{2,11}$ and $G_{6,15}$. We observe that:

$$ $\displaystyle G_{2,11}=-\frac{\lambda_{1}}{-\lambda_{1}+\phi_{1}G_{3,10}},\quad G_{3,10}=-\lambda_{1}\bar{\operatorname{tr}}\,\Sigma_{1}(\Sigma_{1}G_{2,11}+\lambda_{1}I_{d})^{-1}$ (113) $\displaystyle\implies G_{2,11}=\frac{1}{1+\phi_{1}\bar{\operatorname{tr}}\,\Sigma_{1}(\Sigma_{1}G_{2,11}+\lambda_{1}I_{d})^{-1}},$ (114) $\displaystyle G_{6,15}=-\frac{\lambda_{1}}{-\lambda_{1}+\phi_{1}G_{7,14}},\quad G_{7,14}=-\lambda_{1}\bar{\operatorname{tr}}\,\Sigma_{1}\left(\Sigma_{1}G_{6,15}+\lambda_{1}I_{d}\right)^{-1}$ (115) $\displaystyle\implies G_{6,15}=\frac{1}{1+\phi_{1}\bar{\operatorname{tr}}\,\Sigma_{1}\left(\Sigma_{1}G_{6,15}+\lambda_{1}I_{d}\right)^{-1}}.$ (116) $$

We recognize that we must have the identification $e_{1}=G_{2,11}=G_{6,15}$, where $e_{1}\geq 0$. Therefore:

$$ $\displaystyle e_{1}$ $\displaystyle=\frac{1}{1+\phi_{1}\bar{\operatorname{tr}}\,\Sigma_{1}\left(\Sigma_{1}e_{1}+\lambda_{1}I_{d}\right)^{-1}},$ (117) $\displaystyle\text{i.e., }\kappa_{1}$ $\displaystyle=\lambda_{1}+\kappa_{1}\phi_{1}\bar{\operatorname{df}}^{(1)}_{1}(\kappa_{1}).$ (118) $$

Additionally:

$$ $\displaystyle G_{2,15}$ $\displaystyle=\frac{\lambda_{1}\phi_{1}G_{3,14}}{(-\lambda_{1}+\phi_{1}G_{3,10})(-\lambda_{1}+\phi_{1}G_{7,14})}=\phi_{1}e_{1}^{2}\frac{G_{3,14}}{\lambda_{1}},$ (119) $\displaystyle\frac{G_{3,14}}{\lambda_{1}}$ $\displaystyle=\bar{\operatorname{tr}}\,(\Sigma_{1}G_{2,11}+\lambda_{1}I_{d})^{-2}(\Sigma_{1}G_{2,15}+\lambda_{1}\Theta)\Sigma_{1}$ (120) $\displaystyle=\frac{G_{2,15}}{e_{1}^{2}}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})+\frac{\lambda_{1}}{e_{1}^{2}}\bar{\operatorname{tr}}\,(\Sigma_{1}+\kappa_{1}I_{d})^{-2}\Theta\Sigma_{1},$ (121) $\displaystyle\implies G_{2,15}$ $\displaystyle=\phi_{1}G_{2,15}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})+\lambda_{1}\phi_{1}\bar{\operatorname{tr}}\,(\Sigma_{1}+\kappa_{1}I_{d})^{-2}\Theta\Sigma_{1},$ (122) $\displaystyle\text{i.e., }G_{2,15}$ $\displaystyle=\frac{\lambda_{1}\phi_{1}}{1-\phi_{1}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})}\bar{\operatorname{tr}}\,(\Sigma_{1}+\kappa_{1}I_{d})^{-2}\Theta\Sigma_{1}.$ (123) $$

Hence:

$$ $\displaystyle G_{0,17}$ $\displaystyle=\kappa_{1}^{2}\bar{\operatorname{tr}}\,\left(\Sigma_{1}+\kappa_{1}I_{d}\right)^{-2}\Theta\Sigma_{1}+\kappa_{1}^{2}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})\frac{G_{2,15}}{\lambda_{1}}$ (124) $\displaystyle=\kappa_{1}^{2}\bar{\operatorname{tr}}\,\left(\Sigma_{1}+\kappa_{1}I_{d}\right)^{-2}\Theta\Sigma_{1}+\kappa_{1}^{2}\frac{\phi_{1}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})}{1-\phi_{1}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})}\bar{\operatorname{tr}}\,(\Sigma_{1}+\kappa_{1}I_{d})^{-2}\Theta\Sigma_{1}$ (125) $\displaystyle=\left(1+\frac{\phi_{1}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})}{1-\phi_{1}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})}\right)\kappa_{1}^{2}\bar{\operatorname{tr}}\,\left(\Sigma_{1}+\kappa_{1}I_{d}\right)^{-2}\Theta\Sigma_{1}.$ (126) $$

In conclusion:

$$ $\displaystyle B_{1}(\widehat{f}_{1})=\frac{\kappa_{1}^{2}\bar{\operatorname{tr}}\,\left(\Sigma_{1}+\kappa_{1}I_{d}\right)^{-2}\Theta\Sigma_{1}}{1-\phi_{1}\bar{\operatorname{df}}_{2}^{(1)}(\kappa_{1})}.$ (127) $$

Following similar steps for $B_{2}(\widehat{f}_{2})$, we get:

$$ $\displaystyle B_{2}(\widehat{f}_{2})=\frac{\kappa_{2}^{2}\bar{\operatorname{tr}}\,\left(\Sigma_{2}+\kappa_{2}I_{d}\right)^{-2}(\Theta+\Delta)\Sigma_{2}}{1-\phi_{2}\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})}.$ (128) $$

We observe that in the unregularized case (i.e., $\lambda_{s}=0$), $\kappa_{s}=0$. In this setting, $B_{s}(\widehat{f}_{s})=0$ as expected.
∎

Figure: Figure 5: $ODD$, $EDD$, and $ADD$ phase diagrams for classical ridge regression. We plot the bias amplification phase diagrams with respect to $\phi$ (rate of features to samples), as predicted by our theory for ridge regression without random projections (Theorems [1](#Thmtheorem1), [4](#A14.EGx25)). Dashed black lines indicate theoretical predictions. We consider isotropic covariance matrices: $\Sigma_{1}=2I_{d},\Sigma_{2}=I_{d}$, $\Theta=2I_{d}$, $\Delta=I_{d}$. Additionally, $n=1\times 10^{4},\sigma_{1}^{2}=\sigma_{2}^{2}=1$. We further choose $\lambda=\lambda_{1}=\lambda_{2}=1\times 10^{-6}$ to approximate the minimum-norm interpolator. We show that bias amplification can occur even in the balanced data setting, i.e., when $p_{1}=p_{2}=1/2$, without spurious correlations.
Refer to caption: /html/2410.17263/assets/x5.png

### Appendix D Proof of Theorem 1

###### Proof.

We define $M=X^{\top}X+n\lambda I_{d}$. Note that one has:

$$ $\displaystyle\widehat{w}$ $\displaystyle=X^{\dagger}Y=X^{\dagger}(X_{1}w^{*}_{1}+E_{1},X_{2}w^{*}_{2}+E_{2})=M^{-1}(M_{1}w^{*}_{1}+X_{1}^{\top}E_{1}+M_{2}w^{*}_{2}+X_{2}^{\top}E_{2}).$ (129) $$

We deduce that $R_{s}(\widehat{f})=B_{s}(\widehat{f})+V_{s}(\widehat{f})$, where:

$$ $\displaystyle B_{s}(\widehat{f})$ $\displaystyle=\mathbb{E}\,\|M^{-1}M_{s^{\prime}}w^{*}_{s^{\prime}}+M^{-1}M_{s}w^{*}_{s}-w^{*}_{s}\|_{\Sigma_{s}}^{2},$ (130) $\displaystyle V_{s}(\widehat{f})$ $\displaystyle=\mathbb{E}\,\|M^{-1}(X_{1}^{\top}E_{1}+X_{2}^{\top}E_{2})\|_{\Sigma_{s}}^{2}$ (131) $\displaystyle=\mathbb{E}\,\|M^{-1}X_{1}^{\top}E_{1}\|_{\Sigma_{s}}^{2}+\mathbb{E}\,\|M^{-1}X_{2}^{\top}E_{2}\|_{\Sigma_{s}}^{2},$ (132) $$

with $s^{\prime}=\begin{cases}2,&s=1\\
1,&s=2\end{cases}$.

###### Proof.

#### D.1 Variance Terms

Note that $V_{s}(\widehat{f})$ of the test error of $\widehat{f}$ evaluated on group $s$ is given by:

$$ $\displaystyle V_{s}(\widehat{f})$ $\displaystyle=\sigma_{1}^{2}\mathbb{E}\,\operatorname{tr}\,X_{1}M^{-1}\Sigma_{s}M^{-1}X_{1}^{\top}+\sigma_{2}^{2}\mathbb{E}\,\operatorname{tr}\,X_{2}M^{-1}\Sigma_{s}M^{-1}X_{2}^{\top}$ (133) $\displaystyle=\sigma_{1}^{2}\mathbb{E}\,\operatorname{tr}\,M^{-1}M_{1}M^{-1}\Sigma_{s}+\sigma_{2}^{2}\mathbb{E}\,\operatorname{tr}\,M^{-1}M_{2}M^{-1}\Sigma_{s}.$ (134) $$

We can re-express this as:

$$ $\displaystyle nV_{s}(\widehat{f})=\sigma_{1}^{2}\mathbb{E}\operatorname{tr}\,(H+\lambda I_{d})^{-1}H_{1}(H+\lambda I_{d})^{-1}\Sigma_{s}+\sigma_{2}^{2}\mathbb{E}\,\operatorname{tr}\,(H+\lambda I_{d})^{-1}H_{2}(H+\lambda I_{d})^{-1}\Sigma_{s},$ (135) $$

where $H=H_{1}+H_{2}$, $H_{s}=X_{s}^{\top}X_{s}/n$, and $X_{s}=Z_{s}\Sigma_{s}^{1/2}$ with $Z_{1}\in\mathbb{R}^{n_{1}\times d}$ and $Z_{2}\in\mathbb{R}^{n_{2}\times d}$ being independent random matrices with IID entries from ${\cal N}(0,1)$.

WLOG, we focus on $\operatorname{tr}\,(H+\lambda I_{d})^{-1}H_{2}(H+\lambda I_{d})^{-1}\Sigma_{s}$. The matrix of interest has a linear pencil representation given by (with zero-based indexing):

$$ $\displaystyle(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}(H_{2}/\lambda)(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Sigma_{s}=Q^{-1}_{1,8},$ (136) $$

where the linear pencil $Q$ is defined as follows:

$$ $Q=$ $\left(\begin{array}[]{ccccccccccccccc}I_{d}&0&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}^{\top}&0&0&0&0&0&0&0&0&0&0&0&0\\ -\Sigma_{2}^{\frac{1}{2}}&I_{d}&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&I_{d}&-\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&0&0&0&0&0&0\\ \Sigma_{2}^{\frac{1}{2}}&0&0&0&I_{d}&\Sigma_{1}^{\frac{1}{2}}&0&0&-\Sigma_{s}&0&0&0&0&0&0\\ 0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}^{\top}&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}&0&0&0&0&0&0&0\\ 0&0&0&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&I_{d}&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&I_{d}&\Sigma_{1}^{\frac{1}{2}}&0&0&\Sigma_{2}^{\frac{1}{2}}&0&0\\ 0&0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}^{\top}&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}&0&0&0\\ 0&0&0&0&0&0&0&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&I_{d}&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}^{\top}&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}\\ 0&0&0&0&0&0&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&0&I_{d}\\ \end{array}\right)$ . (137) $$

Using OVFPT, we deduce that, in the limit given by equation [9](#S3.E9), the following holds:

$$ $\displaystyle\mathbb{E}\,\bar{\operatorname{tr}}\,(H_{1}+H_{2}+\lambda I_{d})^{-1}H_{2}(H_{1}+H_{2}+\lambda I_{d})^{-1}\Sigma_{s}=\frac{G_{1,23}}{\lambda},$ (138) $$

with:

$$ $\displaystyle\frac{G_{1,23}}{\lambda}=\lambda^{-1}\bar{\operatorname{tr}}\,p_{2}\Sigma_{2}(\lambda\Sigma_{s}G_{0,15}+\lambda G_{0,27}I_{d}-p_{1}\Sigma_{1}G_{0,15}G_{5,24}+p_{1}\Sigma_{1}G_{0,27}G_{5,20})$ (139) $\displaystyle\cdot(p_{1}\Sigma_{1}G_{5,20}+p_{2}\Sigma_{2}G_{0,15}+\lambda I_{d})^{-2}.$ (140) $$

By identifying identical entries of $\overline{Q}^{-1}$, we must have that $\frac{G_{5,20}}{\lambda}=\frac{G_{6,21}}{\lambda}=\frac{G_{10,25}}{\lambda},\frac{G_{0,15}}{\lambda}=\frac{G_{2,17}}{\lambda}=\frac{G_{13,28}}{\lambda}$. For $G_{6,21}$ and $G_{2,17}$, we observe that:

$$ $\displaystyle G_{6,21}=-\frac{\lambda}{-\lambda+\phi G_{7,20}},\quad G_{7,20}=-\lambda\bar{\operatorname{tr}}\,\Sigma_{1}\left(p_{1}\Sigma_{1}G_{6,21}+p_{2}\Sigma_{2}G_{2,17}+\lambda I_{d}\right)^{-1}$ (141) $\displaystyle\implies G_{6,21}=\frac{1}{1+\phi\bar{\operatorname{tr}}\,\Sigma_{1}\left(p_{1}\Sigma_{1}G_{6,21}+p_{2}\Sigma_{2}G_{2,17}+\lambda I_{d}\right)^{-1}},$ (142) $\displaystyle G_{2,17}=-\frac{\lambda}{-\lambda+\phi G_{3,15}},\quad G_{3,15}=-\lambda\bar{\operatorname{tr}}\,\Sigma_{2}\left(p_{1}\Sigma_{1}G_{6,21}+p_{2}\Sigma_{2}G_{2,17}+\lambda I_{d}\right)^{-1}$ (143) $\displaystyle\implies G_{2,17}=\frac{1}{1+\phi\bar{\operatorname{tr}}\,\Sigma_{2}\left(p_{1}\Sigma_{1}G_{6,21}+p_{2}\Sigma_{2}G_{2,17}+\lambda I_{d}\right)^{-1}}.$ (144) $$

We define $\eta_{1}=\frac{G_{6,21}}{\lambda},\eta_{2}=\frac{G_{2,17}}{\lambda}$, with $\eta_{1}\geq 0,\eta_{2}\geq 0$. Therefore:

$$ $\displaystyle\eta_{s}$ $\displaystyle=\frac{1}{\lambda+\phi\bar{\operatorname{tr}}\,\Sigma_{s}K^{-1}},$ (145) $$

where $K=\eta_{1}p_{1}\Sigma_{1}+\eta_{2}p_{2}\Sigma_{2}+I_{d}$. Additionally, by identifying identical entries of $\overline{Q}^{-1}$, we must have that $G_{5,24}=G_{6,25},G_{0,27}=G_{2,28}$. We observe that:

$$ $\displaystyle G_{10,25}$ $\displaystyle=\frac{-\lambda}{-\lambda+\phi G_{11,24}},$ (146) $\displaystyle G_{6,25}$ $\displaystyle=\frac{\lambda\phi G_{7,24}}{(-\lambda+\phi G_{7,20})(-\lambda+\phi G_{11,24})}=\phi\lambda^{2}\eta_{1}^{2}\frac{G_{7,24}}{\lambda},$ (147) $\displaystyle\frac{G_{7,24}}{\lambda}$ $\displaystyle=\lambda^{-2}\bar{\operatorname{tr}}\,K^{-2}(p_{1}\Sigma_{1}G_{6,25}+p_{2}\Sigma_{2}G_{2,28}-\lambda\Sigma_{s})\Sigma_{1},$ (148) $\displaystyle\implies G_{6,25}$ $\displaystyle=\phi\eta_{1}^{2}\bar{\operatorname{tr}}\,K^{-2}(p_{1}\Sigma_{1}G_{6,25}+p_{2}\Sigma_{2}G_{2,28}-\lambda\Sigma_{s})\Sigma_{1},$ (149) $\displaystyle G_{13,28}$ $\displaystyle=\frac{-\lambda}{-\lambda+\phi G_{14,27}},$ (150) $\displaystyle G_{2,28}$ $\displaystyle=\frac{\lambda\phi G_{3,27}}{(-\lambda+\phi G_{3,15})(-\lambda+\phi G_{14,27})}=\phi\lambda^{2}\eta_{2}^{2}\frac{G_{3,27}}{\lambda},$ (151) $\displaystyle\frac{G_{3,27}}{\lambda}$ $\displaystyle=\lambda^{-2}\bar{\operatorname{tr}}\,K^{-2}(p_{1}\Sigma_{1}G_{6,25}+p_{2}\Sigma_{2}G_{2,28}-\lambda\Sigma_{s})\Sigma_{2},$ (152) $\displaystyle\implies G_{2,28}$ $\displaystyle=\phi\eta_{2}^{2}\bar{\operatorname{tr}}\,K^{-2}(p_{1}\Sigma_{1}G_{6,25}+p_{2}\Sigma_{2}G_{2,28}-\lambda\Sigma_{s})\Sigma_{2}.$ (153) $$

We now define $v^{(s)}_{1}=-G_{6,25},v^{(s)}_{2}=-G_{2,28}$, with $v^{(s)}_{1}\geq 0,v^{(s)}_{2}\geq 0$. Therefore, $v^{(s)}_{1},v^{(s)}_{2}$ obey the following system of equations:

$$ $\displaystyle v^{(s)}_{k}$ $\displaystyle=\phi\eta_{k}^{2}\bar{\operatorname{tr}}\,K^{-2}(v^{(s)}_{1}p_{1}\Sigma_{1}+v^{(s)}_{2}p_{2}\Sigma_{2}+\lambda\Sigma_{s})\Sigma_{k}.$ (154) $$

We further define $u^{(s)}_{k}=\frac{v^{(s)}_{k}}{\lambda}$. Putting all the pieces together:

$$ $\displaystyle\frac{G_{1,23}}{\lambda}=\lambda^{-1}\bar{\operatorname{tr}}\,p_{2}\Sigma_{2}\big{(}\eta_{2}\Sigma_{s}-u^{(s)}_{2}I_{d}+p_{1}\Sigma_{1}(\eta_{2}u^{(s)}_{1}-\eta_{1}u^{(s)}_{2})\big{)}K^{-2}.$ (155) $$

By symmetry, in conclusion:

$$ $\displaystyle V_{s}(\widehat{f})=V_{s}^{(1)}(\widehat{f})+V_{s}^{(2)}(\widehat{f}),$ (156) $\displaystyle V_{s}^{(k)}(\widehat{f})=\lambda^{-1}\phi\sigma_{k}^{2}\bar{\operatorname{tr}}\,p_{k}\Sigma_{k}\big{(}\eta_{k}\Sigma_{s}-u^{(s)}_{k}I_{d}+p_{k^{\prime}}\Sigma_{k^{\prime}}(\eta_{k}u^{(s)}_{k^{\prime}}-\eta_{k^{\prime}}u^{(s)}_{k})\big{)}K^{-2},$ (157) $$

with $k^{\prime}=\begin{cases}2,&k=1\\
1,&k=2\end{cases}$.

We now corroborate our result in the limit $p_{2}\to 1$ (i.e., $p_{1}\to 0$) and $s=2$. We observe that:

$$ $\displaystyle\phi\to\phi_{2}$ $\displaystyle,\lambda\to\lambda_{2},$ (158) $\displaystyle V_{2}^{(1)}(\widehat{f})$ $\displaystyle=0,$ (159) $\displaystyle\frac{V_{2}^{(2)}(\widehat{f})}{\lambda^{-1}\phi_{2}\sigma_{2}^{2}}$ $\displaystyle=\bar{\operatorname{tr}}\,\Sigma_{2}(\eta_{2}\Sigma_{2}-u_{2}^{(2)}I_{d})K^{-2}$ (160) $\displaystyle v_{2}^{(2)}$ $\displaystyle=\phi_{2}\eta_{2}^{2}\bar{\operatorname{tr}}\,K^{-2}(v_{2}^{(2)}\Sigma_{2}+\lambda_{2}\Sigma_{2})\Sigma_{2}$ (161) $\displaystyle=\phi_{2}(v_{2}^{(2)}+\lambda_{2})\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2}),$ (162) $\displaystyle u_{2}^{(2)}$ $\displaystyle=\frac{\phi_{2}\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})}{1-\phi_{2}\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})},$ (163) $\displaystyle\frac{V_{2}^{(2)}(\widehat{f})}{\lambda^{-1}\phi_{2}\sigma_{2}^{2}}$ $\displaystyle=\kappa_{2}\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})-u_{2}^{(2)}\bar{\operatorname{tr}}\,\Sigma_{2}(\eta_{2}\Sigma_{2}+I_{d})^{-2}$ (164) $\displaystyle=\kappa_{2}\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})-\kappa_{2}^{2}u_{2}^{(2)}\bar{\operatorname{tr}}\,\Sigma_{2}(\Sigma_{2}+\kappa_{2}I_{d})^{-2}$ (165) $\displaystyle=\kappa_{2}\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})-\kappa_{2}u_{2}^{(2)}(\bar{\operatorname{df}}_{1}^{(2)}(\kappa_{2})-\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2}))$ (166) $\displaystyle=\kappa_{2}(1+u_{2}^{(2)})\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})-\kappa_{2}u_{2}^{(2)}\bar{\operatorname{df}}_{1}^{(2)}(\kappa_{2})$ (167) $\displaystyle=\frac{\kappa_{2}-\kappa_{2}\phi_{2}\bar{\operatorname{df}}_{1}^{(2)}(\kappa_{2})}{1-\phi_{2}\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})}\cdot\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})$ (168) $\displaystyle=\frac{\lambda\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})}{1-\phi_{2}\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})},$ (169) $\displaystyle V_{2}^{(2)}(\widehat{f})$ $\displaystyle=\frac{\sigma_{2}^{2}\phi_{2}\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})}{1-\phi_{2}\bar{\operatorname{df}}_{2}^{(2)}(\kappa_{2})},$ (170) $$

which exactly recovers the result for $V_{2}(\widehat{f}_{2})$ as expected.

#### D.2 Bias Terms

Recall that:

$$ $\displaystyle B_{s}(\widehat{f})$ $\displaystyle=\mathbb{E}\,\|M^{-1}M_{s^{\prime}}w^{*}_{s^{\prime}}+M^{-1}M_{s}w^{*}_{s}-w^{*}_{s}\|_{\Sigma_{s}}^{2}.$ (171) $$

Now, observe that $M^{-1}M_{1}w^{*}_{1}-w^{*}_{1}=M^{-1}M_{1}w^{*}_{1}-M^{-1}Mw^{*}_{1}=-M^{-1}M_{2}w^{*}_{1}-n\lambda M^{-1}w_{1}^{*}$. Let $\delta=w^{*}_{2}-w^{*}_{1}$. Then:

$$ $\displaystyle B_{s}(\widehat{f})$ $\displaystyle=\mathbb{E}\|M^{-1}M_{s^{\prime}}(-1)^{s-1}\delta-n\lambda M^{-1}w_{s}^{*}\|^{2}_{\Sigma_{s}}$ (172) $\displaystyle=\mathbb{E}\operatorname{tr}\,\delta^{\top}M_{s^{\prime}}M^{-1}\Sigma_{s}M^{-1}M_{s^{\prime}}\delta$ (173) $\displaystyle-2(-1)^{s-1}n\lambda\mathbb{E}\operatorname{tr}\,\delta^{\top}M_{s^{\prime}}M^{-1}\Sigma_{s}M^{-1}w_{s}^{*}$ (174) $\displaystyle+n^{2}\lambda^{2}\mathbb{E}\operatorname{tr}\,(w_{s}^{*})^{\top}M^{-1}\Sigma_{s}M^{-1}w_{s}^{*}$ (175) $\displaystyle=B_{s}^{(1)}(\widehat{f})-2(-1)^{s-1}B_{s}^{(2)}(\widehat{f})+B_{s}^{(3)}(\widehat{f}),$ (176) $$

where:

$$ $\displaystyle B_{s}^{(1)}(\widehat{f})$ $\displaystyle=\mathbb{E}\bar{\operatorname{tr}}\,(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}(H_{s^{\prime}}/\lambda)\Delta(H_{s^{\prime}}/\lambda)(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Sigma_{s},$ (177) $\displaystyle B_{s}^{(2)}(\widehat{f})$ $\displaystyle=\mathbb{E}\operatorname{tr}\,\delta^{\top}(H_{s^{\prime}}/\lambda)(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Sigma_{s}(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}w_{s}^{*},$ (178) $\displaystyle B_{s}^{(3)}(\widehat{f})$ $\displaystyle=\mathbb{E}\bar{\operatorname{tr}}\,(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Theta_{s}(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Sigma_{s}.$ (179) $$

Because $\delta$ and $w_{1}^{*}$ are independent and sampled from zero-centered distributions:

$$ $\displaystyle B_{1}^{(2)}(\widehat{f})$ $\displaystyle=0,$ (180) $\displaystyle B_{2}^{(2)}(\widehat{f})$ $\displaystyle=\mathbb{E}\bar{\operatorname{tr}}\,(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Delta(H_{1}/\lambda)(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Sigma_{2}.$ (181) $$

WLOG, for $B_{s}^{(1)}$, we focus on the case $s=1$. The matrix of interest has a linear pencil representation given by (with zero-based indexing):

$$ $\displaystyle(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}(H_{2}/\lambda)\Delta(H_{2}/\lambda)(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Sigma_{1}=Q^{-1}_{1,16},$ (182) $$

where the linear pencil $Q$ is defined as follows:

$$ $Q=$ $\left(\begin{array}[]{ccccccccccccccccc}I_{d}&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ -\Delta&I_{d}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}^{\top}&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&I_{d}&-\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&\Sigma_{2}^{\frac{1}{2}}&0&0&I_{d}&\Sigma_{1}^{\frac{1}{2}}&0&0&-\Sigma_{1}&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}^{\top}&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&I_{d}&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&I_{d}&\Sigma_{1}^{\frac{1}{2}}&0&0&\Sigma_{2}^{\frac{1}{2}}&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}^{\top}&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&I_{d}&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}^{\top}&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}&0\\ 0&0&0&0&0&0&0&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&0&I_{d}&\Sigma_{2}^{\frac{1}{2}}\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{d}\\ \end{array}\right)$ . (183) $$

Using OVFPT, we deduce that, in the limit given by equation [9](#S3.E9), the following holds:

$$ $\displaystyle\mathbb{E}\bar{\operatorname{tr}}\,(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}(H_{2}/\lambda)\Delta(H_{2}/\lambda)(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Sigma_{1}=G_{1,33},$ (184) $$

with:

$$ $\displaystyle\begin{split}&G_{1,33}\\ &=\lambda^{-1}\bar{\operatorname{tr}}\,p_{2}\Sigma_{2}\Delta(p_{2}\Sigma_{2}G^{2}_{2,19}(\lambda-p_{1}G_{6,27})\Sigma_{1}-G_{2,30}(p_{1}\Sigma_{1}G_{6,23}+\lambda I_{d})^{2})\\ &\cdot(p_{1}\Sigma_{1}G_{6,23}+p_{2}\Sigma_{2}G_{2,19}+\lambda I_{d})^{-2}.\end{split}$ (185) $$

By identifying identical entries of $\overline{Q}^{-1}$, we must have that $\eta_{1}=\frac{G_{6,23}}{\lambda}=\frac{G_{7,24}}{\lambda}=\frac{G_{11,28}}{\lambda},\eta_{2}=\frac{G_{2,19}}{\lambda}=\frac{G_{3,20}}{\lambda}=\frac{G_{14,31}}{\lambda}$. For $G_{7,24}$ and $G_{3,20}$, we observe that:

$$ $\displaystyle G_{7,24}=-\frac{\lambda}{-\lambda+\phi G_{8,23}},\quad G_{8,23}=-\lambda\bar{\operatorname{tr}}\,\Sigma_{1}\left(p_{1}\Sigma_{1}G_{7,24}+p_{2}\Sigma_{2}G_{3,20}+\lambda I_{d}\right)^{-1}$ (186) $\displaystyle\implies G_{7,24}=\frac{1}{1+\phi\bar{\operatorname{tr}}\,\Sigma_{1}\left(p_{1}\Sigma_{1}G_{7,24}+p_{2}\Sigma_{2}G_{3,20}+\lambda I_{d}\right)^{-1}},$ (187) $\displaystyle G_{3,20}=-\frac{\lambda}{-\lambda+\phi G_{4,19}},\quad G_{4,19}=-\lambda\bar{\operatorname{tr}}\,\Sigma_{2}\left(p_{1}\Sigma_{1}G_{7,24}+p_{2}\Sigma_{2}G_{3,20}+\lambda I_{d}\right)^{-1}$ (188) $\displaystyle\implies G_{3,20}=\frac{1}{1+\phi\bar{\operatorname{tr}}\,\Sigma_{2}\left(p_{1}\Sigma_{1}G_{7,24}+p_{2}\Sigma_{2}G_{3,20}+\lambda I_{d}\right)^{-1}}.$ (189) $$

By again identifying identical entries of $\overline{Q}^{-1}$, we further have that $v^{(1)}_{1}=-G_{6,27}=-G_{7,28},v^{(1)}_{2}=-G_{2,30}=-G_{3,31}$. We observe that:

$$ $\displaystyle G_{7,28}$ $\displaystyle=\phi\lambda^{2}\eta_{1}^{2}\frac{G_{8,27}}{\lambda},$ (190) $\displaystyle\frac{G_{8,27}}{\lambda}$ $\displaystyle=\lambda^{-2}\bar{\operatorname{tr}}\,K^{-2}(p_{1}\Sigma_{1}G_{7,28}+p_{2}\Sigma_{2}G_{3,31}-\lambda\Sigma_{1})\Sigma_{1}$ (191) $\displaystyle\implies v^{(1)}_{1}$ $\displaystyle=\phi\eta_{1}^{2}\bar{\operatorname{tr}}\,K^{-2}(v^{(s)}_{1}p_{1}\Sigma_{1}+v^{(s)}_{2}p_{2}\Sigma_{2}+\lambda\Sigma_{1})\Sigma_{1},$ (192) $\displaystyle G_{3,31}$ $\displaystyle=\phi\lambda^{2}\eta_{2}^{2}\frac{G_{4,30}}{\lambda},$ (193) $\displaystyle\frac{G_{4,30}}{\lambda}$ $\displaystyle=\lambda^{-2}\bar{\operatorname{tr}}\,K^{-2}(p_{1}\Sigma_{1}G_{7,28}+p_{2}\Sigma_{2}G_{3,31}-\lambda\Sigma_{1})\Sigma_{2},$ (194) $\displaystyle\implies v^{(1)}_{2}$ $\displaystyle=\phi\eta_{2}^{2}\bar{\operatorname{tr}}\,K^{-2}(v^{(s)}_{1}p_{1}\Sigma_{1}+v^{(s)}_{2}p_{2}\Sigma_{2}+\lambda\Sigma_{1})\Sigma_{2}.$ (195) $$

Putting all the pieces together:

$$ $\displaystyle B_{1}^{(1)}(\widehat{f})$ $\displaystyle=\bar{\operatorname{tr}}\,p_{2}\Sigma_{2}\Delta(p_{2}\eta_{2}^{2}\Sigma_{2}(1+p_{1}u^{(s)}_{1})\Sigma_{1}+u^{(s)}_{2}(p_{1}\eta_{1}\Sigma_{1}+I_{d})^{2})K^{-2}.$ (196) $$

In conclusion:

$$ $\displaystyle B_{s}^{(1)}(\widehat{f})$ $\displaystyle=\bar{\operatorname{tr}}\,p_{s^{\prime}}\Sigma_{s^{\prime}}\Delta(p_{s^{\prime}}\eta_{s^{\prime}}^{2}\Sigma_{s^{\prime}}(1+p_{s}u^{(s)}_{s})\Sigma_{s}+u^{(s)}_{s^{\prime}}(p_{s}\eta_{s}\Sigma_{s}+I_{d})^{2})K^{-2}.$ (197) $$

Now, switching focus to $B_{2}^{(2)}(\widehat{f})$, the matrix of interest has a linear pencil representation given by (with zero-based indexing):

$$ $\displaystyle(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Delta(H_{1}/\lambda)(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Sigma_{2}=Q^{-1}_{0,15},$ (198) $$

where the linear pencil $Q$ is defined as follows:

$$ $Q=$ $\left(\begin{array}[]{cccccccccccccccc}I_{d}&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&0&0&0&0&0&0&0&0&0\\ -\Delta&I_{d}&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}^{\top}&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&I_{d}&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&0&0&0&0&0&0\\ 0&0&\Sigma_{1}^{\frac{1}{2}}&0&0&I_{d}&\Sigma_{2}^{\frac{1}{2}}&0&0&-\Sigma_{2}&0&0&0&0&0&0\\ 0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}^{\top}&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}&0&0&0&0&0&0&0\\ 0&0&0&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&I_{d}&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&I_{d}&\Sigma_{1}^{\frac{1}{2}}&0&0&\Sigma_{2}^{\frac{1}{2}}&0&0\\ 0&0&0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}^{\top}&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}&0&0&0\\ 0&0&0&0&0&0&0&0&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&I_{d}&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}^{\top}&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}\\ 0&0&0&0&0&0&0&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&0&I_{d}\\ \end{array}\right)$ . (199) $$

Like before, the following holds:

$$ $\displaystyle\mathbb{E}\bar{\operatorname{tr}}\,(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Delta(H_{1}/\lambda)(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Sigma_{2}=G_{1,25},$ (200) $$

with:

$$ $\displaystyle\begin{split}&G_{1,25}\\ &=\bar{\operatorname{tr}}\,p_{1}\Sigma_{1}\Delta(\lambda\Sigma_{2}G_{2,18}+\lambda G_{2,26}I_{d}-p_{2}\Sigma_{2}G_{2,18}G_{6,29}+p_{2}\Sigma_{2}G_{2,26}G_{6,22})\\ &\cdot(p_{1}\Sigma_{1}G_{2,18}+p_{2}\Sigma_{2}G_{6,22}+\lambda I_{d})^{-2}\end{split}$ (201) $$

By identifying identical entries of $\overline{Q}^{-1}$, we must have that $\eta_{1}=\frac{G_{2,18}}{\lambda}=\frac{G_{3,19}}{\lambda}=\frac{G_{11,27}}{\lambda},\eta_{2}=\frac{G_{6,22}}{\lambda}=\frac{G_{7,23}}{\lambda}=\frac{G_{14,30}}{\lambda}$. For $G_{3,19}$ and $G_{7,23}$, we observe that:

$$ $\displaystyle G_{3,19}=-\frac{\lambda}{-\lambda+\phi G_{4,18}},\quad G_{4,18}=-\lambda\bar{\operatorname{tr}}\,\Sigma_{1}\left(p_{1}\Sigma_{1}G_{3,19}+p_{2}\Sigma_{2}G_{7,23}+\lambda I_{d}\right)^{-1}$ (202) $\displaystyle\implies G_{3,19}=\frac{1}{1+\phi\bar{\operatorname{tr}}\,\Sigma_{1}\left(p_{1}\Sigma_{1}G_{3,19}+p_{2}\Sigma_{2}G_{7,23}+\lambda I_{d}\right)^{-1}},$ (203) $\displaystyle G_{7,23}=-\frac{\lambda}{-\lambda+\phi G_{8,22}},\quad G_{8,22}=-\lambda\bar{\operatorname{tr}}\,\Sigma_{2}\left(p_{1}\Sigma_{1}G_{3,19}+p_{2}\Sigma_{2}G_{7,23}+\lambda I_{d}\right)^{-1}$ (204) $\displaystyle\implies G_{7,23}=\frac{1}{1+\phi\bar{\operatorname{tr}}\,\Sigma_{2}\left(p_{1}\Sigma_{1}G_{3,19}+p_{2}\Sigma_{2}G_{7,23}+\lambda I_{d}\right)^{-1}}.$ (205) $$

By again identifying identical entries of $\overline{Q}^{-1}$, we further have that $v^{(2)}_{1}=-G_{2,26}=-G_{3,27},v^{(2)}_{2}=-G_{6,29}=-G_{7,30}$. We observe that:

$$ $\displaystyle G_{3,27}$ $\displaystyle=\phi\lambda^{2}\eta_{1}^{2}\frac{G_{4,26}}{\lambda},$ (206) $\displaystyle\frac{G_{4,26}}{\lambda}$ $\displaystyle=\lambda^{-2}\bar{\operatorname{tr}}\,K^{-2}(p_{1}\Sigma_{1}G_{3,27}+p_{2}\Sigma_{2}G_{7,30}-\lambda\Sigma_{2})\Sigma_{1},$ (207) $\displaystyle\implies v^{(2)}_{1}$ $\displaystyle=\phi\eta_{1}^{2}\bar{\operatorname{tr}}\,K^{-2}(v^{(2)}_{1}p_{1}\Sigma_{1}+v^{(2)}_{2}p_{2}\Sigma_{2}+\lambda\Sigma_{2})\Sigma_{1},$ (208) $\displaystyle G_{7,30}$ $\displaystyle=\phi\lambda^{2}\eta_{2}^{2}\frac{G_{8,29}}{\lambda},$ (209) $\displaystyle\frac{G_{8,29}}{\lambda}$ $\displaystyle=\lambda^{-2}\bar{\operatorname{tr}}\,K^{-2}(p_{1}\Sigma_{1}G_{3,27}+p_{2}\Sigma_{2}G_{7,30}-\lambda\Sigma_{2})\Sigma_{2},$ (210) $\displaystyle\implies v^{(2)}_{2}$ $\displaystyle=\phi\eta_{2}^{2}\bar{\operatorname{tr}}\,K^{-2}(v^{(2)}_{1}p_{1}\Sigma_{1}+v^{(2)}_{2}p_{2}\Sigma_{2}+\lambda\Sigma_{2})\Sigma_{2}.$ (211) $$

Putting all the pieces together:

$$ $\displaystyle B_{2}^{(1)}(\widehat{f})$ $\displaystyle=0,$ (212) $\displaystyle B_{2}^{(2)}(\widehat{f})$ $\displaystyle=\bar{\operatorname{tr}}\,p_{1}\Sigma_{1}\Delta\big{(}\eta_{1}\Sigma_{2}-u_{1}^{(2)}I_{d}+p_{2}\Sigma_{2}(\eta_{1}u_{2}^{(2)}-\eta_{2}u_{1}^{(2)})\big{)}K^{-2}.$ (213) $$

Finally, switching focus to $B_{1}^{(3)}(\widehat{f})$, the matrix of interest has a linear pencil representation given by (with zero-based indexing):

$$ $\displaystyle(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Theta(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Sigma_{1}=Q^{-1}_{1,8},$ (214) $$

where the linear pencil $Q$ is defined as follows:

$$ $Q=$ $\left(\begin{array}[]{ccccccccccccccc}I_{d}&0&\Sigma_{1}^{\frac{1}{2}}&0&0&\Sigma_{2}^{\frac{1}{2}}&0&0&-\Sigma_{1}&0&0&0&0&0&0\\ -\Theta&I_{d}&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}^{\top}&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}&0&0&0&0&0&0&0&0&0&0\\ -\Sigma_{1}^{\frac{1}{2}}&0&0&0&I_{d}&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}^{\top}&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}&0&0&0&0&0&0&0\\ -\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&0&0&I_{d}&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&I_{d}&\Sigma_{1}^{\frac{1}{2}}&0&0&\Sigma_{2}^{\frac{1}{2}}&0&0\\ 0&0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}^{\top}&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{1}&0&0&0\\ 0&0&0&0&0&0&0&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&I_{d}&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}^{\top}&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}\sqrt{n}}Z_{2}\\ 0&0&0&0&0&0&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&0&I_{d}\\ \end{array}\right)$ . (215) $$

The following holds:

$$ $\displaystyle(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Theta(H_{1}/\lambda+H_{2}/\lambda+I_{d})^{-1}\Sigma_{1}=G_{1,23},$ (216) $$

with $G_{1,23}=\lambda\bar{\operatorname{tr}}\,\Theta(-p_{1}\Sigma_{1}G_{2,24}-p_{2}\Sigma_{2}G_{5,27}+\lambda\Sigma_{1})(p_{1}\Sigma_{1}G_{2,17}+p_{2}\Sigma_{2}G_{5,20}+\lambda I_{d})^{-2}$.

By identifying identical entries of $\overline{Q}^{-1}$ and following similar steps as before, we must have the identification $\eta_{1}=\frac{G_{2,17}}{\lambda},\eta_{2}=\frac{G_{5,20}}{\lambda}$, as well as $v^{(1)}_{1}=-G_{2,24},v^{(1)}_{2}=-G_{5,27}$. Therefore, in conclusion:

$$ $\displaystyle B_{s}^{(3)}(\widehat{f})=\bar{\operatorname{tr}}\,\Theta_{s}(p_{1}u_{1}^{(s)}\Sigma_{1}+p_{2}u_{2}^{(s)}\Sigma_{2}+\Sigma_{s})K^{-2}.$ (217) $$

In the limit $p_{s}\to 1$ (i.e., $p_{s^{\prime}}\to 0$), we observe that:

$$ $\displaystyle\phi$ $\displaystyle\to\phi_{s},\lambda\to\lambda_{s},$ (218) $\displaystyle B_{s}^{(1)}(\widehat{f})$ $\displaystyle\to 0,$ (219) $\displaystyle B_{s}^{(2)}(\widehat{f})$ $\displaystyle\to 0,$ (220) $\displaystyle B_{s}^{(3)}(\widehat{f})$ $\displaystyle=\bar{\operatorname{tr}}\,\Theta_{s}(u_{s}^{(s)}+1)\Sigma_{s}K^{-2},$ (221) $\displaystyle v_{s}^{(s)}$ $\displaystyle=\phi_{s}\eta_{s}^{2}\bar{\operatorname{tr}}\,K^{-2}(v_{s}^{(s)}+\lambda_{s})\Sigma_{s}^{2}$ (222) $\displaystyle=\phi_{s}(v_{s}^{(s)}+\lambda_{s})\bar{\operatorname{df}}_{2}^{(s)}(\kappa_{s})$ (223) $\displaystyle u_{s}^{(s)}$ $\displaystyle=\frac{\phi_{s}\bar{\operatorname{df}}_{2}^{(s)}(\kappa_{s})}{1-\phi_{s}\bar{\operatorname{df}}_{2}^{(s)}(\kappa_{s})},$ (224) $\displaystyle B_{s}^{(3)}(\widehat{f})$ $\displaystyle=\frac{\kappa_{s}^{2}\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\Sigma_{s}+\kappa_{s}I_{d})^{-2}}{1-\phi_{s}\bar{\operatorname{df}}_{2}^{(s)}(\kappa_{s})},$ (225) $\displaystyle B_{s}(\widehat{f})$ $\displaystyle\to B_{s}^{(3)}(\widehat{f}),$ (226) $$

which matches up exactly with $B_{s}(\widehat{f}_{s})$ as expected.
∎

### Appendix E Proof of Theorem 2

###### Proof.

The gradient of the loss $L$ is given by:

$$ $\displaystyle\nabla L(\eta)$ $\displaystyle=\sum_{s}S^{\top}X_{s}^{\top}(X_{s}S\eta-Y_{s})/n+\lambda\eta=\sum_{s}S^{\top}M_{s}S\eta-\sum_{s}S^{\top}X_{s}^{\top}Y_{s}/n+\lambda\eta$ $\displaystyle=H\eta-\sum_{s}S^{\top}X_{s}^{\top}Y_{s}/n,$ $$

where $H=S^{\top}MS+\lambda I_{m}\in\mathbb{R}^{m\times m}$, with $M=M_{1}+M_{2}$ and $M_{s}=X_{s}^{\top}X_{s}/n$.
Thus, setting $R=H^{-1}$, we may write:

$$ $\displaystyle\widehat{w}$ $\displaystyle=S\widehat{\eta}=SRS^{\top}(X_{1}^{\top}Y_{1}+X_{2}^{\top}Y_{2})/n$ $\displaystyle=SRS^{\top}(M_{1}w^{*}_{1}+M_{2}w^{*}_{2})+SRS^{\top}X_{1}^{\top}E_{1}/n+SRS^{\top}X_{2}^{\top}E_{2}/n.$ $$

We deduce the following bias-variance decomposition:

$$ $\displaystyle\mathbb{E}\|\widehat{w}-w^{*}_{s}\|_{\Sigma_{s}}^{2}$ $\displaystyle=B_{s}(\widehat{f})+V_{s}(\widehat{f}),\text{ where }$ $\displaystyle V_{s}(\widehat{f})$ $\displaystyle=V^{(1)}_{s}(\widehat{f})+V^{(2)}_{s}(\widehat{f}),\text{ with }V^{(j)}_{s}(\widehat{f})=\sigma_{j}^{2}\phi\mathbb{E}\bar{\operatorname{tr}}\,M_{j}SRS^{\top}\Sigma_{s}SRS^{\top},$ $\displaystyle B_{s}(\widehat{f})$ $\displaystyle=\mathbb{E}\|SRS^{\top}(M_{1}w^{*}_{1}+M_{2}w^{*}_{2})-w^{*}_{s}\|_{\Sigma_{s}}^{2}.$ $$

We can further decompose $B_{s}(\widehat{f})$, first considering the case $s=1$. We define $\delta=w_{2}^{*}-w_{1}^{*}$.

$$ $\displaystyle\mathbb{E}\|SRS^{\top}(M_{1}w^{*}_{1}+M_{2}w^{*}_{2})-w^{*}_{1}\|_{\Sigma_{1}}^{2}$ $\displaystyle=\mathbb{E}\|(SRS^{\top}(M_{1}+M_{2})-I_{d})w^{*}_{1}+SRS^{\top}M_{2}\delta\|_{\Sigma_{1}}^{2}$ $\displaystyle=\mathbb{E}\|(SRS^{\top}M-I_{d})w_{1}^{*}\|_{\Sigma_{1}}^{2}+\mathbb{E}\|SRS^{\top}M_{2}\delta\|_{\Sigma_{1}}^{2}$ $\displaystyle=\mathbb{E}\bar{\operatorname{tr}}\,\Theta(MSRS^{\top}-I_{d})\Sigma_{1}(SRS^{\top}M-I_{d})+\mathbb{E}\bar{\operatorname{tr}}\,\Delta M_{2}SRS^{\top}\Sigma_{1}SRS^{\top}M_{2}$ $\displaystyle=\mathbb{E}\bar{\operatorname{tr}}\,\Theta\Sigma_{1}+\mathbb{E}\bar{\operatorname{tr}}\,\Theta MSRS^{\top}\Sigma_{1}SRS^{\top}M-2\mathbb{E}\bar{\operatorname{tr}}\,\Theta\Sigma_{1}SRS^{\top}M+\mathbb{E}\bar{\operatorname{tr}}\,\Delta M_{2}SRS^{\top}\Sigma_{1}SRS^{\top}M_{2}.$ $$

We can similarly decompose $B_{2}$:

$$ $\displaystyle\mathbb{E}\|SRS^{\top}(M_{1}w^{*}_{1}+M_{2}w^{*}_{2})-w^{*}_{2}\|_{\Sigma_{2}}^{2}$ $\displaystyle=\mathbb{E}\|SRS^{\top}(M_{1}w^{*}_{1}+M_{2}w^{*}_{2})-w^{*}_{2}\|_{\Sigma_{2}}^{2}$ $\displaystyle=\mathbb{E}\|(SRS^{\top}(M_{1}+M_{2})-I_{d})w^{*}_{2}-SRS^{\top}M_{1}\delta\|_{\Sigma_{2}}^{2}$ $\displaystyle=\mathbb{E}\|(SRS^{\top}M-I_{d})w_{2}^{*}\|_{\Sigma_{2}}^{2}+\mathbb{E}\|SRS^{\top}M_{1}\delta\|_{\Sigma_{2}}^{2}-2\mathbb{E}\operatorname{tr}\,(w_{2}^{*})^{\top}(MSRS^{\top}-I_{d})\Sigma_{2}SRS^{\top}M_{1}\delta$ $\displaystyle=\mathbb{E}\bar{\operatorname{tr}}\,\Theta_{2}(MSRS^{\top}-I_{d})\Sigma_{2}(SRS^{\top}M-I_{d})+\mathbb{E}\bar{\operatorname{tr}}\,\Delta M_{1}SRS^{\top}\Sigma_{2}SRS^{\top}M_{1}$ $\displaystyle-2\mathbb{E}\bar{\operatorname{tr}}\,\Delta(MSRS^{\top}-I_{d})\Sigma_{2}SRS^{\top}M_{1}$ $\displaystyle=\mathbb{E}\bar{\operatorname{tr}}\,\Theta_{2}\Sigma_{2}+\mathbb{E}\bar{\operatorname{tr}}\,\Theta_{2}MSRS^{\top}\Sigma_{2}SRS^{\top}M-2\mathbb{E}\bar{\operatorname{tr}}\,\Theta_{2}\Sigma_{2}SRS^{\top}M$ $\displaystyle+\mathbb{E}\bar{\operatorname{tr}}\,\Delta M_{1}SRS^{\top}\Sigma_{2}SRS^{\top}M_{1}-2\mathbb{E}\bar{\operatorname{tr}}\,\Delta MSRS^{\top}\Sigma_{2}SRS^{\top}M_{1}+2\mathbb{E}\bar{\operatorname{tr}}\,\Delta\Sigma_{2}SRS^{\top}M_{1}.$ $$

Furthermore, we observe that:

$$ $\displaystyle\mathbb{E}\bar{\operatorname{tr}}\,AMSRS^{\top}BSRS^{\top}M$ (227) $\displaystyle=\mathbb{E}\bar{\operatorname{tr}}\,AM_{1}SRS^{\top}BSRS^{\top}M_{1}+\mathbb{E}\bar{\operatorname{tr}}\,AM_{2}SRS^{\top}BSRS^{\top}M_{2}+2\mathbb{E}\bar{\operatorname{tr}}\,AM_{1}SRS^{\top}BSRS^{\top}M_{2},$ (228) $\displaystyle\mathbb{E}\bar{\operatorname{tr}}\,ASRS^{\top}M=\mathbb{E}\bar{\operatorname{tr}}\,ASRS^{\top}M_{1}+\mathbb{E}\bar{\operatorname{tr}}\,ASRS^{\top}M_{2}.$ (229) $$

Hence, we desire deterministic equivalents for the following expressions:

$$ $\displaystyle r_{j}^{(1)}(A)$ $\displaystyle=AS\overline{R}S^{\top}\overline{M}_{j},$ (230) $\displaystyle r_{j}^{(2)}(A,B)$ $\displaystyle=A\overline{M}_{j}S\overline{R}S^{\top}BS\overline{R}S^{\top},$ (231) $\displaystyle r_{j}^{(3)}(A,B)$ $\displaystyle=A\overline{M}_{j}S\overline{R}S^{\top}BS\overline{R}S^{\top}\overline{M}_{j},$ (232) $\displaystyle r_{j}^{(4)}(A,B)$ $\displaystyle=A\overline{M}_{j}S\overline{R}S^{\top}BS\overline{R}S^{\top}\overline{M}_{j^{\prime}},$ (233) $$

where:

$$ $\displaystyle\overline{M}_{j}=\Sigma_{j}^{1/2}Z^{\top}_{j}Z_{j}\Sigma_{j}^{1/2},\overline{R}=(S^{\top}\overline{M}S+I_{m})^{-1},\overline{M}=\overline{M}_{1}+\overline{M}_{2},$ (234) $\displaystyle\overline{M}_{j}=M_{j}/\lambda,\overline{R}=\lambda R,\overline{M}=M/\lambda.$ (235) $$

In summary:

$$ $\displaystyle V_{s}^{(j)}(\widehat{f})$ $\displaystyle=\sigma_{j}^{2}\phi\lambda^{-1}\mathbb{E}\bar{\operatorname{tr}}\,r_{j}^{(2)}(I_{d},\Sigma_{s}),$ (236) $\displaystyle B_{s}(\widehat{f})$ $\displaystyle=\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}$ (237) $\displaystyle+\mathbb{E}\bar{\operatorname{tr}}\,r_{1}^{(3)}(\Theta_{s},\Sigma_{s})+\mathbb{E}\bar{\operatorname{tr}}\,r_{2}^{(3)}(\Theta_{s},\Sigma_{s})+2\mathbb{E}\bar{\operatorname{tr}}\,r_{1}^{(4)}(\Theta_{s},\Sigma_{s})$ (238) $\displaystyle-2\mathbb{E}\bar{\operatorname{tr}}\,r_{1}^{(1)}(\Theta_{s}\Sigma_{s})-2\mathbb{E}\bar{\operatorname{tr}}\,r_{2}^{(1)}(\Theta_{s}\Sigma_{s})$ (239) $\displaystyle+\mathbb{E}\bar{\operatorname{tr}}\,r_{s^{\prime}}^{(3)}(\Delta,\Sigma_{s})$ (240) $\displaystyle-2\begin{cases}0,&s=1,\\ \mathbb{E}\bar{\operatorname{tr}}\,r_{1}^{(3)}(\Delta,\Sigma_{2})+\mathbb{E}\bar{\operatorname{tr}}\,r_{2}^{(4)}(\Delta,\Sigma_{2})-\mathbb{E}\bar{\operatorname{tr}}\,r_{1}^{(1)}(\Delta\Sigma_{2}),&s=2\end{cases}.$ (241) $$

###### Proof.

#### E.1 Computing 𝔼 ​ tr ¯ ​ r j ( 1 ) 𝔼 ¯ tr superscript subscript 𝑟 𝑗 1 \mathbb{E}\bar{\operatorname{tr}}\,r_{j}^{(1)}

WLOG, we focus on $r_{1}^{(1)}$. The matrix of interest has a linear pencil representation given by (with zero-based indexing):

$$ $\displaystyle r_{1}^{(1)}=Q_{1,10}^{-1},$ (242) $$

where the linear pencil $Q$ is defined as follows:

$$ $Q=\leavevmode\resizebox{346.89731pt}{}{$\left(\begin{array}[]{ccccccccccc}I_{d}&0&-S&0&0&0&0&0&0&0&0\\ -A&I_{d}&0&0&0&0&0&0&0&0&0\\ 0&0&I_{m}&S^{\top}&0&0&0&0&0&0&0\\ 0&0&0&I_{d}&-\Sigma_{1}^{\frac{1}{2}}&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&0\\ 0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}}Z_{1}^{\top}&0&0&0&0&0\\ 0&0&0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}}Z_{1}&0&0&0&0\\ -\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&0&I_{d}&0&0&0&\Sigma_{1}^{\frac{1}{2}}\\ 0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}}Z_{2}^{\top}&0&0\\ 0&0&0&0&0&0&0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}}Z_{2}&0\\ -\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&0&0&0&0&I_{d}&0\\ 0&0&0&0&0&0&0&0&0&0&I_{d}\\ \end{array}\right)$}.$ (243) $$

Using the tools of OVFPT, the following holds:

$$ $\displaystyle\mathbb{E}\bar{\operatorname{tr}}\,r_{1}^{(1)}=G_{1,21},$ (244) $$

with:

$$ $\displaystyle G_{1,21}=\bar{\operatorname{tr}}\,\gamma p_{1}\Sigma_{1}AG_{2,13}G_{5,16}(\gamma G_{2,13}(p_{1}\Sigma_{1}G_{5,16}+p_{2}G_{8,19})+\lambda I_{d})^{-1}.$ (245) $$

For $G_{5,16}$ and $G_{8,19}$, we observe that:

$$ $\displaystyle G_{5,16}=\frac{-\lambda}{-\lambda+\phi G_{6,15}},\quad G_{6,15}=-\lambda\gamma G_{2,13}\bar{\operatorname{tr}}\,\Sigma_{1}(\gamma G_{2,13}(p_{1}\Sigma_{1}G_{5,16}+p_{2}\Sigma_{2}G_{8,19})+\lambda I_{d})^{-1},$ (246) $\displaystyle\implies G_{5,16}=\frac{1}{1+\psi G_{2,13}\bar{\operatorname{tr}}\,\Sigma_{1}(\gamma G_{2,13}(p_{1}\Sigma_{1}G_{5,16}+p_{2}\Sigma_{2}G_{8,19})+\lambda I_{d})^{-1}},$ (247) $\displaystyle G_{8,19}=\frac{-\lambda}{-\lambda+\phi G_{9,18}},\quad G_{9,18}=-\lambda\gamma G_{2,13}\bar{\operatorname{tr}}\,\Sigma_{2}(\gamma G_{2,13}(p_{1}\Sigma_{1}G_{5,16}+p_{2}\Sigma_{2}G_{8,19})+\lambda I_{d})^{-1},$ (248) $\displaystyle G_{8,19}=\frac{1}{1+\psi G_{2,13}\bar{\operatorname{tr}}\,\Sigma_{2}(\gamma G_{2,13}(p_{1}\Sigma_{1}G_{5,16}+p_{2}\Sigma_{2}G_{8,19})+\lambda I_{d})^{-1}}.$ (249) $$

We define $e_{1}=G_{5,16},e_{2}=G_{8,19}$, with $e_{1}\geq 0,e_{2}\geq 0$. We further observe that:

$$ $\displaystyle G_{2,13}$ $\displaystyle=\frac{1}{1+G_{3,11}},$ (250) $\displaystyle G_{3,11}$ $\displaystyle=\bar{\operatorname{tr}}\,(p_{1}\Sigma_{1}G_{5,16}+p_{2}\Sigma_{2}G_{8,19})(\gamma G_{2,13}(p_{1}\Sigma_{1}G_{5,16}+p_{2}\Sigma_{2}G_{8,19})+\lambda I_{d})^{-1}.$ (251) $$

We define $\tau=G_{2,13}\geq 0$. We further define $L=p_{1}e_{1}\Sigma_{1}+p_{2}e_{2}\Sigma_{2},K=\gamma\tau L+\lambda I_{d}$. Therefore, we have the following system of equations:

$$ $\displaystyle e_{s}$ $\displaystyle=\frac{1}{1+\psi\tau\bar{\operatorname{tr}}\,\Sigma_{s}K^{-1}},\tau=\frac{1}{1+\bar{\operatorname{tr}}\,LK^{-1}}.$ (252) $$

In conclusion:

$$ $\displaystyle\mathbb{E}\bar{\operatorname{tr}}\,r_{j}^{(1)}=p_{j}\gamma e_{j}\tau\bar{\operatorname{tr}}\,A\Sigma_{j}K^{-1}.$ (253) $$

#### E.2 Computing 𝔼 ​ tr ¯ ​ r j ( 2 ) 𝔼 ¯ tr superscript subscript 𝑟 𝑗 2 \mathbb{E}\bar{\operatorname{tr}}\,r_{j}^{(2)}

WLOG, we focus on $r_{1}^{(2)}$. The matrix of interest has a linear pencil representation given by (with zero-based indexing):

$$ $\displaystyle r_{1}^{(2)}=-Q^{-1}_{1,13},$ (254) $$

where the linear pencil $Q$ is defined as follows:

$$ $\rotatebox{270.0}{$Q=$ \leavevmode\resizebox{433.62pt}{}{$\left(\begin{array}[]{cccccccccccccccccccc}I_{d}&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ -A&I_{d}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&I_{d}&-\frac{1}{\sqrt{\lambda}}Z_{1}^{\top}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}}Z_{1}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&I_{d}&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&I_{d}&-S&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&I_{m}&S^{\top}&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&I_{d}&-\Sigma_{2}^{\frac{1}{2}}&0&0&B&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}}Z_{2}^{\top}&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}}Z_{2}&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&I_{d}&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&I_{d}&-S&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&I_{m}&S^{\top}&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&I_{d}&-\Sigma_{1}^{\frac{1}{2}}&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}}Z_{1}^{\top}&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}}Z_{1}&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&I_{d}&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}}Z_{2}^{\top}&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}}Z_{2}\\ 0&0&0&0&0&0&0&0&0&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&0&0&0&I_{d}\\ \end{array}\right)$}}.$ (255) $$

The following holds:

$$ $\displaystyle\mathbb{E}\bar{\operatorname{tr}}\,r_{1}^{(2)}=-G_{1,33},$ (256) $$

with:

$$ $\displaystyle G_{1,33}$ $\displaystyle=-p_{1}\bar{\operatorname{tr}}\,A\Sigma_{1}P_{1}P^{-1}_{2},$ (257) $\displaystyle P_{1}$ $\displaystyle=\gamma\lambda BG_{3,23}G_{6,26}G_{12,32}-\gamma p_{2}\Sigma_{2}G_{3,23}G_{6,26}G_{9,38}G_{12,32}$ (258) $\displaystyle+\gamma p_{2}\Sigma_{2}G_{3,35}G_{6,26}G_{9,29}G_{12,32}+\lambda G_{3,23}G_{6,32}I_{d}+\lambda G_{3,15}G_{12,32}I_{d},$ (259) $\displaystyle P_{2}$ $\displaystyle=(\gamma G_{6,26}(p_{1}\Sigma_{1}G_{3,23}+\gamma p_{2}\Sigma_{2}G_{9,29})+\lambda I_{d})$ (260) $\displaystyle\cdot(\gamma G_{12,32}(p_{1}\Sigma_{1}G_{15,35}+p_{2}\Sigma_{2}G_{18,38})+\lambda I_{d}).$ (261) $$

Following similar steps as before and recognizing identifications, we arrive at that:

$$ $\displaystyle e_{1}$ $\displaystyle=G_{3,23}=G_{15,35},$ (262) $\displaystyle e_{2}$ $\displaystyle=G_{9,29}=G_{18,38},$ (263) $\displaystyle\tau$ $\displaystyle=G_{6,26}=G_{12,32}.$ (264) $$

We now focus on the remaining terms. We observe that:

$$ $\displaystyle G_{3,35}$ $\displaystyle=\phi e_{1}^{2}\frac{G_{4,14}}{\lambda}$ (265) $\displaystyle\frac{G_{4,14}}{\lambda}$ $\displaystyle=\gamma\bar{\operatorname{tr}}\,\Sigma_{1}(\gamma\tau^{2}(p_{1}\Sigma_{1}G_{3,35}+p_{2}\Sigma_{2}G_{9,38}-\lambda B)-\lambda G_{6,32}I_{d})K^{-2},$ (266) $\displaystyle G_{9,38}$ $\displaystyle=\phi e_{2}^{2}\frac{G_{10,37}}{\lambda},$ (267) $\displaystyle\frac{G_{10,37}}{\lambda}$ $\displaystyle=\gamma\bar{\operatorname{tr}}\,\Sigma_{2}(\gamma\tau^{2}(p_{1}\Sigma_{1}G_{3,35}+p_{2}\Sigma_{2}G_{9,38}-\lambda B)-\lambda G_{6,32}I_{d})K^{-2}.$ (268) $$

We define $u_{1}=-\frac{G_{3,35}}{\lambda},u_{2}=-\frac{G_{9,38}}{\lambda}$, with $u_{1}\leq 0,u_{2}\leq 0$. We further define $D=p_{1}u_{1}\Sigma_{1}+p_{2}u_{2}\Sigma_{2}+B$. We now observe that:

$$ $\displaystyle G_{6,32}$ $\displaystyle=-\frac{G_{7,31}}{(G_{7,25}+1)(G_{13,31}+1)}=-\tau^{2}G_{7,31},$ (269) $\displaystyle G_{7,31}$ $\displaystyle=-\bar{\operatorname{tr}}\,(\gamma G_{6,32}L^{2}+\lambda^{2}D)K^{-2}.$ (270) $$

Defining $\rho=G_{6,32}$, we must have the following system of equations:

$$ $\displaystyle u_{s}$ $\displaystyle=\psi e_{s}^{2}\bar{\operatorname{tr}}\,\Sigma_{s}(\gamma\tau^{2}D+\rho I_{d})K^{-2},$ (271) $\displaystyle\rho$ $\displaystyle=\tau^{2}\bar{\operatorname{tr}}\,(\gamma\rho L^{2}+\lambda^{2}D)K^{-2}.$ (272) $$

In conclusion:

$$ $\displaystyle P_{2}$ $\displaystyle=K^{2},$ (273) $\displaystyle-P_{1}$ $\displaystyle=\lambda\gamma e_{1}\tau^{2}B+\lambda\gamma\tau^{2}p_{2}\Sigma_{2}(e_{1}u_{2}-e_{2}u_{1})+\lambda e_{1}\rho I_{d}-\lambda^{2}u_{1}\tau I_{d},$ (274) $\displaystyle\mathbb{E}\bar{\operatorname{tr}}\,r_{j}^{(2)}$ $\displaystyle=\lambda p_{j}\gamma\bar{\operatorname{tr}}\,A\Sigma_{j}(\gamma e_{j}\tau^{2}B+\gamma\tau^{2}p_{j^{\prime}}\Sigma_{j^{\prime}}(e_{j}u_{j^{\prime}}-e_{j^{\prime}}u_{j})+e_{j}\rho I_{d}-\lambda u_{j}\tau I_{d})K^{-2}.$ (275) $$

#### E.3 Computing 𝔼 ​ tr ¯ ​ r j ( 3 ) 𝔼 ¯ tr superscript subscript 𝑟 𝑗 3 \mathbb{E}\bar{\operatorname{tr}}\,r_{j}^{(3)}

WLOG, we focus on $r_{1}^{(3)}$. The matrix of interest has a linear pencil representation given by (with zero-based indexing):

$$ $\displaystyle r_{1}^{(3)}=Q^{-1}_{1,20},$ (276) $$

where the linear pencil $Q$ is defined as follows:

$$ $\rotatebox{270.0}{$Q=$ \leavevmode\resizebox{433.62pt}{}{$\left(\begin{array}[]{ccccccccccccccccccccc}I_{d}&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ -A&I_{d}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&I_{d}&-\frac{1}{\sqrt{\lambda}}Z_{1}^{\top}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}}Z_{1}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&I_{d}&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&I_{d}&-S&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&I_{m}&S^{\top}&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&I_{d}&-\Sigma_{2}^{\frac{1}{2}}&0&0&B&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}}Z_{2}^{\top}&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}}Z_{2}&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&I_{d}&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&I_{d}&-S&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&I_{m}&S^{\top}&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&I_{d}&-\Sigma_{1}^{\frac{1}{2}}&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}}Z_{1}^{\top}&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}}Z_{1}&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&I_{d}&0&0&0&\Sigma_{1}^{\frac{1}{2}}\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}}Z_{2}^{\top}&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}}Z_{2}&0\\ 0&0&0&0&0&0&0&0&0&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&0&0&0&I_{d}&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{d}\\ \end{array}\right)$}}.$ (277) $$

It holds that $\mathbb{E}\bar{\operatorname{tr}}\,r_{1}^{(3)}=G_{1,41}$. We immediately observe that:

$$ $\displaystyle e_{1}$ $\displaystyle=G_{3,24},G_{15,36},$ (278) $\displaystyle e_{2}$ $\displaystyle=G_{9,30},G_{18,39},$ (279) $\displaystyle\tau$ $\displaystyle=G_{6,27},G_{12,33},$ (280) $\displaystyle u_{1}$ $\displaystyle=-\frac{G_{3,36}}{\lambda},$ (281) $\displaystyle u_{2}$ $\displaystyle=-\frac{G_{9,39}}{\lambda},$ (282) $\displaystyle\rho$ $\displaystyle=G_{6,33}.$ (283) $$

In conclusion:

$$ $\displaystyle\mathbb{E}\bar{\operatorname{tr}}\,r_{j}^{(3)}=p_{j}\bar{\operatorname{tr}}\,A\Sigma_{j}(\gamma e_{j}^{2}p_{j}\Sigma_{j}(\gamma\tau^{2}u_{j^{\prime}}p_{j^{\prime}}\Sigma_{j^{\prime}}+\gamma\tau^{2}B+\rho I_{d})+u_{j}(\gamma e_{j^{\prime}}\tau p_{j^{\prime}}\Sigma_{j^{\prime}}+\lambda I_{d})^{2})K^{-2}.$ (284) $$

#### E.4 Computing 𝔼 ​ tr ¯ ​ r j ( 4 ) 𝔼 ¯ tr superscript subscript 𝑟 𝑗 4 \mathbb{E}\bar{\operatorname{tr}}\,r_{j}^{(4)}

WLOG, we focus on $r_{1}^{(4)}$. The matrix of interest has a linear pencil representation given by (with zero-based indexing):

$$ $\displaystyle r_{1}^{(4)}=Q^{-1}_{1,20},$ (285) $$

where the linear pencil $Q$ is defined as follows:

$$ $Q=$ $\left(\begin{array}[]{ccccccccccccccccccccc}I_{d}&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ -A&I_{d}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&I_{d}&-\frac{1}{\sqrt{\lambda}}Z_{1}^{\top}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}}Z_{1}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&I_{d}&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&I_{d}&-S&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&I_{m}&S^{\top}&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&I_{d}&-\Sigma_{2}^{\frac{1}{2}}&0&0&B&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}}Z_{2}^{\top}&0&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}}Z_{2}&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&I_{d}&0&0&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&I_{d}&-S&0&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&I_{m}&S^{\top}&0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&I_{d}&-\Sigma_{1}^{\frac{1}{2}}&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}}Z_{1}^{\top}&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{n_{1}}&-\frac{1}{\sqrt{\lambda}}Z_{1}&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&-\Sigma_{1}^{\frac{1}{2}}&0&0&0&0&I_{d}&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{d}&-\frac{1}{\sqrt{\lambda}}Z_{2}^{\top}&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{n_{2}}&-\frac{1}{\sqrt{\lambda}}Z_{2}&0\\ 0&0&0&0&0&0&0&0&0&0&0&-\Sigma_{2}^{\frac{1}{2}}&0&0&0&0&0&0&0&I_{d}&\Sigma_{2}^{\frac{1}{2}}\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&I_{d}\\ \end{array}\right)$ . (286) $$

It holds that $\mathbb{E}\bar{\operatorname{tr}}\,r_{1}^{(4)}=G_{1,41}$. We immediately observe that:

$$ $\displaystyle e_{1}$ $\displaystyle=G_{3,24},G_{15,36},$ (287) $\displaystyle e_{2}$ $\displaystyle=G_{9,30},G_{18,39},$ (288) $\displaystyle\tau$ $\displaystyle=G_{6,27},G_{12,33},$ (289) $\displaystyle u_{1}$ $\displaystyle=-\frac{G_{3,36}}{\lambda},$ (290) $\displaystyle u_{2}$ $\displaystyle=-\frac{G_{9,39}}{\lambda},$ (291) $\displaystyle\rho$ $\displaystyle=G_{6,33}.$ (292) $$

In conclusion:

$$ $\displaystyle\mathbb{E}\bar{\operatorname{tr}}\,r_{j}^{(4)}$ $\displaystyle=p_{j}\gamma p_{j^{\prime}}\bar{\operatorname{tr}}\,\Sigma_{j}\Sigma_{j^{\prime}}A(\gamma\tau^{2}(Be_{j}e_{j^{\prime}}-p_{j}\Sigma_{j}e_{j}^{2}u_{j^{\prime}}-p_{j^{\prime}}\Sigma_{j^{\prime}}e_{j^{\prime}}^{2}u_{j})$ (293) $\displaystyle-\lambda\tau(e_{j}u_{j^{\prime}}+e_{j^{\prime}}u_{j})I_{d}+e_{j}e_{j^{\prime}}\rho I_{d})K^{-2}.$ (294) $$

∎

### Appendix F Theorem 3

###### Definition 5 .

Let $(e_{1},e_{2},\tau_{1},\tau_{2},u_{1},u_{2},\rho_{1},\rho_{2})$ is be unique positive solution to the following system of fixed-point equations:

$$ $\displaystyle e_{s}=\frac{1}{1+\psi_{s}\tau_{s}\bar{\operatorname{tr}}\,\Sigma_{s}(\gamma\tau_{s}e_{s}\Sigma_{s}+\lambda_{s}I_{d})^{-1}},\quad\tau_{s}=\frac{1}{1+\bar{\operatorname{tr}}\,e_{s}\Sigma_{s}(\gamma\tau_{s}e_{s}\Sigma_{s}+\lambda_{s}I_{d})^{-1}},$ (295) $\displaystyle u_{s}=\psi_{s}e_{s}^{2}\bar{\operatorname{tr}}\,\Sigma_{s}(\gamma\tau_{s}^{2}(u_{s}+1)\Sigma_{s}+\rho_{s}I_{d})(\gamma\tau_{s}e_{s}\Sigma_{s}+\lambda_{s}I_{d})^{-2},$ (296) $\displaystyle\rho_{s}=\tau_{s}^{2}\bar{\operatorname{tr}}\,(\gamma\rho_{s}(e_{s}\Sigma_{s})^{2}+\lambda_{s}^{2}(u_{s}+1)\Sigma_{s})(\gamma\tau_{s}e_{s}\Sigma_{s}+\lambda_{s}I_{d})^{-2}.$ (297) $$

For deterministic $d\times d$ PSD matrices $A$ and $B$, we define the following auxiliary quantities:

$$ $\displaystyle h_{j}^{(1)}(A)$ $\displaystyle:=\gamma e_{j}\tau_{j}\bar{\operatorname{tr}}\,A\Sigma_{j}(\gamma\tau_{j}e_{j}\Sigma_{j}+\lambda_{j}I_{d})^{-1},$ (298) $\displaystyle h_{j}^{(2)}(A)$ $\displaystyle:=\gamma\bar{\operatorname{tr}}\,A\Sigma_{j}(\gamma e_{j}\tau_{j}^{2}\Sigma_{j}+e_{j}\rho_{j}I_{d}-\lambda_{j}u_{j}\tau_{j}I_{d})(\gamma\tau_{j}e_{j}\Sigma_{j}+\lambda_{j}I_{d})^{-2},$ (299) $\displaystyle h_{j}^{(3)}(A)$ $\displaystyle:=\bar{\operatorname{tr}}\,A\Sigma_{j}(\gamma e_{j}^{2}\Sigma_{j}(\gamma\tau_{j}^{2}\Sigma_{j}+\rho_{j}I_{d})+\lambda_{j}^{2}u_{j}I_{d})(\gamma\tau_{j}e_{j}\Sigma_{j}+\lambda_{j}I_{d})^{-2}.$ (300) $$

Under Assumptions [3](#Thmassumption3) and [2](#Thmassumption2), it holds that:

$$ $\displaystyle R_{s}(\widehat{f}_{s})$ $\displaystyle\simeq B_{s}(\widehat{f}_{s})+V_{s}(\widehat{f}_{s}),\text{ with }V_{s}(\widehat{f}_{s})=\lim_{p_{s}\to 1}V_{s}(\widehat{f}),\quad B_{s}(\widehat{f}_{s})=\lim_{p_{s}\to 1}B_{s}(\widehat{f}).$ (301) $$

More explicitly:

$$ $\displaystyle V_{s}(\widehat{f}_{s})$ $\displaystyle=\sigma_{s}^{2}\phi_{s}h_{s}^{(2)}(I_{d}),\quad B_{s}(\widehat{f}_{s})=\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}+h_{s}^{(3)}(\Theta_{s})-2h_{s}^{(1)}(\Theta_{s}\Sigma_{s}).$ (302) $$

###### Proof.

Theorem [3](#Thmtheorem3) follows from Theorem [2](#A14.EGx13) in the limit $p_{s}\to 1$ (i.e., $p_{s^{\prime}}\to 0$).
∎

###### Definition 5 .

###### Proof.

### Appendix G Solving Fixed-Point Equations for Theorem 1

#### G.1 Proportional Covariance Matrices

When $\lambda\to 0^{+}$, it is not possible to analytically solve the fixed-point equations for the constants in Definition [3](#Thmdefinition3) for general $\Sigma_{1},\Sigma_{2}$. As such, we consider a more tractable case where the covariance matrices are proportional, i.e., $\Sigma_{1}=a_{1}\Sigma$ and $\Sigma_{2}=a_{2}\Sigma$, for some $\Sigma\in\mathbb{R}^{d\times d}$.

We define $\theta=\frac{\lambda}{\gamma\tau(a_{1}p_{1}e_{1}+a_{2}p_{2}e_{2})}$ and $\eta=\bar{\operatorname{tr}}\,\Sigma(\Sigma+\theta I_{d})^{-1}$. Then, we have that:

$$ $\displaystyle e_{s}^{\prime}$ $\displaystyle=1+\psi\tau\bar{\operatorname{tr}}\,\Sigma_{s}K^{-1}=1+\frac{\phi a_{s}\eta}{a_{1}p_{1}e_{1}+a_{2}p_{2}e_{2}},$ (303) $\displaystyle\tau^{\prime}$ $\displaystyle=1+\bar{\operatorname{tr}}\,LK^{-1}=1+(\eta/\gamma)\tau^{\prime}=\frac{1}{1-\eta/\gamma}.$ (304) $$

If $\theta_{0}=0$, then $\eta_{0}=1$. Therefore, $e_{s}^{\prime}\to 1+\frac{\phi a_{s}}{a_{1}p_{1}e_{1}+a_{2}p_{2}e_{2}}$, which is a quadratic fixed-point equation. Accounting for the constraint that $e_{s}>0$, the fixed-point equation requires that $\phi<1$. Moreover, $\tau\to 1-1/\gamma$, which requires that $\gamma>1$. We further observe that $\rho\to(\tau^{2}\bar{\operatorname{tr}}\,\gamma L^{2}K^{-2})\rho$, which implies that $\rho\to 0$. We can then see that, for $c\in\{a_{1},a_{2}\}$:

$$ $\displaystyle u_{s}$ $\displaystyle\to\phi\gamma^{2}\tau^{2}e_{s}^{2}a_{s}(a_{1}p_{1}u_{1}+a_{2}p_{2}u_{2}+c)\bar{\operatorname{tr}}\,\Sigma^{2}K^{-2}$ (305) $\displaystyle=\frac{\phi e_{s}^{2}a_{s}(a_{1}p_{1}u_{1}+a_{2}p_{2}u_{2}+c)}{(a_{1}p_{1}e_{1}+a_{2}p_{2}e_{2})^{2}},$ (306) $$

which is a linear fixed-point equation in $u_{s}$.

In contrast, if $\theta_{0}>1$, we have $e_{s}^{\prime}=1+\frac{\psi\tau a_{s}\eta\theta}{\lambda}$ and the equation:

$$ $\displaystyle\gamma\theta$ $\displaystyle=\frac{\lambda}{(1-\eta/\gamma)\left(\frac{a_{1}p_{1}}{1+\frac{\psi(1-\eta/\gamma)a_{1}\eta\theta}{\lambda}}+\frac{a_{2}p_{2}}{1+\frac{\psi(1-\eta/\gamma)a_{2}\eta\theta}{\lambda}}\right)},$ (307) $$

which is a quartic equation in $\eta$.

#### G.2 The General Regularized Case

We now consider the case where the covariance structure is the same for both groups, i.e $\Sigma_{1}=\Sigma_{2}=\Sigma$. In this setting, it is clear that $e_{1}=e_{2}=e$ and $u_{1}=u_{2}=u$, where $(\tau,e,u,\rho)$ now satisfy:

$$ $\displaystyle 1/e$ $\displaystyle=1+\psi\tau\bar{\operatorname{tr}}\,\Sigma K^{-1},\quad 1/\tau=1+\bar{\operatorname{tr}}\,K_{0}K^{-1},\text{ where }K_{0}:=e\Sigma,\,K:=\gamma\tau K_{0}+\lambda I_{d},$ (308) $\displaystyle u$ $\displaystyle=\psi e^{2}\bar{\operatorname{tr}}\,\Sigma_{1}(\gamma\tau^{2}L^{\prime}+\rho I_{d})K^{-2},\,\,\rho=\tau^{2}\bar{\operatorname{tr}}\,(\gamma\rho K_{0}^{2}+\lambda^{2}L^{\prime})K^{-2},\,\,L^{\prime}:=(1+u)\Sigma.$ (309) $$

###### Lemma 2 .

The scalars $u$ and $\rho^{\prime}=\rho/(\gamma\tau^{2})$ solve the following pair of linear equations:

$$ $\displaystyle\begin{split}u&=\phi I_{2,2}(\theta)(1+u)+\phi I_{1,2}(\theta)\rho^{\prime},\\ \gamma\rho^{\prime}&=I_{2,2}(\theta)\rho^{\prime}+\theta^{2}I_{1,2}(\theta)(1+u).\end{split}$ (310) $$

Furthermore, the solutions can be explicitly represented as:

$$ $\displaystyle\begin{split}u&=\frac{\phi z}{\gamma-\phi z-I_{2,2}(\theta)},\quad\rho^{\prime}=\frac{\theta^{2}I_{2,2}(\theta)}{\gamma-\phi z-I_{2,2}(\theta)},\end{split}$ (311) $$

where $z=I_{2,2}(\theta)(\gamma-I_{2,2}(\theta))+\theta^{2}I_{1,2}(\theta)^{2}.$

In particular, in the limit $\gamma\to\infty$, it holds that:

$$ $\displaystyle\theta\simeq\kappa,\quad\rho^{\prime}\to 0,\quad u\simeq\frac{\phi I_{2,2}(\kappa)}{1-\phi I_{2,2}(\kappa)}\simeq\frac{\operatorname{df}_{2}(\kappa)/n}{1-\operatorname{df}_{2}(\kappa)/n},$ (312) $$

where $\kappa>0$ is uniquely satisfies the fixed-point equation $\kappa-\lambda=\kappa\operatorname{tr}\,\Sigma(\Sigma+\kappa I_{d})^{-1}/n$.

###### Proof.

The equations defining these scalars are:

$$ $\displaystyle u$ $\displaystyle=\psi e^{2}\bar{\operatorname{tr}}\,\Sigma(\gamma\tau^{2}L^{\prime}+\rho I_{d})K^{-2},$ (313) $\displaystyle\rho$ $\displaystyle=\tau^{2}\bar{\operatorname{tr}}\,(\gamma\rho K_{0}^{2}+\lambda^{2}L^{\prime})K^{-2},$ (314) $$

where $K_{0}=e\Sigma$, $K=\gamma\tau K_{0}+\lambda I_{d}$, and $L^{\prime}:=u\Sigma+B$. Further, since $B=\Sigma$, we have $L^{\prime}=(1+u)\Sigma$. Now, we can rewrite the previous equations like so

$$ $\displaystyle u$ $\displaystyle=\psi e^{2}\bar{\operatorname{tr}}\,\Sigma(\gamma\tau^{2}(1+u)\Sigma+\rho I_{d})K^{-2}=\phi\gamma^{2}\tau^{2}e^{2}(1+u)\bar{\operatorname{tr}}\,\Sigma^{2}K^{-2}+\phi\gamma e^{2}\rho\bar{\operatorname{tr}}\,\Sigma K^{-2},$ $\displaystyle\rho$ $\displaystyle=\tau^{2}\bar{\operatorname{tr}}\,(\gamma\rho e^{2}\Sigma^{2}+\lambda^{2}(1+u)\Sigma)K^{-2}=\gamma\tau^{2}e^{2}\rho\bar{\operatorname{tr}}\,\Sigma^{2}K^{-2}+\lambda^{2}\tau^{2}(1+u)\bar{\operatorname{tr}}\,\Sigma K^{-2}.$ $$

This can be equivalently written as:

$$ $\displaystyle u$ $\displaystyle=\phi(1+u)\gamma^{2}\tau^{2}e^{2}\bar{\operatorname{tr}}\,\Sigma^{2}K^{-2}+\phi\rho^{\prime}\gamma^{2}\tau^{2}e^{2}\bar{\operatorname{tr}}\,\Sigma K^{-2},$ (315) $\displaystyle\gamma\rho^{\prime}$ $\displaystyle=\rho^{\prime}\gamma^{2}\tau^{2}e^{2}\bar{\operatorname{tr}}\,\Sigma^{2}K^{-2}+(1+u)\lambda^{2}\bar{\operatorname{tr}}\,\Sigma K^{-2}.$ (316) $$

Now, observe that:

$$ $\displaystyle\tau^{2}e^{2}\bar{\operatorname{tr}}\,\Sigma^{2}K^{-2}$ $\displaystyle=\bar{\operatorname{tr}}\,\Sigma^{2}(\Sigma+\theta I_{d})^{-2}/\gamma^{2}=I_{2,2}(\theta)/\gamma^{2},$ (317) $\displaystyle\tau^{2}e^{2}\bar{\operatorname{tr}}\,\Sigma K^{-2}$ $\displaystyle=\bar{\operatorname{tr}}\,\Sigma(\Sigma+\theta I_{d})^{-2}/\gamma^{2}=I_{1,2}(\theta)/\gamma^{2},$ (318) $\displaystyle\lambda^{2}\bar{\operatorname{tr}}\,\Sigma K^{-2}$ $\displaystyle=\theta^{2}\bar{\operatorname{tr}}\,\Sigma(\Sigma+\theta I_{d})^{-2}=\theta^{2}I_{1,2}(\theta),$ (319) $\displaystyle e^{2}\bar{\operatorname{tr}}\,\Sigma K^{-2}$ $\displaystyle=\bar{\operatorname{tr}}\,\Sigma(\Sigma+\theta I_{d})^{-2}/(\gamma\tau)^{2}=I_{1,2}(\theta)/(\gamma\tau)^{2},$ (320) $\displaystyle\tau^{2}\bar{\operatorname{tr}}\,\Sigma K^{-2}$ $\displaystyle=\bar{\operatorname{tr}}\,\Sigma(\Sigma+\theta I_{d})^{-2}/(\gamma e)^{2}=I_{1,2}(\theta)/(\gamma e)^{2},$ (321) $$

where we have used the definition $\theta=\lambda/(\gamma\tau e)$.
Thus, $u$ and $\rho$ have limiting values $u$ and $\rho$ respectively, which solve the system of linear equations:

$$ $\displaystyle u$ $\displaystyle=\psi\gamma\cdot\gamma^{-2}I_{2,2}(\theta)(1+u)+\psi\gamma\cdot\gamma^{-2}I_{1,2}\rho^{\prime}=\phi I_{2,2}(\theta)(1+u)+\phi I_{1,2}(\theta)\rho^{\prime},$ $\displaystyle\gamma\rho^{\prime}$ $\displaystyle=I_{2,2}(\theta)\rho^{\prime}+\theta^{2}I_{1,2}(\theta)(1+u)=I_{2,2}(\theta)\rho^{\prime}+\theta^{2}I_{1,2}(\theta)(1+u),$ $$

where we have used the identity $\phi\gamma=\psi$. These correspond exactly to the equations given in the lemma. This proves the first part.

For the second part, indeed, $\tau=1-\eta_{0}/\gamma\to 1$ in the limit $\gamma\to\infty$, and so $\theta\simeq\lambda/(\gamma e)$ which verifies the equation:

$$ $\theta\simeq\lambda+\lambda\psi\bar{\operatorname{tr}}\,\Sigma(\gamma e\Sigma+\lambda)^{-1}=\lambda+\phi\cdot\frac{\lambda}{\gamma e}\bar{\operatorname{tr}}\,\Sigma(\Sigma+\frac{\lambda}{\gamma e}I_{d})^{-1}\simeq\lambda+\theta\operatorname{tr}\,\Sigma(\Sigma+\theta I_{d})^{-1}/n,$ $$

i.e., $\theta\simeq\lambda+\theta\operatorname{df}_{1}(\theta)/n$ and $\theta>0$.
By comparing with the equation $\kappa-\lambda=\kappa\operatorname{df}_{1}(\kappa)/n$ satisfied by $\kappa>0$ in Definition [4](#Thmdefinition4), we conclude $\theta\simeq\kappa$.

Now, equation [310](#A7.E310) becomes $\rho^{\prime}=0$, and $u=\phi I_{2,2}(\kappa)(1+u)$, i.e.,

$$ $u=\frac{\phi I_{2,2}(\kappa)}{1-\phi I_{2,2}(\kappa)}\simeq\frac{\operatorname{df}_{2}(\kappa)/n}{1-\operatorname{df}_{2}(\kappa)/n},$ $$

as claimed.
∎

###### Lemma 2 .

###### Proof.

#### G.3 Unregularized Limit

Define the following auxiiliary quantities:

$$ $\displaystyle\theta:=\frac{\lambda}{\gamma\tau e},\quad\chi:=\frac{\lambda}{\tau},\quad\kappa:=\frac{\lambda}{e}.$ (322) $$

where $\tau$, $e$, $u$, and $\rho$ are as previously defined.

###### Lemma 3 .

In the limit $\lambda\to 0^{+}$, we have the following analytic formulae:

$$ $\displaystyle\chi$ $\displaystyle\to\chi_{0}=(1-\psi)_{+}\cdot\gamma\theta_{0},$ (323) $\displaystyle\kappa$ $\displaystyle\to\kappa_{0}=(\psi-1)_{+}\cdot\theta_{0}/\phi,$ (324) $\displaystyle\tau$ $\displaystyle\to\tau_{0}=1-\eta_{0}/\gamma,$ (325) $\displaystyle e$ $\displaystyle\to e_{0}=1-\phi\eta_{0}.$ (326) $$

###### Proof.

Observe that $K_{0}=e\Sigma$ and $K=\gamma\tau K_{0}+\lambda I_{d}=\gamma\tau e\cdot(\Sigma+\theta I_{d})$.
Defining $\eta:=I_{1,1}(\theta)$, one can then rewrite the equations defining $e$ and $\tau$ as follows:

$$ $\displaystyle e^{\prime}$ $\displaystyle=\frac{\lambda}{e}=\lambda+\psi\tau\lambda\bar{\operatorname{tr}}\,\Sigma K^{-1}=\lambda+\frac{\psi\tau\lambda}{\gamma\tau e}\bar{\operatorname{tr}}\,\Sigma(\Sigma+\theta I_{d})^{-1}=\lambda+\phi\eta e^{\prime},$ (327) $\displaystyle\tau^{\prime}$ $\displaystyle=\frac{\lambda}{\tau}=\lambda+\lambda\bar{\operatorname{tr}}\,K_{0}K^{-1}=\lambda+\frac{\lambda e}{\gamma\tau e}\bar{\operatorname{tr}}\,\Sigma(\Sigma+\theta I_{d})^{-1}=\lambda+(\eta/\gamma)\tau^{\prime}.$ (328) $$

We deduce that:

$$ $\displaystyle e^{\prime}$ $\displaystyle=\frac{\lambda}{1-\phi\eta},\quad\tau^{\prime}=\frac{\lambda}{1-\eta/\gamma},\quad\tau^{\prime}e^{\prime}=\lambda\gamma\theta.$ (329) $$

In particular, the above means that $\eta\leq\min(\gamma,1/\phi)$. The last part of equations equation [329](#A7.E329) can be rewritten as follows:

$$ $\displaystyle\frac{\lambda}{(1-\phi\eta)(1-\eta/\gamma)}=\gamma\theta,\text{ i.e }\phi\eta^{2}-(\phi\gamma+1)\eta+\gamma-\frac{\lambda}{\theta}=0.$ (330) $$

This is a quadratic equation for $\eta$ as a function of $\lambda$ and $\theta$, with roots

$$ $\displaystyle\eta^{\pm}=\frac{\phi\gamma+1\pm\sqrt{(\phi\gamma+1)^{2}-4(\phi\gamma-(\phi/\theta)\lambda)}}{2\phi}=\frac{\psi+1\pm\sqrt{(\psi+1)^{2}-4(\psi-\phi/\theta^{\prime})}}{2\phi}.$ (331) $$

Now, for small $\lambda>0$ and $\psi\neq 1$, we can do a Taylor expansion to get:

$$ $\displaystyle\eta^{\pm}$ $\displaystyle\simeq\frac{\psi+1\pm|\psi-1|}{2\phi}\pm\frac{1}{\theta|\psi-1|}\lambda+O(\lambda^{2}).$ $$

More explicitly:

$$ $\displaystyle\eta^{+}$ $\displaystyle\simeq O(\lambda^{2})\begin{cases}1/\phi+\lambda/((1-\psi)\theta),&\mbox{ if }\psi<1,\\ \gamma+\lambda/((\psi-1)\theta),&\mbox{ if }\psi>1.\end{cases}$ $\displaystyle\eta^{-}$ $\displaystyle\simeq O(\lambda^{2})+\begin{cases}\gamma-\lambda/((1-\psi)\theta),&\mbox{ if }\psi<1,\\ 1/\phi-\lambda/((\psi-1)\theta),&\mbox{ if }\psi>1,\\ \end{cases}$ $$

Because $\eta\leq\min(1,1/\phi,\gamma)$, we must have the expansion:

$$ $\displaystyle\begin{split}\eta&\simeq O(\lambda^{2})+\begin{cases}\gamma-\lambda/((1-\psi)\theta),&\mbox{ if }\psi<1,\\ 1/\phi+\lambda/((\psi-1)\theta),&\mbox{ if }\psi>1,\end{cases}\\ &=\eta_{0}-\frac{1}{(1-\psi)\theta_{0}}\lambda+O(\lambda^{2}),\end{split}$ (332) $$

provided $\theta_{0}>0$, i.e $\eta_{0}\neq 1$.
in this regime, we obtain:

$$ $\displaystyle\tau^{\prime}=\frac{\lambda}{1-\eta/\gamma}$ $\displaystyle\simeq\begin{cases}\lambda/(1-1+\lambda/((1-\psi)\gamma\theta_{0}))=(1-\psi)\gamma\theta_{0},&\mbox{ if }\psi\leq 1,\\ \lambda/(1-1/\psi+o(1))\to 0,&\mbox{ if }\psi>1,\end{cases}$ $\displaystyle e^{\prime}=\frac{\lambda}{1-\phi\eta}$ $\displaystyle\simeq\begin{cases}\lambda/(1-\psi+o(1))\to 0,&\mbox{ if }\psi\leq 1,\\ \lambda/(1-1+\lambda\phi/((\psi-1)\theta_{0})\to(\psi-1)\theta_{0}/\phi,&\mbox{ if }\psi>1,\end{cases}$ $\displaystyle\tau=1-\eta/\gamma$ $\displaystyle\simeq 1-\eta_{0}/\gamma=(1-1/\psi)_{+},$ $\displaystyle e=1-\phi\eta$ $\displaystyle\simeq 1-\phi\eta_{0}=(1-\psi)_{+}.$ $$

On the other hand, if $\theta_{0}=0$ (which only happens if $\psi<1$ and $\gamma>1$ OR $\psi\geq 1$ and $\phi\leq 1$), it is easy to see from equation [329](#A7.E329) that we must have $\tau^{\prime}\to 0$, $e^{\prime}\to 0$, $\tau\to 1-1/\gamma$, $e\to 1-\phi\geq 0$.
∎

###### Lemma 3 .

###### Proof.

### Appendix H Corollary 1

As a highly special case of Theorem [2](#A14.EGx13), we recover Corollary [1](#A14.EGx147), which aligns with Proposition 4 from (Bach, ). Theorem [2](#A14.EGx13) is a non-trivial generalization of Proposition 4.

###### Corollary 1 .

Under Assumptions [3](#Thmassumption3) and [2](#Thmassumption2), it holds in the unregularized setting $\lambda_{s}\to 0^{+}$ that

$$ $\displaystyle B_{s}(\widehat{f}_{s})$ $\displaystyle=\begin{cases}\frac{\theta_{0}\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\Sigma_{s}+\theta_{0}I_{d})^{-1}}{1-\psi_{s}},&\gamma,\psi_{s}<1\\ 0,&\psi_{s}<1,\gamma\geq 1\text{ or }1\leq\psi_{s}\leq\gamma\\ \frac{\theta_{0}^{2}\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\Sigma_{s}+\theta_{0}I_{d})^{-2}}{1-\phi_{s}I_{2,2}(\theta_{0})}+\frac{\theta_{0}\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\Sigma_{s}+\theta_{0}I_{d})^{-1}}{\psi_{s}-1},&\psi_{s}\geq 1,\psi_{s}\geq\gamma\end{cases},$ (333) $\displaystyle V_{s}(\widehat{f}_{s})$ $\displaystyle=\begin{cases}\frac{\sigma_{s}^{2}\psi_{s}}{1-\psi_{s}},&\gamma,\psi_{s}<1\\ \frac{\sigma_{s}^{2}\phi_{s}}{1-\phi_{s}},&\psi_{s}<1,\gamma\geq 1\text{ or }1\leq\psi_{s}\leq\gamma\\ \frac{\sigma_{s}^{2}\phi_{s}I_{2,2}(\theta_{0})}{1-\phi_{s}I_{2,2}(\theta_{0})}+\frac{\sigma_{s}^{2}}{\psi_{s}-1},&\psi_{s}\geq 1,\psi_{s}\geq\gamma\end{cases},$ (334) $$

where $I_{a,b}(\theta_{0})=\bar{\operatorname{tr}}\,\Sigma^{a}(\Sigma+\theta_{0}I_{d})^{-b}$ for any positive integers $a,b$; and $\theta_{0}$ is the unique solution to the following non-linear equation:

$$ $\displaystyle I_{1,1}(\theta_{0})=\begin{cases}\gamma,&\gamma,\psi_{s}<1\\ 1,&\psi_{s}<1,\gamma\geq 1\text{ or }1\leq\psi_{s}\leq\gamma\\ 1/\phi_{s},&\psi_{s}\geq 1,\psi_{s}\geq\gamma\end{cases}.$ (335) $$

###### Proof.

Define $e^{\prime}=1/e_{s}\geq 0$, $\tau^{\prime}=1/\tau_{s}\geq 0$, $\theta=\lambda_{s}\tau^{\prime}e^{\prime}/\gamma$, and $\eta=I_{1,1}(\theta)\in[0,1]$. One can then express $e^{\prime}$ and $\tau^{\prime}$ as:

$$ $\displaystyle e^{\prime}$ $\displaystyle=1+\psi\tau_{s}\bar{\operatorname{tr}}\,\Sigma(\gamma\tau_{s}e_{s}\Sigma+\lambda_{s}I_{d})^{-1}=1+\phi_{s}\eta e^{\prime},$ (336) $\displaystyle\tau^{\prime}$ $\displaystyle=1+\bar{\operatorname{tr}}\,e_{s}\Sigma(\gamma\tau_{s}e_{s}\Sigma+\lambda_{s}I_{d})^{-1}=1+(\eta/\gamma)\tau^{\prime}.$ (337) $$

We deduce that:

$$ $\displaystyle e^{\prime}$ $\displaystyle=\frac{1}{1-\phi_{s}\eta},$ (338) $\displaystyle\tau^{\prime}$ $\displaystyle=\frac{1}{1-\eta/\gamma},$ (339) $\displaystyle\lambda\tau^{\prime}e^{\prime}$ $\displaystyle=\gamma\theta.$ (340) $$

We define the following limiting values:

$$ $\displaystyle\lim_{\lambda_{s}\to 0^{+}}\theta\to\theta_{0},$ $\displaystyle\lim_{\lambda_{s}\to 0^{+}}\eta\to\eta_{0},$ (341) $\displaystyle\lim_{\lambda_{s}\to 0^{+}}e_{s}\to e_{0},$ $\displaystyle\lim_{\lambda_{s}\to 0^{+}}\tau_{s}\to\tau_{0},$ (342) $\displaystyle\lim_{\lambda_{s}\to 0^{+}}u_{s}\to u_{0},$ $\displaystyle\lim_{\lambda_{s}\to 0^{+}}\rho_{s}\to\rho_{0}.$ (343) $$

There are now two cases to consider.

###### Corollary 1 .

###### Proof.

#### H.1 Case 1: θ 0 = 0 subscript 𝜃 0 0 \theta_{0}=0

This implies $\eta_{0}=1$. Therefore, by simple computation, $e_{0}=1/e^{\prime}_{0}=1-\phi_{s}\eta_{0}=1-\phi_{s}$ and $\tau_{0}=1/\tau^{\prime}_{0}=1-1/\gamma$. This requires $\phi_{s}\leq 1$ and $\gamma\geq 1$.

#### H.2 Case 2: θ 0 > 0 subscript 𝜃 0 0 \theta_{0}>0

Equation [340](#A8.E340) can be re-written as:

$$ $\displaystyle\frac{\lambda_{s}}{(1-\phi_{s}\eta)(1-\eta/\gamma)}=\gamma\theta,\text{ i.e., }\phi_{s}\eta^{2}-(\psi_{s}+1)\eta+\gamma-\frac{\lambda_{s}}{\theta}=0.$ (344) $$

We solve this quadratic equation for $\eta$, arriving at the solutions:

$$ $\displaystyle\eta^{\pm}=\frac{\psi_{s}+1\pm\sqrt{(\psi_{s}+1)^{2}-4(\psi_{s}-(\phi_{s}/\theta)\lambda_{s})}}{2\phi_{s}}=\frac{\psi_{s}+1\pm\sqrt{(\psi_{s}+1)^{2}-4(\psi_{s}-(\phi_{s}/\theta)\lambda_{s})}}{2\phi_{s}}.$ (345) $$

Taking the limit of $\eta^{\pm}$ as $\lambda_{s}\to 0^{+}$ gives:

$$ $\displaystyle\begin{split}\eta^{+}\to\frac{\psi_{s}+1+|\psi_{s}-1|}{2\phi_{s}}&=\begin{cases}\psi_{s}/\phi_{s}=\gamma,&\mbox{ if }\psi_{s}\geq 1,\\ 1/\phi_{s},&\mbox{ if }\psi_{s}<1,\end{cases}\\ \eta^{-}\to\frac{\psi_{s}+1-|\psi_{s}-1|}{2\phi_{s}}&=\begin{cases}1/\phi_{s},&\mbox{ if }\psi_{s}\geq 1,\\ \psi_{s}/\phi_{s}=\gamma,&\mbox{ if }\psi_{s}<1.\end{cases}\end{split}$ (346) $$

Recall that we have the following constraints:

- •
$e^{\prime}\geq 0,\tau^{\prime}\geq 0$.
- •
$\eta\in[0,1]$.

We can show that $\eta_{0}=1/\phi_{s}$ is incompatible with $\psi_{s}<1$. Indeed, otherwise we would have $\tau^{\prime}_{0}=1/(1-\eta_{0}/\gamma)=1/(1-1/\psi_{s})<0$. Similarly, if $\psi_{s}>1$, we would have $e_{0}=1-\phi_{s}\gamma=1-\psi_{s}<0$. Therefore, $\eta_{0}=\eta^{-}$. Furthermore, if $\psi_{s},\gamma<1$, it must be that $\theta_{0}>0$ and $\eta_{0}=\gamma$. Instead, if $\psi_{s}<1,\gamma\geq 1$, we must have that $\phi_{s}\leq 1$, and therefore, $\theta_{0}=0$ and $\eta_{0}=1$. Similarly, if $\psi_{s}\geq 1,\gamma\geq 1$, and $\phi_{s}\leq 1$ (i.e., $1\leq\psi_{s}\leq\gamma$), we must have that $\theta_{0}=0$ and $\eta_{0}=1$. In all other cases where $\psi_{s}\geq 1$, it must be that $\eta_{0}=1/\phi_{s}$ (which additionally requires $\phi_{s}\geq 1$ or $\psi_{s}\geq\gamma$). Succinctly:

$$ $\displaystyle\eta_{0}=\begin{cases}\gamma,&\gamma,\psi_{s}<1\\ 1,&\psi_{s}<1,\gamma\geq 1\text{ or }1\leq\psi_{s}\leq\gamma\\ 1/\phi_{s},&\psi_{s}\geq 1,\psi_{s}\geq\gamma\end{cases}.$ (347) $$

Plugging this into equation [338](#A8.E338) and equation [339](#A8.E339) gives:

$$ $\displaystyle e_{0}$ $\displaystyle=1-\phi_{s}\eta_{0}=1-\phi_{s}I_{1,1}(\theta_{0}),$ (348) $\displaystyle\tau_{0}$ $\displaystyle=1-\eta_{0}/\gamma=1-I_{1,1}(\theta_{0})/\gamma.$ (349) $$

We will now solve for $u_{0}$ and $\rho_{0}/\tau_{0}^{2}$. We can re-write $u_{s}$ and $\rho_{s}/\tau_{s}^{2}$ as:

$$ $\displaystyle\rho_{s}/\tau_{s}^{2}$ $\displaystyle=\gamma^{-1}(\rho_{s}/\tau_{s}^{2})I_{2,2}(\theta)+\theta^{2}(u_{s}+1)I_{1,2}(\theta),$ (350) $\displaystyle\tau_{s}^{2}u_{s}$ $\displaystyle=\tau_{s}^{2}\phi_{s}(u_{s}+1)I_{2,2}(\theta)+\phi_{s}\gamma^{-1}\rho_{s}I_{1,2}(\theta).$ (351) $$

Solving for $u_{0}$ and $\rho_{0}/\tau_{0}^{2}$ yields:

$$ $\displaystyle u_{0}$ $\displaystyle=\frac{\phi\zeta}{\gamma-\phi\zeta-I_{2,2}(\theta_{0})},\,\rho_{0}/\tau_{0}^{2}=\frac{\gamma\theta_{0}^{2}I_{2,2}(\theta_{0})}{\gamma-\phi\zeta-I_{1,2}(\theta_{0})},$ (352) $\displaystyle\text{where }\zeta$ $\displaystyle=I_{2,2}(\theta_{0})(\gamma-I_{2,2}(\theta_{0}))+\theta_{0}^{2}I_{1,2}(\theta_{0})^{2}.$ (353) $$

We can then see for the variance term that:

$$ $\displaystyle V_{s}(\widehat{f}_{s})$ $\displaystyle=\sigma_{s}^{2}\phi_{s}\gamma\bar{\operatorname{tr}}\,\Sigma_{s}(\gamma e_{s}\tau_{s}^{2}\Sigma_{s}+e_{s}\rho_{s}I_{d}-\lambda_{s}u_{s}\tau_{s}I_{d})(\gamma\tau_{s}e_{s})^{-2}(\Sigma_{s}+\theta I_{d})^{-2}$ (354) $\displaystyle=\sigma_{s}^{2}\phi_{s}(1/e_{s})\bar{\operatorname{tr}}\,\Sigma_{s}^{2}(\Sigma_{s}+\theta I_{d})^{-2}+(\sigma_{s}^{2}\phi_{s}/\gamma)(1/e_{s})(\rho_{s}/\tau_{s}^{2})\bar{\operatorname{tr}}\,\Sigma_{s}(\Sigma_{s}+\theta I_{d})^{-2}$ (355) $\displaystyle-\sigma_{s}^{2}\phi_{s}(u_{s})(1/e_{s})\theta\bar{\operatorname{tr}}\,\Sigma_{s}(\Sigma_{s}+\theta I_{d})^{-2}$ (356) $\displaystyle=\sigma_{s}^{2}\phi_{s}I_{2,2}(\theta)/e_{s}+\sigma_{s}^{2}\phi_{s}(\rho_{s}/\tau_{s}^{2})I_{1,2}(\theta)/(\gamma e_{s})-\sigma_{s}^{2}\phi_{s}u_{s}\theta I_{1,2}(\theta)/e_{s}$ (357) $\displaystyle\to\frac{\sigma_{s}^{2}\phi_{s}I_{2,2}(\theta_{0})-\sigma_{s}^{2}\phi_{s}u_{0}\theta_{0}I_{1,2}(\theta_{0})}{1-\phi_{s}I_{1,1}(\theta_{0})}+\frac{\sigma_{s}^{2}\phi_{s}\rho_{0}/\tau_{0}^{2}}{\gamma(1-\phi_{s}I_{1,1}(\theta_{0}))}$ (358) $\displaystyle=-\frac{\sigma_{s}^{2}\phi_{s}\xi}{\phi_{s}\xi+I_{2,2}(\theta_{0})-\gamma},$ (359) $$

where $\xi=I_{1,1}^{2}(\theta_{0})-2I_{1,1}(\theta_{0})I_{2,2}(\theta_{0})+I_{2,2}(\theta_{0})\gamma$ and we have used the fact that $I_{1,2}(\theta)=(I_{1,1}(\theta)-I_{2,2}(\theta))/\theta$. Plugging in $I_{1,1}(\theta_{0})=\eta_{0}$, we have that:

$$ $\displaystyle V_{s}(\widehat{f}_{s})$ $\displaystyle\to\begin{cases}\frac{\sigma_{s}^{2}\psi_{s}}{1-\psi_{s}},&\gamma,\psi_{s}<1\\ \frac{\sigma_{s}^{2}\phi_{s}}{1-\phi_{s}},&\psi_{s}<1,\gamma\geq 1\text{ or }1\leq\psi_{s}\leq\gamma\\ \frac{\sigma_{s}^{2}\phi_{s}I_{2,2}(\theta_{0})}{1-\phi_{s}I_{2,2}(\theta_{0})}+\frac{\sigma_{s}^{2}}{\psi_{s}-1},&\psi_{s}\geq 1,\psi_{s}\geq\gamma\end{cases},$ (360) $$

where we have used that $I_{2,2}(\theta_{0})=I_{2,2}(0)=1$ in the second case.

Likewise, for the bias term, we obtain:

$$ $\displaystyle B_{s}(\widehat{f}_{s})$ $\displaystyle=\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}+\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\gamma e_{s}^{2}\Sigma_{s}(\gamma\tau_{s}^{2}\Sigma_{s}+\rho_{s}I_{d})+\lambda_{s}^{2}u_{s}I_{d})(\gamma\tau_{s}e_{s}\Sigma_{s}+\lambda_{s}I_{d})^{-2}$ (361) $\displaystyle-2\gamma e_{s}\tau_{s}\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}^{2}(\gamma\tau_{s}e_{s}\Sigma_{s}+\lambda_{s}I_{d})^{-1}$ (362) $\displaystyle\to\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\Sigma_{s}^{2}+2\theta_{0}\Sigma_{s}+\theta_{0}^{2}I_{d})(\Sigma_{s}+\theta_{0}I_{d})^{-2}$ (363) $\displaystyle+\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\Sigma_{s}^{2})(\Sigma_{s}+\theta_{0}I_{d})^{-2}$ (364) $\displaystyle+\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}((\rho_{0}/\tau_{0}^{2})\Sigma_{s}/\gamma)(\Sigma_{s}+\theta_{0}I_{d})^{-2}$ (365) $\displaystyle+\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\theta_{0}^{2}u_{0}I_{d})(\Sigma_{s}+\theta_{0}I_{d})^{-2}$ (366) $\displaystyle+\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(-2\Sigma_{s}^{2}-2\theta_{0}\Sigma_{s})(\Sigma_{s}+\theta_{0}I_{d})^{-2}$ (367) $\displaystyle=\theta_{0}^{2}(u_{0}+1)\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\Sigma_{s}+\theta_{0}I_{d})^{-2}+(1/\gamma)(\rho_{0}/\tau_{0}^{2})\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}^{2}(\Sigma_{s}+\theta_{0}I_{d})^{-2}.$ (368) $$

Again, plugging in $I_{1,1}(\theta_{0})=\eta_{0}$, we have that:

$$ $\displaystyle B_{s}(\widehat{f}_{s})$ $\displaystyle\to\begin{cases}\frac{\theta_{0}\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\Sigma_{s}+\theta_{0}I_{d})^{-1}}{1-\psi_{s}},&\gamma,\psi_{s}<1\\ \frac{\theta_{0}^{2}\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\Sigma_{s}+\theta_{0}I_{d})^{-2}}{1-\phi_{s}}=0,&\psi_{s}<1,\gamma\geq 1\text{ or }1\leq\psi_{s}\leq\gamma\\ \frac{\theta_{0}^{2}\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\Sigma_{s}+\theta_{0}I_{d})^{-2}}{1-\phi_{s}I_{2,2}(\theta_{0})}+\frac{\theta_{0}\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\Sigma_{s}+\theta_{0}I_{d})^{-1}}{\psi_{s}-1},&\psi_{s}\geq 1,\psi_{s}\geq\gamma\end{cases},$ (369) $$

where we have used that $\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}^{2}(\Sigma_{s}+\theta_{0}I_{d})^{-2}=\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\Sigma_{s}+\theta_{0}I_{d})^{-1}-\theta_{0}\bar{\operatorname{tr}}\,\Theta_{s}\Sigma_{s}(\Sigma_{s}+\theta_{0}I_{d})^{-2}$ and in the second case, $\theta_{0}=0$ and $I_{2,2}(\theta_{0})=1$.
∎

### Appendix I Experimental Details

#### I.1 Synthetic Experiments

Across all experiments on synthetic data, we choose $n=400$. We further use 5 runs to estimate test risks (e.g., $\mathbb{E}R_{s}(\widehat{f}),\mathbb{E}R_{s}(\widehat{f}_{s})$), and 5 runs to capture the variance of the estimators, for a total of 25 runs. We use 10,000 samples to estimate test risks.

#### I.2 Colored MNIST Experiment

Train-test split. Colored MNIST has a total of 60k instances. We use the prescribed 0.67-0.33 train-test split. We do not perform validation of hyperparameters, which we mostly adopt(^3^33[https://colab.research.google.com/github/reiinakano/invariant-risk-minimization/blob/master/invariant_risk_minimization_colored_mnist.ipynb](https://colab.research.google.com/github/reiinakano/invariant-risk-minimization/blob/master/invariant_risk_minimization_colored_mnist.ipynb)).

Model architecture. Our CNN architecture consists of: (1) a convolutional layer (3 in-channels, 20 out-channels, kernel size of 5, stride of 1); (2) a max pooling layer (kernel size of 2, stride of 2); (3) a second convolutional layer (20 in-channels, 50 out-channels, kernel size of 5, stride of 1); (4) a second max-pooling layer (kernel size of 2, stride of 2); (5) a fully-connected layer ($\mathbb{R}^{800}\to\mathbb{R}^{500}$); and (6) a second fully-connected layer ($\mathbb{R}^{500}\to\mathbb{R}^{1}$).

Model training. We train each model with a batch size of 250 for a single epoch. We use a cross-entropy loss and the Adam optimizer with learning rate 0.01. We run all experiments on a single Quadro GP100. We report our results over 10 random seeds in Figure [6](#A9.F6).

Figure: Figure 6: Our theory predicts that disparate label noise between groups deamplifies bias on Colored MNIST. The error bars capture the standard deviation computed over 10 random seeds.
Refer to caption: /html/2410.17263/assets/x6.png

### Appendix J Bias Amplification Plots

Figure: Figure 7: We empirically demonstrate that bias amplification occurs and validate our theory (Theorems [2](#A14.EGx13) and [3](#Thmtheorem3)) for $ODD$, $EDD$, and $ADD$ under the setup described in Section [4.1](#S4.SS1). The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We plot $ODD$ and $EDD$ on the same scale for easy comparison, and include a black dashed line at $ADD=1$ to contrast bias amplification vs. deamplification. The error bars capture the range of the estimators over 25 random seeds.
Refer to caption: /html/2410.17263/assets/x7.png

Figure: Figure 8: We empirically demonstrate that bias amplification occurs and validate our theory (Theorems [2](#A14.EGx13) and [3](#Thmtheorem3)) for $ODD$, $EDD$, and $ADD$ under the setup described in Section [4.1](#S4.SS1). The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We plot $ODD$ and $EDD$ on the same scale for easy comparison, and include a black dashed line at $ADD=1$ to contrast bias amplification vs. deamplification. The error bars capture the range of the estimators over 25 random seeds.
Refer to caption: /html/2410.17263/assets/x10.png

Figure: Figure 9: We empirically demonstrate that bias amplification occurs and validate our theory (Theorems [2](#A14.EGx13) and [3](#Thmtheorem3)) for $ODD$, $EDD$, and $ADD$ under the setup described in Section [4.1](#S4.SS1). The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We plot $ODD$ and $EDD$ on the same scale for easy comparison, and include a black dashed line at $ADD=1$ to contrast bias amplification vs. deamplification. The error bars capture the range of the estimators over 25 random seeds.
Refer to caption: /html/2410.17263/assets/x13.png

### Appendix K Power-Law Covariance

To better understand how $\phi$ and the noise ratio $c$ affect bias amplification, we derive explicit phase transitions in the bias amplification profile of ridge regression with random projections in terms of these quantities. We consider the setting of power-law covariance, as it is analytically tractable and can be translated to the case of wide neural networks (Caponnetto & de Vito, ; Cui et al., ; Maloney et al., ), where the exponents can be empirically gauged. Let the eigenvalues $\lambda_{k}^{(s)}$ of $\Sigma_{s}$ have power-law decay, i.e., $\lambda_{k}^{(s)}=k^{-\beta_{s}}$, for all $k$ and some positive constants $\beta_{1}$ and $\beta_{2}$. WLOG, we will assume $\beta_{1}>\beta_{2}$. Note that $\beta_{s}$ controls the effective dimension and ultimately the difficulty of fitting the noiseless part of the signal from group $s$. If $\beta_{s}$ is large, then all the information is concentrated in a few features, and so the learning problem is easier. We similarly assume that the eigenvalues $\mu_{k}$ of $\Delta$ have power-law decay $\mu_{k}=k^{-\alpha}$, for all $k$ and constant $\alpha>0$. Finally, we consider balanced groups (i.e., $p_{1}=p_{2}=1/2$). Under this setup, we have the following corollary.

###### Corollary 2 .

Suppose $\phi<p_{2}$ and $\gamma>1$. Under the assumptions of Theorem [2](#A14.EGx13) and Assumption [4](#Thmassumption4), as $\lambda\to 0^{+}$, we have the following approximate analytical phase transitions in the bias amplification profile of ridge regression with random projections:

$$ $\displaystyle\lim_{\begin{subarray}{c}d,n_{1},n_{2}\to\infty\\ \phi_{1,2}\to 2\phi\end{subarray}}ADD\to\frac{c}{|c-1|},$ $\displaystyle\quad\lim_{c\to 0^{+}}\lim_{\begin{subarray}{c}d,n_{1},n_{2}\to\infty\\ \phi_{1,2}\to 2\phi\end{subarray}}ADD\to 0,$ (370) $\displaystyle\lim_{c\to\infty}\lim_{\begin{subarray}{c}d,n_{1},n_{2}\to\infty\\ \phi_{1,2}\to 2\phi\end{subarray}}ADD\to 1,$ $\displaystyle\quad\lim_{c\to 1}\lim_{\begin{subarray}{c}d,n_{1},n_{2}\to\infty\\ \phi_{1,2}\to 2\phi\end{subarray}}ADD\to\infty.$ (371) $$

We relegate the proof to Appendix [L](#A12) and empirically validate this result in Figure [10](#A12.F10). The phase transitions reveal that bias amplification peaks near $c=1$, bias reduction peaks when $c\to 0^{+}$, and bias amplification does not occur when $c\to\infty$. Furthermore, the right tail of the $ODD$ profile (which is proportional to $c$) is higher than the left tail (i.e., 0) for larger $c$, which aligns with our empirical findings in Section [4.1](#S4.SS1). Interestingly, in the proof of Corollary [2](#A14.EGx162), we observe that the bias term depends on $\bar{\operatorname{tr}}\,\Delta\Sigma_{s}$; therefore, the setting $\forall k,\lambda_{k}^{(s)}\geq 1/\mu_{k}$ (e.g., common in learning from synthetic data (Dohmatob et al., )) can prevent the bias term from vanishing or even cause it to explode. This may explain why iteratively training models on synthetic data (i.e., data previously generated by the models) amplifies unfairness (Wyllie et al., ).

###### Corollary 2 .

### Appendix L Proof of Corollary 2

###### Proof.

We begin by computing the $ODD$. We define $u^{(s)}_{j}=u_{j}$ for $B=\Sigma_{s}$. When $\lambda\to 0^{+}$, we can re-express the constants in Definition [3](#Thmdefinition3) in terms of the limiting spectral densities of the covariance matrices:

$$ $\displaystyle e_{1}=\frac{1}{1+\phi\int_{0}^{\infty}\frac{1}{p_{1}e_{1}+p_{2}e_{2}r}\mathrm{d}\nu(r)},e_{2}=\frac{1}{1+\phi\int_{0}^{\infty}\frac{r}{p_{1}e_{1}+p_{2}e_{2}r}\mathrm{d}\nu(r)},$ (372) $\displaystyle\tau=\frac{1}{1+\frac{1}{\gamma\tau}}=1-1/\gamma,\rho=0,$ (373) $\displaystyle u^{(1)}_{1}=\phi e_{1}^{2}\int_{0}^{\infty}\frac{u^{(1)}_{1}p_{1}+u^{(1)}_{2}p_{2}r+1}{(p_{1}e_{1}+p_{2}e_{2}r)^{2}}\mathrm{d}\nu(r),u^{(1)}_{2}=\phi e_{2}^{2}\int_{0}^{\infty}\frac{u^{(1)}_{1}p_{1}r+u^{(1)}_{2}p_{2}r^{2}+r}{(p_{1}e_{1}+p_{2}e_{2}r)^{2}}\mathrm{d}\nu(r),$ (374) $\displaystyle u^{(2)}_{1}=\phi e_{1}^{2}\int_{0}^{\infty}\frac{u^{(2)}_{1}p_{1}+u^{(2)}_{2}p_{2}r+r}{(p_{1}e_{1}+p_{2}e_{2}r)^{2}}\mathrm{d}\nu(r),u^{(2)}_{2}=\phi e_{2}^{2}\int_{0}^{\infty}\frac{u^{(2)}_{1}p_{1}r+u^{(2)}_{2}p_{2}r^{2}+r^{2}}{(p_{1}e_{1}+p_{2}e_{2}r)^{2}}\mathrm{d}\nu(r).$ (375) $$

Since $\beta_{1}>\beta_{2}$, $-\beta_{2}-(-\beta_{1})>0$. As such, for $d\to\infty$, the ratios $r_{k}=\lambda_{k}^{(2)}/\lambda_{k}^{(1)}$ have the approximate limiting distribution $\nu=\delta_{r=\infty}$, i.e., a Dirac atom at infinity. Thus:

$$ $\displaystyle e_{1}=1,e_{2}=1-\frac{\phi}{p_{2}}=1-\phi_{2},\tau=1-1/\gamma,\rho=0,$ (376) $\displaystyle u^{(1)}_{1}=0,u^{(1)}_{2}=0,u^{(2)}_{1}=0,u^{(2)}_{2}=\frac{\phi}{p_{2}(p_{2}-\phi)}.$ (377) $$

Now, we can re-express the variance terms as:

$$ $\displaystyle V_{1}(\widehat{f})$ $\displaystyle=\phi\sigma_{1}^{2}\int_{0}^{\infty}\frac{p_{1}}{(p_{1}+p_{2}e_{2}r)^{2}}\mathrm{d}\nu(r)+\phi\sigma_{2}^{2}\int_{0}^{\infty}\frac{p_{2}e_{2}r}{(p_{1}+p_{2}e_{2}r)^{2}}\mathrm{d}\nu(r)=0,$ (378) $\displaystyle V_{2}(\widehat{f})$ $\displaystyle=\phi\sigma_{1}^{2}\int_{0}^{\infty}\frac{p_{1}r+p_{1}p_{2}u_{2}^{(2)}r}{(p_{1}+p_{2}e_{2}r)^{2}}\mathrm{d}\nu(r)+\phi\sigma_{2}^{2}\int_{0}^{\infty}\frac{p_{2}e_{2}r^{2}}{(p_{1}e_{1}+p_{2}e_{2}r)^{2}}\mathrm{d}\nu(r)=\frac{\sigma_{2}^{2}\phi}{p_{2}-\phi}.$ (379) $$

Likewise, we can re-express the bias terms as:

$$ $\displaystyle B_{1}(\widehat{f})=\int_{0}^{\infty}\int_{0}^{\infty}\int_{0}^{\infty}\frac{a\delta e_{2}^{2}p_{2}^{2}r^{2}}{(e_{1}p_{1}+e_{2}p_{2}r)^{2}}\mathrm{d}\mu(r,a)d\pi(\delta)=\int_{0}^{\infty}\int_{0}^{\infty}a\delta\mathrm{\,}d\mu(a)d\pi(\delta),B_{2}(\widehat{f})=0.$ (380) $$

In this calculation, we observe that the adversarial setting $\forall k,\lambda_{k}^{(1)}\geq 1/\mu_{k}$ can prevent the bias term from vanishing. Putting these pieces together and recalling that $p_{2}=1/2$:

$$ $\displaystyle ODD\to\left|V_{1}(\widehat{f})-V_{2}(\widehat{f})\right|=\frac{2\phi\sigma_{1}^{2}}{1-2\phi}c.$ (381) $$

We now compute the $EDD$. We can once again re-express the constants in Definition [5](#Thmdefinition5) in terms of the limiting spectral densities of the covariance matrices:

$$ $\displaystyle e_{s}=\frac{1}{1+\phi_{s}/e_{s}}=1-\phi_{s},\tau_{s}=1-1/\gamma.$ (382) $$

Because $\theta_{0}=\lim_{\lambda\to 0^{+}}\lambda_{s}/(\gamma e_{s}\tau_{s})=0$, by Corollary [1](#A14.EGx147), $B_{s}(\widehat{f}_{s})=0$ and $V_{s}(\widehat{f}_{s})=\frac{\sigma_{2}^{2}\phi_{s}}{1-\phi_{s}}$.

Therefore, because $\phi=p_{s}\phi_{s}$:

$$ $\displaystyle EDD\to\left|V_{1}(\widehat{f}_{1})-V_{2}(\widehat{f}_{2})\right|=\frac{2\phi}{1-2\phi}\left|\sigma_{1}^{2}-\sigma_{2}^{2}\right|=\frac{2\phi\sigma_{1}^{2}}{1-2\phi}\left|c-1\right|,$ (383) $\displaystyle ADD\to\frac{c}{|c-1|}.$ (384) $$

∎

Figure: Figure 10: Our theory predicts that bias amplification is larger for higher noise ratios than lower noise ratios. We observe that Corollary [2](#A14.EGx162) closely predicts the bias amplification profile with respect to the noise ratio $c$. The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We plot $ODD$ and $EDD$ on the same scale for easy comparison, and include a black dashed line at $ADD=1$ to contrast bias amplification vs. deamplification. The error bars capture the range of the estimators over 25 random seeds. We consider the setup described in [K](#A11) with $\psi=0.5$, $\phi=0.2$, and $\lambda=1\times 10^{-6}$.
Refer to caption: /html/2410.17263/assets/x16.png

###### Proof.

### Appendix M Bias Amplification During Training

Figure: Figure 11: Our theory reveals that there may be an optimal regularization penalty to deamplify bias. We empirically demonstrate that bias amplification can be heavily affected by $\lambda$ and validate our theory (Theorems [2](#A14.EGx13) and [3](#Thmtheorem3)) for $ODD$, $EDD$, and $ADD$ under setup described in Section [4.2](#S4.SS2). The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We include a black dashed line at $ADD=1$ to contrast bias amplification vs. deamplification. The error bars capture the range of the estimators over 25 random seeds.
Refer to caption: /html/2410.17263/assets/x17.png

### Appendix N Minority-Group Error Plots

Figure: Figure 12: We empirically demonstrate that minority-group bias is affected by extraneous features. We validate our theory (Theorems [2](#A14.EGx13) and [3](#Thmtheorem3)) for together $R_{1},R_{2}$ (i.e., single model learned for both groups) and separate $R_{1},R_{2}$ (i.e., separate model learned per group) under the setup described in Section [4.2](#S4.SS2). The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We include a black dashed line at $ADD=1$ to contrast bias amplification vs. deamplification. All y-axes are on the same scale for easy comparison. The error bars capture the range of the estimators over 25 random seeds.
Refer to caption: /html/2410.17263/assets/x20.png

Figure: Figure 13: We empirically demonstrate that minority-group bias is affected by extraneous features. We validate our theory (Theorems [2](#A14.EGx13) and [3](#Thmtheorem3)) for together $R_{1},R_{2}$ (i.e., single model learned for both groups) and separate $R_{1},R_{2}$ (i.e., separate model learned per group) under the setup described in Section [4.2](#S4.SS2). The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We include a black dashed line at $ADD=1$ to contrast bias amplification vs. deamplification. All y-axes are on the same scale for easy comparison. The error bars capture the range of the estimators over 25 random seeds.
Refer to caption: /html/2410.17263/assets/x23.png

Figure: Figure 14: We empirically demonstrate that minority-group bias is affected by extraneous features. We validate our theory (Theorems [2](#A14.EGx13) and [3](#Thmtheorem3)) for together $R_{1},R_{2}$ (i.e., single model learned for both groups) and separate $R_{1},R_{2}$ (i.e., separate model learned per group) under the setup described in Section [4.2](#S4.SS2). The solid lines capture empirical values while the corresponding lower-opacity dashed lines represent what our theory predicts. We include a black dashed line at $ADD=1$ to contrast bias amplification vs. deamplification. All y-axes are on the same scale for easy comparison. The error bars capture the range of the estimators over 25 random seeds.
Refer to caption: /html/2410.17263/assets/x26.png

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Is the bias amplification phenomena even in a low dimensional model, such as classical linear regression, or it is a special phenomenon on a high dimensional limit?

2. In Figure 1, the EDD plot has an extreme phase transition in the lower left corner, can the authors provide more discussion about this behavior? Moreover, the extreme case seems to mismatch with the ADD plot on the right-hand side. 

3. According to Figure 11, most of the metrics are monotone with respect to lambda, how do you conclude the optimal lambda in this case?
soundness: 3
strengths: The proposed theory is solid with the exact formulation of the limiting risk, and it provides a rigorous foundation on the bias amplification phenomenon and its relationship with various components in machine learning models. The discovery is novel and the contribution is important to deepen our understanding of the bias amplification in ML.
summary: This paper considers the theoretical foundation of the bias amplification phenomenon. On a Gaussian mixture data generating distribution and ridge regression model, the authors derive the exact form of the bias and variance formula under a high dimensional limit. Using the theory, the authors show the relationship between bias amplification and the model size, label noise, number of features, regularization, and training dynamics.
weaknesses: 1. Although the authors present the exact formulation of the risk in the main text, it is complicated to understand the implications of those formulas. It would be helpful to include more discussion to explain each term to better understand the results. 

2. The paper's main contribution is to examine the bias amplification phenomenon using the formula. However, a formal statement about how different components affect the bias amplification is lacking. I would suggest the authors write them in formal theorems. 

3. It is unclear how these theoretical findings relate to real-world deep learning models, I would suggest the authors verify the conclusion about the label noise and model size on MNIST and CNN as well.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - I have mixed feelings about the analysis of bias amplification. On one hand, the results are somewhat intuitive if we assume two groups, one with $y = x$ and the other with $y = -x$. On the other hand, the theoretical results are based on isotropic covariance, which is a simplified scenario. What would happen if the covariance structure were non-isotropic, for instance, if the covariance between $X_i$ and $X_j$ fixes or exponentially decays? 

- In real-world datasets, we often lack information about the detailed properties of each group such as $\Sigma_1$. How can we assess whether a machine learning model trained on such data is prone to bias amplification?

- Even if we identify bias amplification and attribute it to heterogeneous conditional dependencies, how can we design a practical criterion to select the optimal $\lambda$ or runtime? Additionally, could the heterogeneity itself be leveraged to develop a better machine learning model?
soundness: 3
strengths: This paper examines bias amplification in ridge regression with random projections. Theoretical findings not only predict known bias amplification effects but also reveal new amplification phenomena under isotropic covariance. These findings are further validated through empirical analysis on a complex semi-synthetic image dataset.
summary: Machine learning models can capture and amplify biases present in data. This paper develops an analytical theory within the context of ridge regression to better understand and address these biases. The theory provides insights into phenomena like bias amplification and minority-group bias, revealing that optimal regularization penalties or training times may help mitigate bias. Their theoretical predictions are supported by empirical validation on diverse synthetic and semi-synthetic datasets.
weaknesses: - It appears that the analysis of bias amplification heavily relies on an implicit assumption about the conditional dependence heterogeneity (e.g., $w_2\neq w_1$) between the two groups. What would happen if this assumption were relaxed or removed?

- The manuscript does not sufficiently justify the definition of “bias amplification.” I think more theoretical backing is needed for this definition. 

- More theoretical justification is needed to demonstrate that ridge regression with random projections serves as an effective approximation for neural networks (NN). Additionally, in Line 172, the one-hidden-layer NN model seems lacks a nonlinear activation function?

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: MAJOR
1) As mentioned above, the authors should acknowledge and discuss the connection of their work to arxiv:2205.15935. Ignoring such similarity to a previous work is unacceptable for an ICLR paper. 
2) It seems plausible that much of the observed bias phenomenology in this paper is a direct effect of the double-descent / interpolation peak phenomenology, which is known to shift with the degree of over-parametrization of the network and to cause a critical sensitivity to input noise (hindering the generalization). Could the authors discuss this connection more in detail, and clarify if a different cause is at play? Is the linear activation assumption a confounding effect regarding this?
3) The imputation of the effect of early stopping in the training dynamics from the effect of a properly scaled regularization (1/t) might lead to simplistic conclusions. Can the authors discuss, for example, the connection to this previous work arXiv:2405.18296? In general, I believe mentioning optimal training time, and training dynamics both in the abstract and in the paragraph titles without specifying that this extrapolation hinges on a very simplistic approximation is misleading. 
4) Can the authors mention how, for example, their results can be used to "inform strategies to evaluate and mitigate unfairness in machine learning"?

MINORS
1) Not clear if in Fig. 1 the setting is with balanced groups, or if there is another figure for this.
2) The explanations e.g. in lines 311-317, 372-375, 400-403, 443-453, are hardly understandable if you are not extremely familiar with the model parametrization. Maybe a more effective way of presenting the results would be to state in words what setup is being considered, before looking at the parameter values.
3) I don't think it is fair practice to hide the error bars of Figure 3 in the supplementary. Moreover, the associated theoretical prediction seems very qualitative.


AFTER REVISION
Given the implemented changes, I will raise my score to a 6.
soundness: 3
strengths: I believe the topic analyzed in this paper is very relevant, and theoretical contributions in this setting are crucial to understanding how to mitigate the pressing issue of bias amplification. In this context, it is interesting to understand and discuss whether the usual recipe of overparametrization could be more harmful than expected. 
The authors' analysis seems technically sound, although the methods required the introduction of some modeling assumptions that might be simplistic in some cases.
summary: The authors introduce a simple modeling framework for understanding the phenomenon of bias amplification, i.e. the observation of abnormal disparate model performance on communities unequally represented in the dataset. The setup encompasses a regression with a linear model (both a single-layer network and a linear two-layer network are considered) on a dataset with multiple modalities and different associated rules. This framework allows them to capture some key aspects of bias amplification and to make qualitative predictions on the possible regimes where over-parametrization can help or hinder the model's unbiasedness. The analytical results are derived with the tools of "operator-valued free probability theory", and corroborated with some empirical observations on standard benchmarks.
weaknesses: My main concern about this work is its extreme similarity, in goals, modeling setup, and some of the conclusions, with previous work ("Bias-inducing geometries: an exactly solvable data model with fairness implications" https://arxiv.org/abs/2205.15935), which is ignored in the present paper. Notice that this same reference is for example cited in ref. (Bell & Sagun, 2023) within this paper. While I can recognize some differences, reasonably motivating the present extension, there should be a clear discussion of the connections and differences.

Moreover, the authors do not discuss at length the limitations of some of their assumptions. For example, using a linear network as a proxy of a deep network, or using regularization intensity to approximate early stopping and to understand the training dynamics (without looking at the training dynamics). Although the assumptions might be "standard", the limitations should be discussed also in the present work.

Finally, some of the paragraphs on the results in figures are hardly readable because constantly refer to different values of the model parameters, often without providing a clear explanation of the "meaning" of the different regimes.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Please see the **Weaknesses** section.
soundness: 3
strengths: **Originality**: This paper has originality. There are many works on model bias, but most of them are empirical work. A unified statistical theory of model bias is underexplored. Compared to previous work on model bias theory, this paper focuses on disparities in test error across majority and minority groups with different data distributions when models are trained on data from both groups. This paper does use existing tools based on random matrix theory from previous work, but the theory is not a special case of the existing ones. Therefore, I think the paper has its originality. 


**Quality**: This paper has good quality. The analysis using operator-valued free probability theory (OVFPT) and high-dimensional scaling limits look technically sound to me. Fixed-point equations and a detailed exploration of phase transitions (e.g., bias amplification at specific proportions of features to samples) are mathematically rigorous. Further, besides the theoretical analysis, this empirical section demonstrates how ridge regression models with random projections can amplify bias, even in balanced data without spurious correlations. This is an interesting finding. It offers a clear, empirical validation of theoretical predictions, demonstrating the impact of model size, label noise, and training dynamics on bias amplification. 

**Clarity**: The structure is clear. From preliminaries in section 2 to a detailed model analysis in section 3, and empirical study in section 4, they all manifest a well-organized and thoughtful presentation. 

**Significance**: This paper studies an important problem and provides a good contribution to the field. To be specific, this article tackles the critical issue of bias in machine learning, which can lead to unequal performance across demographic groups. It presents a new theoretical framework using operator-valued free probability theory to explain and predict bias amplification, unifying previous findings and offering a clearer understanding of bias mechanisms. The theory is validated in diverse empirical settings, showing practical relevance. Overall, this work makes a significant contribution by providing a structured approach to understanding and mitigating bias, with implications for fairer machine learning practices and future research directions.
summary: The paper develops a rigorous statistical framework with theory to explain and analyze model bias, particularly focusing on one of the most popular statistical methods, ridge regression, both with and without random projections (which model neural networks in a simplified regime). Its key contributions include:

Unified Theory of Bias: The paper provides a unifying theory to understand how model design choices and data distribution properties lead to bias, including phenomena like bias amplification and minority-group error. This framework explains how bias can emerge and be amplified in different feature and parameter regimes.

Bias Amplification: It analyzes how training a single model on combined data from different groups can amplify bias more than training separate models on each group’s data, even without group imbalance or spurious correlations. The theory also suggests that early stopping and tuning regularization can reduce this amplification.

Minority Group Error: The paper investigates how overparameterization and extraneous features negatively affect test performance for minority groups. It explicitly connects model size to error amplification for these groups.
Empirical Validation: The theoretical predictions are extensively validated using synthetic and semi-synthetic datasets, and the results align with empirical findings reported in prior literature.

Overall, the paper offers novel statistical theory and new insights into model bias and proposes avenues for mitigating it through informed model design and training choices.
weaknesses: There are no significant weaknesses in this paper. However, the following minor weaknesses can be improved. 

1. The connection between the theoretical analysis in this paper and the prior work in OVFPT is not discussed clearly enough. This paper mentions that the theory cannot be derived as a special case in Adlame et al., Tripuraneni et al., Lee et al., but does not provide a more involved explanation on why this is the case. It would thus be great if a paragraph discussing this point can be added.

2. Section 4.2 nicely addresses how regularization and training time influence bias amplification, but it lacks a bit of practical guidance on applying these insights in real scenario. After discussing the effects of regularization and training dynamics on bias amplification, the authors could add a bit more remarks or comments. This would be an ideal place to provide concrete guidance on setting regularization parameters or selecting optimal training durations for different dataset sizes and feature-sample ratios. For instance, they could suggest regularization ranges or early stopping criteria when $\psi$ is close to or above 1.

Meta-review: ### Summary

The paper proposes a theoretical framework for analyzing bias amplification in simple neural networks. 
Despite the framework’s simplifications, it reproduces several empirical observations from previous studies and predicts new ones.
The results are further validated with numerical simulations. 

### Strengths

Despite its simplicity, the paper reconciles various findings in the literature and explores areas previous models couldn’t, particularly overparameterization. 
Furthermore, the analysis relies on different techniques than previous theoretical papers, providing an additional technical contribution.
The results and predictions are validated with extensive numerical results.

### Weakness

The paper is highly technical and requires the reader to first familiarize themselves with the formalism before fully appreciating the results. Additionally, the theoretical framework is limited as it relies on a linear model. While random projections are used, the framework is likely insufficient to capture certain complex features that emerge from non-linearities, potentially limiting its applicability in real-world settings. 

### Reason for acceptance

The problem addressed by this paper is very relevant and there are very limited theoretical frameworks that allow us to understand these phenomena. This paper extends the range of phenomena that theory could address and it therefore represents a valuable contribution for the conference.

Decision: Accept (Poster)