Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory

## Abstract

Abstract Synthetic data has gained attention for training large language models, but poor-quality data can harm performance (see, e.g., Shumailov et al. ( 2023 ); Seddik et al. ( 2024 ) ). A potential solution is data pruning, which retains only high-quality data based on a score function (human or machine feedback). Previous work Feng et al. ( 2024 ) analyzed models trained on synthetic data as sample size increases. Using random matrix theory, we generalize this analysis and derive the performance of a binary classifier trained on a mix of real and pruned synthetic data in a high dimensional setting. Our findings identify conditions where synthetic data could improve performance, focusing on the quality of the generative model and verification strategy. We also show a smooth phase transition in synthetic label noise, contrasting with prior works on sharp transition in infinite sample limits. Our extensive experimental setup validates our theoretical results.

## 1 Introduction

The landscape of large language models (LLMs) is evolving rapidly, with a growing trend towards training models on a combination of real and synthetic data. This synthetic data is often generated by previously trained models (Allal et al., ; Ben Allal et al., ; Abdin et al., ). However, the quality of these generators can significantly impact the performance of newly trained models, potentially leading to model collapse (Shumailov et al., ), a phenomenon in which the model drastically degrades in performance.

Model collapse has been extensively studied, both empirically (Guo et al., ) and theoretically (Seddik et al., ), highlighting the potential risks associated with training on synthetic data. To mitigate these risks, researchers have proposed various strategies, including the verification of AI-synthesized data (Feng et al., ). This approach aligns with the widely adopted Reinforcement Learning from Human Feedback (RLHF) technique (Kaufmann et al., ). Feng et al. () provided theoretical support for this strategy by analyzing synthetic data as Gaussian mixtures with noisy labels, using linear binary classifiers and scalar parameters to control verifier quality. Their findings reveal a sharp performance transition: under infinite synthetic sample size conditions, model accuracy shifts from zero accuracy (due to errors in synthetic data and verification) to optimal performance as these errors decrease.

While current theoretical studies primarily focus on label noise in synthetic data (Dohmatob et al., ; Gerstgrasser et al., ; Feng et al., ), they often overlook potential distribution shifts in the feature space between real and synthetic data. This gap is particularly relevant in practical scenarios where generative models are trained on finite real data sets, potentially leading to imperfect learning of the underlying distribution.

Our paper addresses this gap by proposing a statistical model that accounts for both distribution shifts in the feature space and label noise. In our model, we induce distribution shifts in the feature space by supposing that the statistics of synthetic data are empirical estimates of the underlying real data statistics. In a finite sample size regime, these estimates may be biased, resulting in distribution shifts between real and synthetic data. Leveraging random matrix theory, we derive the theoretical performance of a binary classifier trained on a mixture of real and pruned (i.e., verified) synthetic data in a high-dimensional setting. Our analysis provides conditions under which synthetic data improves performance, emphasizing the critical roles of the generative model’s quality and the efficacy of the synthetic data verification strategy. Lastly, we show that the sharp phase transition phenomenon identified in (Feng et al., ) in the infinite sample size limit is a particular case of a general result, where smooth phase transitions can take place.

#### Summary of contributions.

Our contributions are four fold:

- •
We introduce a statistical model for studying synthetic data that accounts for label and feature noise, extending beyond previous models that only consider label noise.
- •
By leveraging random matrix theory, we characterize the performance of a binary classifier trained on a mixture of real and synthetic data in a high-dimensional setting.
- •
When training only on synthetic data, we find a smooth phase transition in classifier performance, generalizing the work of Feng et al. () on sharp transitions in infinite sample size limit.
- •
We validate our results with extensive experiments (toy example and realistic LLM setups).

## 2 Related Work

The use of synthesized data for model training has gained significant traction in recent years, particularly with the widespread adoption of large language models (LLMs) that rely on large amounts of data in their training stages. Several studies have explored the impact of synthesized data on model performance, revealing both its advantages and limitations. A primary concern is the phenomenon of model collapse Shumailov et al. (), where the iterative use of generated data for model training results in a degradation of model quality. This issue has been explored theoretically and empirically across multiple studies (e.g. Alemohammad et al. (); LeBrun et al. (); Bohacek & Farid (); Seddik et al. (); Dohmatob et al. ()).

Seddik et al. () investigated model collapse in recursive training settings, where new models are trained on data generated by previous models. They demonstrate that recursive training on purely synthetic data inevitably leads to performance degradation. However, they show that mixing real and synthetic data can attenuate model collapse, though the proportion of real data must remain high to maintain model performance. Their findings support the idea that synthesized data alone cannot sustain model quality across iterations without a significant quantity of real data.

Gerstgrasser et al. () argue that model collapse can be avoided entirely if data is accumulated rather than replaced across iterations. Their empirical studies on language models, diffusion models, and variational autoencoders indicate that accumulating both real and synthetic data helps maintain model performance over time, breaking the recursive degradation loop that leads to collapse.

The most relevant work to our study is Feng et al. () where the authors examined the effects of synthesized data on model performance in a non-recursive setting, using the concept of reinforcement through feedback to select high quality synthetic data. Their theoretical results, based on Gaussian mixture models, showed that adding feedback significantly improves the robustness of models trained on synthesized data. However, their setup assumes that only labels, not features, are noisy. Additionally, their focus is primarily on scenarios where only the number of data points,
$n$, grows to infinity. Other (practical) scenarios where for instance the feature dimension,
$p$, grows at a fixed ratio with $n$ are not covered.

Our work extends the Gaussian mixture model setup to include both noisy features and labels, which is a more realistic scenario when training on synthesized data. Additionally, we consider a high-dimensional regime where both
$p$ and
$n$ grow to infinity with a fixed ratio, a setup often used in Random Matrix Theory (RMT). This allows us to study the interaction between feature dimension, pruner error, and data size in a more comprehensive manner. Our approach also accounts for the presence of mixed data—original and synthetic—providing a more realistic framework for studying the effect of synthetic data in practical applications.

## 3 Theoretical Setup

#### Real data.

We suppose that real data consists of $n$ $p$-dimensional i.i.d. vectors ${\bm{x}}_{1},\ldots,{\bm{x}}_{n}\in{\mathbb{R}}^{p}$ sampled from a Gaussian mixture of two distinct isotropic clusters ${\mathcal{C}}_{1}$ and ${\mathcal{C}}_{2}$ of means $\pm{\bm{\mu}}$ with ${\bm{\mu}}\in{\mathbb{R}}^{p}$. Essentially, for $a\in\{1,2\}$, each data vector ${\bm{x}}_{i}\in{\mathcal{C}}_{a}$ has a corresponding label $y_{i}=(-1)^{a}$ and is sampled as

$$ $\displaystyle{\bm{x}}_{i}=y_{i}{\bm{\mu}}+{\bm{z}}_{i},\quad{\bm{z}}_{i}\sim{ \mathcal{N}}({\bm{0}},{\mathbf{I}}_{p}).$ (1) $$

#### Generative model.

To generate synthetic data, we consider the generative model corresponding to maximum likelihood which consists of estimating the underlying first and second-order statistics of the real data with their empirical estimates.
In particular, we suppose that we are given a subset $\hat{n}\leq n$ of the real dataset $({\bm{x}}_{i},y_{i})_{i=1}^{n}$ on which we can estimate the statistics. This setup allows us to model a situation where new real data samples might be available to train next-generation models and the parameter $\hat{n}$ offers control over the generative model quality. The statistics for generating synthetic data are therefore computed using the following estimates

$$ $\displaystyle\hat{\bm{\mu}}=\frac{1}{\hat{n}}\sum_{i=1}^{\hat{n}}y_{i}{\bm{x}} _{i},\quad\hat{\mathbf{C}}=\frac{1}{\hat{n}}\sum_{i=1}^{\hat{n}}\left(y_{i}{ \bm{x}}_{i}-\hat{\bm{\mu}}\right)\left(y_{i}{\bm{x}}_{i}-\hat{\bm{\mu}}\right) ^{\top}.$ (2) $$

#### Synthetic data.

We consider that synthetic data is generated as $m$ i.i.d. vectors $\tilde{\bm{x}}_{1},\ldots,\tilde{\bm{x}}_{m}\in{\mathbb{R}}^{p}$ with corresponding (noisy) labels $\tilde{y}_{1},\ldots,\tilde{y}_{m}=\pm 1$ such that $\tilde{\bm{x}}_{i}\in\tilde{\mathcal{C}}_{a}$ with true label $\bar{y}_{i}=(-1)^{a}$ for $a\in\{1,2\}$ is sampled as ($\tilde{\mathcal{C}}_{1}$ and $\tilde{\mathcal{C}}_{2}$ denote the synthetic clusters)

$$ $\displaystyle\tilde{\bm{x}}_{i}=\bar{y}_{i}\hat{\bm{\mu}}+\hat{\mathbf{C}}^{ \frac{1}{2}}\tilde{\bm{z}}_{i},\quad\tilde{\bm{z}}_{i}\sim{\mathcal{N}}({\bm{0 }},{\mathbf{I}}_{p}),$ (3) $$

and the labels $\tilde{y}_{i}$ are generated such that ${\mathbb{P}}\{\tilde{y}_{i}=\bar{y}_{i}\}=1-\varepsilon$ where $\varepsilon\geq 0$ controls label noise. Essentially, the quality of synthetic data depends on the sample size $\hat{n}$ and the label noise rate $\varepsilon$.

Figure: Figure 1: Illustration of the Marchenko-Pastur law: The histogram of eigenvalues of the empirical covariance matrix $\hat{{\mathbf{C}}}$ (as per equation (2)) using different values of $\hat{n}$. The histograms correspond to $p=500$ and $\hat{n}=500$ (in blue) and $\hat{n}=5\times 10^{4}$ (in red). The line plots depict the limiting Marchenko-Pastur law. As $\hat{n}$ grows, the distribution of eigenvalues shrinks towards $1$.
Refer to caption: x1.png

In the asymptotic regime where $\hat{n}\to\infty$ with $\frac{p}{\hat{n}}\to 0$, we can generate synthetic samples that follow asymptotically the exact same distribution as of the real ones, and therefore only label noise is relevant to the quality of the synthetic data. However, in the regime when both $\hat{n},p\to\infty$ with $\frac{p}{\hat{n}}\to\hat{\eta}>0$, while the estimation of ${\bm{\mu}}$ with $\hat{\bm{\mu}}$ remains consistent, the estimation of the covariance is not. In fact, in this regime $\|\hat{\mathbf{C}}-{\mathbf{I}}_{p}\|\not\to 0$ and the eigenvalues of $\hat{\mathbf{C}}$ spread in the vicinity of $1$ which is described in the limit by the Marchenko-Pastur law (Marchenko & Pastur, ) as depicted in Fig. 1. Eventually, such inconsistency in estimating the second moment in high dimensions yields a distribution shift between synthetic and real data, which might cause a drop in performance when training a new model on synthetic data generated with $\hat{\bm{\mu}}$ and $\hat{\mathbf{C}}$. In the remainder, we describe precisely how the performance of a simple classifier is affected in this regime.

#### Objective.

Our goal throughout the paper is to study the effect of synthetic data when training on a mixture of the $n$ real and $m$ synthetic data described above, i.e., with the following objective function

$$ ${\mathcal{L}}({\bm{w}}):=\underbrace{\frac{1}{n+m}\sum_{i=1}^{n}\ell({\bm{x}}_ {i},y_{i};{\bm{w}})}_{\text{real data}}+\underbrace{\frac{1}{n+m}\sum_{i=1}^{m }q_{i}\ell(\tilde{\bm{x}}_{i},\tilde{y}_{i};{\bm{w}})}_{\text{synthetic data}},$ (4) $$

where $\ell$ is some convex loss function and the $q_{i}$’s are data pruning parameters ($q_{i}\in\{0,1\}$), indicating whether to select or drop the $i^{th}$ synthetic sample $(\tilde{\bm{x}}_{i},\tilde{y}_{i})$. In particular, the $q_{i}$’s are Bernoulli random variables conditionally on $\tilde{y}_{i}\neq\bar{y}_{i}$ or $\tilde{y}_{i}=\bar{y}_{i}$ (we recall that $\bar{y}_{i}$’s denote the true labels of the synthetic samples) with conditional probabilities

$$ $\displaystyle\rho:={\mathbb{P}}\{q_{i}=1\mid\tilde{y}_{i}\neq\bar{y}_{i}\}, \quad\phi:={\mathbb{P}}\{q_{i}=1\mid\tilde{y}_{i}=\bar{y}_{i}\},$ (5) $$

which control the pruner accuracy (as discussed in (Feng et al., )). As we mentioned previously, we suppose training on $n\geq\hat{n}$ real data, modeling a situation where new real samples are available with $\hat{n}$ controlling the generative model quality in generating faithful synthetic features(^1^11Technically, our results hold irrespective of the statistical dependencies between the data used to train the generative model in equation 2 or the classifier in equation 6.).

#### L 2 superscript 𝐿 2 L^{2} italic_L start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT -loss.

In the remainder of the paper we take $\ell$ to be the regularized squared loss as it allows us to obtain a closed-form solution for the optimization problem in equation 4, hence, a more tractable analysis. Specifically, we take $\ell({\bm{x}},y;{\bm{w}})=({\bm{w}}^{\top}{\bm{x}}-y)^{2}+\gamma\|{\bm{w}}\|^{2}$ where $\gamma\geq 0$ is a regularisation parameter, which yields the following closed-form solution

$$ $\displaystyle{\bm{w}}=\frac{1}{N}{\mathbf{Q}}{\mathbf{X}}{\bm{y}},\quad{ \mathbf{Q}}=\left(\frac{1}{N}{\mathbf{X}}{\mathbf{X}}^{\top}+\gamma{\mathbf{I} }_{p}\right)^{-1}.$ (6) $$

where $N=n+m$, the matrix ${\mathbf{X}}=({\bm{x}}_{1},\ldots,{\bm{x}}_{n},q_{1}\tilde{\bm{x}}_{1},\ldots,
q_{m}\tilde{\bm{x}}_{m})\in{\mathbb{R}}^{p\times N}$ is the concatenation of both real and (pruned) synthetic features, and the vector ${\bm{y}}=(y_{1},\ldots,y_{n},\tilde{y}_{1},\ldots,\tilde{y}_{m})\in{\mathbb{R}
}^{N}$ is the concatenation of real and (noisy) synthetic labels.

## 4 Main Results

In this section, we present and discuss the main results obtained through the analysis of the classifier model defined in equation 6. We start by specifying the supposed growth rate assumptions.

###### Assumption 4.1 (Growth Rate) .

We consider a high-dimensional regime where $p,n,\hat{n},m\to\infty$ and we recall $N=n+m$ such that:

1) $\frac{p}{N}\to\eta\in[0,\infty)$,
2) $\frac{p}{\hat{n}}\to\hat{\eta}\in[0,\infty)$,
3) $\frac{n}{N}\to\pi\in[0,1]$,
4) $\|{\bm{\mu}}\|={\mathcal{O}}(1)$.

###### Assumption 4.1 (Growth Rate) .

#### Role of the assumptions.

The above assumptions are central to understanding the nuances between real and synthetic data (as constructed above) in a high-dimensional regime. Essentially,

- •
Assumptions 1), 2), and 3) define the scaling of data dimension $p$ and the different sample sizes ($n$ real data, $m$ synthetic data, and $\hat{n}$ real samples used to train the generative model). In particular, we suppose that all these dimensions scale linearly relative to each other, which corresponds to the classical RMT regime. This setting is more general than the infinite sample size regime in the sense that the former can be recovered by taking $\eta,\hat{\eta}\to 0$. Specifically, the parameter $\hat{\eta}$ controls the generative model quality, where lower values indicate better generative model quality.
Plus, the parameter $\pi$ corresponds to the proportion of the real samples in the data mixture. For instance, $\pi=0$ models a setting where the training is done only on synthetic samples, and $0<\pi<1$ highlights the fact that the number $n$ of real and $m$ of synthetic samples are of the same order, therefore, making our results scalable to any possible proportion $\pi$.
- •
The fourth condition about the magnitude of the mean vector ${\bm{\mu}}$ reflects the fact that the classification problem should neither be trivial ($\|{\bm{\mu}}\|\gg 1$) nor impossible ($\|{\bm{\mu}}\|\to$ 0) as the dimension of data grows large. For instance, assuming $\|{\bm{\mu}}\|$ of order $O(\sqrt{p})$ would not be relevant as $p\to\infty$ since the classification problem becomes trivial in this regime. We refer the reader to (Couillet & Benaych-Georges, ) for a more general formulation and justifications of this assumption under an extended $k$-class Gaussian mixture model.

Having stated the main assumptions, we are now in place to present our main technical findings on the performance of the classifier model trained on a mixture of real and synthetic data. As a corollary, we also cover the case where the model is trained solely on synthetic data and showcase a generalization of the result obtained by Feng et al. ().

### 4.1 Partially Synthetic: Mixture of Real and Synthetic Data

We start by analyzing the general case of training on a mix of real and synthetic data. As we described in the previous section, the statistics of synthetic data are empirical estimates of the ones of real data. Under Assumption 4.1, the estimation of ${\bm{\mu}}$ with $\hat{\bm{\mu}}$ remains consistent, while the estimation of the underlying real data covariance (i.e., ${\mathbf{I}}_{p}$ in our setting) with $\hat{\mathbf{C}}$ is inconsistent as we previously discussed. As a result, studying the theoretical performance of the classifier in equation 6 demands deploying tools from random matrix theory that refines the estimation of scalar quantities depending on large random matrices. In our case, the scalar quantity of interest corresponds to the model’s accuracy which depends on the random matrices $\hat{\mathbf{C}}$ and ${\mathbf{X}}{\mathbf{X}}^{\top}$ as per equation 6.

Figure: Figure 2: Behavior of $(\delta_{r}^{*},\delta_{s}^{*},\delta_{g}^{*})$ in terms of the ratio $\frac{p}{n}$. For small ratio $\frac{p}{n}$, the values of $\delta_{r}^{*},\delta_{s}^{*},\delta_{g}^{*}$ are close to 0 0. $(\delta_{r}^{*},\delta_{s}^{*},\delta_{g}^{*})$ are computed by iterating the system 7 starting from random values.
Refer to caption: x2.png

In our analysis of the classifier’s theoretical performance, we found that the effect of high-dimension (and that of distribution shift between real and synthetic samples) is described by three scalar quantities $(\delta_{r}^{*},\delta_{s}^{*},\delta_{g}^{*})$ which are defined as the unique solution of the following fixed point system which is derived from Lemma D.1 in the Appendix.

$$ $\begin{cases}\delta_{g}=\frac{\alpha(1-\pi)}{1+\delta_{s}}\cdot\frac{\hat{\eta }}{\gamma+\frac{\pi}{1+\delta_{r}}+\frac{\alpha(1-\pi)}{(1+\delta_{s})(1+ \delta_{g})}},\\ \delta_{r}=\frac{\eta}{\hat{\eta}}\cdot\frac{1+\delta_{s}}{\alpha(1-\pi)} \delta_{g},\\ \delta_{s}=\frac{\alpha\delta_{r}}{1+\delta_{g}}.\end{cases}$ (7) $$

where $\alpha=\phi(1-\varepsilon)+\rho\varepsilon$. These quantities will be used subsequently in our results. Intuitively, $\delta_{r}^{*}$ captures the contribution of real data, $\delta_{s}^{*}$ corresponds to the contribution of synthetic data, and $\delta_{g}^{*}$ corresponds to the influence of the generative model. In an infinite sample size regime where $n,m,\hat{n}\to\infty$ while the dimension $p$ is kept fixed, $(\delta_{r}^{*},\delta_{s}^{*},\delta_{g}^{*})=(0,0,0)$ as per Fig. 2, while under Assumption 4.1 these quantities are non zero yielding a counterintuitive behavior in high-dimension. For convenience, we further define a set of scalar quantities that will prove usefull in the next result.

$$ $\displaystyle\alpha$ $\displaystyle=\mathbb{E}[q_{i}]=\phi(1-\varepsilon)+\rho\varepsilon,\quad \lambda=\mathbb{E}[q_{i}\tilde{y}_{i}]=\phi(1-\varepsilon)-\rho\varepsilon,$ $\displaystyle a$ $\displaystyle=\frac{\pi}{1+\delta_{r}^{*}}+\frac{\alpha(1-\pi)}{1+\delta_{s}^{ *}},\quad b=\gamma+\frac{\pi}{1+\delta_{r}^{*}}+\frac{\alpha(1-\pi)}{(1+\delta _{s}^{*})(1+\delta_{g}^{*})},\quad c=\frac{\pi}{1+\delta_{r}^{*}}+\frac{ \lambda(1-\pi)}{1+\delta_{s}^{*}},$ $\displaystyle a_{1}$ $\displaystyle=\frac{\pi\eta}{(1+\delta_{r}^{*})^{2}h_{2}b^{2}},\quad b_{1}= \frac{\alpha(1-\pi)\eta}{(1+\delta_{s}^{*})^{2}(1+\delta_{g})^{2}h_{2}b^{2}}, \quad b_{2}=\frac{\alpha(1-\pi)\eta}{(1+\delta_{s}^{*})^{2}(1+\delta_{g}^{*})^ {4}h_{2}b^{2}},$ $\displaystyle h_{1}$ $\displaystyle=1-a_{1}-b_{2},\quad h_{2}=1-\left(\frac{\alpha(1-\pi)}{(1+\delta _{s}^{*})(1+\delta_{g}^{*})}\right)^{2}\frac{\hat{\eta}}{b^{2}}.$ $$

The first set of parameters $(\alpha,\lambda,a,b,c)$ pop out from the expectation of the classifier’s decision function while the remaining quantities are related to second-order statistics. Essentially, the main relevant quantities to our analysis are $\hat{\eta}$ and $\varepsilon$ which characterize the quality of synthetic data, with $\phi$ and $\rho$ characterizing the verification process.
In an idealized scenario, we would have $\hat{\eta}=\varepsilon=0$ which reflects the fact that there is no distribution shift nor label noise, while $\phi=1-\rho=1$ corresponds to a perfect (oracle) verification process.
Our main goal is to study how these parameters influence the classifier’s performance hence providing the conditions that make synthetic data relevant for performance boost. The main result brought by this paper is therefore stated as follows.

###### Theorem 4.2 (Theoretical performance) .

Let ${\bm{w}}$ be the Ridge classifier as defined in equation 6 and suppose that Assumption 4.1 holds. The decision function ${\bm{w}}^{\top}{\bm{x}}$, on some (real) test sample ${\bm{x}}\in{\mathcal{C}}_{a}$, with corresponding label $y=(-1)^{a}$ and independent of ${\mathbf{X}}$, satisfies

$$ $\displaystyle{\bm{w}}^{\top}{\bm{x}}\,\,\operatorname{\xrightarrow{{\mathcal{D }}}}\,\,{\mathcal{N}}\left(y\cdot\mu,\,\nu-m^{2}\right),$ $$

where $\mu=\frac{c\|{\bm{\mu}}\|^{2}}{b+a\|{\bm{\mu}}\|^{2}}$ and

$$ $\displaystyle\nu$ $\displaystyle=\frac{c\|{\bm{\mu}}\|^{2}}{h_{1}(b+a\|{\bm{\mu}}\|^{2})^{2}} \left(c(1+b_{1}-b_{2})\|{\bm{\mu}}\|^{2}+\frac{c}{h_{2}}-2\left(a_{1}+\frac{ \lambda b_{1}}{\alpha}\right)(b+a\|{\bm{\mu}}\|^{2})\right)+\frac{a_{1}+b_{1}} {h_{1}}.$ $$

Moreover, the asymptotic test accuracy of the classifier is given by $\Phi\left((\nu-m^{2})^{-\frac{1}{2}}m\right)$ where $\Phi(x)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{x}e^{-\frac{t^{2}}{2}}dt$.

Figure: Figure 3: Scatter plots correspond to empirical test accuracy while lines correspond to the theoretical counterpart as per Theorem 4.2. The parameters used in this experiments are: $n=\hat{n}=1000$, $\|{\bm{\mu}}\|=0.7$ and $\gamma=1$, $(\rho,\phi)=(0,1)$ for Oracle supervision and $(\rho,\phi)=(1,0.5)$ for the Weak supervision. The parameter $\varepsilon$ is variable depending on the proportion of synthetic data by taking it equal to the misclassification error corresponding to training a classifier on synthetic data only. As theoretically anticipated, a boost of performance is observed with synthetic data supervision while distribution shift affects negatively the performance.
Refer to caption: x3.png

Theorem 4.2 states that the decision function of the classifier in equation 6 is asymptotically equivalent to the thresholding of two monovariate Gaussian random variables with respective means $\mu$ and $-\mu$ and standard deviation $\nu$, where the statistics $\mu$ and $\nu$ are expressed in terms of the scalar quantities defined above. Here, $\mu$ represents the signal strength while $\nu$ highlights the classifier’s uncertainty or dispersion. To provide some insights into the implications of this theorem, we start by examining it in a low-dimensional regime where $p$ is kept fixed while $n,m,\hat{n}\to\infty$. In this case, we have $\eta,\hat{\eta}\to 0$ and $\delta_{r}^{*},\delta_{s}^{*},\delta_{g}^{*}\to 0$ which yields

$$ $\displaystyle a$ $\displaystyle=\pi+\alpha(1-\pi),\quad b=\gamma=\pi+\alpha(1-\pi),\quad c=\pi+ \lambda(1-\pi),$ $$

and $a_{1}=b_{1}=b_{2}=0$ with $h_{1}=h_{2}=1$. As such, the accuracy of the classifier increases with $\lambda$, i.e., when the synthetic labels are verified (large $\frac{\phi}{\rho}$) or less noisy (small $\varepsilon$). This is in line with the findings of Feng et al. () while extended by our result to training on a mix of real and synthetic data. However, when the dimension scales linearly with the different sample sizes, the values of $\delta_{r}^{*},\delta_{s}^{*},\delta_{g}^{*}\not\to 0$ yielding a lower signal strength $\mu$ and higher variance $\nu^{2}$. This highlights the fact that in high-dimension, even if the synthetic labels are not noisy or equivalently well verified, there is a performance drop due to the feature distribution shift between real and synthetic data.

Fig. 3 depicts the empirical test accuracy and the theoretical prediction as per Theorem 4.2 when varying the proportion of synthetic data. As theoretically anticipated, adding synthetic data does not boost the classifier’s performance unless it is verified accurately (oracle supervision versus weak supervision). Moreover, our results show the effect of the distribution shift which heavily affects performance in the case of weak supervision (Fig. 3 right).

###### Theorem 4.2 (Theoretical performance) .

### 4.2 Fully Synthetic: Training on Synthetic Data

In this section, we study the fully synthetic setting which corresponds to training solely on synthetic data (i.e. $n=0$ in equation 6). For simplicity, we consider only label noise and ignore feature noise in the synthetic data. Essentially, this allows us to exhibit the smooth phase transition of the classifier’s accuracy in terms of label noise, which extends the result of Feng et al. (). Specifically, we obtain the following corollary of theorem 4.2.

###### Corollary 4.3 (Performance when training only on synthetic data) .

Let ${\bm{w}}_{s}$ be the Ridge classifier described in equation 6 trained only on synthetic data with only label noise (i.e., $\hat{\mathbf{C}}={\mathbf{I}}_{p}$). Under Assumption 4.1, the decision function ${\bm{w}}_{s}^{\top}{\bm{x}}$ on a test sample ${\bm{x}}\in{\mathcal{C}}_{a}$ with corresponding label $y=(-1)^{a}$ and independent of ${\mathbf{X}}$, satisfies

$$ $\displaystyle{\bm{w}}_{s}^{\top}{\bm{x}}\,\,\operatorname{\xrightarrow{{ \mathcal{D}}}}\,\,{\mathcal{N}}\left(y\cdot\mu_{s},\,\nu_{s}-\mu_{s}^{2}\right),$ $$

where

$$ $\displaystyle\mu_{s}=\frac{\phi(1-\varepsilon)-\rho\varepsilon}{\alpha\|{\bm{ \mu}}\|^{2}+\alpha+\gamma(1+\delta_{s})}\|{\bm{\mu}}\|^{2},$ $\displaystyle\nu_{s}=\frac{\lambda^{2}\|{\bm{\mu}}\|^{2}}{h(\alpha\|{\bm{\mu}} \|^{2}+\alpha+\gamma(1+\delta_{s}))}\left(\frac{\|{\bm{\mu}}\|^{2}+1}{\alpha\| {\bm{\mu}}\|^{2}+\alpha+\gamma(1+\delta_{s})}-\frac{2(1-h)}{\alpha}\right)+ \frac{1-h}{h},$ $$

with

$$ $\eta_{s}=\lim_{p\to\infty}\frac{p}{m},\quad h=1-\frac{\alpha\eta_{s}}{(\alpha+ \gamma(1+\delta))^{2}},\quad\delta_{s}=\frac{\eta_{s}\alpha-\alpha-\gamma+ \sqrt{(\alpha+\gamma-\eta_{s}\alpha)^{2}+4\eta_{s}\alpha\gamma}}{2\gamma}.$ $$

Figure: Figure 4: Phase transition in terms of label noise as predicted by Corollary 4.3. The critical value for $\varepsilon$ is predicted at $\varepsilon^{*}=(1+\frac{\rho}{\phi})^{-1}$. We fix $p=100$ and vary $m$. The remaining parameters are $\|{\bm{\mu}}\|=1$, $\rho=0.3$ and $\phi=0.8$, i.e. $\varepsilon^{*}=0.73$.
Refer to caption: x4.png

Corollary 4.3 provides an explicit formulation of Theorem 4.2 with synthetic data only and ignoring distribution shift (yielding an explicit expression of $\delta_{s}$).
This setting provides a clearer interpretation of the effect of label noise since the classifier’s performance is directly related to the quantity $\lambda=\phi(1-\varepsilon)-\rho\varepsilon$. The breaking point of the classifier’s performance occurs at $\lambda=0$, which corresponds to the accuracy of random guessing, yielding to the critical value of label noise $\varepsilon^{*}=(1+\frac{\rho}{\phi})^{-1}$. This critical value is equivalent to the one obtained by Feng et al. (), however, we extend their result to the high-dimensional setting which exhibits a smoother phase transition as depicted in Fig. 4. Essentially, the sharp phase transition of Feng et al. () is covered by our result by taking $\eta_{s}\to 0$. In this sense, the predicted smooth transition better mirrors real-world scenarios where finite sample sizes introduce gradual changes in performance rather than abrupt shifts. This makes our theoretical findings more applicable and reliable for practical scenarios.

###### Corollary 4.3 (Performance when training only on synthetic data) .

## 5 Experiments

In this section, we present our experiments conducted on different real-world tasks and datasets in order to illustrate our theoretical findings presented in the previous section.

### 5.1 Experimental settings

#### Amazon Reviews.

We use the Amazon Reviews datasets (Blitzer et al. ()) which include several binary classification tasks corresponding to positive versus negative reviews of books, electronics and kitchen. We apply the standard scaler from sklearn (Pedregosa et al., ) and estimate $\|{\bm{\mu}}\|$ with the normalized data. The synthetic data is generated following the described generative scheme (see equation 2). We use the Ridge classifier in equation 6 for this data.

#### MNIST.

We also conducted experiments on the MNIST (LeCun & Cortes ()) dataset to illustrate our theoretical insights, by training a simple neural network with one-hidden layer and ReLU activation function. Concerning the synthetic data, we used different values of $\hat{n}$ to generate new samples in order to highlight the importance of the generation quality, and introduced a label noise $\varepsilon$ to emphasize on the importance of the pruning. Figure 5 shows some examples of MNIST-like synthetic data that has been generated and used in our experiments.

Figure: Figure 5: Illustration of two different generation schemes for the MNIST data. Top figure: Generating MNIST-like data samples by only estimating the mean of each class $\hat{\bm{\mu}}_{a}$ for $a\in[10]$ and without estimating the covariance matrix, i.e samples here are generated through the distribution ${\mathcal{N}}(\hat{\bm{\mu}}_{a},{\mathbf{I}}_{p})$. Bottom figure: Generating samples by estimating both the mean and covariance of each class, as of our considered generative model defined in equation 2.
Refer to caption: x5.png

#### LLM Safety Alignment.

We also investigated the impact of synthetic text data for the task of alignment of LLMs with direct preference optimization on safety datasets, using the same approach as in (Alami et al., ). We finetune the Falcon 2-11B Instruct model Malartic et al. () on $n=5000$ human data from the Anthropic’s HH-RLHF dataset(^2^22[https://huggingface.co/datasets/yimingzhang/hh-rlhf-safety](https://huggingface.co/datasets/yimingzhang/hh-rlhf-safety)), which correspond to real data, while synthetic data are extracted from the PKU safe RLHF dataset(^3^33[https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF](https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF)) which are generated using Alpaca3-70B(^4^44[https://huggingface.co/PKU-Alignment/alpaca-70b-reproduced-llama-3](https://huggingface.co/PKU-Alignment/alpaca-70b-reproduced-llama-3)). We increase the amount of synthetic data by injecting gradually five batches of $7000$ samples per batch, to study the performance of the fine-tuned model as we add more synthetic data. In this experiment, we focus only on label noise by randomly perturbing the synthetic dataset. Each entry from the synthetic dataset includes a prompt $x^{(j)}$, a safe response $y_{s_{w}}^{(j)}$ (safety-accepted response), and a less safe response $y_{s_{l}}^{(j)}$ (safety-rejected response). We, therefore, perturbed this dataset by swapping safe and less safe responses with a probability $\varepsilon$ (label noise), and selecting the prompts according to a verifier of parameters $\rho$ and $\phi$ described earlier in this paper.

For the evaluation, we use the ALERT dataset(^5^55[https://github.com/Babelscape/ALERT/blob/master/data/alert.jsonl](https://github.com/Babelscape/ALERT/blob/master/data/alert.jsonl)) (Tedeschi et al. ()) to test the safety of responses of the finetuned model after being judged by LLama-Guard-3-8B (Dubey et al., ). As in (Alami et al., ), we compute the safety score as the percentage of safe answers labeled by Llama-Guard-3-8B. We report the results in figure 8 for strong supervision $(\rho,\phi)=(0.2,0.9)$ and weak supervision $(\rho,\phi)=(0.5,0.5)$ for both $\varepsilon=0.1$ and $\varepsilon=0.5$.

#### LLM Q&A Safety Generation.

This experiment aims to evaluate the impact of synthetically generated prompts (i.e. feature noise). To construct the generative model for this experiment, we fine-tune an LLM $(M)$ with supervised fine-tuning (SFT) on pairs of question-answer (Q&A) sentences extracted from a safety dataset. Initially, we fine-tune $M$ on $12k$ human annotated Q&A as safe or unsafe (Ji et al., ), yielding a fine-tuned model on human data denoted as $M_{h}$. Then, $M_{h}$ is considered as the generative model to generate a large dataset of synthetic Q&A prompts (around $120k$ samples) that were further annotated as safe/unsafe using Mistral-Nemo(^6^66[https://mistral.ai/news/mistral-nemo/](https://mistral.ai/news/mistral-nemo/)) and Qwen2-7B-Instruct (Yang et al., ), which incorporate a further label noise. To verify the data, we use Llama-Guard-3.1 (Dubey et al., ). We conducted this experiment using two LLMs (M) of different sizes (to vary the generative model quality) which are the Llama-3.1-8B and Gemma-2-2B-it (Team et al., ) instruct models.

### 5.2 Effect of label noise

Figure: Figure 6: Results of the Amazon Reviews setting: Test Accuracy with the proportion of synthetic data evaluated on Amazon Review Blitzer et al. () dataset. The number of real data sample used is $n=800$, the dimension is $p=400$, $\gamma=10^{-1}$ and $\varepsilon=0.2$ (fixed). The pruning parameters are $(\rho,\phi)=(0,1)$ for Oracle supervision and $(\rho,\phi)=(1,1)$ for No supervision.
Refer to caption: x7.png

Figure: Figure 7: Results of the MNIST setting: Training an NN with one hidden layer and ReLU activation function on a mixture of real ($n=500$) and varying the proportion of synthetic Gaussian data.
Refer to caption: x8.png

Figures 6, 7 (left plot) 8 reflect the effect of label noise. Essentially, as theoretically anticipated, the trained models do not benefit from synthetic data unless it is accurately verified. Specifically, in the case of weak supervision, model performance drops significantly, and the improvement from using synthetic data is only visible with very high synthetic sample sizes. On the contrary, with strong supervision, we observe a monotonous performance boost as the proportion of synthetic data increases.

### 5.3 Effect of Feature noise

Figure: Figure 8: Results of LLM Safety Alignment: Strong supervision corresponds to $(\rho,\phi)=(0.2,0.9)$ and weak supervision to$(\rho,\phi)=(0.5,0.5)$.
Refer to caption: x9.png

Figure: Figure 9: Results of LLM Q&A Safety Generation: Evaluation of two LLMs trained as presented in section 5.1 is depicted for both (left) $M$= Llama3.1-8B-Instruct and (right) $M$= Gemma-2-2B-it. The test accuracy is computed over the testing dataset extracted from Ji et al. (), with $2.8k$ Q&A samples. The results shown are the average over 3 runs.
Refer to caption: x10.png

In this section, we discuss the experiments related to feature noise. In Fig. 7 (right), we depict the performance of a one-hidden layer MLP trained on a mix of real and synthetic MNIST data following our theoretical framework. As we can observe from the figure, the performance boost from synthetic data heavily depends on the generative model quality as predicted by our theoretical results. We further observe the same trend using LLMs as depicted in Fig. 9, where we observe that the synthetic data generated by Llama3.1-8B-Instruct yields a better performance boost compared to Gemma-2-2B-it as we increase the amount of synthetic samples.

## 6 Conclusion and Limitations

In this work, we conducted a comprehensive theoretical and empirical analysis of models trained on a mixture of real and synthetic data with verification. By leveraging random matrix theory, we identified critical factors such as distribution shifts and label noise that significantly impact model performance. Our findings demonstrate that synthetic data can enhance model accuracy under specific conditions, particularly when the generative model is of high quality and the verification process is accurate. Additionally, we extended previous research by showing that performance transitions are smooth rather than sharp when synthetic data is incorporated in high-dimensional settings.

Despite these advancements, our current setting is limited to label verification of synthetic data. Incorporating feature verification represents a promising extension for future research, which could provide further insights into the reliability and effectiveness of synthetic data in model training. Another possible extension of our work is to study distributions beyond the Gaussian model and analyze how higher-order statistics can be incorporated into our current framework.

In conclusion, this work provides a foundational understanding of the conditions under which synthetic data can be beneficial for model training in high-dimensional settings. By integrating both theoretical insights and empirical validations, this study provides new insights into the effective utilization of synthetic data, paving the way for more resilient and performant AI models.

## Appendix

## Appendix A Useful Lemmas

#### Notation:

For $a\in\{1,2\}$, we denote by ${\mathbb{I}}_{a}=\{i\mid{\bm{x}}_{i}\in{\mathcal{C}}_{a}\}$, i.e, the set of indices of vectors belonging to class ${\mathcal{C}}_{a}$. Furthermore, we denote $\Sigma={\bm{\mu}}{\bm{\mu}}^{\top}+{\mathbf{I}}_{p}=\mathbb{E}\left[{\bm{x}}{
\bm{x}}^{\top}\right]$ for ${\bm{x}}\in{\mathcal{C}}_{a}$, and $\Sigma_{\beta}={\bm{\mu}}_{\beta}{\bm{\mu}}_{\beta}^{\top}+{\mathbf{C}}$

Here we will list the most useful lemmas and results used in our analysis.

### A.1 General lemmas

###### Lemma A.1 (Inverse identity) .

For invertible matrices ${\mathbf{A}}$ and ${\mathbf{B}}$, we have that:

$$ ${\mathbf{A}}^{-1}-{\mathbf{B}}^{-1}={\mathbf{A}}^{-1}({\mathbf{B}}-{\mathbf{A} }){\mathbf{B}}^{-1}$ $$

###### Lemma A.2 (Woodbury) .

For ${\mathbf{A}}\in{\mathbb{R}}^{p\times p}$, ${\mathbf{U}},{\mathbf{V}}\in{\mathbb{R}}^{p\times k}$, such that both ${\mathbf{A}}$ and ${\mathbf{A}}+{\mathbf{U}}{\mathbf{V}}^{\top}$ are invertible, we have:

$$ $\left({\mathbf{A}}+{\mathbf{U}}{\mathbf{V}}^{\top}\right)^{-1}={\mathbf{A}}^{- 1}-{\mathbf{A}}^{-1}{\mathbf{U}}\left({\mathbf{I}}_{k}+{\mathbf{V}}^{\top}{ \mathbf{A}}^{-1}{\mathbf{U}}\right)^{-1}{\mathbf{V}}^{\top}{\mathbf{A}}^{-1}$ $$

A particular case of this lemma A.2, in the case of $k=1$, is called Sherman-Morisson’s identity.

###### Lemma A.3 (Sherman-Morisson) .

For ${\mathbf{A}}\in{\mathbb{R}}^{p\times p}$ invertible and ${\bm{u}},{\bm{v}}\in{\mathbb{R}}^{p}$, ${\mathbf{A}}+{\bm{u}}{\bm{v}}^{\top}$ is invertible if and only if : $1+{\bm{v}}^{\top}{\mathbf{A}}{\bm{u}}\neq 0$, and:

$$ $({\mathbf{A}}+{\bm{u}}{\bm{v}}^{\top})^{-1}={\mathbf{A}}^{-1}-\frac{{\mathbf{A }}^{-1}{\bm{u}}{\bm{v}}^{\top}{\mathbf{A}}^{-1}}{1+{\bm{v}}^{\top}{\mathbf{A}} ^{-1}{\bm{u}}}$ $$

Besides,

$$ $({\mathbf{A}}+{\bm{u}}{\bm{v}}^{\top})^{-1}{\bm{u}}=\frac{{\mathbf{A}}^{-1}{ \bm{u}}}{1+{\bm{v}}^{\top}{\mathbf{A}}^{-1}{\bm{u}}}$ $$

###### Lemma A.1 (Inverse identity) .

###### Lemma A.2 (Woodbury) .

###### Lemma A.3 (Sherman-Morisson) .

### A.2 Deterministic equivalents

Let us state here the deterministic equivalent of the resolvent matrix ${\mathbf{Q}}$ defined in the general model’s equation (6) for any general covariance matrix ${\mathbf{C}}$ and mean ${\bm{\mu}}_{\beta}=\beta{\bm{\mu}}+{\bm{\mu}}^{\perp}$ that define the statistic of the synthetic data, as in equation 13.

###### Lemma A.4 (Deterministic equivalent of ${\mathbf{Q}}$ ) .

Under the 4.1 assumptions listed above in the main paper, a deterministic equivalent for ${\mathbf{Q}}\equiv{\mathbf{Q}}(\gamma)$, denoted $\bar{\mathbf{Q}}$, is given by:

$$ $\bar{\mathbf{Q}}=\left(\frac{\pi({\bm{\mu}}{\bm{\mu}}^{\top}+{\mathbf{I}}_{p}) }{1+\delta}+\frac{\alpha(1-\pi)({\bm{\mu}}_{\beta}{\bm{\mu}}_{\beta}^{\top}+{ \mathbf{C}})}{1+\delta_{S}}+\gamma{\mathbf{I}}_{p}\right)^{-1}$ $$

where:

$$ $\pi=\frac{n}{n+m},\quad\alpha=\phi(1-\varepsilon)+\rho\varepsilon,\quad\delta= \frac{1}{N}\operatorname{Tr}(\bar{\mathbf{Q}}),\quad\delta_{S}=\frac{\alpha}{N }\operatorname{Tr}({\mathbf{C}}\bar{\mathbf{Q}})$ $$

###### Proof.

We want to find $\bar{\mathbf{Q}}$ such that for all bounded ${\bm{a}},{\bm{b}}\in{\mathbb{R}}^{p}$:

$$ ${\bm{a}}^{\top}(\mathbb{E}[{\mathbf{Q}}]-\bar{\mathbf{Q}}){\bm{b}}\to 0$ $$

Let $\bar{\mathbf{Q}}=({\mathbf{S}}+\gamma{\mathbf{I}}_{p})^{-1}$. We want to determine an ${\mathbf{S}}$ that satisfies the above property. We have that:

$$ $\displaystyle\mathbb{E}[{\mathbf{Q}}]-\bar{\mathbf{Q}}$ $\displaystyle=\mathbb{E}[{\mathbf{Q}}({\mathbf{S}}-\frac{1}{N}{\mathbf{V}}{ \mathbf{V}}^{\top})\bar{\mathbf{Q}}]\quad(\text{lemma}\ref{lemma:inverse- identity})$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\mathbb{E}[{\mathbf{Q}}({\mathbf{S}}-{ \bm{v}}_{i}{\bm{v}}_{i}^{\top})\bar{\mathbf{Q}}]$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{n}\mathbb{E}[{\mathbf{Q}}({\mathbf{S}}-{ \bm{x}}_{i}{\bm{x}}_{i}^{\top})\bar{\mathbf{Q}}]+\frac{1}{N}\sum_{i=1}^{m} \mathbb{E}[{\mathbf{Q}}({\mathbf{S}}-q_{i}\tilde{\bm{x}}_{i}\tilde{\bm{x}}_{i} ^{\top})\bar{\mathbf{Q}}]$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{n}\mathbb{E}[{\mathbf{Q}}{\mathbf{S}}- \frac{1}{1+\delta_{R}}{\mathbf{Q}}_{-{\bm{x}}_{i}}{\bm{x}}_{i}{\bm{x}}_{i}^{ \top}]\bar{\mathbf{Q}}+\frac{1}{N}\sum_{i=1}^{m}\mathbb{E}[{\mathbf{Q}}{ \mathbf{S}}-\frac{q_{i}}{1+\delta_{S}}{\mathbf{Q}}_{-\tilde{\bm{x}}_{i}}\tilde {\bm{x}}_{i}\tilde{\bm{x}}_{i}^{\top}]\bar{\mathbf{Q}}$ $\displaystyle=\pi\mathbb{E}[{\mathbf{Q}}{\mathbf{S}}-\frac{1}{1+\delta_{R}}{ \mathbf{Q}}_{-{\bm{x}}_{i}}{\bm{x}}_{i}{\bm{x}}_{i}^{\top}]\bar{\mathbf{Q}}+(1 -\pi)\mathbb{E}[{\mathbf{Q}}{\mathbf{S}}-\frac{q_{i}}{1+\delta_{S}}{\mathbf{Q} }_{-\tilde{\bm{x}}_{i}}\tilde{\bm{x}}_{i}\tilde{\bm{x}}_{i}^{\top}]\bar{ \mathbf{Q}}$ $\displaystyle=\pi\mathbb{E}[{\mathbf{Q}}_{-{\bm{x}}_{i}}({\mathbf{S}}-\frac{{ \bm{x}}_{i}{\bm{x}}_{i}^{\top}}{1+\delta_{R}})\bar{\mathbf{Q}}]+(1-\pi)\mathbb {E}[{\mathbf{Q}}_{-\tilde{\bm{x}}_{i}}({\mathbf{S}}-\frac{q_{i}\tilde{\bm{x}}_ {i}\tilde{\bm{x}}_{i}^{\top}}{1+\delta_{S}})\bar{\mathbf{Q}}]+{\mathcal{O}}(N^ {-1})$ $$

Thus, it suffices to have:

$$ ${\mathbf{S}}=\frac{\pi({\bm{\mu}}{\bm{\mu}}^{\top}+{\mathbf{I}}_{p})}{1+\delta _{R}}+\frac{\alpha(1-\pi)({\bm{\mu}}_{\beta}{\bm{\mu}}_{\beta}^{\top}+{\mathbf {C}})}{1+\delta_{S}}$ $$

to get the desired property.
∎

###### Lemma A.5 (Deterministic equivalent of ${\mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}$ ) .

Let ${\mathbf{A}}\in{\mathbb{R}}^{p\times p}$ be any deterministic symmetric semi-definite matrix. We have that:

$$ ${\mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}\leftrightarrow\bar{\mathbf{Q}}{\mathbf{A} }\bar{\mathbf{Q}}+\frac{\pi}{N(1+\delta)^{2}}\operatorname{Tr}(\Sigma\bar{ \mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}})\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{ Q}}]+\frac{\alpha(1-\pi)}{N(1+\delta_{S})^{2}}\operatorname{Tr}(\Sigma_{\beta} \bar{\mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}})\mathbb{E}[{\mathbf{Q}}\Sigma_{ \beta}{\mathbf{Q}}]$ $$

Thus, we get that for ${\mathbf{A}}=\Sigma$, and for ${\mathbf{A}}=\Sigma_{\beta}$:

$$ $\displaystyle{\mathbf{Q}}\Sigma{\mathbf{Q}}\leftrightarrow\bar{\mathbf{Q}} \Sigma\bar{\mathbf{Q}}+\frac{\pi}{N(1+\delta)^{2}}\operatorname{Tr}((\Sigma \bar{\mathbf{Q}})^{2})\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]+\frac{\alpha( 1-\pi)}{N(1+\delta_{S})^{2}}\operatorname{Tr}(\Sigma_{\beta}\bar{\mathbf{Q}} \Sigma\bar{\mathbf{Q}})\mathbb{E}[{\mathbf{Q}}\Sigma_{\beta}{\mathbf{Q}}]$ $\displaystyle{\mathbf{Q}}\Sigma_{\beta}{\mathbf{Q}}\leftrightarrow\bar{\mathbf {Q}}\Sigma_{\beta}\bar{\mathbf{Q}}+\frac{\pi}{N(1+\delta)^{2}}\operatorname{Tr }(\Sigma\bar{\mathbf{Q}}\Sigma_{\beta}\bar{\mathbf{Q}})\mathbb{E}[{\mathbf{Q}} \Sigma{\mathbf{Q}}]+\frac{\alpha(1-\pi)}{N(1+\delta_{S})^{2}}\operatorname{Tr} ((\Sigma_{\beta}\bar{\mathbf{Q}})^{2})\mathbb{E}[{\mathbf{Q}}\Sigma_{\beta}{ \mathbf{Q}}]$ $$

And by denoting:

$$ $\displaystyle a_{1}=\frac{\pi}{N(1+\delta)^{2}}\operatorname{Tr}((\Sigma\bar{ \mathbf{Q}})^{2}),\quad b_{1}=\frac{\alpha(1-\pi)}{N(1+\delta_{S})^{2}} \operatorname{Tr}(\Sigma_{\beta}\bar{\mathbf{Q}}\Sigma\bar{\mathbf{Q}}),$ $\displaystyle a_{2}=\frac{\pi}{N(1+\delta)^{2}}\operatorname{Tr}(\Sigma_{\beta }\bar{\mathbf{Q}}\Sigma\bar{\mathbf{Q}}),\quad b_{2}=\frac{\alpha(1-\pi)}{N(1+ \delta_{S})^{2}}\operatorname{Tr}((\Sigma_{\beta}\bar{\mathbf{Q}})^{2})$ $\displaystyle h=(1-b_{2})(1-a_{1})-a_{2}b_{1}$ $$

We get that:

$$ $\displaystyle{\mathbf{Q}}\Sigma{\mathbf{Q}}\leftrightarrow\frac{1-b_{2}}{h} \bar{\mathbf{Q}}\Sigma\bar{\mathbf{Q}}+\frac{b_{1}}{h}\bar{\mathbf{Q}}\Sigma_{ \beta}\bar{\mathbf{Q}},$ $\displaystyle{\mathbf{Q}}\Sigma_{\beta}{\mathbf{Q}}\leftrightarrow\frac{a_{2}} {h}\bar{\mathbf{Q}}\Sigma\bar{\mathbf{Q}}+\frac{1-a_{1}}{h}\bar{\mathbf{Q}} \Sigma_{\beta}\bar{\mathbf{Q}}.$ $$

###### Proof.

Recall that:

$$ $\bar{\mathbf{Q}}(\gamma)=\left(\frac{\pi\Sigma}{1+\delta}+\frac{\alpha(1-\pi) \Sigma_{\beta}}{1+\delta_{S}}+\gamma{\mathbf{I}}_{p}\right)^{-1}$ $$

Let us denote by : ${\mathbf{S}}=\frac{\pi\Sigma}{1+\delta}+\frac{\alpha(1-\pi)\Sigma_{\beta}}{1+
\delta_{S}}$, so that: $\bar{\mathbf{Q}}=\left({\mathbf{S}}+\gamma{\mathbf{I}}_{p}\right)^{-1}$.
We have that:

$$ $\displaystyle\mathbb{E}[{\mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}]$ $\displaystyle=\mathbb{E}[\bar{\mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}]+\mathbb{E}[ ({\mathbf{Q}}-\bar{\mathbf{Q}}){\mathbf{A}}{\mathbf{Q}}]$ $\displaystyle=\bar{\mathbf{Q}}\mathbb{E}[{\mathbf{A}}{\mathbf{Q}}]+\mathbb{E}[ ({\mathbf{Q}}-\bar{\mathbf{Q}}){\mathbf{A}}{\mathbf{Q}}]$ $\displaystyle=\bar{\mathbf{Q}}\left(\mathbb{E}[{\mathbf{A}}\bar{\mathbf{Q}}]+ \mathbb{E}[{\mathbf{A}}({\mathbf{Q}}-\bar{\mathbf{Q}})]\right)+\mathbb{E}[({ \mathbf{Q}}-\bar{\mathbf{Q}}){\mathbf{A}}{\mathbf{Q}}]$ $\displaystyle=\bar{\mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}}+\mathbb{E}[({ \mathbf{Q}}-\bar{\mathbf{Q}}){\mathbf{A}}{\mathbf{Q}}]$ $\displaystyle=\bar{\mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}}+\mathbb{E}[{\mathbf {Q}}\left({\mathbf{S}}-\frac{1}{N}{\mathbf{V}}{\mathbf{V}}^{\top}\right){ \mathbf{A}}{\mathbf{Q}}]$ $\displaystyle=\bar{\mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}}+\mathbb{E}[{\mathbf {Q}}{\mathbf{S}}\bar{\mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}]-\frac{1}{N}\sum_{i=1 }^{N}\mathbb{E}[{\mathbf{Q}}{\bm{v}}_{i}{\bm{v}}_{i}^{\top}\bar{\mathbf{Q}}{ \mathbf{A}}{\mathbf{Q}}]$ $\displaystyle=\bar{\mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}}+\mathbb{E}[{\mathbf {Q}}{\mathbf{S}}\bar{\mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}]-\pi\mathbb{E}[{ \mathbf{Q}}{\bm{x}}_{i}{\bm{x}}_{i}^{\top}\bar{\mathbf{Q}}{\mathbf{A}}{\mathbf {Q}}]-(1-\pi)\mathbb{E}[{\mathbf{Q}}q_{i}\tilde{\bm{x}}_{i}\tilde{\bm{x}}_{i}^ {\top}\bar{\mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}]$ $$

And we have that:

$$ $\displaystyle\mathbb{E}[{\mathbf{Q}}{\bm{x}}_{i}{\bm{x}}_{i}^{\top}\bar{ \mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}]=\frac{1}{1+\delta}\mathbb{E}[{\mathbf{Q}} _{-x_{i}}{\bm{x}}_{i}{\bm{x}}_{i}^{\top}\bar{\mathbf{Q}}{\mathbf{A}}{\mathbf{Q }}]$ $\displaystyle=\frac{1}{1+\delta}\mathbb{E}\left[{\mathbf{Q}}_{-x_{i}}{\bm{x}}_ {i}{\bm{x}}_{i}^{\top}\bar{\mathbf{Q}}{\mathbf{A}}\left({\mathbf{Q}}_{-x_{i}}- \frac{{\mathbf{Q}}_{-x_{i}}{\bm{x}}_{i}{\bm{x}}_{i}^{\top}{\mathbf{Q}}_{-x_{i} }}{N(1+\delta)}\right)\right]$ $\displaystyle=\frac{1}{1+\delta}\mathbb{E}[{\mathbf{Q}}_{-x_{i}}{\bm{x}}_{i}{ \bm{x}}_{i}^{\top}\bar{\mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}_{-x_{i}}]-\frac{1}{ N(1+\delta)^{2}}\mathbb{E}[{\mathbf{Q}}_{-x_{i}}{\bm{x}}_{i}{\bm{x}}_{i}^{\top }\bar{\mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}_{-x_{i}}{\bm{x}}_{i}{\bm{x}}_{i}^{ \top}{\mathbf{Q}}_{-x_{i}}]$ $\displaystyle=\frac{1}{1+\delta}\mathbb{E}[{\mathbf{Q}}\Sigma\bar{\mathbf{Q}}{ \mathbf{A}}{\mathbf{Q}}]-\frac{1}{N(1+\delta)^{2}}\operatorname{Tr}(\Sigma\bar {\mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}})\mathbb{E}[{\mathbf{Q}}_{-x_{i}}{\bm{ x}}_{i}{\bm{x}}_{i}^{\top}{\mathbf{Q}}_{-x_{i}}]$ $\displaystyle=\frac{1}{1+\delta}\mathbb{E}[{\mathbf{Q}}\Sigma\bar{\mathbf{Q}}{ \mathbf{A}}{\mathbf{Q}}]-\frac{1}{N(1+\delta)^{2}}\operatorname{Tr}(\Sigma\bar {\mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}})\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf {Q}}]$ $$

And:

$$ $\displaystyle\mathbb{E}[q_{i}{\mathbf{Q}}\tilde{\bm{x}}_{i}\tilde{\bm{x}}_{i}^ {\top}\bar{\mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}]=\frac{1}{1+\delta_{S}}\mathbb{ E}[q_{i}{\mathbf{Q}}_{-\tilde{x}_{i}}\tilde{\bm{x}}_{i}\tilde{\bm{x}}_{i}^{ \top}\bar{\mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}]$ $\displaystyle=\frac{1}{1+\delta_{S}}\mathbb{E}\left[q_{i}{\mathbf{Q}}_{-\tilde {x}_{i}}\tilde{\bm{x}}_{i}\tilde{\bm{x}}_{i}^{\top}\bar{\mathbf{Q}}{\mathbf{A} }\left({\mathbf{Q}}_{-\tilde{x}_{i}}-\frac{q_{i}{\mathbf{Q}}_{-\tilde{x}_{i}} \tilde{\bm{x}}_{i}\tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}_{-\tilde{x}_{i}}}{N(1+ \delta_{S})}\right)\right]$ $\displaystyle=\frac{1}{1+\delta_{S}}\mathbb{E}[q_{i}{\mathbf{Q}}_{-\tilde{x}_{ i}}\tilde{\bm{x}}_{i}\tilde{\bm{x}}_{i}^{\top}\bar{\mathbf{Q}}{\mathbf{A}}{ \mathbf{Q}}_{-\tilde{x}_{i}}]-\frac{1}{N(1+\delta_{S})^{2}}\mathbb{E}[q_{i}{ \mathbf{Q}}_{-\tilde{x}_{i}}\tilde{\bm{x}}_{i}\tilde{\bm{x}}_{i}^{\top}\bar{ \mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}_{-\tilde{x}_{i}}\tilde{\bm{x}}_{i}\tilde{ \bm{x}}_{i}^{\top}{\mathbf{Q}}_{-\tilde{x}_{i}}]$ $\displaystyle=\frac{\alpha}{1+\delta_{S}}\mathbb{E}[{\mathbf{Q}}\Sigma_{\beta} \bar{\mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}]-\frac{\alpha}{N(1+\delta_{S})^{2}} \operatorname{Tr}(\Sigma_{\beta}\bar{\mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}}) \mathbb{E}[{\mathbf{Q}}_{-\tilde{x}_{i}}\tilde{\bm{x}}_{i}\tilde{\bm{x}}_{i}^{ \top}{\mathbf{Q}}_{-x_{i}}]$ $\displaystyle=\frac{\alpha}{1+\delta_{S}}\mathbb{E}[{\mathbf{Q}}\Sigma_{\beta} \bar{\mathbf{Q}}{\mathbf{A}}{\mathbf{Q}}]-\frac{\alpha}{N(1+\delta_{S})^{2}} \operatorname{Tr}(\Sigma_{\beta}\bar{\mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}}) \mathbb{E}[{\mathbf{Q}}\Sigma_{\beta}{\mathbf{Q}}]$ $$

Which concludes the proof by summing all these separate terms.
∎

###### Corollary A.6 (Trace identities) .

Using the above lemma A.5, we get that:

$$ $\frac{1}{N}\operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]) =\frac{(1+\delta)^{2}}{\pi h}\left(a_{1}(1-b_{2})+a_{2}b_{1}\right),\quad\frac {1}{N}\operatorname{Tr}(\Sigma_{\beta}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q} }])=\frac{(1+\delta_{S})^{2}}{\alpha(1-\pi)h}b_{1}$ $$

And in the case of isotropic covariance matrix: ${\mathbf{C}}=\sigma^{2}{\mathbf{I}}_{p}$:

$$ $\frac{1}{N}\operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]) =\frac{\eta}{h\theta^{2}}(1-b_{2}+\sigma^{2}b_{1}),\quad\frac{1}{N} \operatorname{Tr}(\Sigma_{\beta}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}])= \frac{\sigma^{2}}{N}\operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}\Sigma{ \mathbf{Q}}])$ $$

###### Lemma A.4 (Deterministic equivalent of 𝐐 𝐐 {\mathbf{Q}} bold_Q ) .

###### Proof.

###### Lemma A.5 (Deterministic equivalent of 𝐐𝐀𝐐 𝐐𝐀𝐐 {\mathbf{Q}}{\mathbf{A}}{\mathbf{Q}} bold_QAQ ) .

###### Proof.

###### Corollary A.6 (Trace identities) .

### A.3 Resolvent identities

Let ${\mathbf{Q}}$ be the resolvent matrix defined in equation (6). Denote by ${\mathbf{Q}}_{-{\bm{v}}_{i}}$ the resolvent matrix obtained from the dataset ${\mathbf{V}}$ by removing the $i^{th}$ sample ${\bm{v}}_{i}$, i.e:

$$ ${\mathbf{Q}}_{-{\bm{v}}_{i}}=\left({\mathbf{Q}}^{-1}-\frac{1}{N}{\bm{v}}_{i}{ \bm{v}}_{i}^{\top}\right)^{-1}$ $$

Then, using lemma A.3, we have that:

$$ ${\mathbf{Q}}={\mathbf{Q}}_{-{\bm{v}}_{i}}-\frac{{\mathbf{Q}}_{-{\bm{v}}_{i}} \frac{1}{N}{\bm{v}}_{i}{\bm{v}}_{i}^{\top}{\mathbf{Q}}_{-i}}{1+\frac{1}{N}{\bm {v}}_{i}^{\top}{\mathbf{Q}}_{-{\bm{v}}_{i}}{\bm{v}}_{i}},$ $$

and,

$$ ${\mathbf{Q}}{\bm{x}}_{i}=\frac{{\mathbf{Q}}_{-{\bm{x}}_{i}}{\bm{x}}_{i}}{1+ \delta},\quad{\mathbf{Q}}\tilde{\bm{x}}_{i}=\frac{{\mathbf{Q}}_{-\tilde{\bm{x} }_{i}}\tilde{\bm{x}}_{i}}{1+\delta_{S}},$ (8) $$

where:

$$ $\delta=\frac{1}{N}\operatorname{Tr}(\Sigma\bar{\mathbf{Q}})=\frac{1}{N} \operatorname{Tr}(\bar{\mathbf{Q}}),\quad\delta_{S}=\left(\phi(1-\varepsilon)+ \rho\varepsilon\right)\frac{1}{N}\operatorname{Tr}(\Sigma_{\beta}\bar{\mathbf{ Q}})=\frac{\alpha}{N}\operatorname{Tr}({\mathbf{C}}\bar{\mathbf{Q}})$ (9) $$

Let us recall the expression of $\bar{\mathbf{Q}}$ defined in lemma A.4:

$$ $\displaystyle\bar{\mathbf{Q}}$ $\displaystyle=\left(\frac{\pi({\bm{\mu}}{\bm{\mu}}^{\top}+{\mathbf{I}}_{p})}{1 +\delta}+\frac{\alpha(1-\pi)({\bm{\mu}}_{\beta}{\bm{\mu}}_{\beta}^{\top}+{ \mathbf{C}})}{1+\delta_{S}}+\gamma{\mathbf{I}}_{p}\right)^{-1}$ $\displaystyle=\left({\mathbf{A}}+{\mathbf{U}}{\mathbf{U}}^{\top}\right)^{-1}$ $$

where:

$$ ${\mathbf{A}}=\frac{\alpha(1-\pi)}{1+\delta_{S}}{\mathbf{C}}+\left(\gamma+\frac {\pi}{1+\delta}\right){\mathbf{I}}_{p},\quad{\mathbf{U}}=\left(\sqrt{\frac{\pi }{1+\delta}}{\bm{\mu}},\sqrt{\frac{\alpha(1-\pi)}{1+\delta_{S}}}{\bm{\mu}}_{ \beta}\right)$ (10) $$

Since ${\mathbf{C}}$ is symmetric and real valued, then it is diagonalizable, and can be written as:

$$ ${\mathbf{C}}={\mathbf{P}}{\mathbf{D}}{\mathbf{P}}^{\top}$ $$

where: ${\mathbf{P}}^{-1}={\mathbf{P}}^{\top}$ is the matrix containing the eigenvectors of ${\mathbf{C}}$ in its columns, and ${\mathbf{D}}=Diag((d_{i})_{i=1}^{p})$ the diagonal matrix of the eigenvalues of ${\mathbf{C}}$. Hence, ${\mathbf{A}}$ can be written as:

$$ ${\mathbf{A}}={\mathbf{P}}\Delta{\mathbf{P}}^{\top},\quad\Delta=Diag\left( \gamma+\frac{\pi}{1+\delta}+\frac{\alpha(1-\pi)}{1+\delta_{S}}d_{i}\right)_{i= 1}^{p}$ (11) $$

And using Woodbury’s identity in lemma A.2, we get that:

$$ $\bar{\mathbf{Q}}={\mathbf{A}}^{-1}-{\mathbf{A}}^{-1}{\mathbf{U}}\left({\mathbf {I}}_{2}+{\mathbf{U}}^{\top}{\mathbf{A}}^{-1}{\mathbf{U}}\right)^{-1}{\mathbf{ U}}^{\top}{\mathbf{A}}^{-1}$ $$

where: ${\mathbf{A}}^{-1}={\mathbf{P}}\Delta^{-1}{\mathbf{P}}^{\top}$ and $\Delta^{-1}=Diag\left(\frac{1}{\gamma+\frac{\pi}{1+\delta}+\frac{\alpha(1-\pi)
}{1+\delta_{S}}d_{i}}\right)_{i=1}^{p}$.
Let ${\mathbf{M}}=\left({\mathbf{I}}_{2}+{\mathbf{U}}^{\top}{\mathbf{A}}^{-1}{
\mathbf{U}}\right)^{-1}$, and denote by $M_{i,j}$ its coordinate in the $i^{\text{th}}$ row and $j^{\text{th}}$ column. We have that:

$$ $\displaystyle\bar{\mathbf{Q}}$ $\displaystyle={\mathbf{A}}^{-1}-{\mathbf{A}}^{-1}{\mathbf{U}}{\mathbf{M}}{ \mathbf{U}}^{\top}{\mathbf{A}}^{-1}$ $\displaystyle={\mathbf{A}}^{-1}-{\mathbf{A}}^{-1}\left(\zeta_{1}{\bm{\mu}}{\bm {\mu}}^{\top}+\zeta_{2}{\bm{\mu}}_{\beta}{\bm{\mu}}_{\beta}^{\top}+\zeta_{3}({ \bm{\mu}}{\bm{\mu}}_{\beta}^{\top}+{\bm{\mu}}_{\beta}{\bm{\mu}}^{\top})\right) {\mathbf{A}}^{-1}$ $$

where:

$$ $\zeta_{1}=\frac{\pi M_{1,1}}{1+\delta},\quad\zeta_{2}=\frac{\alpha(1-\pi)M_{2, 2}}{1+\delta_{S}},\quad\zeta_{3}=\sqrt{\frac{\alpha\pi(1-\pi)}{(1+\delta)(1+ \delta_{S})}}M_{1,2}$ $$

Thus,

$$ $\bar{\mathbf{Q}}={\mathbf{A}}^{-1}-{\mathbf{A}}^{-1}\left(\zeta_{1}{\bm{\mu}}{ \bm{\mu}}^{\top}+\zeta_{2}{\bm{\mu}}_{\beta}{\bm{\mu}}_{\beta}^{\top}+\zeta_{3 }({\bm{\mu}}{\bm{\mu}}_{\beta}^{\top}+{\bm{\mu}}_{\beta}{\bm{\mu}}^{\top}) \right){\mathbf{A}}^{-1}$ (12) $$

We can further show that:

$$ $\displaystyle M_{1,1}=\frac{1}{\text{det}(M^{-1})}\left(1+\frac{\alpha(1-\pi)} {1+\delta_{S}}{\bm{\mu}}_{\beta}^{\top}{\mathbf{A}}^{-1}{\bm{\mu}}_{\beta}\right)$ $\displaystyle M_{1,2}=\frac{1}{\text{det}(M^{-1})}\left(-\sqrt{\frac{\alpha\pi (1-\pi)}{(1+\delta)(1+\delta_{S})}}{\bm{\mu}}^{\top}{\mathbf{A}}^{-1}{\bm{\mu} }_{\beta}\right)$ $\displaystyle M_{2,1}=\frac{1}{\text{det}(M^{-1})}\left(-\sqrt{\frac{\alpha\pi (1-\pi)}{(1+\delta)(1+\delta_{S})}}{\bm{\mu}}^{\top}{\mathbf{A}}^{-1}{\bm{\mu} }_{\beta}\right)$ $\displaystyle M_{2,2}=\frac{1}{\text{det}(M^{-1})}\left(1+\frac{\pi}{1+\delta} {\bm{\mu}}^{\top}{\mathbf{A}}^{-1}{\bm{\mu}}\right)$ $\displaystyle\text{det}(M^{-1})=\left(1+\frac{\pi}{1+\delta}{\bm{\mu}}^{\top}{ \mathbf{A}}^{-1}{\bm{\mu}}\right)\left(1+\frac{\alpha(1-\pi)}{1+\delta_{S}}{ \bm{\mu}}_{\beta}^{\top}{\mathbf{A}}^{-1}{\bm{\mu}}_{\beta}\right)-\frac{ \alpha\pi(1-\pi)}{(1+\delta)(1+\delta_{S})}({\bm{\mu}}^{\top}{\mathbf{A}}^{-1} {\bm{\mu}}_{\beta})^{2}$ $$

###### Lemma A.7 (Delta) .

The parameters $\delta$ and $\delta_{S}$ defined in equation 9, are given by the following identities:

$$ $\delta=\frac{1}{N}\sum_{i=1}^{p}\frac{1}{\gamma+\frac{\pi}{1+\delta}+\frac{ \alpha(1-\pi)}{1+\delta_{S}}d_{i}},\quad\delta_{S}=\frac{\alpha}{N}\sum_{i=1}^ {p}\frac{d_{i}}{\gamma+\frac{\pi}{1+\delta}+\frac{\alpha(1-\pi)}{1+\delta_{S}} d_{i}}$ $$

where: $(d_{i})_{i=1}^{p}$ are the eigenvalues of the covariance matrix ${\mathbf{C}}$.

###### Proof.

Let: ${\mathbf{M}}=\left({\mathbf{I}}_{2}+{\mathbf{U}}^{\top}{\mathbf{A}}^{-1}{
\mathbf{U}}\right)^{-1}$, and denote by $M_{i,j}$ its coordinate in the $i^{\text{th}}$ row and $j^{\text{th}}$ column. We have that using the expression of $\bar{\mathbf{Q}}$ in equation 12:

$$ $\displaystyle\bar{\mathbf{Q}}={\mathbf{A}}^{-1}-{\mathbf{A}}^{-1}\left(\zeta_{ 1}{\bm{\mu}}{\bm{\mu}}^{\top}+\zeta_{2}{\bm{\mu}}_{\beta}{\bm{\mu}}_{\beta}^{ \top}+\zeta_{3}({\bm{\mu}}{\bm{\mu}}_{\beta}^{\top}+{\bm{\mu}}_{\beta}{\bm{\mu }}^{\top})\right){\mathbf{A}}^{-1}$ $$

Then:

$$ $\displaystyle\delta$ $\displaystyle=\frac{1}{N}\operatorname{Tr}(\bar{\mathbf{Q}})$ $\displaystyle=\frac{1}{N}\operatorname{Tr}({\mathbf{A}}^{-1})-\frac{1}{N} \operatorname{Tr}({\mathbf{A}}^{-1}\left(\zeta_{1}{\bm{\mu}}{\bm{\mu}}^{\top}+ \zeta_{2}{\bm{\mu}}_{\beta}{\bm{\mu}}_{\beta}^{\top}+\zeta_{3}({\bm{\mu}}{\bm{ \mu}}_{\beta}^{\top}+{\bm{\mu}}_{\beta}{\bm{\mu}}^{\top})\right){\mathbf{A}}^{ -1})$ $$

We have that, when $N\to\infty$:

$$ $\displaystyle\frac{1}{N}\operatorname{Tr}({\mathbf{A}}^{-1}{\bm{\mu}}{\bm{\mu} }^{\top}{\mathbf{A}}^{-1})=\frac{1}{N}{\bm{\mu}}^{\top}({\mathbf{A}}^{-1})^{2} {\bm{\mu}}={\mathcal{O}}(N^{-1})$ $$

since $\|{\bm{\mu}}\|={\mathcal{O}}(N^{-1})$ by assumption 4.1. The same applies for ${\bm{\mu}}_{\beta}$. Thus:

$$ $\displaystyle\delta$ $\displaystyle=\frac{1}{N}\operatorname{Tr}({\mathbf{A}}^{-1})-\frac{1}{N}\zeta _{1}{\bm{\mu}}^{\top}({\mathbf{A}}^{-1})^{2}{\bm{\mu}}-\frac{1}{N}\zeta_{2}{ \bm{\mu}}_{\beta}^{\top}({\mathbf{A}}^{-1})^{2}{\bm{\mu}}_{\beta}-\frac{2}{N} \zeta_{3}{\bm{\mu}}^{\top}({\mathbf{A}}^{-1})^{2}{\bm{\mu}}_{\beta}$ $\displaystyle=\frac{1}{N}\operatorname{Tr}({\mathbf{A}}^{-1})+{\mathcal{O}}(N^ {-1})$ $\displaystyle=\frac{1}{N}\operatorname{Tr}(\Delta^{-1})+{\mathcal{O}}(N^{-1})$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{p}\frac{1}{\gamma+\frac{\pi}{1+\delta}+ \frac{\alpha(1-\pi)}{1+\delta_{S}}d_{i}}+{\mathcal{O}}(N^{-1})$ $$

Hence we have the desired result for $\delta$ in the regime $N\gg 1$ which we considered in our assumption 4.1.
Similarly for $\delta_{S}$, we have that:

$$ $\displaystyle\frac{1}{\alpha}\delta_{S}$ $\displaystyle=\frac{1}{n}\operatorname{Tr}({\mathbf{C}}\bar{\mathbf{Q}})$ $\displaystyle=\frac{1}{N}\operatorname{Tr}({\mathbf{C}}{\mathbf{A}}^{-1})- \frac{1}{N}\zeta_{1}{\bm{\mu}}^{\top}{\mathbf{A}}^{-1}{\mathbf{C}}{\mathbf{A}} ^{-1}{\bm{\mu}}-\frac{1}{N}\zeta_{2}{\bm{\mu}}_{\beta}^{\top}{\mathbf{A}}^{-1} {\mathbf{C}}{\mathbf{A}}^{-1}{\bm{\mu}}_{\beta}-\frac{2}{N}\zeta_{3}{\bm{\mu}} ^{\top}{\mathbf{A}}^{-1}{\mathbf{C}}{\mathbf{A}}^{-1}{\bm{\mu}}_{\beta}$ $\displaystyle=\frac{1}{N}\operatorname{Tr}({\mathbf{C}}{\mathbf{A}}^{-1})+{ \mathcal{O}}(N^{-1})$ $\displaystyle=\frac{1}{N}\operatorname{Tr}({\mathbf{D}}\Delta^{-1})+{\mathcal{ O}}(N^{-1})$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{p}\frac{d_{i}}{\gamma+\frac{\pi}{1+\delta }+\frac{\alpha(1-\pi)}{1+\delta_{S}}d_{i}}+{\mathcal{O}}(N^{-1})$ $$

Which concludes our proof.
∎

Now let us compute the trace identities that will be useful in the next sections.

###### Lemma A.8 (Trace identities) .

We have the following trace identities:

$$ $\displaystyle\frac{1}{N}\operatorname{Tr}((\Sigma\bar{\mathbf{Q}})^{2})=\frac{ 1}{N}\sum_{i=1}^{p}\frac{1}{\left(\gamma+\frac{\pi}{1+\delta}+\frac{\alpha(1- \pi)}{1+\delta_{S}}d_{i}\right)^{2}},\quad\frac{1}{N}\operatorname{Tr}((\Sigma _{\beta}\bar{\mathbf{Q}})^{2})=\frac{1}{N}\sum_{i=1}^{p}\left(\frac{d_{i}}{ \gamma+\frac{\pi}{1+\delta}+\frac{\alpha(1-\pi)}{1+\delta_{S}}d_{i}}\right)^{2},$ $\displaystyle\frac{1}{N}\operatorname{Tr}(\Sigma\bar{\mathbf{Q}}\Sigma_{\beta} \bar{\mathbf{Q}})=\frac{1}{N}\sum_{i=1}^{p}\frac{d_{i}}{\left(\gamma+\frac{\pi }{1+\delta}+\frac{\alpha(1-\pi)}{1+\delta_{S}}d_{i}\right)^{2}}$ $$

###### Proof.

We have that:

$$ $\displaystyle\frac{1}{N}\operatorname{Tr}((\Sigma\bar{\mathbf{Q}})^{2})$ $\displaystyle=\frac{1}{N}\operatorname{Tr}\left(({\bm{\mu}}{\bm{\mu}}^{\top}+{ \mathbf{I}}_{p})\bar{\mathbf{Q}}({\bm{\mu}}{\bm{\mu}}^{\top}+{\mathbf{I}}_{p}) \bar{\mathbf{Q}}\right)$ $\displaystyle=\frac{1}{N}\operatorname{Tr}(\bar{\mathbf{Q}}^{2})+{\mathcal{O}} (N^{-1})$ $\displaystyle=\frac{1}{N}\operatorname{Tr}(({\mathbf{A}}^{-1})^{2})+{\mathcal{ O}}(N^{-1})$ $\displaystyle=\frac{1}{N}\operatorname{Tr}(({\mathbf{P}}\Delta^{-1}{\mathbf{P} }^{\top})^{2})+{\mathcal{O}}(N^{-1})$ $\displaystyle=\frac{1}{N}\operatorname{Tr}((\Delta^{-1})^{2})+{\mathcal{O}}(N^ {-1})$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{p}\frac{1}{\left(\gamma+\frac{\pi}{1+ \delta}+\frac{\alpha(1-\pi)}{1+\delta_{S}}d_{i}\right)^{2}}+{\mathcal{O}}(N^{- 1})$ $$

Thus we demonstrated the first identity. For the second one, we have that:

$$ $\displaystyle\frac{1}{N}\operatorname{Tr}((\Sigma_{\beta}\bar{\mathbf{Q}})^{2})$ $\displaystyle=\frac{1}{N}\operatorname{Tr}\left(({\bm{\mu}}_{\beta}{\bm{\mu}}_ {\beta}^{\top}+{\mathbf{C}})\bar{\mathbf{Q}}({\bm{\mu}}_{\beta}{\bm{\mu}}_{ \beta}^{\top}+{\mathbf{C}})\bar{\mathbf{Q}}\right)$ $\displaystyle=\frac{1}{N}\operatorname{Tr}(({\mathbf{C}}\bar{\mathbf{Q}})^{2}) +{\mathcal{O}}(N^{-1})$ $\displaystyle=\frac{1}{N}\operatorname{Tr}(({\mathbf{C}}{\mathbf{A}}^{-1})^{2} )+{\mathcal{O}}(N^{-1})$ $\displaystyle=\frac{1}{N}\operatorname{Tr}(({\mathbf{D}}\Delta^{-1})^{2})+{ \mathcal{O}}(N^{-1})$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{p}\left(\frac{d_{i}}{\gamma+\frac{\pi}{1+ \delta}+\frac{\alpha(1-\pi)}{1+\delta_{S}}d_{i}}\right)^{2}++{\mathcal{O}}(N^{ -1})$ $$

And the same spirit of the proof applies to the last identity.
∎

###### Lemma A.7 (Delta) .

###### Proof.

###### Lemma A.8 (Trace identities) .

###### Proof.

## Appendix B Random Matrix Analysis of the general model

In the generalized model, we consider that the synthetic data follow the following distribution:

$$ $\tilde{\bm{x}}_{i}\sim{\mathcal{N}}({\bm{\mu}}_{\beta},{\mathbf{C}}),\quad{\bm {\mu}}_{\beta}=\beta{\bm{\mu}}+{\bm{\mu}}^{\perp}$ (13) $$

where $\beta\in{\mathbb{R}}$ defines the alignment of the synthetic mean with the mean of real data, and ${\bm{\mu}}^{\perp}$ is a vector orthogonal to ${\bm{\mu}}$.
Now we will analyze here the performance of the classifier given by equation (6), and prove a generalized theorem B.1 in the paper.

$$ ${\bm{w}}_{q}=\frac{1}{N}{\mathbf{Q}}(\gamma)\left({\mathbf{X}}{\bm{y}}+\tilde{ \mathbf{X}}{\mathbf{D}}({\bm{q}})\tilde{\bm{y}}\right),\quad{\mathbf{Q}}( \gamma)=\left(\frac{1}{N}{\mathbf{V}}{\mathbf{V}}^{\top}+\gamma{\mathbf{I}}_{p }\right)^{-1}.$ $$

The performance of (6) are fully determined by the first two order moments: $\mathbb{E}[{\bm{w}}_{q}^{\top}{\bm{x}}]$ and $\mathbb{E}[({\bm{w}}_{q}^{\top}{\bm{x}})^{2}]$.

### B.1 Test expectation:

We have that:

$$ ${\bm{w}}_{q}=\frac{1}{N}\sum_{i=1}^{n}{\mathbf{Q}}y_{i}{\bm{x}}_{i}+\frac{1}{N }\sum_{i=1}^{m}{\mathbf{Q}}q_{i}\tilde{y}_{i}\tilde{\bm{x}}_{i}$ $$

Let ${\bm{x}}\sim{\mathcal{N}}((-1)^{a}{\bm{\mu}},{\mathbf{I}}_{p})$ be a test sample independent of all the training samples $({\bm{v}}_{i})_{i=1}^{N}$. Then:

$$ $\displaystyle\mathbb{E}[{\bm{w}}_{q}^{\top}{\bm{x}}]$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{n}\mathbb{E}[y_{i}{\bm{x}}_{i}^{\top}{ \mathbf{Q}}{\bm{x}}]+\frac{1}{N}\sum_{i=1}^{m}\mathbb{E}[q_{i}\tilde{y}_{i} \tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}]$ $$

#### First sum:

We have that, using the same lemma A.3:

$$ $\displaystyle\frac{1}{N}\sum_{i=1}^{n}\mathbb{E}[y_{i}{\bm{x}}_{i}^{\top}{ \mathbf{Q}}{\bm{x}}]$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{n}\frac{1}{1+\delta}\mathbb{E}[y_{i}{\bm{ x}}_{i}^{\top}{\mathbf{Q}}_{-{\bm{x}}_{i}}{\bm{x}}]$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{n}\frac{1}{1+\delta}\mathbb{E}[{\bm{x}}_{ i}]^{\top}\mathbb{E}[{\mathbf{Q}}_{-{\bm{x}}_{i}}]\mathbb{E}[x]$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{n}\frac{(-1)^{a}}{1+\delta}{\bm{\mu}}^{ \top}\bar{\mathbf{Q}}{\bm{\mu}}$ $\displaystyle=\frac{(-1)^{a}\pi}{1+\delta}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{ \bm{\mu}}$ $$

Thus,

$$ $\frac{1}{N}\sum_{i=1}^{n}\mathbb{E}[y_{i}{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x }}]=\frac{(-1)^{a}\pi}{1+\delta}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}$ (14) $$

#### Second sum:

Using the same lemma A.3:

$$ $\displaystyle\frac{1}{N}\sum_{i=1}^{m}\mathbb{E}[q_{i}\tilde{y}_{i}\tilde{\bm{ x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}]$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{m}\frac{1}{1+\delta_{S}}\mathbb{E}[q_{i} \tilde{y}_{i}\tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}_{-\tilde{\bm{x}}_{i}}{\bm{x }}]$ $\displaystyle=\frac{1}{N(1+\delta_{S})}\sum_{i=1}^{m}\mathbb{E}[q_{i}\tilde{y} _{i}]\mathbb{E}[\tilde{\bm{x}}_{i}]^{\top}\mathbb{E}[{\mathbf{Q}}_{-\tilde{\bm {x}}_{i}}]\mathbb{E}[{\bm{x}}]$ $\displaystyle=\frac{(-1)^{a}}{N(1+\delta_{S})}\sum_{i=1}^{m}\lambda{\bm{\mu}}_ {\beta}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}$ $\displaystyle=\frac{(-1)^{a}\lambda(1-\pi)}{1+\delta_{S}}{\bm{\mu}}_{\beta}^{ \top}\bar{\mathbf{Q}}{\bm{\mu}}$ $$

where (here $y_{i}$ means the true label of $\tilde{\bm{x}}_{i}$):

$$ $\displaystyle\mathbb{E}[q_{i}\tilde{y}_{i}]$ $\displaystyle=y_{i}{\mathbb{P}}[q_{i}=1\mid\tilde{y}_{i}=y_{i}]-y_{i}{\mathbb{ P}}[q_{i}=1\mid\tilde{y}_{i}\neq y_{i}]$ $\displaystyle=y_{i}(\phi(1-\varepsilon)-\rho\varepsilon)=\lambda y_{i}$ $$

Therefore,

$$ $\mathbb{E}[{\bm{w}}_{q}^{\top}{\bm{x}}]=(-1)^{a}\left(\frac{\pi}{1+\delta}{\bm {\mu}}^{\top}+\frac{\lambda(1-\pi)}{1+\delta_{S}}{\bm{\mu}}_{\beta}^{\top} \right)\bar{\mathbf{Q}}{\bm{\mu}}$ (15) $$

### B.2 Test variance:

To determine the variance of ${\bm{w}}_{q}^{\top}{\bm{x}}$, it only remains to compute its second order. We have that:

$$ $\displaystyle\mathbb{E}[({\bm{w}}_{q}^{\top}{\bm{x}})^{2}]=\frac{1}{N^{2}} \mathbb{E}\left[\left(\sum_{i=1}^{n}y_{i}{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x }}+\sum_{j=1}^{m}q_{j}\tilde{y}_{j}\tilde{\bm{x}}_{j}^{\top}{\mathbf{Q}}{\bm{x }}\right)^{2}\right]$ $\displaystyle=\frac{1}{N^{2}}\mathbb{E}\left[\left(\sum_{i=1}^{n}y_{i}{\bm{x}} _{i}^{\top}{\mathbf{Q}}{\bm{x}}\right)^{2}\right]+\frac{1}{N^{2}}\mathbb{E} \left[\left(\sum_{j=1}^{m}q_{j}\tilde{y}_{j}\tilde{\bm{x}}_{j}^{\top}{\mathbf{ Q}}{\bm{x}}\right)^{2}\right]+\frac{2}{N^{2}}\mathbb{E}\left[\left(\sum_{i=1}^ {n}y_{i}{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}\right)\left(\sum_{j=1}^{m}q_{j }\tilde{y}_{j}\tilde{\bm{x}}_{j}^{\top}{\mathbf{Q}}{\bm{x}}\right)\right]$ $$

Let us compute each sum on its own, and then group the results at the end.

#### First sum:

We have that:

$$ $\displaystyle\frac{1}{N^{2}}\mathbb{E}\left[\left(\sum_{i=1}^{n}y_{i}{\bm{x}}_ {i}^{\top}{\mathbf{Q}}{\bm{x}}\right)^{2}\right]$ $\displaystyle=\frac{1}{N^{2}}\sum_{i=1}^{n}\sum_{k=1}^{n}\mathbb{E}[y_{i}y_{j} {\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}{\bm{x}}_{k}^{\top}{\mathbf{Q}}{\bm{x}}]$ $\displaystyle=\frac{1}{N^{2}}\sum_{i\neq k}^{n}\mathbb{E}[y_{i}y_{k}{\bm{x}}_{ i}^{\top}{\mathbf{Q}}{\bm{x}}{\bm{x}}_{k}^{\top}{\mathbf{Q}}{\bm{x}}]+\frac{1} {N^{2}}\sum_{i=1}^{n}\mathbb{E}[{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}{\bm{x} }_{i}^{\top}{\mathbf{Q}}{\bm{x}}]$ $$

- For $i\neq k$, we have that:

$$ $\displaystyle\mathbb{E}[y_{i}y_{k}{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}{\bm{ x}}_{k}^{\top}{\mathbf{Q}}{\bm{x}}]=\mathbb{E}[y_{i}y_{k}{\bm{x}}_{i}^{\top}{ \mathbf{Q}}{\bm{x}}{\bm{x}}^{\top}{\mathbf{Q}}{\bm{x}}_{k}]$ $\displaystyle=\mathbb{E}[y_{i}y_{k}{\bm{x}}_{i}^{\top}{\mathbf{Q}}\Sigma{ \mathbf{Q}}{\bm{x}}_{k}]$ $\displaystyle=\frac{1}{(1+\delta)^{2}}\mathbb{E}[y_{i}y_{k}{\bm{x}}_{i}^{\top} {\mathbf{Q}}_{-{\bm{x}}_{i}}\Sigma{\mathbf{Q}}_{-{\bm{x}}_{k}}{\bm{x}}_{k}]$ $\displaystyle=\frac{1}{(1+\delta)^{2}}\mathbb{E}\left[y_{i}y_{k}{\bm{x}}_{i}^{ \top}\left({\mathbf{Q}}_{-{\bm{x}}_{i,k}}-\frac{\frac{1}{N}{\mathbf{Q}}_{-{\bm {x}}_{i,k}}{\bm{x}}_{k}{\bm{x}}_{k}^{\top}{\mathbf{Q}}_{-{\bm{x}}_{i,k}}}{1+ \delta}\right)\Sigma\left({\mathbf{Q}}_{-{\bm{x}}_{i,k}}-\frac{\frac{1}{N}{ \mathbf{Q}}_{-{\bm{x}}_{i,k}}{\bm{x}}_{i}{\bm{x}}_{i}^{\top}{\mathbf{Q}}_{-{ \bm{x}}_{i,k}}}{1+\delta}\right){\bm{x}}_{k}\right]$ $\displaystyle=\frac{1}{(1+\delta)^{2}}(A_{1}-A_{2}-A_{3}+A_{4})$ $$

And we have that:

$$ $\displaystyle A_{1}$ $\displaystyle=\mathbb{E}[y_{i}y_{k}{\bm{x}}_{i}^{\top}{\mathbf{Q}}_{-{\bm{x}}_ {i,k}}\Sigma{\mathbf{Q}}_{-{\bm{x}}_{i,k}}{\bm{x}}_{k}]$ $\displaystyle={\bm{\mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{ \mu}}$ $\displaystyle={\bm{\mu}}^{\top}\left(\frac{1-b_{2}}{h}\bar{\mathbf{Q}}\Sigma \bar{\mathbf{Q}}+\frac{b_{1}}{h}\bar{\mathbf{Q}}\Sigma_{\beta}\bar{\mathbf{Q}} \right){\bm{\mu}}$ $$

And:

$$ $\displaystyle A_{2}$ $\displaystyle=\frac{1}{N(1+\delta)}\mathbb{E}[y_{i}y_{k}{\bm{x}}_{i}^{\top}{ \mathbf{Q}}_{-{\bm{x}}_{i,k}}{\bm{x}}_{k}{\bm{x}}_{k}^{\top}{\mathbf{Q}}_{-{ \bm{x}}_{i,k}}\Sigma{\mathbf{Q}}_{-{\bm{x}}_{i,k}}{\bm{x}}_{k}]$ $\displaystyle=\frac{\operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}\Sigma{ \mathbf{Q}}])}{N(1+\delta)}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}$ $$

Since, by concentration laws:

$$ $\displaystyle\frac{1}{N}{\bm{x}}_{k}^{\top}{\mathbf{Q}}_{-{\bm{x}}_{i,k}} \Sigma{\mathbf{Q}}_{-{\bm{x}}_{i,k}}{\bm{x}}_{k}$ $\displaystyle=\frac{1}{N}\mathbb{E}[{\bm{x}}_{k}^{\top}{\mathbf{Q}}_{-{\bm{x}} _{i,k}}\Sigma{\mathbf{Q}}_{-{\bm{x}}_{i,k}}{\bm{x}}_{k}]$ $\displaystyle=\frac{1}{N}\mathbb{E}[\operatorname{Tr}({\bm{x}}_{k}{\bm{x}}_{k} ^{\top}{\mathbf{Q}}_{-{\bm{x}}_{i,k}}\Sigma{\mathbf{Q}}_{-{\bm{x}}_{i,k}})]$ $\displaystyle=\frac{1}{N}\operatorname{Tr}(\mathbb{E}[{\bm{x}}_{k}{\bm{x}}_{k} ^{\top}{\mathbf{Q}}_{-{\bm{x}}_{i,k}}\Sigma{\mathbf{Q}}_{-{\bm{x}}_{i,k}}])$ $\displaystyle=\frac{1}{N}\operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}_{-{ \bm{x}}_{i,k}}\Sigma{\mathbf{Q}}_{-{\bm{x}}_{i,k}}])$ $\displaystyle=\frac{1}{N}\operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}\Sigma {\mathbf{Q}}])$ $$

And we can easily verify that:

$$ $A_{3}=A_{2},\quad A_{4}={\mathcal{O}}(N^{-1})$ $$

Thus,

$$ $\displaystyle\frac{1}{N^{2}}\sum_{i\neq k}^{n}\mathbb{E}[y_{i}y_{k}{\bm{x}}_{i }^{\top}{\mathbf{Q}}{\bm{x}}{\bm{x}}_{k}^{\top}{\mathbf{Q}}{\bm{x}}]$ $\displaystyle=\frac{n^{2}-n}{N^{2}}\left({\bm{\mu}}^{\top}\mathbb{E}[{\mathbf{ Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}-\frac{2\operatorname{Tr}(\Sigma\mathbb{E}[{ \mathbf{Q}}\Sigma{\mathbf{Q}}])}{N(1+\delta)}{\bm{\mu}}^{\top}\bar{\mathbf{Q}} {\bm{\mu}}\right)$ $\displaystyle=\frac{\pi^{2}}{(1+\delta)^{2}}\left({\bm{\mu}}^{\top}\mathbb{E}[ {\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}-\frac{2\operatorname{Tr}(\Sigma \mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}])}{N(1+\delta)}{\bm{\mu}}^{\top}\bar {\mathbf{Q}}{\bm{\mu}}\right)$ $$

- And then, for $i\in\{1,...,n\}$:

$$ $\displaystyle\mathbb{E}[{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}{\bm{x}}_{i}^{ \top}{\mathbf{Q}}{\bm{x}}]$ $\displaystyle=\mathbb{E}[{\bm{x}}_{i}^{\top}{\mathbf{Q}}\Sigma{\mathbf{Q}}{\bm {x}}_{i}]$ $\displaystyle=\frac{1}{(1+\delta)^{2}}\mathbb{E}[{\bm{x}}_{i}^{\top}{\mathbf{Q }}_{-{\bm{x}}_{i}}\Sigma{\mathbf{Q}}_{-{\bm{x}}_{i}}{\bm{x}}_{i}]$ $\displaystyle=\frac{1}{(1+\delta)^{2}}\operatorname{Tr}(\mathbb{E}[{\bm{x}}_{i }{\bm{x}}_{i}^{\top}{\mathbf{Q}}_{-{\bm{x}}_{i}}\Sigma{\mathbf{Q}}_{-{\bm{x}}_ {i}}])$ $\displaystyle=\frac{1}{(1+\delta)^{2}}\operatorname{Tr}(\Sigma\mathbb{E}[{ \mathbf{Q}}\Sigma{\mathbf{Q}}])$ $$

Thus:

$$ $\frac{1}{N^{2}}\sum_{i=1}^{n}\mathbb{E}[{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x} }{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}]=\frac{\pi}{N(1+\delta)^{2}} \operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}])$ $$

Hence, the first sum gives us:

$$ $\frac{1}{N^{2}}\mathbb{E}\left[\left(\sum_{i=1}^{n}y_{i}{\bm{x}}_{i}^{\top}{ \mathbf{Q}}{\bm{x}}\right)^{2}\right]=\frac{\pi^{2}}{(1+\delta)^{2}}\left({\bm {\mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}-\frac{2 \operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}])}{N(1+ \delta)}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}\right)+\frac{\pi}{N(1+ \delta)^{2}}\operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}])$ (16) $$

#### Second sum:

We have that:

$$ $\displaystyle\frac{1}{N^{2}}\mathbb{E}\left[\left(\sum_{i=1}^{m}q_{i}\tilde{y} _{i}\tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}\right)^{2}\right]$ $\displaystyle=\frac{1}{N^{2}}\sum_{i,j=1}^{m}\mathbb{E}[q_{i}q_{j}\tilde{y}_{i }\tilde{y}_{j}\tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}\tilde{\bm{x}}_{j}^ {\top}{\mathbf{Q}}{\bm{x}}]$ $\displaystyle=\frac{1}{N^{2}}\sum_{i\neq j}\mathbb{E}[q_{i}q_{j}\tilde{y}_{i} \tilde{y}_{j}\tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}\tilde{\bm{x}}_{j}^{ \top}{\mathbf{Q}}{\bm{x}}]+\frac{1}{N^{2}}\sum_{i=1}^{m}\mathbb{E}[q_{i}\tilde {\bm{x}}_{i}^{\top}{\bm{x}}\tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}]$ $$

- For $i\neq j\in\{1,...,m\}$, we have that:

$$ $\displaystyle\mathbb{E}[q_{i}q_{j}\tilde{y}_{i}\tilde{y}_{j}\tilde{\bm{x}}_{i} ^{\top}{\mathbf{Q}}{\bm{x}}\tilde{\bm{x}}_{j}^{\top}{\mathbf{Q}}{\bm{x}}]= \mathbb{E}[q_{i}q_{j}\tilde{y}_{i}\tilde{y}_{j}\tilde{\bm{x}}_{i}^{\top}{ \mathbf{Q}}\Sigma{\mathbf{Q}}{\bm{x}}_{j}]$ $\displaystyle=\frac{1}{(1+\delta_{S})^{2}}\mathbb{E}[q_{i}q_{j}\tilde{y}_{i} \tilde{y}_{j}\tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}_{-\tilde{\bm{x}}_{i}}\Sigma {\mathbf{Q}}_{-\tilde{\bm{x}}_{j}}{\bm{x}}_{j}]$ $\displaystyle=\frac{1}{(1+\delta_{S})^{2}}\mathbb{E}\left[q_{i}q_{j}\tilde{y}_ {i}\tilde{y}_{j}\tilde{\bm{x}}_{i}^{\top}\left({\mathbf{Q}}_{-\tilde{\bm{x}}_{ i,j}}-\frac{\frac{1}{N}{\mathbf{Q}}_{-\tilde{\bm{x}}_{i,j}}q_{j}\tilde{\bm{x}} _{j}\tilde{\bm{x}}_{j}^{\top}{\mathbf{Q}}_{-\tilde{\bm{x}}_{i,j}}}{1+\delta_{S }}\right)\Sigma\left({\mathbf{Q}}_{-\tilde{\bm{x}}_{i,j}}-\frac{\frac{1}{N}{ \mathbf{Q}}_{-\tilde{\bm{x}}_{i,j}}q_{i}\tilde{\bm{x}}_{i}\tilde{\bm{x}}_{i}^{ \top}{\mathbf{Q}}_{-\tilde{\bm{x}}_{i,j}}}{1+\delta_{S}}\right)\tilde{\bm{x}}_ {j}\right]$ $\displaystyle=\frac{1}{(1+\delta_{S})^{2}}(A_{1}-A_{2}-A_{3}+A_{4})$ $$

And, we have that:

$$ $\displaystyle A_{1}$ $\displaystyle=\mathbb{E}[q_{i}q_{j}\tilde{y}_{i}\tilde{y}_{j}\tilde{\bm{x}}_{i }^{\top}{\mathbf{Q}}_{-\tilde{\bm{x}}_{i,j}}\Sigma{\mathbf{Q}}_{-\tilde{\bm{x} }_{i,j}}\tilde{\bm{x}}_{j}]$ $\displaystyle=\lambda^{2}\mathbb{E}[y_{i}\tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}} _{-\tilde{\bm{x}}_{i,j}}\Sigma{\mathbf{Q}}_{-\tilde{\bm{x}}_{i,j}}y_{j}\tilde{ \bm{x}}_{j}]$ $\displaystyle=\lambda^{2}{\bm{\mu}}_{\beta}^{\top}\mathbb{E}[{\mathbf{Q}} \Sigma{\mathbf{Q}}]{\bm{\mu}}_{\beta}$ $$

And:

$$ $\displaystyle A_{2}$ $\displaystyle=\frac{1}{N(1+\delta_{S})}\mathbb{E}[q_{i}q_{j}\tilde{y}_{i} \tilde{y}_{j}\tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}_{-\tilde{\bm{x}}_{i,j}} \tilde{\bm{x}}_{j}\tilde{\bm{x}}_{j}^{\top}{\mathbf{Q}}_{-\tilde{\bm{x}}_{i,j} }\Sigma{\mathbf{Q}}_{-\tilde{\bm{x}}_{i,j}}\tilde{\bm{x}}_{j}]$ $\displaystyle=\frac{1}{N(1+\delta_{S})}\operatorname{Tr}(\Sigma_{\beta}\mathbb {E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}])\mathbb{E}[q_{i}q_{j}\tilde{y}_{i}\tilde{y }_{j}\tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}_{-\tilde{\bm{x}}_{i,j}}\tilde{\bm{x }}_{j}]$ $\displaystyle=\frac{\lambda^{2}}{N(1+\delta_{S})}\operatorname{Tr}(\Sigma_{ \beta}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]){\bm{\mu}}_{\beta}^{\top}\bar {\mathbf{Q}}{\bm{\mu}}_{\beta}$ $$

And, we can easily observe that:

$$ $A_{3}=A_{2},\quad A_{4}={\mathcal{O}}(N^{-1})$ $$

Thus:

$$ $\displaystyle\frac{1}{N^{2}}\sum_{i\neq j}\mathbb{E}[q_{i}q_{j}\tilde{y}_{i} \tilde{y}_{j}\tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}\tilde{\bm{x}}_{j}^{ \top}{\mathbf{Q}}{\bm{x}}]=\frac{\lambda^{2}(1-\pi)^{2}}{(1+\delta_{S})^{2}} \left({\bm{\mu}}_{\beta}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{ \mu}}_{\beta}-\frac{2}{N(1+\delta_{S})}\operatorname{Tr}(\Sigma_{\beta}\mathbb {E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]){\bm{\mu}}_{\beta}^{\top}\bar{\mathbf{Q}}{ \bm{\mu}}_{\beta}\right)$ $$

- And for $i\in\{1,...,m\}$:

$$ $\displaystyle\mathbb{E}[q_{i}\tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}} \tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}]$ $\displaystyle=\frac{1}{(1+\delta_{S})^{2}}\mathbb{E}[q_{i}\tilde{\bm{x}}_{i}^{ \top}{\mathbf{Q}}_{-\tilde{\bm{x}}_{i}}\Sigma{\mathbf{Q}}_{-\tilde{\bm{x}}_{i} }\tilde{\bm{x}}_{i}]$ $\displaystyle=\frac{\alpha}{(1+\delta_{S})^{2}}\mathbb{E}[\tilde{\bm{x}}_{i}^{ \top}{\mathbf{Q}}_{-\tilde{\bm{x}}_{i}}\Sigma{\mathbf{Q}}_{-\tilde{\bm{x}}_{i} }\tilde{\bm{x}}_{i}]$ $\displaystyle=\frac{\alpha}{(1+\delta_{S})^{2}}\operatorname{Tr}(\mathbb{E}[ \tilde{\bm{x}}_{i}\tilde{\bm{x}}_{i}^{\top}]\mathbb{E}[{\mathbf{Q}}_{-\tilde{ \bm{x}}_{i}}\Sigma{\mathbf{Q}}_{-\tilde{\bm{x}}_{i}}])$ $\displaystyle=\frac{\alpha}{(1+\delta_{S})^{2}}\operatorname{Tr}(\Sigma_{\beta }\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}])$ $$

Hence, by grouping the terms, the second sum gives us:

$$ $\displaystyle\frac{1}{N^{2}}\mathbb{E}\left[\left(\sum_{i=1}^{m}q_{i}\tilde{y} _{i}\tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}\right)^{2}\right]$ (17) $\displaystyle=\frac{\lambda^{2}(1-\pi)^{2}}{(1+\delta_{S})^{2}}\left({\bm{\mu} }_{\beta}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}_{\beta}- \frac{2}{N(1+\delta_{S})}\operatorname{Tr}(\Sigma_{\beta}\mathbb{E}[{\mathbf{Q }}\Sigma{\mathbf{Q}}]){\bm{\mu}}_{\beta}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{ \beta}\right)+\frac{\alpha(1-\pi)}{N(1+\delta_{S})^{2}}\operatorname{Tr}( \Sigma_{\beta}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}])$ (18) $$

#### Third sum:

Let us now compute the remaining term in the sum that is given by:

$$ $\displaystyle\frac{2}{N^{2}}\sum_{i=1}^{n}\sum_{j=1}^{m}\mathbb{E}[y_{i}{\bm{x }}_{i}^{\top}{\mathbf{Q}}{\bm{x}}q_{j}\tilde{y}_{j}\tilde{\bm{x}}_{j}^{\top}{ \mathbf{Q}}{\bm{x}}]$ $$

Let $i\in\{1,...,n\}$ and $j\in\{1,...,m\}$, we have that:

$$ $\displaystyle\mathbb{E}[y_{i}q_{j}\tilde{y}_{j}{\bm{x}}_{i}^{\top}{\mathbf{Q}} {\bm{x}}{\bm{x}}^{\top}{\mathbf{Q}}\tilde{\bm{x}}_{j}]=\mathbb{E}[y_{i}q_{j} \tilde{y}_{j}{\bm{x}}_{i}^{\top}{\mathbf{Q}}\Sigma{\mathbf{Q}}\tilde{\bm{x}}_{ j}]$ $\displaystyle=\frac{1}{(1+\delta)(1+\delta_{S})}\mathbb{E}[y_{i}q_{j}\tilde{y} _{j}{\bm{x}}_{i}^{\top}{\mathbf{Q}}_{-{\bm{x}}_{i}}\Sigma{\mathbf{Q}}_{-\tilde {\bm{x}}_{j}}\tilde{\bm{x}}_{j}]$ $\displaystyle=\frac{1}{(1+\delta)(1+\delta_{S})}\mathbb{E}\left[y_{i}q_{j} \tilde{y}_{j}{\bm{x}}_{i}^{\top}\left({\mathbf{Q}}_{-ij}-\frac{{\mathbf{Q}}_{- ij}q_{j}\tilde{\bm{x}}_{j}\tilde{\bm{x}}_{j}^{\top}{\mathbf{Q}}_{-ij}}{N(1+ \delta_{S})}\right)\Sigma\left({\mathbf{Q}}_{-ij}-\frac{{\mathbf{Q}}_{-ij}{\bm {x}}_{i}{\bm{x}}_{i}^{\top}{\mathbf{Q}}_{-ij}}{N(1+\delta)}\right)\tilde{\bm{x }}_{j}\right]$ $\displaystyle=\frac{1}{(1+\delta)(1+\delta_{S})}\left(A_{1}-A_{2}-A_{3}+A_{4}\right)$ $$

We have that:

$$ $\displaystyle A_{1}$ $\displaystyle=\mathbb{E}[y_{i}q_{j}\tilde{y}_{j}{\bm{x}}_{i}^{\top}{\mathbf{Q} }_{-ij}\Sigma{\mathbf{Q}}_{-ij}\tilde{\bm{x}}_{j}]=\lambda\mathbb{E}[y_{i}{\bm {x}}_{i}^{\top}{\mathbf{Q}}_{-ij}\Sigma{\mathbf{Q}}_{-ij}\tilde{y}_{j}{\bm{x}} _{j}]$ $\displaystyle=\lambda{\bm{\mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q} }]{\bm{\mu}}_{\beta}$ $$

And:

$$ $\displaystyle A_{2}$ $\displaystyle=\frac{1}{N(1+\delta)}\mathbb{E}[y_{i}q_{j}\tilde{y}_{j}{\bm{x}}_ {i}^{\top}{\mathbf{Q}}_{-ij}\Sigma{\mathbf{Q}}_{-ij}{\bm{x}}_{i}{\bm{x}}_{i}^{ \top}{\mathbf{Q}}_{-ij}\tilde{\bm{x}}_{j}]$ $\displaystyle=\frac{\lambda}{N(1+\delta)}\mathbb{E}[y_{i}{\bm{x}}_{i}^{\top}{ \mathbf{Q}}_{-ij}\Sigma{\mathbf{Q}}_{-ij}{\bm{x}}_{i}{\bm{x}}_{i}^{\top}{ \mathbf{Q}}_{-ij}y_{j}\tilde{\bm{x}}_{j}]$ $\displaystyle=\frac{\lambda}{N(1+\delta)}\operatorname{Tr}(\Sigma\mathbb{E}[{ \mathbf{Q}}\Sigma{\mathbf{Q}}]){\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{\beta}$ $$

And also:

$$ $\displaystyle A_{3}$ $\displaystyle=\frac{1}{N(1+\delta_{S})}\mathbb{E}[y_{i}q_{j}\tilde{y}_{j}{\bm{ x}}_{i}^{\top}{\mathbf{Q}}_{-ij}\tilde{\bm{x}}_{j}\tilde{\bm{x}}_{j}^{\top}{ \mathbf{Q}}_{-ij}\Sigma{\mathbf{Q}}_{-ij}\tilde{\bm{x}}_{j}]$ $\displaystyle=\frac{\lambda}{N(1+\delta_{S})}\operatorname{Tr}(\Sigma_{\beta} \mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]){\bm{\mu}}^{\top}\bar{\mathbf{Q}}{ \bm{\mu}}_{\beta}$ $$

Hence:

$$ $\displaystyle\frac{2}{N^{2}}\sum_{i=1}^{n}\sum_{j=1}^{m}\mathbb{E}[y_{i}{\bm{x }}_{i}^{\top}{\mathbf{Q}}{\bm{x}}q_{j}\tilde{y}_{j}\tilde{\bm{x}}_{j}^{\top}{ \mathbf{Q}}{\bm{x}}]$ (19) $\displaystyle=\frac{2\lambda\pi(1-\pi)}{(1+\delta)(1+\delta_{S})}\left({\bm{ \mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}_{\beta}-\frac {1}{N}\operatorname{Tr}\left(\left(\frac{\Sigma}{1+\delta}+\frac{\Sigma_{\beta }}{1+\delta_{S}}\right)\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]\right){\bm{ \mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{\beta}\right)$ (20) $$

#### Grouping all the sums:

Denote by : $T_{1}=\frac{1}{N}\operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf
{Q}}])$, then: $T_{2}=\frac{1}{N}\operatorname{Tr}(\Sigma_{\beta}\mathbb{E}[{\mathbf{Q}}\Sigma
{\mathbf{Q}}])$.
Now let us group the terms in $T$ in the three sums, and those that do not depend on $T$.
We get that:

$$ $\displaystyle\mathbb{E}[({\bm{w}}^{\top}{\bm{x}})^{2}]=\frac{\pi^{2}}{(1+ \delta)^{2}}{\bm{\mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{ \mu}}+\frac{\lambda^{2}(1-\pi)^{2}}{(1+\delta_{S})^{2}}{\bm{\mu}}_{\beta}^{ \top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}_{\beta}+\frac{2 \lambda\pi(1-\pi)}{(1+\delta)(1+\delta_{S})}{\bm{\mu}}^{\top}\mathbb{E}[{ \mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}_{\beta}$ $\displaystyle+T_{1}\left(\frac{\pi}{(1+\delta)^{2}}-\frac{2\pi^{2}}{(1+\delta) ^{3}}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}-\frac{2\lambda\pi(1-\pi)}{(1+ \delta)^{2}(1+\delta_{S})}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{\beta}\right)$ $\displaystyle+T_{2}\left(\frac{\alpha(1-\pi)}{(1+\delta_{S})^{2}}-\frac{2 \lambda^{2}(1-\pi)^{2}}{(1+\delta_{S})^{3}}{\bm{\mu}}_{\beta}^{\top}\bar{ \mathbf{Q}}{\bm{\mu}}_{\beta}-\frac{2\lambda\pi(1-\pi)}{(1+\delta)(1+\delta_{S })^{2}}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{\beta}\right)$ $\displaystyle=\frac{\pi^{2}}{(1+\delta)^{2}}{\bm{\mu}}^{\top}\mathbb{E}[{ \mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}+\frac{\lambda^{2}(1-\pi)^{2}}{(1+ \delta_{S})^{2}}{\bm{\mu}}_{\beta}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf {Q}}]{\bm{\mu}}_{\beta}+\frac{2\lambda\pi(1-\pi)}{(1+\delta)(1+\delta_{S})}{ \bm{\mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}_{\beta}$ $\displaystyle+\frac{\pi T_{1}}{(1+\delta)^{2}}\left(1-\frac{2\pi}{1+\delta}{ \bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}-\frac{2\lambda(1-\pi)}{1+\delta_{S} }{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{\beta}\right)$ $\displaystyle+\frac{(1-\pi)T_{2}}{(1+\delta_{S})^{2}}\left(\alpha-\frac{2 \lambda^{2}(1-\pi)}{1+\delta_{S}}{\bm{\mu}}_{\beta}^{\top}\bar{\mathbf{Q}}{\bm {\mu}}_{\beta}-\frac{2\lambda\pi}{1+\delta}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{ \bm{\mu}}_{\beta}\right)$ $$

This leads to the following theorem:

###### Theorem B.1 (Gaussianity of the General model) .

Let ${\bm{w}}_{q}$ be the Mixed classifier as defined in equation 6 and suppose that Assumption 4.1 holds. The decision function ${\bm{w}}_{q}^{\top}{\bm{x}}$, on some test sample ${\bm{x}}\in{\mathcal{C}}_{a}$ independent of ${\mathbf{X}}$, satisfies:

$$ $\displaystyle{\bm{w}}_{q}^{\top}{\bm{x}}\,\,\operatorname{\xrightarrow{{ \mathcal{D}}}}\,\,{\mathcal{N}}\left((-1)^{a}m_{q},\,\nu_{q}-m_{q}^{2}\right),$ $$

where:

$$ $\displaystyle m_{q}$ $\displaystyle=\left(\frac{\pi}{1+\delta}{\bm{\mu}}^{\top}+\frac{\lambda(1-\pi) }{1+\delta_{S}}{\bm{\mu}}_{\beta}^{\top}\right)\bar{\mathbf{Q}}{\bm{\mu}},$ $\displaystyle\nu_{q}$ $\displaystyle=\frac{\pi^{2}}{(1+\delta)^{2}}{\bm{\mu}}^{\top}\mathbb{E}[{ \mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}+\frac{\lambda^{2}(1-\pi)^{2}}{(1+ \delta_{S})^{2}}{\bm{\mu}}_{\beta}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf {Q}}]{\bm{\mu}}_{\beta}+\frac{2\lambda\pi(1-\pi)}{(1+\delta)(1+\delta_{S})}{ \bm{\mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}_{\beta}$ $\displaystyle+\frac{\pi T_{1}}{(1+\delta)^{2}}\left(1-\frac{2\pi}{1+\delta}{ \bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}-\frac{2\lambda(1-\pi)}{1+\delta_{S} }{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{\beta}\right)$ $\displaystyle+\frac{(1-\pi)T_{2}}{(1+\delta_{S})^{2}}\left(\alpha-\frac{2 \lambda^{2}(1-\pi)}{1+\delta_{S}}{\bm{\mu}}_{\beta}^{\top}\bar{\mathbf{Q}}{\bm {\mu}}_{\beta}-\frac{2\lambda\pi}{1+\delta}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{ \bm{\mu}}_{\beta}\right).$ $$

where:

$$ $T_{1}=\frac{1}{N}\operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf {Q}}]),\quad T_{2}=\frac{1}{N}\operatorname{Tr}(\Sigma_{\beta}\mathbb{E}[{ \mathbf{Q}}\Sigma{\mathbf{Q}}]),\quad\lambda=\phi(1-\varepsilon)-\rho\varepsilon$ $$

###### Theorem B.1 (Gaussianity of the General model) .

## Appendix C Particular case: Isotropic covariance matrix

Here, we consider a simple covariance matrix of the form ${\mathbf{C}}=\sigma^{2}{\mathbf{I}}_{p}$ for some $\sigma>0$. So

$$ $\delta_{S}=\alpha\sigma^{2}\delta$ (21) $$

### C.1 Resolvent identities in the case of 𝐂 = σ 2 ⁢ 𝐈 p 𝐂 superscript 𝜎 2 subscript 𝐈 𝑝 {\mathbf{C}}=\sigma^{2}{\mathbf{I}}_{p} bold_C = italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT bold_I start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT

We have that by lemma A.4:

$$ $\displaystyle\bar{\mathbf{Q}}$ $\displaystyle=\left(\frac{\pi}{1+\delta}({\bm{\mu}}{\bm{\mu}}^{\top}+{\mathbf{ I}}_{p})+\frac{\alpha(1-\pi)}{1+\alpha\sigma^{2}\delta}({\bm{\mu}}_{\beta}{\bm {\mu}}_{\beta}^{\top}+\sigma^{2}{\mathbf{I}}_{p})+\gamma{\mathbf{I}}_{p}\right )^{-1}$ $\displaystyle=\left(\frac{\pi}{1+\delta}{\bm{\mu}}{\bm{\mu}}^{\top}+\frac{ \alpha(1-\pi)}{1+\alpha\sigma^{2}\delta}{\bm{\mu}}_{\beta}{\bm{\mu}}_{\beta}^{ \top}+\theta{\mathbf{I}}_{p}\right)^{-1},$ $$

where:

$$ $\theta=\gamma+\frac{\pi}{1+\delta}+\frac{\alpha\sigma^{2}(1-\pi)}{1+\alpha \sigma^{2}\delta}$ (22) $$

Define by:

$$ $\displaystyle{\mathbf{R}}_{1}=\left(\frac{\alpha(1-\pi)}{1+\alpha\sigma^{2} \delta}{\bm{\mu}}_{\beta}{\bm{\mu}}_{\beta}^{\top}+\theta{\mathbf{I}}_{p} \right)^{-1},\quad{\mathbf{R}}_{2}=\left(\frac{\pi}{1+\delta}{\bm{\mu}}{\bm{ \mu}}^{\top}+\theta{\mathbf{I}}_{p}\right)^{-1},$ (23) $$

such that:

$$ $\bar{\mathbf{Q}}=\left(\frac{\pi}{1+\delta}{\bm{\mu}}{\bm{\mu}}^{\top}+{ \mathbf{R}}_{1}^{-1}\right)^{-1}=\left(\frac{\alpha(1-\pi)}{1+\alpha\sigma^{2} \delta}{\bm{\mu}}_{\beta}{\bm{\mu}}_{\beta}^{\top}+{\mathbf{R}}_{2}^{-1}\right )^{-1}$ $$

Thus, using lemma A.3:

$$ $\bar{\mathbf{Q}}{\bm{\mu}}=\frac{{\mathbf{R}}_{1}{\bm{\mu}}}{1+\frac{\pi}{1+ \delta}{\bm{\mu}}^{\top}{\mathbf{R}}_{1}{\bm{\mu}}},\quad\bar{\mathbf{Q}}{\bm{ \mu}}_{\beta}=\frac{{\mathbf{R}}_{2}{\bm{\mu}}_{\beta}}{1+\frac{\alpha(1-\pi)} {1+\alpha\sigma^{2}\delta}{\bm{\mu}}_{\beta}^{\top}{\mathbf{R}}_{2}{\bm{\mu}}_ {\beta}}$ (24) $$

And:

$$ $\bar{\mathbf{Q}}={\mathbf{R}}_{1}-\frac{\pi{\mathbf{R}}_{1}{\bm{\mu}}{\bm{\mu} }^{\top}{\mathbf{R}}_{1}}{1+\delta+\pi{\bm{\mu}}^{\top}{\mathbf{R}}_{1}{\bm{ \mu}}}={\mathbf{R}}_{2}-\frac{\alpha(1-\pi){\mathbf{R}}_{2}{\bm{\mu}}_{\beta}{ \bm{\mu}}_{\beta}^{\top}{\mathbf{R}}_{2}}{1+\alpha\sigma^{2}\delta+\alpha(1- \pi){\bm{\mu}}_{\beta}^{\top}{\mathbf{R}}_{2}{\bm{\mu}}_{\beta}}$ (25) $$

###### Lemma C.1 (Delta) .

The parameter $\delta$ as defined in equation 9, is given by the following identity:

$$ $\delta=\frac{\eta}{\theta}=\frac{\eta}{\gamma+\frac{\pi}{1+\delta}+\frac{ \alpha\sigma^{2}(1-\pi)}{1+\alpha\sigma^{2}\delta}}$ $$

Which gives us a third order equation:

$$ $\alpha\sigma^{2}\gamma\delta^{3}+\left(\gamma+\alpha\sigma^{2}(1+\gamma-\eta) \right)\delta^{2}+\left(\gamma+\pi-\eta+\alpha\sigma^{2}(1-\pi-\eta)\right) \delta-\eta=0$ $$

###### Lemma C.2 (Resolvent identities) .

Using the first identity in Sherman-Morisson’s lemma A.3, we have that the expressions of ${\mathbf{R}}_{1}$ and ${\mathbf{R}}_{2}$ are given by:

$$ ${\mathbf{R}}_{1}=\frac{1}{\theta}{\mathbf{I}}_{p}-\frac{\alpha(1-\pi){\bm{\mu} }_{\beta}{\bm{\mu}}_{\beta}^{\top}}{\theta^{2}(1+\alpha\sigma^{2}\delta)+ \theta\alpha(1-\pi)\|{\bm{\mu}}_{\beta}\|^{2}},\quad{\mathbf{R}}_{2}=\frac{1}{ \theta}{\mathbf{I}}_{p}-\frac{\pi{\bm{\mu}}{\bm{\mu}}^{\top}}{\theta^{2}(1+ \delta)+\theta\pi\|{\bm{\mu}}\|^{2}}$ $$

And we also have the following identities:

$$ ${\mathbf{R}}_{1}{\bm{\mu}}_{\beta}=\frac{{\bm{\mu}}_{\beta}}{\theta+\frac{ \alpha(1-\pi)}{1+\delta_{S}}\|{\bm{\mu}}_{\beta}\|^{2}},\quad{\mathbf{R}}_{2}{ \bm{\mu}}=\frac{{\bm{\mu}}}{\theta+\frac{\pi}{1+\delta}\|{\bm{\mu}}\|^{2}}$ $$

$$ $\displaystyle{\bm{\mu}}^{\top}{\mathbf{R}}_{1}{\bm{\mu}}=\frac{\|{\bm{\mu}}\|^ {2}}{\theta}\left(1-\frac{\alpha(1-\pi)\beta^{2}\|{\bm{\mu}}\|^{2}}{\theta(1+ \delta_{S})+\alpha(1-\pi)\|{\bm{\mu}}_{\beta}\|^{2}}\right)=\frac{\|{\bm{\mu}} \|^{2}}{\theta}\frac{\theta(1+\delta_{S})+\alpha(1-\pi)(1-\beta^{2})\|{\bm{\mu }}^{\perp}\|^{2}}{\theta(1+\delta_{S})+\alpha(1-\pi)\|{\bm{\mu}}_{\beta}\|^{2}}$ $$

$$ $\displaystyle{\bm{\mu}}^{\top}{\mathbf{R}}_{2}{\bm{\mu}}_{\beta}=\frac{\beta(1 +\delta)\|{\bm{\mu}}\|^{2}}{\theta(1+\delta)+\pi\|{\bm{\mu}}\|^{2}}$ $$

$$ ${\bm{\mu}}_{\beta}^{\top}{\mathbf{R}}_{2}{\bm{\mu}}_{\beta}=\frac{1}{\theta} \left(\|{\bm{\mu}}_{\beta}\|^{2}-\frac{\pi\beta^{2}\|{\bm{\mu}}\|^{4}}{\theta( 1+\delta)+\pi\|{\bm{\mu}}\|^{2}}\right)$ $$

$$ $\displaystyle{\bm{\mu}}^{\top}{\mathbf{R}}_{1}^{2}{\bm{\mu}}$ $\displaystyle=\frac{\|{\bm{\mu}}\|^{2}}{\theta^{2}}\left(1-\frac{2\alpha(1-\pi )\beta^{2}\|{\bm{\mu}}\|^{2}}{\theta(1+\delta_{S})+\alpha(1-\pi)\|{\bm{\mu}}_{ \beta}\|^{2}}+\frac{\alpha^{2}(1-\pi)^{2}\beta^{2}\|{\bm{\mu}}\|^{2}\|{\bm{\mu }}_{\beta}\|^{2}}{(\theta(1+\delta_{S})+\alpha(1-\pi)\|{\bm{\mu}}_{\beta}\|^{2 })^{2}}\right)$ $\displaystyle=\frac{\|{\bm{\mu}}\|^{2}}{\theta^{2}}+\frac{\alpha(1-\pi)\beta^{ 2}\|{\bm{\mu}}\|^{4}}{\theta^{2}(\theta(1+\delta_{S})+\alpha(1-\pi)\|{\bm{\mu} }_{\beta}\|^{2})}\left(\frac{\alpha(1-\pi)\|{\bm{\mu}}_{\beta}\|^{2}}{\theta(1 +\delta_{S})+\alpha(1-\pi)\|{\bm{\mu}}_{\beta}\|^{2}}-2\right)$ $$

$$ $\displaystyle{\bm{\mu}}_{\beta}^{\top}{\mathbf{R}}_{2}{\mathbf{R}}_{1}{\bm{\mu}}$ $\displaystyle=\frac{\beta\|{\bm{\mu}}\|^{2}}{\theta^{2}}(1-\frac{\alpha(1-\pi) \|{\bm{\mu}}_{\beta}\|^{2}}{\theta(1+\delta_{S})+\alpha(1-\pi)\|{\bm{\mu}}_{ \beta}\|^{2}}-\frac{\pi\|{\bm{\mu}}\|^{2}}{\theta(1+\delta)+\pi\|{\bm{\mu}}\|^ {2}}$ $\displaystyle+\frac{\alpha\pi(1-\pi)\beta^{2}\|{\bm{\mu}}\|^{4}}{(\theta(1+ \delta)+\pi\|{\bm{\mu}}\|^{2})(\theta(1+\delta_{S})+\alpha(1-\pi)\|{\bm{\mu}}_ {\beta}\|^{2})})$ $$

$$ $\displaystyle{\bm{\mu}}_{\beta}^{\top}{\mathbf{R}}_{2}^{2}{\bm{\mu}}_{\beta}$ $\displaystyle=\frac{\|{\bm{\mu}}_{\beta}\|^{2}}{\theta^{2}}+\frac{\pi\beta^{2} \|{\bm{\mu}}\|^{4}}{\theta^{2}(\theta(1+\delta)+\pi\|{\bm{\mu}}\|^{2})}\left( \frac{\pi\|{\bm{\mu}}\|^{2}}{\theta(1+\delta)+\pi\|{\bm{\mu}}\|^{2}}-2\right)$ $\displaystyle=\frac{\|{\bm{\mu}}_{\beta}\|^{2}}{\theta^{2}}-\frac{\pi\beta^{2} \|{\bm{\mu}}\|^{4}\left(\pi\|{\bm{\mu}}\|^{2}+2\theta(1+\delta)\right)}{\theta ^{2}(\theta(1+\delta)+\pi\|{\bm{\mu}}\|^{2})^{2}}$ $$

###### Lemma C.3 (Trace identities) .

Let $i\in\{1,...,n\}$, and $j\in\{1,...,m\}$, such that: $\Sigma=\mathbb{E}[{\bm{x}}_{i}{\bm{x}}_{i}^{\top}]={\bm{\mu}}{\bm{\mu}}^{\top}
+{\mathbf{I}}_{p}$ and $\Sigma_{\beta}=\mathbb{E}[\tilde{\bm{x}}_{j}\tilde{\bm{x}}_{j}^{\top}]={\bm{
\mu}}_{\beta}{\bm{\mu}}_{\beta}^{\top}+\sigma^{2}{\mathbf{I}}_{p}$.
We can prove that:

$$ $\displaystyle\frac{1}{N}\operatorname{Tr}((\Sigma\bar{\mathbf{Q}})^{2})=\frac{ \eta}{\theta^{2}},\quad\frac{1}{N}\operatorname{Tr}((\Sigma_{\beta}\bar{ \mathbf{Q}})^{2})=\frac{\eta\sigma^{4}}{\theta^{2}},\quad\frac{1}{N} \operatorname{Tr}(\Sigma_{\beta}\bar{\mathbf{Q}}\Sigma\bar{\mathbf{Q}})=\frac{ \eta\sigma^{2}}{\theta^{2}}$ $$

The performance of ${\bm{w}}_{q}$ in (6) is fully determined by the first two order moments: $\mathbb{E}[{\bm{w}}_{q}^{\top}{\bm{x}}]$ and $\mathbb{E}[({\bm{w}}_{q}^{\top}{\bm{x}})^{2}]$.

###### Lemma C.1 (Delta) .

###### Lemma C.2 (Resolvent identities) .

###### Lemma C.3 (Trace identities) .

### C.2 Test Expectation

We have that using the calculus in the past section:

$$ $\mathbb{E}[{\bm{w}}_{q}^{\top}{\bm{x}}]=(-1)^{a}\left(\frac{\pi}{1+\delta}{\bm {\mu}}^{\top}+\frac{\lambda(1-\pi)}{1+\alpha\sigma^{2}\delta}{\bm{\mu}}_{\beta }^{\top}\right)\bar{\mathbf{Q}}{\bm{\mu}}$ (26) $$

And finally we use lemma C.2 and the following identities to obtain the result:

$$ ${\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}=\frac{{\bm{\mu}}^{\top}{\mathbf{R} }_{1}{\bm{\mu}}}{1+\frac{\pi}{1+\delta}{\bm{\mu}}^{\top}{\mathbf{R}}_{1}{\bm{ \mu}}},\quad{\bm{\mu}}_{\beta}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}=\frac{{\bm{\mu }}_{\beta}^{\top}{\mathbf{R}}_{2}{\bm{\mu}}}{1+\frac{\alpha(1-\pi)}{1+\delta_{ S}}{\bm{\mu}}_{\beta}^{\top}{\mathbf{R}}_{2}{\bm{\mu}}_{\beta}}$ $$

### C.3 Test Variance

To determine the variance of ${\bm{w}}_{q}^{\top}{\bm{x}}$, it only remains to compute its second order. We have that:

$$ $\displaystyle\mathbb{E}[({\bm{w}}_{q}^{\top}{\bm{x}})^{2}]=\frac{1}{N^{2}} \mathbb{E}\left[\left(\sum_{i=1}^{n}y_{i}{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x }}+\sum_{j=1}^{m}q_{j}\tilde{y}_{j}\tilde{\bm{x}}_{j}^{\top}{\mathbf{Q}}{\bm{x }}\right)^{2}\right]$ $\displaystyle=\frac{1}{N^{2}}\mathbb{E}\left[\left(\sum_{i=1}^{n}y_{i}{\bm{x}} _{i}^{\top}{\mathbf{Q}}{\bm{x}}\right)^{2}\right]+\frac{1}{N^{2}}\mathbb{E} \left[\left(\sum_{j=1}^{m}q_{j}\tilde{y}_{j}\tilde{\bm{x}}_{j}^{\top}{\mathbf{ Q}}{\bm{x}}\right)^{2}\right]+\frac{2}{N^{2}}\mathbb{E}\left[\left(\sum_{i=1}^ {n}y_{i}{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}\right)\left(\sum_{j=1}^{m}q_{j }\tilde{y}_{j}\tilde{\bm{x}}_{j}^{\top}{\mathbf{Q}}{\bm{x}}\right)\right]$ $$

And using the same computations in the past section, we get:

#### First sum:

The first sum gives us:

$$ $\frac{1}{N^{2}}\mathbb{E}\left[\left(\sum_{i=1}^{n}y_{i}{\bm{x}}_{i}^{\top}{ \mathbf{Q}}{\bm{x}}\right)^{2}\right]=\frac{\pi^{2}}{(1+\delta)^{2}}\left({\bm {\mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}-\frac{2 \operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}])}{N(1+ \delta)}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}\right)+\frac{\pi}{N(1+ \delta)^{2}}\operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}])$ $$

#### Second sum:

the second sum gives us:

$$ $\displaystyle\frac{1}{N^{2}}\mathbb{E}\left[\left(\sum_{i=1}^{m}q_{i}\tilde{y} _{i}\tilde{\bm{x}}_{i}^{\top}{\mathbf{Q}}{\bm{x}}\right)^{2}\right]$ $\displaystyle=\frac{\lambda^{2}(1-\pi)^{2}}{(1+\delta_{S})^{2}}\left({\bm{\mu} }_{\beta}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}_{\beta}- \frac{2}{N(1+\delta_{S})}\operatorname{Tr}(\Sigma_{\beta}\mathbb{E}[{\mathbf{Q }}\Sigma{\mathbf{Q}}]){\bm{\mu}}_{\beta}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{ \beta}\right)+\frac{\alpha(1-\pi)}{N(1+\delta_{S})^{2}}\operatorname{Tr}( \Sigma_{\beta}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}])$ $$

#### Third sum:

The third sum is given by:

$$ $\displaystyle\frac{2}{N^{2}}\sum_{i=1}^{n}\sum_{j=1}^{m}\mathbb{E}[y_{i}{\bm{x }}_{i}^{\top}{\mathbf{Q}}{\bm{x}}q_{j}\tilde{y}_{j}\tilde{\bm{x}}_{j}^{\top}{ \mathbf{Q}}{\bm{x}}]$ $\displaystyle=\frac{2\lambda\pi(1-\pi)}{(1+\delta)(1+\delta_{S})}\left({\bm{ \mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}_{\beta}-\frac {1}{N}\operatorname{Tr}\left(\left(\frac{\Sigma}{1+\delta}+\frac{\Sigma_{\beta }}{1+\delta_{S}}\right)\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]\right){\bm{ \mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{\beta}\right)$ $$

#### Grouping all the sums:

Denote by : $T=\frac{1}{N}\operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}])$, then: $\frac{1}{N}\operatorname{Tr}(\Sigma_{\beta}\mathbb{E}[{\mathbf{Q}}\Sigma{
\mathbf{Q}}])=\sigma^{2}T$.
Now let us group the terms in $T$ in the three sums, and those that do not depend on $T$.
We get that:

$$ $\displaystyle\mathbb{E}[({\bm{w}}^{\top}{\bm{x}})^{2}]=\frac{\pi^{2}}{(1+ \delta)^{2}}{\bm{\mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{ \mu}}+\frac{\lambda^{2}(1-\pi)^{2}}{(1+\delta_{S})^{2}}{\bm{\mu}}_{\beta}^{ \top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}_{\beta}+\frac{2 \lambda\pi(1-\pi)}{(1+\delta)(1+\delta_{S})}{\bm{\mu}}^{\top}\mathbb{E}[{ \mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}_{\beta}$ $\displaystyle+T\left(\frac{\pi}{(1+\delta)^{2}}-\frac{2\pi^{2}}{(1+\delta)^{3} }{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}-\frac{2\lambda\pi(1-\pi)}{(1+ \delta)^{2}(1+\delta_{S})}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{\beta}\right)$ $\displaystyle+\sigma^{2}T\left(\frac{\alpha(1-\pi)}{(1+\delta_{S})^{2}}-\frac{ 2\lambda^{2}(1-\pi)^{2}}{(1+\delta_{S})^{3}}{\bm{\mu}}_{\beta}^{\top}\bar{ \mathbf{Q}}{\bm{\mu}}_{\beta}-\frac{2\lambda\pi(1-\pi)}{(1+\delta)(1+\delta_{S })^{2}}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{\beta}\right)$ $\displaystyle=\frac{\pi^{2}}{(1+\delta)^{2}}{\bm{\mu}}^{\top}\mathbb{E}[{ \mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}+\frac{\lambda^{2}(1-\pi)^{2}}{(1+ \delta_{S})^{2}}{\bm{\mu}}_{\beta}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf {Q}}]{\bm{\mu}}_{\beta}+\frac{2\lambda\pi(1-\pi)}{(1+\delta)(1+\delta_{S})}{ \bm{\mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}_{\beta}$ $\displaystyle+\frac{\pi T}{(1+\delta)^{2}}\left(1-\frac{2\pi}{1+\delta}{\bm{ \mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}-\frac{2\lambda(1-\pi)}{1+\delta_{S}}{ \bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{\beta}\right)$ $\displaystyle+\frac{(1-\pi)\sigma^{2}T}{(1+\delta_{S})^{2}}\left(\alpha-\frac{ 2\lambda^{2}(1-\pi)}{1+\delta_{S}}{\bm{\mu}}_{\beta}^{\top}\bar{\mathbf{Q}}{ \bm{\mu}}_{\beta}-\frac{2\lambda\pi}{1+\delta}{\bm{\mu}}^{\top}\bar{\mathbf{Q} }{\bm{\mu}}_{\beta}\right)$ $$

And we can compute this since we have that:

$$ $\displaystyle{\bm{\mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{ \mu}}=\frac{1}{h}\left((1-b_{2})\left(({\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{ \mu}})^{2}+{\bm{\mu}}^{\top}\bar{\mathbf{Q}}^{2}{\bm{\mu}}\right)+b_{1}\left(( {\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{\beta})^{2}+\sigma^{2}{\bm{\mu}}^ {\top}\bar{\mathbf{Q}}^{2}{\bm{\mu}}\right)\right),$ $\displaystyle{\bm{\mu}}_{\beta}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q} }]{\bm{\mu}}_{\beta}=\frac{1}{h}\left((1-b_{2})\left[({\bm{\mu}}_{\beta}^{\top }\bar{\mathbf{Q}}{\bm{\mu}})^{2}+{\bm{\mu}}_{\beta}^{\top}\bar{\mathbf{Q}}^{2} {\bm{\mu}}_{\beta}\right]+b_{1}\left[({\bm{\mu}}_{\beta}^{\top}\bar{\mathbf{Q} }{\bm{\mu}}_{\beta})^{2}+\sigma^{2}{\bm{\mu}}_{\beta}^{\top}\bar{\mathbf{Q}}^{ 2}{\bm{\mu}}_{\beta}\right]\right)$ $\displaystyle{\bm{\mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{ \mu}}_{\beta}=\frac{1}{h}\left((1-b_{2})\left[{\bm{\mu}}^{\top}\bar{\mathbf{Q} }{\bm{\mu}}.{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{\beta}+{\bm{\mu}}^{ \top}\bar{\mathbf{Q}}^{2}{\bm{\mu}}_{\beta}\right]+b_{1}\left[{\bm{\mu}}^{\top }\bar{\mathbf{Q}}{\bm{\mu}}_{\beta}.{\bm{\mu}}_{\beta}^{\top}\bar{\mathbf{Q}}{ \bm{\mu}}_{\beta}+\sigma^{2}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}^{2}{\bm{\mu}}_{ \beta}\right]\right)$ $\displaystyle{\bm{\mu}}_{\beta}\bar{\mathbf{Q}}{\bm{\mu}}_{\beta}=\frac{{\bm{ \mu}}_{\beta}^{\top}{\mathbf{R}}_{2}{\bm{\mu}}_{\beta}}{1+\frac{\alpha(1-\pi)} {1+\delta_{S}}{\bm{\mu}}_{\beta}{\mathbf{R}}_{2}{\bm{\mu}}_{\beta}},\quad{\bm{ \mu}}^{\top}\bar{\mathbf{Q}}^{2}{\bm{\mu}}=\frac{{\bm{\mu}}^{\top}{\mathbf{R}} _{1}^{2}{\bm{\mu}}}{\left(1+\frac{\pi}{1+\delta}{\bm{\mu}}^{\top}{\mathbf{R}}_ {1}{\bm{\mu}}\right)^{2}},\quad{\bm{\mu}}_{\beta}^{\top}\bar{\mathbf{Q}}^{2}{ \bm{\mu}}_{\beta}=\frac{{\bm{\mu}}_{\beta}^{\top}{\mathbf{R}}_{2}^{2}{\bm{\mu} }_{\beta}}{\left(1+\frac{\alpha(1-\pi)}{1+\delta_{S}}{\bm{\mu}}_{\beta}^{\top} {\mathbf{R}}_{2}{\bm{\mu}}_{\beta}\right)^{2}},$ $\displaystyle{\bm{\mu}}^{\top}\bar{\mathbf{Q}}^{2}{\bm{\mu}}_{\beta}=\frac{{ \bm{\mu}}_{\beta}^{\top}{\mathbf{R}}_{2}{\mathbf{R}}_{1}{\bm{\mu}}}{\left(1+ \frac{\pi}{1+\delta}{\bm{\mu}}^{\top}{\mathbf{R}}_{1}{\bm{\mu}}\right)\left(1+ \frac{\alpha(1-\pi)}{1+\delta_{S}}{\bm{\mu}}_{\beta}^{\top}{\mathbf{R}}_{2}{ \bm{\mu}}_{\beta}\right)}$ $$

###### Theorem C.4 (Gaussianity of the 6 model for ${\mathbf{C}}=\sigma^{2}{\mathbf{I}}_{p}$ ) .

Let ${\bm{w}}_{q}$ be the Mixed classifier as defined in equation 6 and suppose that Assumption 4.1 holds. The decision function ${\bm{w}}_{q}^{\top}{\bm{x}}$, on some test sample ${\bm{x}}\in{\mathcal{C}}_{a}$ independent of ${\mathbf{X}}$, satisfies:

$$ $\displaystyle{\bm{w}}_{q}^{\top}{\bm{x}}\,\,\operatorname{\xrightarrow{{ \mathcal{D}}}}\,\,{\mathcal{N}}\left((-1)^{a}m_{q},\,\nu_{q}-m_{q}^{2}\right),$ $$

where:

$$ $\displaystyle m_{q}$ $\displaystyle=\left(\frac{\pi}{1+\delta}{\bm{\mu}}^{\top}+\frac{\lambda(1-\pi) }{1+\delta_{S}}{\bm{\mu}}_{\beta}^{\top}\right)\bar{\mathbf{Q}}{\bm{\mu}},$ $\displaystyle\nu_{q}$ $\displaystyle=\frac{\pi^{2}}{(1+\delta)^{2}}{\bm{\mu}}^{\top}\mathbb{E}[{ \mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}+\frac{\lambda^{2}(1-\pi)^{2}}{(1+ \delta_{S})^{2}}{\bm{\mu}}_{\beta}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf {Q}}]{\bm{\mu}}_{\beta}+\frac{2\lambda\pi(1-\pi)}{(1+\delta)(1+\delta_{S})}{ \bm{\mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu}}_{\beta}$ $\displaystyle+\frac{\pi T}{(1+\delta)^{2}}\left(1-\frac{2\pi}{1+\delta}{\bm{ \mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}-\frac{2\lambda(1-\pi)}{1+\delta_{S}}{ \bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}_{\beta}\right)$ $\displaystyle+\frac{(1-\pi)\sigma^{2}T}{(1+\delta_{S})^{2}}\left(\alpha-\frac{ 2\lambda^{2}(1-\pi)}{1+\delta_{S}}{\bm{\mu}}_{\beta}^{\top}\bar{\mathbf{Q}}{ \bm{\mu}}_{\beta}-\frac{2\lambda\pi}{1+\delta}{\bm{\mu}}^{\top}\bar{\mathbf{Q} }{\bm{\mu}}_{\beta}\right).$ $$

With:

$$ $\displaystyle\lambda=\phi(1-\varepsilon)-\rho\varepsilon,\quad\delta_{S}= \alpha\sigma^{2}\delta$ $$

###### Theorem C.4 (Gaussianity of the 6 model for 𝐂 = σ 2 ⁢ 𝐈 p 𝐂 superscript 𝜎 2 subscript 𝐈 𝑝 {\mathbf{C}}=\sigma^{2}{\mathbf{I}}_{p} bold_C = italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT bold_I start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ) .

## Appendix D Random matrix analysis of distribution shift

We will now quantify the performance of the classifier obtained through mixing some real data and synthetic data sampled according to the schema described in 2. Hence, the matrix $\bar{\mathbf{Q}}$, defined in lemma A.4, is no longer deterministic as we take the covariance matrix $\hat{{\mathbf{C}}}=\frac{1}{\hat{n}}\sum_{i=1}^{\hat{n}}({\bm{x}}_{i}-y_{i}
\hat{{\bm{\mu}}})({\bm{x}}_{i}-y_{i}\hat{{\bm{\mu}}})^{\top}$. For simplicity, and without loss of generality, we consider $\hat{n}$ Gaussian vectors $({\bm{z}}_{i})_{i=1}^{\hat{n}}\sim{\mathcal{N}}(0,{\mathbf{I}}_{p})$ that are independent of $({\bm{x}}_{i})_{i=1}^{\hat{n}}$, and write:

$$ $\hat{{\bm{\mu}}}={\bm{\mu}}_{\beta}={\bm{\mu}}+\frac{1}{\hat{n}}\sum_{i=1}^{ \hat{n}}{\bm{z}}_{i},\quad\hat{{\mathbf{C}}}=\frac{1}{\hat{n}}\sum_{i=1}^{\hat {n}}{\bm{z}}_{i}{\bm{z}}_{i}^{\top}$ $$

Note that we can ignore the error of estimation of $\hat{{\bm{\mu}}}$ because we have that:

$$ $\mathbb{E}\left[\frac{1}{\hat{n}}\sum_{i=1}^{\hat{n}}{\bm{z}}_{i}\right]=0, \quad\mathbb{E}\left[\left(\frac{1}{\hat{n}}\sum_{i=1}^{\hat{n}}{\bm{z}}_{i} \right)\left(\frac{1}{\hat{n}}\sum_{j=1}^{\hat{n}}{\bm{z}}_{j}\right)^{\top} \right]=\frac{1}{\hat{n}}{\mathbf{I}}_{p}$ $$

Hence, when we have a sufficiently large $\hat{n}$, we will assume that: $\hat{{\bm{\mu}}}={\bm{\mu}}$ (the estimation error is on ${\mathcal{O}}(\hat{n}^{-1})$).

### D.1 Deterministic equivalents:

The resolvent matrix to be considered in this setting is the one defined in lemma A.4 but with $\hat{{\mathbf{C}}}$:

$$ $\displaystyle\bar{\mathbf{Q}}(\gamma)$ $\displaystyle=\left(\left(\frac{\pi}{1+\delta}+\frac{\alpha(1-\pi)}{1+\delta_{ S}}\right){\bm{\mu}}{\bm{\mu}}^{\top}+\frac{\alpha(1-\pi)}{1+\delta_{S}}\hat{{ \mathbf{C}}}+\left(\gamma+\frac{\pi}{1+\delta}\right){\mathbf{I}}_{p}\right)^{ -1}$ $\displaystyle=\left(\left(\frac{\pi}{1+\delta}+\frac{\alpha(1-\pi)}{1+\delta_{ S}}\right){\bm{\mu}}{\bm{\mu}}^{\top}+\frac{\alpha(1-\pi)}{(1+\delta_{S})\hat{ n}}\sum_{i=1}^{\hat{n}}{\bm{z}}_{i}{\bm{z}}_{i}^{\top}+\left(\gamma+\frac{\pi} {1+\delta}\right){\mathbf{I}}_{p}\right)^{-1}$ $$

where:

$$ $\delta=\frac{1}{N}\operatorname{Tr}(\bar{\mathbf{Q}}),\quad\delta_{S}=\frac{ \alpha}{N}\operatorname{Tr}(\hat{{\mathbf{C}}}\bar{\mathbf{Q}})$ $$

Let us denote by $\bar{\mathbf{Q}}_{-i}$ the resolvent matrix gotten by removing its dependence on the vector ${\bm{z}}_{i}$. In other words:

$$ $\bar{\mathbf{Q}}_{-i}=\left(\bar{\mathbf{Q}}-\frac{\alpha(1-\pi)}{\hat{n}(1+ \delta_{S})}{\bm{z}}_{i}{\bm{z}}_{i}^{\top}\right)^{-1},\quad\bar{\mathbf{Q}}= \left(\bar{\mathbf{Q}}_{-i}+\frac{\alpha(1-\pi)}{\hat{n}(1+\delta_{S})}{\bm{z} }_{i}{\bm{z}}_{i}^{\top}\right)^{-1}$ $$

By Sherman-Morisson’s lemma A.3, we have that:

$$ $\bar{\mathbf{Q}}=\bar{\mathbf{Q}}_{-i}-\frac{\frac{\alpha(1-\pi)}{\hat{n}(1+ \delta_{S})}\bar{\mathbf{Q}}_{-i}{\bm{z}}_{i}{\bm{z}}_{i}^{\top}\bar{\mathbf{Q }}_{-i}}{1+\frac{\alpha(1-\pi)}{\hat{n}(1+\delta_{S})}{\bm{z}}_{i}^{\top}\bar{ \mathbf{Q}}_{-i}{\bm{z}}_{i}}$ $$

And:

$$ $\bar{\mathbf{Q}}{\bm{z}}_{i}=\frac{\bar{\mathbf{Q}}_{-i}{\bm{z}}_{i}}{1+\frac{ \alpha(1-\pi)}{\hat{n}(1+\delta_{S})}{\bm{z}}_{i}^{\top}\bar{\mathbf{Q}}_{-i}{ \bm{z}}_{i}}=\frac{\bar{\mathbf{Q}}_{-i}{\bm{z}}_{i}}{1+\bar{\delta}}$ $$

where:

$$ $\bar{\delta}=\frac{\alpha(1-\pi)}{1+\delta_{S}}\frac{1}{\hat{n}}\operatorname{ Tr}(\bar{\mathbf{Q}})$ (27) $$

Since the covariance estimate in equation 2 is stochastic, the matrix $\bar{\mathbf{Q}}$ is no longer deterministic when replacing ${\mathbf{C}}$ with $\hat{{\mathbf{C}}}$. Hence, we will give a further deterministic equivalent to $\bar{\mathbf{Q}}$ in the following lemma.

###### Lemma D.1 (Second Deterministic equivalent) .

A deterministic equivalent of $\bar{\mathbf{Q}}$ is given by:

$$ $\bar{\bar{\mathbf{Q}}}=\left(\left(\frac{\pi}{1+\delta}+\frac{\alpha(1-\pi)}{1 +\delta_{S}}\right){\bm{\mu}}{\bm{\mu}}^{\top}+\left(\gamma+\frac{\pi}{1+ \delta}+\frac{\alpha(1-\pi)}{(1+\delta_{S})(1+\bar{\delta})}\right){\mathbf{I} }_{p}\right)^{-1}$ $$

where $\bar{\delta}$ can be found as a fixed point using the following identity:

$$ $\bar{\delta}=\frac{\alpha(1-\pi)}{(1+\delta_{S})}\frac{1}{\hat{n}} \operatorname{Tr}(\bar{\bar{\mathbf{Q}}})=\frac{\alpha(1-\pi)}{(1+\delta_{S})} \frac{\frac{p}{\hat{n}}}{\gamma+\frac{\pi}{1+\delta}+\frac{\alpha(1-\pi)}{(1+ \delta_{S})(1+\bar{\delta})}},\quad\delta=\frac{1}{N}\operatorname{Tr}(\bar{ \bar{\mathbf{Q}}})=\frac{\hat{n}}{N}\frac{(1+\delta_{S})}{\alpha(1-\pi)}\bar{\delta}$ $$

$$ $\delta_{S}=\frac{\alpha}{N}\operatorname{Tr}(\mathbb{E}[\hat{{\mathbf{C}}}\bar {\mathbf{Q}}])=\frac{\alpha\delta}{1+\bar{\delta}}$ $$

Now we will prove the deterministic equivalent given by lemma D.1.

###### Lemma D.1 (Second Deterministic equivalent) .

### Proof of lemma D.1 :

Let us denote $\bar{\mathbf{Q}}=\left(\frac{\alpha(1-\pi)}{1+\delta_{S}}\frac{1}{\hat{n}}\sum
_{i=1}^{\hat{n}}{\bm{z}}_{i}{\bm{z}}_{i}^{\top}+{\mathbf{A}}\right)^{-1}$. Let also $\bar{\bar{\mathbf{Q}}}$ be the deterministic equivalent of $\bar{\mathbf{Q}}$. It can be written as: $\bar{\bar{\mathbf{Q}}}=\left({\mathbf{S}}+{\mathbf{A}}\right)^{-1}$. We want to find some ${\mathbf{S}}$ such that for all ${\bm{a}},{\bm{b}}\in{\mathbb{R}}^{p}$:

$$ ${\bm{a}}^{\top}\mathbb{E}[\bar{\mathbf{Q}}]{\bm{b}}\to{\bm{a}}^{\top}\bar{\bar {\mathbf{Q}}}{\bm{b}}$ $$

We have that:

$$ $\displaystyle\mathbb{E}[\bar{\mathbf{Q}}]-\bar{\bar{\mathbf{Q}}}$ $\displaystyle=\mathbb{E}\left[\bar{\mathbf{Q}}({\mathbf{S}}-\frac{\alpha(1-\pi )}{1+\delta_{S}}\frac{1}{\hat{n}}\sum_{i=1}^{\hat{n}}{\bm{z}}_{i}{\bm{z}}_{i}^ {\top})\bar{\bar{\mathbf{Q}}}\right]$ $\displaystyle=\frac{1}{\hat{n}}\sum_{i=1}^{\hat{n}}\mathbb{E}[\bar{\mathbf{Q}} ({\mathbf{S}}-\frac{\alpha(1-\pi)}{1+\delta_{S}}{\bm{z}}_{i}{\bm{z}}_{i}^{\top })\bar{\bar{\mathbf{Q}}}]$ $\displaystyle=\frac{1}{\hat{n}}\sum_{i=1}^{\hat{n}}\mathbb{E}\left[\bar{ \mathbf{Q}}{\mathbf{S}}-\frac{\alpha(1-\pi)}{1+\delta_{S}}\frac{1}{1+\bar{ \delta}}\bar{\mathbf{Q}}_{-i}{\bm{z}}_{i}{\bm{z}}_{i}^{\top}\right]\bar{\bar{ \mathbf{Q}}}$ $\displaystyle=\frac{1}{\hat{n}}\sum_{i=1}^{\hat{n}}\mathbb{E}\left[\bar{ \mathbf{Q}}_{-i}\left({\mathbf{S}}-\frac{\alpha(1-\pi)}{1+\delta_{S}}\frac{1}{ 1+\bar{\delta}}{\bm{z}}_{i}{\bm{z}}_{i}^{\top}\right)\right]\bar{\bar{\mathbf{ Q}}}+{\mathcal{O}}(\hat{n}^{-1})$ $$

Hence, it suffices to have ${\mathbf{S}}=\mathbb{E}[\frac{\alpha(1-\pi)}{1+\delta_{S}}\frac{1}{1+\bar{
\delta}}{\bm{z}}_{i}{\bm{z}}_{i}^{\top}]=\frac{\alpha(1-\pi)}{1+\delta_{S}}
\frac{1}{1+\bar{\delta}}{\mathbf{I}}_{p}$, and thus:

### D.2 Deterministic equivalent of 𝐐 ¯ ⁢ 𝐀 ⁢ 𝐐 ¯ ¯ 𝐐 𝐀 ¯ 𝐐 \bar{\mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}} over¯ start_ARG bold_Q end_ARG bold_A over¯ start_ARG bold_Q end_ARG :

Let ${\mathbf{A}}\in{\mathbb{R}}^{p\times p}$ be some deterministic matrix. We have that:

$$ $\displaystyle\mathbb{E}[\bar{\mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}}]$ $\displaystyle=\bar{\bar{\mathbf{Q}}}{\mathbf{A}}\bar{\bar{\mathbf{Q}}}+\mathbb {E}[(\bar{\mathbf{Q}}-\bar{\bar{\mathbf{Q}}}){\mathbf{A}}\bar{\mathbf{Q}}]$ $\displaystyle=\bar{\bar{\mathbf{Q}}}{\mathbf{A}}\bar{\bar{\mathbf{Q}}}+\mathbb {E}[\bar{\mathbf{Q}}(\bar{\bar{\mathbf{Q}}}^{-1}-\bar{\mathbf{Q}}^{-1})\bar{ \bar{\mathbf{Q}}}{\mathbf{A}}\bar{\mathbf{Q}}]$ $\displaystyle=\bar{\bar{\mathbf{Q}}}{\mathbf{A}}\bar{\bar{\mathbf{Q}}}+\frac{ \alpha(1-\pi)}{(1+\delta_{S})}\mathbb{E}[\bar{\mathbf{Q}}\left(\frac{1}{1+\bar {\delta}}-\frac{1}{\hat{n}}\sum_{i=1}^{\hat{n}}{\bm{z}}_{i}{\bm{z}}_{i}^{\top} \right)\bar{\bar{\mathbf{Q}}}{\mathbf{A}}\bar{\mathbf{Q}}]$ $\displaystyle=\bar{\bar{\mathbf{Q}}}{\mathbf{A}}\bar{\bar{\mathbf{Q}}}+\frac{ \alpha(1-\pi)}{(1+\delta_{S})}\left(\frac{1}{1+\bar{\delta}}\mathbb{E}[\bar{ \mathbf{Q}}\bar{\bar{\mathbf{Q}}}{\mathbf{A}}\bar{\mathbf{Q}}]-\frac{1}{\hat{n }}\sum_{i=1}^{\hat{n}}\mathbb{E}[\bar{\mathbf{Q}}{\bm{z}}_{i}{\bm{z}}_{i}^{ \top}\bar{\bar{\mathbf{Q}}}{\mathbf{A}}\bar{\mathbf{Q}}]\right)$ $$

And we have that for $i\in\{1,...,\hat{n}\}$:

$$ $\displaystyle\mathbb{E}[\bar{\mathbf{Q}}{\bm{z}}_{i}{\bm{z}}_{i}^{\top}\bar{ \bar{\mathbf{Q}}}{\mathbf{A}}\bar{\mathbf{Q}}]$ $\displaystyle=\frac{1}{1+\bar{\delta}}\mathbb{E}[\bar{\mathbf{Q}}_{-i}{\bm{z}} _{i}{\bm{z}}_{i}^{\top}\bar{\bar{\mathbf{Q}}}{\mathbf{A}}\bar{\mathbf{Q}}]$ $\displaystyle=\frac{1}{1+\bar{\delta}}\mathbb{E}\left[\bar{\mathbf{Q}}_{-i}{ \bm{z}}_{i}{\bm{z}}_{i}^{\top}\bar{\bar{\mathbf{Q}}}{\mathbf{A}}\left(\bar{ \mathbf{Q}}_{-i}-\frac{\alpha(1-\pi)}{\hat{n}(1+\delta_{S})(1+\bar{\delta})} \bar{\mathbf{Q}}_{-i}{\bm{z}}_{i}{\bm{z}}_{i}^{\top}\bar{\mathbf{Q}}_{-i} \right)\right]$ $\displaystyle=\frac{1}{1+\bar{\delta}}\mathbb{E}[\bar{\mathbf{Q}}_{-i}{\bm{z}} _{i}{\bm{z}}_{i}^{\top}\bar{\bar{\mathbf{Q}}}{\mathbf{A}}\bar{\mathbf{Q}}_{-i} ]-\frac{\alpha(1-\pi)}{\hat{n}(1+\delta_{S})(1+\bar{\delta})^{2}}\mathbb{E}[ \bar{\mathbf{Q}}_{-i}{\bm{z}}_{i}{\bm{z}}_{i}^{\top}\bar{\bar{\mathbf{Q}}}{ \mathbf{A}}\bar{\mathbf{Q}}_{-i}{\bm{z}}_{i}{\bm{z}}_{i}^{\top}\bar{\mathbf{Q} }_{-i}]$ $\displaystyle=\frac{1}{1+\bar{\delta}}\mathbb{E}[\bar{\mathbf{Q}}\bar{\bar{ \mathbf{Q}}}{\mathbf{A}}\bar{\mathbf{Q}}]-\frac{\alpha(1-\pi)}{\hat{n}(1+ \delta_{S})(1+\bar{\delta})^{2}}\operatorname{Tr}(\bar{\bar{\mathbf{Q}}}{ \mathbf{A}}\bar{\bar{\mathbf{Q}}})\mathbb{E}[\bar{\mathbf{Q}}^{2}]$ $$

Hence by replacing this term in he previous sum, we get the following result.

###### Lemma D.2 (Deterministic equivalent of $\bar{\mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}}$ ) .

Let ${\mathbf{A}}\in{\mathbb{R}}^{p\times p}$ be any deterministic symmetric semi-definite matrix. We have that:

$$ $\bar{\mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}}\leftrightarrow\bar{\bar{\mathbf{Q }}}{\mathbf{A}}\bar{\bar{\mathbf{Q}}}+\left(\frac{\alpha(1-\pi)}{(1+\delta_{S} )(1+\bar{\delta})}\right)^{2}\frac{1}{\hat{n}}\operatorname{Tr}(\bar{\bar{ \mathbf{Q}}}{\mathbf{A}}\bar{\bar{\mathbf{Q}}})\mathbb{E}[\bar{\mathbf{Q}}^{2}]$ $$

In particular, we have that:

$$ $\displaystyle\bar{\mathbf{Q}}^{2}\leftrightarrow\frac{1}{\bar{h}}\bar{\bar{ \mathbf{Q}}}^{2},\quad\bar{\mathbf{Q}}{\bm{\mu}}{\bm{\mu}}^{\top}\bar{\mathbf{ Q}}\leftrightarrow\bar{\bar{\mathbf{Q}}}{\bm{\mu}}{\bm{\mu}}^{\top}\bar{\bar{ \mathbf{Q}}}$ $$

where:

$$ $\displaystyle\bar{h}=1-\left(\frac{\alpha(1-\pi)}{(1+\delta_{S})(1+\bar{\delta })}\right)^{2}\frac{1}{\hat{n}}\operatorname{Tr}(\bar{\bar{\mathbf{Q}}}^{2})$ $$

###### Lemma D.2 (Deterministic equivalent of 𝐐 ¯ ⁢ 𝐀 ⁢ 𝐐 ¯ ¯ 𝐐 𝐀 ¯ 𝐐 \bar{\mathbf{Q}}{\mathbf{A}}\bar{\mathbf{Q}} over¯ start_ARG bold_Q end_ARG bold_A over¯ start_ARG bold_Q end_ARG ) .

### D.3 Useful results:

Here we will list all the results with $\bar{\bar{\mathbf{Q}}}$ that will be useful in this analysis. Let us denote by $a,b$ the following quantities:

$$ $\displaystyle a=\left(\frac{\pi}{1+\delta}+\frac{\alpha(1-\pi)}{1+\delta_{S}} \right),\quad b=\gamma+\frac{\pi}{1+\delta}+\frac{\alpha(1-\pi)}{(1+\delta_{S} )(1+\bar{\delta})}$ $$

such that:

$$ $\bar{\bar{\mathbf{Q}}}=\left(a{\bm{\mu}}{\bm{\mu}}^{\top}+b{\mathbf{I}}_{p} \right)^{-1}$ (28) $$

By Sherman-Morisson’s lemma A.3, we have that:

$$ $\bar{\bar{\mathbf{Q}}}=\frac{1}{b}\left({\mathbf{I}}_{p}-\frac{a{\bm{\mu}}{\bm {\mu}}^{\top}}{b+a\|{\bm{\mu}}\|^{2}}\right),\quad\bar{\bar{\mathbf{Q}}}{\bm{ \mu}}=\frac{{\bm{\mu}}}{b+a\|{\bm{\mu}}\|^{2}}$ (29) $$

We also have that the constants $a_{1},a_{2},b_{1}$ and $b_{2}$ from lemma A.5 become by taking their expectations on ${\bm{z}}$:

###### Lemma D.3 (New values of constants) .

$$ $\displaystyle a_{1}=\frac{\pi}{N(1+\delta)^{2}}\frac{1}{\bar{h}}\operatorname{ Tr}(\bar{\bar{\mathbf{Q}}}^{2}),\quad b_{1}=\frac{\alpha(1-\pi)}{N(1+\delta_{S })^{2}}\frac{1}{\bar{h}(1+\bar{\delta})^{2}}\operatorname{Tr}(\bar{\bar{ \mathbf{Q}}}^{2})$ $\displaystyle a_{2}=\frac{\pi}{N(1+\delta)^{2}}\frac{1}{\bar{h}(1+\bar{\delta} )^{2}}\operatorname{Tr}(\bar{\bar{\mathbf{Q}}}^{2}),\quad b_{2}=\frac{\alpha(1 -\pi)}{N(1+\delta_{S})^{2}}\frac{1}{\bar{h}(1+\bar{\delta})^{4}}\operatorname{ Tr}(\bar{\bar{\mathbf{Q}}}^{2})$ $$

where:

$$ $\frac{1}{N}\operatorname{Tr}(\bar{\bar{\mathbf{Q}}}^{2})=\frac{\eta}{b^{2}}$ (30) $$

###### Lemma D.3 (New values of constants) .

### D.4 Test Expectation:

It only suffices to apply the expectation on ${\bm{z}}_{i}$ to $m_{q}$ obtained with the general model in theorem B.1. Hence:

$$ $\displaystyle\mathbb{E}[{\bm{w}}_{q}^{\top}{\bm{x}}]$ $\displaystyle=(-1)^{a}\left(\frac{\pi}{1+\delta}+\frac{\lambda(1-\pi)}{1+ \delta_{S}}\right)\mathbb{E}[{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}]$ $\displaystyle=(-1)^{a}\left(\frac{\pi}{1+\delta}+\frac{\lambda(1-\pi)}{1+ \delta_{S}}\right){\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}{\bm{\mu}}$ $\displaystyle=(-1)^{a}\left(\frac{\pi}{1+\delta}+\frac{\lambda(1-\pi)}{1+ \delta_{S}}\right)\frac{\|{\bm{\mu}}\|^{2}}{b+a\|{\bm{\mu}}\|^{2}}$ $$

### D.5 Test variance:

Using theorem B.1, we need to apply the expectation on ${\bm{z}}$ to the following second order moment:

$$ $\displaystyle\nu_{q}$ $\displaystyle=\left(\frac{\pi}{1+\delta}+\frac{\lambda(1-\pi)}{1+\delta_{S}} \right)^{2}{\bm{\mu}}^{\top}\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]{\bm{\mu }}+\frac{\pi T_{1}}{(1+\delta)^{2}}\left(1-\frac{2\pi}{1+\delta}{\bm{\mu}}^{ \top}\bar{\mathbf{Q}}{\bm{\mu}}-\frac{2\lambda(1-\pi)}{1+\delta_{S}}{\bm{\mu}} ^{\top}\bar{\mathbf{Q}}{\bm{\mu}}\right)$ $\displaystyle+\frac{(1-\pi)T_{2}}{(1+\delta_{S})^{2}}\left(\alpha-\frac{2 \lambda^{2}(1-\pi)}{1+\delta_{S}}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}- \frac{2\lambda\pi}{1+\delta}{\bm{\mu}}^{\top}\bar{\mathbf{Q}}{\bm{\mu}}\right)$ $$

where:

$$ $T_{1}=\frac{1}{N}\operatorname{Tr}(\Sigma\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf {Q}}]),\quad T_{2}=\frac{1}{N}\operatorname{Tr}(\Sigma_{\beta}\mathbb{E}[{ \mathbf{Q}}\Sigma{\mathbf{Q}}])$ $$

and these two quantities are obtained using corollary A.6 and lemma D.3, which after simplification are given by:

$$ $T_{1}=\frac{(1+\delta)^{2}}{\pi h}a_{1},\quad T_{2}=\frac{(1+\delta_{S})^{2}}{ \alpha(1-\pi)h}b_{1}$ (31) $$

Now we should define the new deterministic equivalent of $\bar{\mathbf{Q}}\Sigma\bar{\mathbf{Q}}$ to obtain an expression of $\mathbb{E}_{\bm{z}}[\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]]$ and to finish this calculus !
Let :

$$ $\bar{h}=1-\left(\frac{\alpha(1-\pi)}{(1+\delta_{S})(1+\bar{\delta})}\right)^{2 }\frac{1}{\hat{n}}\operatorname{Tr}(\bar{\bar{\mathbf{Q}}}^{2})$ (32) $$

Then, using lemma D.2 we have that the following identities stand for any linear form:

$$ $\displaystyle\mathbb{E}[\bar{\mathbf{Q}}\Sigma\bar{\mathbf{Q}}]=\bar{\bar{ \mathbf{Q}}}{\bm{\mu}}{\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}+\frac{1}{\bar{h} }\bar{\bar{\mathbf{Q}}}^{2},\quad\bar{\mathbb{E}}[\bar{\mathbf{Q}}\Sigma_{ \beta}\bar{\mathbf{Q}}]=\bar{\bar{\mathbf{Q}}}{\bm{\mu}}{\bm{\mu}}^{\top}\bar{ \bar{\mathbf{Q}}}+\frac{1}{(1+\bar{\delta})^{2}}\frac{1}{\bar{h}}\bar{\bar{ \mathbf{Q}}}^{2}$ $$

Thus:

$$ $\displaystyle\mathbb{E}_{\bm{z}}[\mathbb{E}[{\mathbf{Q}}\Sigma{\mathbf{Q}}]]$ $\displaystyle=\frac{1-b_{2}}{h}\mathbb{E}[\bar{\mathbf{Q}}\Sigma\bar{\mathbf{Q }}]+\frac{b_{1}}{h}\mathbb{E}[\bar{\mathbf{Q}}\Sigma_{\beta}\bar{\mathbf{Q}}]$ $\displaystyle=\frac{1}{h}\left((1-b_{2})\left(\bar{\bar{\mathbf{Q}}}{\bm{\mu}} {\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}+\frac{1}{\bar{h}}\bar{\bar{\mathbf{Q}} }^{2}\right)+b_{1}\left(\bar{\bar{\mathbf{Q}}}{\bm{\mu}}{\bm{\mu}}^{\top}\bar{ \bar{\mathbf{Q}}}+\frac{1}{\bar{h}(1+\bar{\delta})^{2}}\bar{\bar{\mathbf{Q}}}^ {2}\right)\right)$ $\displaystyle=\frac{1}{h}\left((1+b_{1}-b_{2})\bar{\bar{\mathbf{Q}}}{\bm{\mu}} {\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}+\frac{1}{\bar{h}}\left(1-b_{2}+\frac{b _{1}}{(1+\bar{\delta})^{2}}\right)\bar{\bar{\mathbf{Q}}}^{2}\right)$ $\displaystyle=\frac{1}{h}\left((1+b_{1}-b_{2})\bar{\bar{\mathbf{Q}}}{\bm{\mu}} {\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}+\frac{1}{\bar{h}}\bar{\bar{\mathbf{Q}} }^{2}\right)$ $$

because:

$$ $\frac{b_{1}}{(1+\bar{\delta})^{2}}=b_{2}$ $$

Finally, we get the second order moment:

$$ $\displaystyle\mathbb{E}[({\bm{w}}_{q}^{\top}{\bm{x}})^{2}]=\left(\frac{\pi}{1+ \delta}+\frac{\lambda(1-\pi)}{1+\delta_{S}}\right)^{2}\frac{1}{h}{\bm{\mu}}^{ \top}\left((1+b_{1}-b_{2})\bar{\bar{\mathbf{Q}}}{\bm{\mu}}{\bm{\mu}}^{\top} \bar{\bar{\mathbf{Q}}}+\frac{1}{\bar{h}}\bar{\bar{\mathbf{Q}}}^{2}\right){\bm{ \mu}}$ $\displaystyle+\frac{\pi T_{1}}{(1+\delta)^{2}}\left(1-\frac{2\pi}{1+\delta}{ \bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}{\bm{\mu}}-\frac{2\lambda(1-\pi)}{1+ \delta_{S}}{\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}{\bm{\mu}}\right)$ $\displaystyle+\frac{(1-\pi)T_{2}}{(1+\delta_{S})^{2}}\left(\alpha-\frac{2 \lambda^{2}(1-\pi)}{1+\delta_{S}}{\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}{\bm{ \mu}}-\frac{2\lambda\pi}{1+\delta}{\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}{\bm{ \mu}}\right)$ $\displaystyle=\frac{1}{h}\left(\frac{\pi}{1+\delta}+\frac{\lambda(1-\pi)}{1+ \delta_{S}}\right)^{2}\left((1+b_{1}-b_{2})({\bm{\mu}}^{\top}\bar{\bar{\mathbf {Q}}}{\bm{\mu}})^{2}+\frac{1}{\bar{h}}{\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}^ {2}{\bm{\mu}}\right)$ $\displaystyle+\frac{\pi T_{1}}{(1+\delta)^{2}}\left(1-\frac{2\pi}{1+\delta}{ \bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}{\bm{\mu}}-\frac{2\lambda(1-\pi)}{1+ \delta_{S}}{\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}{\bm{\mu}}\right)$ $\displaystyle+\frac{(1-\pi)T_{2}}{(1+\delta_{S})^{2}}\left(\alpha-\frac{2 \lambda^{2}(1-\pi)}{1+\delta_{S}}{\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}{\bm{ \mu}}-\frac{2\lambda\pi}{1+\delta}{\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}{\bm{ \mu}}\right)$ $\displaystyle=\frac{1}{h}c^{2}\left((1+b_{1}-b_{2})({\bm{\mu}}^{\top}\bar{\bar {\mathbf{Q}}}{\bm{\mu}})^{2}+\frac{1}{\bar{h}}{\bm{\mu}}^{\top}\bar{\bar{ \mathbf{Q}}}^{2}{\bm{\mu}}\right)$ $\displaystyle+\frac{a_{1}}{h}\left(1-2c{\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}} {\bm{\mu}}\right)+\frac{b_{1}}{\alpha h}\left(\alpha-2\lambda c{\bm{\mu}}^{ \top}\bar{\bar{\mathbf{Q}}}{\bm{\mu}}\right)$ $$

Note that:

$$ $\displaystyle{\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q}}}{\bm{\mu}}=\frac{\|{\bm{ \mu}}\|^{2}}{b+a\|{\bm{\mu}}\|^{2}},\quad{\bm{\mu}}^{\top}\bar{\bar{\mathbf{Q} }}^{2}{\bm{\mu}}=\frac{\|{\bm{\mu}}\|^{2}}{\left(b+a\|{\bm{\mu}}\|^{2}\right)^ {2}},\quad c=\left(\frac{\pi}{1+\delta}+\frac{\lambda(1-\pi)}{1+\delta_{S}}\right)$ $$

Therefore:

$$ $\mathbb{E}[({\bm{w}}_{q}^{\top}{\bm{x}})^{2}]=\frac{c\|{\bm{\mu}}\|^{2}}{h(b+a \|{\bm{\mu}}\|^{2})^{2}}\left(c(1+b_{1}-b_{2})\|{\bm{\mu}}\|^{2}+\frac{c}{\bar {h}}-2\left(a_{1}+\frac{\lambda b_{1}}{\alpha}\right)(b+a\|{\bm{\mu}}\|^{2}) \right)+\frac{a_{1}+b_{1}}{h}$ $$

which concludes the proof of the main theorem 4.2 of this paper.

## Appendix E Details about experiments with Safety LLM Alignment with IPO

### E.1 Hyperparameters

**Table 1: Implementation Details for the safety LLM alignment with IPO**
| Parameter | Value |
| --- | --- |
| use_flash_attention_2 | true |
| LoRA Arguments |  |
| lora_r | 128 |
| lora_alpha | 128 |
| lora_dropout | 0.05 |
| preprocessing_num_workers | 12 |
| Trainer Arguments |  |
| bf16 | true |
| beta | 0.01 |
| eval_steps | 100 |
| gradient_accumulation_steps | 4 |
| gradient_checkpointing | true |
| learning_rate | 5.0e-6 |
| log_level | info |
| logging_steps | 10 |
| lr_scheduler_type | cosine |
| max_length | 1024 |
| max_prompt_length | 512 |
| num_train_epochs | 1 |
| optim | paged_adamw_32bit |
| per_device_train_batch_size | 4 |
| per_device_eval_batch_size | 8 |
| seed | 42 |
| warmup_ratio | 0.1 |
| Label_smoothing | 0.001 |

## Appendix F Details about experiments with LLM QA classification

### F.1 Prompting LLMs

As part of this experiment, we had to generate a synthetic QA Dataset. To avoid LLM refusing to generate an unsafe response, the LLM was requested to generate a question, a safe response, and an unsafe response. Figure 10 shows the system prompt used to request from an LLM to generate QA. $<$Topic$>$ is a placeholder referring to a particular risk topic, selected from the list of topics seen in Figure 11, the section written in red. As discussed in the paper, the generated QA will be annotated by LLM, using the prompt presented in Figure 11.

Figure: Figure 10: Prompt for QA generation
Refer to caption: extracted/5920247/figures/prompt_QA.png

Figure: Figure 11: Prompt for LLM annotations
Refer to caption: extracted/5920247/figures/prompt_label.png

### F.2 Hyper-parameters

**Table 2: Fine-tuning for Llama3.1-8B-Instruct**
| Parameter | Value |
| --- | --- |
| LoRA Parameters |  |
| lora_r | 64 |
| lora_alpha | 64 |
| lora_dropout | 0.05 |
| Training Parameters Parameters |  |
| bf16 | true |
| gradient_accumulation_steps | 4 |
| learning_rate | $1.5\times 10^{-6}<lr<2\times 10^{-5}$ |
| lr_scheduler_type | cosine |
| packing | False |
| max_seq_length | 2048 |
| num_train_epochs | 1 |
| optim | adamw_torch |
| per_device_train_batch_size | 2 |
| warmup_ratio | 0.1 |
| seed | 42 |

**Table 3: Fine-tuning for Gemma-2-2B-it**
| Parameter | Value |
| --- | --- |
| LoRA Parameters |  |
| lora_r | 64 |
| lora_alpha | 64 |
| lora_dropout | 0.05 |
| Training Parameters Parameters |  |
| bf16 | true |
| gradient_accumulation_steps | 4 |
| learning_rate | $1\times 10^{-6}<lr<2\times 10^{-5}$ |
| lr_scheduler_type | cosine |
| packing | False |
| max_seq_length | 2048 |
| num_train_epochs | 1 |
| optim | adamw_torch |
| per_device_train_batch_size | 2 |
| warmup_ratio | 0.1 |
| seed | 42 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: Though I think the paper is potentially a very good paper, I still have a few worries as outlined in the **Weaknesses** section of this review and also the questions listed below (I'm open to changing my mind if addressed).

- Clarify relevance of the consider model (see **Wicknesses** section above)
- Clarify connection to literature, especially Jain et al. (2024) "Scaling laws for learning with real and surrogate data" and Dohmatob et al. (2024) "Strong Model Collapse" (see **Wicknesses** section above)
- Can you instantiate your results on a concrete setup and show the corresponding scaling laws (perhaps with some approximations of Gaussian CDF) ?
soundness: 3
strengths: - A nice theoretical setup for analysing the impact of synthetic data and pruning strategies is introduced.
- A complete analytic theory for this toy setting is obtained and its phenomological clearly discussed.
- Theoretical results from previous work Feng et al. (2024) are recovered as special cases.
- The calculations provided in the appendix could be of independent usefulness, beyond the specific problem considered in the paper. I should however immediately nuance this praise by noting that similar calculations have been done in  Liao and Couillet (2019) "A Large Dimensional Analysis of Least Squares Support Vector Machines", for more general kernel-based models.
- The paper is very clearly written.
summary: This work considers the problem of maximizing the usefulness of synthetic data in the modern era of generative modelling whereby at least part of the training data is not from the real data distribution, but synthesized from another AI model. In the solvable setting of a mixture of isotropic Gaussians, the authors consider a linear model whose weights vector is a regression-style ridge estimator. The synthetic data distribution is taken as yet another Gaussian mixture whose parameters (i.e shape) is given by empirical estimates of the ground-truth Gaussian mixture: the amount of data on which these parameters are estimated serves as a proxy for the quality of the synthetic data. Optiinally, this synthetic data is mixed with data from the real data distribution.

In the "proportionate" high-dimensional scaling limit, the authors use classical tools from random matrix theory (RMT) to obtain exact analytic expressions for the classifier and its accuracy. In particular, they paper recovers the theoretical results of Feng. et al (2024) as special case (corresponding to training only on synthetic data).
weaknesses: - **The setup.** Fitting a regression model to use its weights as a linear classifier, nobody does this in practice. The authors should justify why such a model is reasonable / relevant, beyond the fact that it leads to a tractable analysis. Note that the underlying estimator is a linear version of the so-called least-squares support vector machine (LS-SVM) which was analyzed in Liao and Couillet (2019) "A Large Dimensional Analysis of Least Squares Support Vector Machines".
- **(Ir)relevance to practice.** In the asymptotic regime considered in this paper, mixing real and synthetic data only helps when the proportion of real data is bounded-away from zero. Also, the practitional needs to know which samples are real and which samples are synthetic. These constraints might not match what happens in practice. This observations have also been made in Dohmatob et al. (2024) "Strong Model Collapse".
- **Non-optimal mixing.** The weights in mixing strategy considered is a bit naive / sub-optimal. The right thing to do would be to consider a general mixing constant $\alpha \in (0,1)$ (the papers uses $\alpha=1-\alpha=1/2$), and derive a theory as a function of $\alpha$, alongside the other  constants in the theory, namely $\pi$, etc.. In general the optimal value of $\alpha$ will depend on the other constants. See Jain et al. (2024) "Scaling laws for learning with real and surrogate data" and Dohmatob et al. (2024) "Strong Model Collapse"

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1.Figure 1: $\hat{\eta}$ = 0.1, $p$ = 500, and $\hat{n}$ = 500, given that $\hat{\eta}$ = $\frac{p}{\hat{n}}$, is this a typo?              
2. Figure 3, in the figure of weak supervision, when the proportion of synthetic data is higher than 0.8, the performance increases, what is the reason for this? This trend is opposite to that when the proportion of synthetic data is lower than 0.8.  Similar observations also happen in Figure 6.      
3. Can authors explain why $\hat{\eta}$ can represent a distribution shift?
soundness: 3
strengths: 1. This work presents analysis and insights with theoretical justification. 
2. Through multiple tasks, the approach is shown its empirical effectiveness.
summary: Synthetic data has been used in training large language models, but concerns about its quality have emerged in practical applications. This paper works on data pruning to select high-quality data and analyzing the performance of trained on a mixture of real and synthetic data in a high-dimensional setting. The authors provide both theoretical and experimental results for the benefits of using and verifying synthetic data.
weaknesses: 1. The paper lacks a clear definition of the quality of synthetic data. 
2. The derived theory in the paper is strongly assumed based on Gaussian distribution, however, non-Gaussian data are common in text or image data used for LLMs.
3. The paper only focuses on label verification for synthetic data. In real-world applications, synthetic data generation often struggles with feature fidelity, addressing feature consistency or the quality of synthetic data in feature space is more beneficial.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: 1) How does your work compare to Jain et al. [1] specifically? (It’s a different data model, of course, but qualitatively, are there similarities/differences and how do the methods differ?)
2) I am confused about the "No supervision" case in Fig. 6 (the Amazon data). While in all other experiments more synthetic data doesn't help, here the U-shape of the curve stands out. Can you discuss a little more why in this case the red curve starts going up?
soundness: 4
strengths: The paper lays out a coherent program and goes ahead to prove it with relatively sophisticated tools. The message is conveyed very clearly, the extensions over prior work, and in particular Feng et al 24 are clearly demarcated. 
The consequences of theorems are explained well, despite the necessarily heavy math. This paper thus provides further evidence to what was laid out in Feng et al 2024: synthetic data can only boost performance when coupled with verification (pruning).
The paper does a masterful job of presenting the results and the experiments are a nice addition. The paper is also well written and is fun to read.
summary: This paper extends important work by Feng et al. 24 and studies the effect of synthetesized data as part of the training data of models. It has two parts, a theoretical part that significantly extends the analysis of a mixture of two isotropic Gaussians from Feng et al. 24 from the case of infinite data at finite dimension to the case where both dimension and data tend to infinity with fixed ratio. Using tools from RMT they thus uncover a richer structure of transitions in a parameter space defined by the quality of synthetic data, the generator, label noise and verifier strength. A key novel element (which facilitates the analysis) is the assumption that the synthetic data comes from a model that estimates the mean and covariance matrix from initial (real) data (as opposed to a more general estimator). This allows to nicely study distribution shift of the features (not just the labels). Another extension over Feng et al.24 is the analysis of mixtures of real and synthetic data (though this is also done in "Strong Model Collapse" by Dohmatob et al., in a different and more general setting). 4 experiments nicely support the theory.
weaknesses: The main weakness is that a few crucial references are missing. In particular there are three papers mixing real and synthetic data that should be cited, [1] and [2], and [3], a more recent one (there is no fault in not citing the latter, as it is recent, 
but the former date back over 8 months). In particular, a careful brief comparison to the [1] setting and results should be added, and [2] should be mentioned. Another paper that should be cited among the theoretical works for model collapse is [4]. On the other hand, 
your reference to [5] should probably be completely removed as it is misleading (the analysis of combination of real and synthetic data doesn't compare to adding the same amount of real data properly - this has been debunked on several occasions and should not be propagated).

Specifics (mostly minor):
1. line 252: typo "usefull"
2. line 402: "emphasize on" - sounds wrong, reformulate
3. line 406 use \citep for Malartic
4. line 408: "from the the Antropic's" --> "from Antropic's"
5. line 483-484: discuss Llama versus Gemma to make clear that you regard one as a weaker supervisor than the other
6. line 518: "Conclusionn" - typo
7. line 527-28: you might want to discuss that label pruning can induce a distribution shift in the features (because the pruning could be biased towards keeping certain features alive) - see (and cite) the discussion in Feng et al. 2024 (https://arxiv.org/pdf/2406.07515 version bottom of p. 10 - top p 11)

I will be happy to reconsider my score once these have been addressed (and the questions).

[1] Ayush Jain, Andrea Montanari, Eren Sasoglu: Scaling laws for learning with real and surrogate data, https://arxiv.org/abs/2402.04376

[2] Bertrand, Q., Bose, A. J., Duplessis, A., Jiralerspong, M., and Gidel, G. On the stability of iterative retraining of generative models on their own data, ICLR 2024 spotlight

[3] Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, Julia Kempe: Strong Model Collapse, https://arxiv.org/abs/2410.04840

[4] Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, Julia Kempe: A Tale of Tails: Model Collapse as a Change of Scaling Laws, ICML 2024

[5] Gerstgrasser et al. 2024

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In Equation (2), the generative model assumes a prior with symmetric data sharing the same $\hat{\mu}$. How would the theory change if one Gaussian per class was fit instead?
2. On line 129, label noise is introduced as purely random noise. What if the label error depended on the input? For instance, in safety applications, a generative model might introduce label errors in specific regions of the distribution due to bias. Could such input-dependent noise be analyzed?
3. It would be helpful to discuss how label noise and feature noise correspond to various types of errors and shifts in real settings.

Missing details from the experiment:

4. What value of $\hat{n}$ is used across all experiments? The authors mention $\hat{n}$ varies in the MNIST experiment (line 400) but don’t provide further details.
5. In the LLM Safety experiment (line 411), it is stated, "we focus only on label noise." However, since data is generated by another language model, feature noise is likely present. 
6. In Figure 7 (left), both feature noise and label noise should be included, as data are generated by a Gaussian fitted to the distribution. Could the authors provide error bars here? Given the simplicity of the MNIST setup with two-layer networks, this should be feasible.
7. In Figure 8, why does using $(\rho, \phi) = (0.5, 0.5)$ as weak supervision? According to the definition in Equation (5), the data selection remains random regardless of label accuracy, which seems equivalent to no supervision.
8. In the LLM Q&A Safety Generation experiment, is accuracy evaluated by the Llama-Guard-3.1 model? What weak supervision method is employed here? Why do the authors use two models to annotate labels? It would be helpful to include the evaluation results comparing the two labeling models with the “ground truth” from Llama-Guard-3.1 to assess the quality of synthetic labels. Additionally, how is supervised fine-tuning conducted with data containing both safe and unsafe answers?
soundness: 3
strengths: 1. The theoretical section is well-written, offering strong motivations and clear explanations.
2. Analyzing distribution shifts in the feature space is a valuable contribution compared with the previous work.
3. The paper provides a precise analysis of performance when training on mixed real and (selected) synthetic data with random matrix theory, expanding beyond previous work.
summary: This paper builds upon Feng et al. that examined how using synthetic data with selection impacts final performance. Here, the authors extend that research to a setting that considers both distribution shifts in the feature space and the combination of real and synthetic data. This approach provides precise results without requiring an infinite number of synthetic samples. The experiments corroborate some theoretical insights.
weaknesses: My primary concern is the applicability of the theoretical insights in practical settings. The theoretical predictions are based on binary Gaussian mixture data, with synthetic data generated by fitting another Gaussian mixture. While the authors provide performance predictions for models trained on mixed data, it is unclear how well these predictions, derived from Gaussian mixture models, will generalize to real-world data. Specifically, how can we estimate some of the key theoretical variables? How can we ensure that the supervision quality is sufficient to mitigate the negative effects of distribution shift?

Currently, the experimental section could be improved by drawing more explicit connections to the theoretical insights. It is generally expected that better supervision and a more accurate generator would lead to improved performance with selected data. However, what practical insights does the theory provide? Can the theoretical findings be leveraged to inform the design of good data selection methods?

The experiment part is not well-written, with lots of details missing. I will elaborate more in the questions.

Meta-review: The submission considers the impact of using synthetic data (e.g., samples from a pre-trained generative model) to train a model, and how one must incorporate a verification/pruning mechanism. This is investigated first from a theoretical point of view using a Gaussian mixture model, and then experimentally with a more realistic data setup. The analysis is more general than previous work, and recovers the result of Feng et al. (2024) as a special case. Moreover, the empirical analysis corroborates and extends the claims made about Gaussian mixture analysis to the more practical real-world setting. Reviewers also note that the presentation is excellent and more approachable than one might expect given the sophistication of the tools employed.

The main concerns outlined by the reviewers were related to the Gaussian mixture model used in the theoretical analysis. However, I agree with the authors and some of the reviewers that this is not a massive limitation, given the experimental validation and the possibility that the results can be generalised to the sub-Gaussian case.

Decision: Accept (Poster)