Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds

## Abstract

Abstract Retrieval-augmented generation (RAG) has seen many empirical successes in recent years by aiding the LLM with external knowledge. However, its theoretical aspect has remained mostly unexplored. In this paper, we propose the first finite-sample generalization bound for RAG in in-context linear regression and derive an exact bias-variance tradeoff. Our framework views the retrieved texts as query-dependent noisy in-context examples and recovers the classical in-context learning (ICL) and standard RAG as the limit cases. Our analysis suggests that an intrinsic ceiling on generalization error exists on RAG as opposed to the ICL. Furthermore, our framework is able to model retrieval both from the training data and from external corpora by introducing uniform and non-uniform RAG noise. In line with our theory, we show the sample efficiency of ICL and RAG empirically with experiments on common QA benchmarks, such as Natural Questions and TriviaQA.

## 1 Introduction

Retrieval-Augmented Generation (RAG) enhances language models by appending retrieved texts to the input, enabling access to information beyond pretraining. It is widely used in open-domain QA, fact-checking, and knowledge-intensive tasks (Huang et al., 2023; Lewis et al., 2020a; Ramos et al., 2022; Sarto et al., 2022; Zhao et al., 2024a). Retrieval sources typically fall into two categories: (1) labeled dataset, such as training dataset itself (Liu et al., 2021; Izacard et al., 2022; Huang et al., 2024), and (2) generic corpora without labels, such as Wikipedia (Chen et al., 2017). Despite its promise, empirical studies show that increasing the number of retrieved passages can degrade performance, especially when irrelevant or redundant texts are included (Levy et al., 2025, 2024). However, the theoretical aspects for understanding of how retrieval affects generalization remain underexplored.

To study its behavior, we frame RAG as noisy in-context learning (ICL). ICL refers to the ability of language models to adapt given the contextual information without updating model weights (Dong et al., 2024). Under this view, retrieved RAG examples can act as noisy context and its quality depends on the retrieval. This view has motivated the development of many work in in-context retrieval (Luo et al., 2024; Shi et al., 2022), where the goal is to retrieve high-quality demonstrate pairs, which reduces the noise of the retrieval.

From a theoretical standpoint, RAG becomes tractable when framed as structured in-context learning, where the context consists of fixed format demonstration pairs. Prior ICL work has analyzed this regime under clean, i.i.d. examples (Ahn et al., 2023; Zhang et al., 2024). These assumptions do not hold in RAG, where retrieved examples are noisy, and their noise level tends to be inversely correlated to their relevance. Currently, no theoretical framework has been developed to study RAG under this structured ICL formulation. Although retrieved examples close to the query should, in principle, improve the predictive accuracy, their quantitative contribution remains unknown because RAG introduces these examples only at the test time (absent during pretraining), thus imposing a distribution shift. In this work, we bridge this gap by modeling RAG as noisy ICL, where retrieved examples follow a structured but perturbed distribution. In particular, we model the retrieval noise both under the uniform (same across examples) and non-uniform (inversely correlated with the retrieval relevance). This view allows us to quantify the impact of retrieval noise and derive generalization bounds that depend on the number of in-context and RAG examples, and the retrieval distance from queries.

Our contributions are summarized as follows:

- •
We propose a theoretical framework for analyzing RAG and provide the first finite sample bounds for in-context linear regression with RAG. Our bounds show that the improvement from RAG shrinks as you add more retrieved examples, and can even flip to hurt performance, giving concrete guidance on when to stop.
- •
Our framework includes ICL and standard RAG as limit cases, and also models retrieved data under different noise regimes, uniform and non-uniform retrieval noise.
- •
We develop new tools for analyzing the query-dependent RAG data, e.g. a derivation of the expectation for 6th-order Gaussian monomial ([Lemma 3](https://arxiv.org/html/2506.03100v3#Thmlemma3)), which can be useful for future research on RAG.
- •
We conduct experiments for representative models on common QA datasets and demonstrate that early RAG retrieves lie in the uniform noise regime, while later ones shift to non-uniform noise regime, aligning with our theory.

## 2 Related Work

##### Retrieval Augmented Generation

Retrieval-augmented generation (RAG) has emerged as a widely adopted paradigm for enriching LLMs with external knowledge by prepending retrieved passages to the input context (Lewis et al., 2020a; Izacard and Grave, 2020; Borgeaud et al., 2021). From a functional perspective, RAG transforms the model’s input distribution by conditioning generation on retrieved textual evidence, often drawn from large-scale corpora via learned or heuristic retrieval mechanisms (Li et al., 2023; Meng et al., 2024; Chen et al., 2024). While much of the literature focuses on improving retrieval quality, system performance (Asai et al., 2023; Li et al., 2024; Xu et al., 2024), and answer reliability (Xiang et al., 2024; Xu et al., 2024), the theoretical foundations of RAG remain underexplored.

##### In-context Learning (ICL)

ICL obtains its popularity from the original GPT-3 paper (Brown et al., 2020), and becomes widely used in LLM applications (Dong et al., 2024; Min et al., 2021). The recent advance in ICL theory (Ahn et al., 2023; Zhang et al., 2024; Xie et al., 2021) provides a rigorous and versatile framework to study transformers and LLMs. People have use this ICL framework to study novel setting, like out-of-distributions tasks (Wang et al., 2024b) and test-time training (Gozeten et al., 2025). People also have also studied the noisy in-context learning from robustness (Cheng et al., 2025) and calibration perspectives (Zhao et al., 2024b), which are different from our setup.

##### In-context Retrieval

In-context retrieval (Luo et al., 2024) refers to retrieving a set of query-dependent demonstrations than using fixed set of demonstrations. The label of the demonstration pairs can come from various sources, such as in-domain training set (Izacard et al., 2022; Huang et al., 2024; Ye et al., 2023), cross-domain data (Cheng et al., 2023; Shi et al., 2022), automatic LLM generation (Zhang et al., 2022; Li and Qiu, 2023), pseudo-labels from unstructured data (Lyu et al., 2022; Li et al., 2022). In our theoretical analysis and experiments, we focus on the simplest in-context retrieval, in-domain retrieval from the training set, as in (Izacard et al., 2022; Huang et al., 2024). Note that in-context retrieval is a term developed later and some earlier papers discuss ICL with retrieval as retrieving relevant documents without labels (Ram et al., 2023).

## 3 Problem Setup

Our problem setup is similar to (Zhang et al., 2024; Garg et al., 2022), with RAG examples to form the additional in-context examples. It is worth noting that many works focus on ICL at test (inference) time, specifically without parameter updates (Dong et al., 2022). Our work adopts the framework of ICL with warmup, also known as, supervised in-context training. Specifically, we assume that the pretraining data is also formed by in-context examples. Then, during the test time, we formed prompts with in-context examples with additional RAG examples.

##### Notations

We denote $[n]=\{1,\dots,n\}$ for an integer $n\geq 1$. We denote the trace product of two matrices $A,B\in{\mathbb{R}}^{m\times n}$ as $\operatorname{tr}(AB^{\top})$.

##### Pretraining Data

We consider learning over linear regression data. The training data is a set of prompts. Each prompt is of size $m$: $({\boldsymbol{x}}_{1},y_{1},\dots,{\boldsymbol{x}}_{m},y_{m},{\boldsymbol{x}}_
{q})\in\mathbb{R}^{d(m+1)+m}$ where $({\boldsymbol{x}}_{1},y_{1}),\dots,({\boldsymbol{x}}_{m},y_{m})$ form the $m$ demonstration pairs. The goal is to predict $\hat{y}_{q}$ for the query example ${\boldsymbol{x}}_{q}$ to match the true label $y_{q}$. The prompt is embedded in the following form:

$$ ${\textbf{P}}^{\mathrm{pt}}_{m}:=\begin{pmatrix}{\boldsymbol{x}}_{1}&{ \boldsymbol{x}}_{2}&\dots&{\boldsymbol{x}}_{m}&{\boldsymbol{x}}_{q}\\ y_{1}&y_{2}&\dots&y_{m}&0\end{pmatrix}\in\mathbb{R}^{(d+1)\times(m+1)}$ (1) $$

where $({\boldsymbol{x}}_{1},y_{1}),\dots,({\boldsymbol{x}}_{m},y_{m}),({\boldsymbol{
x}}_{q},y_{q})\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{{\sim}}{\cal D}_{\text{
pt}}$ (pt denoting Pretraining). The output follows the linear model:

$$ $\displaystyle y_{i}$ $\displaystyle={\boldsymbol{x}}_{i}^{\top}\beta_{\text{pt}}+\epsilon_{i},\quad \epsilon_{i}\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{{\sim}}{\cal N}(0,\sigma^ {2})\quad\text{under}\quad{\cal D}_{\mathrm{pt}}$ (2) $$

where $i\in[m]\cup\{q\}$, $\beta_{\mathrm{tt}}$ is the weight vector in pretraining, and $\epsilon_{i}$ is the noise for example $i$.

##### Inference Data (with RAG)

During inference/test time, the test prompt ${\textbf{P}}^{\text{tt+rag}}_{m,n}$ (tt denoting test-time) is formed by $m$ in-context pairs $({\boldsymbol{x}}_{1},y_{1}),\dots,({\boldsymbol{x}}_{m},y_{m})$, $n$ retrieval-augmented pairs $({\boldsymbol{x}}_{1}^{\text{rag}},y_{1}^{\text{rag}}),\dots,({\boldsymbol{x}}
_{n}^{\text{rag}},y^{\text{rag}})$, and the query pair ${\boldsymbol{x}}_{q},y_{q}$. The test prompt is embedded in the following form:

$$ ${\textbf{P}}^{\mathrm{tt+rag}}_{m,n}:=\begin{pmatrix}{\boldsymbol{x}}_{1}& \dots&{\boldsymbol{x}}_{m}&{\boldsymbol{x}}_{1}^{\text{rag}}&\dots&{ \boldsymbol{x}}_{n}^{\text{rag}}&{\boldsymbol{x}}_{q}\\ y_{1}&\dots&y_{m}&y_{1}^{\text{rag}}&\dots&y_{n}^{\text{rag}}&0\end{pmatrix} \in\mathbb{R}^{(d+1)\times(m+n+1)}$ (3) $$

The input ${\boldsymbol{x}}$ in each in-context or query pair follows the test-time distribution ${\cal D}_{\text{tt}}$, and the label is:

$$ $\displaystyle y_{i}={\boldsymbol{x}}_{i}^{\top}\beta_{\text{tt}}+\epsilon_{i}, \quad\epsilon_{i}\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{{\sim}}{\cal N}(0, \sigma^{2})\quad\text{under}\quad{\cal D}_{\text{tt}}$ (4) $$

where $i\in[m]\cup\{q\}$, $\epsilon_{i}$ is the noise of example $i$, and $\beta_{\mathrm{tt}}$ is the weight vector during test time.
The input ${\boldsymbol{x}}$ in each RAG pair follows the corresponding RAG distribution ${\cal D}_{\text{rag}}({\boldsymbol{x}}_{q})$:
assume the RAG query ${\boldsymbol{x}}_{i}^{\text{rag}}={\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i}$ is generated around the query example ${\boldsymbol{x}}_{q}$, where ${\boldsymbol{r}}_{i}$ is the offset. The label in the RAG example is given by:

$$ $\displaystyle\quad y_{i}^{\text{rag}}=({\boldsymbol{x}}_{i}^{\text{rag}})^{\top}$ $\displaystyle\beta_{\text{{tt}}}+\epsilon_{i}^{\text{rag}},\quad\epsilon^{ \text{rag}}_{i}\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{{\sim}}{\cal N}(0, \sigma^{2}_{\text{rag},i})\quad\text{under}\quad{\cal D}_{\text{rag}}({ \boldsymbol{x}}_{q})$ (5) $$

where $i\in[n]$, $\epsilon^{\text{rag}}_{i}$ is the noise of the $i$-th RAG example.

For the compactness of writing, we define the following matrices and vectors:

$$ ${\boldsymbol{X}}_{\mathrm{icl}}:=[{\boldsymbol{x}}_{1}^{\top};\dots;{ \boldsymbol{x}}_{m}^{\top}],\;{\boldsymbol{X}}_{\mathrm{rag}}:=[({\boldsymbol{ x}}_{1}^{\text{rag}})^{\top};\dots;({\boldsymbol{x}}_{n}^{\text{rag}})^{\top}] ,\;{\boldsymbol{y}}_{\mathrm{icl}}:=[y_{1};\dots;y_{m}],\;{\boldsymbol{y}}_{ \mathrm{rag}}:=[y_{1}^{\text{rag}};\dots;y_{n}^{\text{rag}}],\;$ $$

$$ $\boldsymbol{\epsilon}_{\mathrm{icl}}:=[\epsilon_{1};\dots;\epsilon_{m}],\; \boldsymbol{\epsilon}_{\mathrm{rag}}:=[\epsilon_{1}^{\text{rag}};\dots; \epsilon_{n}^{\text{rag}}],\;{\boldsymbol{r}}=[{\boldsymbol{r}}_{i}^{\top}; \dots;{\boldsymbol{r}}_{n}^{\top}]$ $$

$$ ${\boldsymbol{X}}=\begin{bmatrix}{\boldsymbol{X}}_{\mathrm{icl}}\\ {\boldsymbol{X}}_{\mathrm{rag}}\end{bmatrix}\in\mathbb{R}^{(m+n)\times d},\;{ \boldsymbol{X}}_{\mathrm{rag}}=\begin{bmatrix}{\boldsymbol{x}}_{q}+{ \boldsymbol{r}}_{1}\\ \vdots\\ {\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{n}\end{bmatrix}\in\mathbb{R}^{n\times d },\;{\boldsymbol{y}}=\begin{bmatrix}{\boldsymbol{y}}_{\mathrm{icl}}\\ {\boldsymbol{y}}_{\mathrm{rag}}\end{bmatrix}\in\mathbbm{R}^{m+n},\;\boldsymbol {\epsilon}=\begin{bmatrix}\boldsymbol{\epsilon}_{\mathrm{icl}}\\ \boldsymbol{\epsilon}_{\mathrm{rag}}\end{bmatrix}\in\mathbbm{R}^{m+n}$ $$

##### Training and Testing

We let ${\boldsymbol{W}}$ be the model parameters, and $F$ be the model.
Given an input prompt ${\textbf{P}}_{m}^{\text{pt}}$ with demonstration pairs, the model predicts $\hat{y}_{q}:=F({\textbf{P}}_{m}^{\text{pt}};{\boldsymbol{W}})$. As a common practice in theoretical studies of LLM for feasible analysis, we use the MSE loss as the evaluation metrics (Zhang et al., 2024; Ahn et al., 2023; Xie et al., 2021). Then, the population loss on the pretraining data is:

$$ ${\cal L}_{\text{pt}}({\boldsymbol{W}}):=\operatorname*{\mathbb{E}}_{({ \boldsymbol{x}}_{1},y_{1}),\dots,({\boldsymbol{x}}_{m},y_{m})({\boldsymbol{x}} _{q},y_{q})\sim{\cal D}_{\mathrm{pt}}}\left[\left(y_{q}-F\left({\textbf{P}}_{m }^{\text{pt}};{\boldsymbol{W}}\right)\right)^{2}\right]$ (6) $$

Its minimizer is denoted as:

$$ $\bar{{\boldsymbol{W}}}^{*}:=\min_{{\boldsymbol{W}}}{\cal L}_{\text{pt}}({ \boldsymbol{W}}).$ (7) $$

To apply the pretrained ${\boldsymbol{W}}^{*}$ from the pretraining context size of $m$ to the test-time context size of $m+n$, we will need to scale it properly (see [Lemma 1](https://arxiv.org/html/2506.03100v3#Thmlemma1)) and use

$$ ${\boldsymbol{W}}^{*}=\frac{m}{m+n}\bar{{\boldsymbol{W}}}^{*}.$ (8) $$

During the test time we evaluate the population loss over the test prompt with RAG examples ${\textbf{P}}_{m,n}^{\text{tt+rag}}$:

$$ ${\cal L}_{\text{tt+rag}}({\boldsymbol{W}}):=\operatorname*{\mathbb{E}}_{\begin {subarray}{c}({\boldsymbol{x}}_{1},y_{1}),\dots,({\boldsymbol{x}}_{m},y_{m})({ \boldsymbol{x}}_{q},y_{q})\sim{\cal D}_{\mathrm{tt}}\\ ({\boldsymbol{x}}^{\text{rag}}_{1},y^{\text{rag}}_{1}),\dots,({\boldsymbol{x}} ^{\text{rag}}_{n},y^{\text{rag}}_{n})\sim{\cal D}_{\mathrm{rag}}({\boldsymbol{ x}}_{q})\end{subarray}}\left[\left(y_{q}-F\left({\textbf{P}}_{m,n}^{\text{tt+ rag}};{\boldsymbol{W}}\right)\right)^{2}\right]$ (9) $$

##### Model Architecture

We study the single-layer linear self-attention model (LSA) as the framework for theoretical analysis, similar to many existing studies (e.g., (Ahn et al., 2023; Zhang et al., 2024)). The prediction of the model $F$ on a prompt P with query ${\boldsymbol{x}}_{q}$ is:

$$ $\hat{y}_{q}:=F({\textbf{P}})=[{\textbf{P}}{\boldsymbol{W}}_{Q}{\boldsymbol{W}} _{K}^{\top}{\textbf{P}}^{\top}{\textbf{P}}{\boldsymbol{W}}_{V}]_{m+n+1,d+1}={ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{X}}^{\top}{\boldsymbol{ y}}$ (10) $$

where the query, key, and value matrices ${\boldsymbol{W}}_{Q},{\boldsymbol{W}},{\boldsymbol{W}}_{V}\in\mathbb{R}^{(d+1)
\times(d+1)}$ are parameterized by ${\boldsymbol{W}}$ in the follow way:

$$ ${\boldsymbol{W}}_{Q}{\boldsymbol{W}}_{K}^{\top}=\begin{bmatrix}{\boldsymbol{W} }&\mathbf{0}_{d\times 1}\\ \mathbf{0}_{1\times d}&0\end{bmatrix},\quad{\boldsymbol{W}}_{V}=\begin{bmatrix }\mathbf{0}_{d\times d}&\mathbf{0}_{d\times 1}\\ \mathbf{0}_{1\times d}&1\end{bmatrix}$ $$

We note that this parameterization is commonly used in the previous works (Ahn et al., 2023; Zhang et al., 2024), and is shown to capture the key properties of in-context learning. Furthermore, (Ahn et al., 2023) shows that the formulation is the optimum converged from pretraining on Gaussian data.

## 4 Theoretical Analysis: Generalization Bound for RAG

To study test-time error and sample complexity in in-context linear regression with RAG examples, we consider two noise regimes: uniform retrieval noise and non-uniform retrieval noise. Uniform retrieval noise assumes the RAG noise $\epsilon_{i}^{\mathrm{rag}}$ for each example $i$ is i.i.d. Since its variance is distance-agnostic, it can model a scenario of retrieval where the noise is prevailing across data points. Non-uniform retrieval noise assumes either the variance or the label-corruption probability grows with the variance of retrieval vector — e.g. $\sigma_{\mathrm{rag},i}^{2}$ increases with $\delta_{i}^{2}$ or probability of making mistakes increases with $\delta_{i}^{2}$. This captures retrieval from datasets where near neighbors often supply the right signal while far ones are potentially noisy or misleading. Because the noise spectrum is now heavy-tailed, adding more RAG examples past a threshold could yield diminishing benefits for RAG examples and even become counter-productive. Framing RAG through these two lenses allows precise clarification about when extra retrieved examples will pay off, and when they will hit the intrinsic ceiling and more retrieved examples don’t help anymore. These are well corroborated by our experimental results on real data (see Section [5](https://arxiv.org/html/2506.03100v3#S5)).

First, we introduce the key data assumptions.

###### Assumption 1 (Gaussian Retrieval Offset) .

We assume the retrieval offset ${\boldsymbol{r}}_{i},\;\forall i\in[n]$ to follow a Gaussian distribution:
${\boldsymbol{r}}_{i}\stackrel{{\scriptstyle\text{ i.i.d. }}}{{\sim}}\mathcal{N
}\left(0,\delta_{i}^{2}I_{d}\right).$

The key property that we want to control for RAG examples is its distance from the query points ${\boldsymbol{x}}_{q}$. However, modeling the queried example directly through the retrieval distance leads to complicated theoretical analysis. Here, we note that the retrieval distance $\|{\boldsymbol{r}}_{i}\|_{2}$ converges to a distribution concentrated in an ${\cal O}(\delta_{i}\sqrt{d})$ ball around the query with respect to $d$ (Cover and Hart, 1967). Thus, controlling the variance of the retrieval offset can alternatively control the retrieval distance. And we make the following additional data assumptions.

###### Assumption 2 (Data Assumption) .

We assume the data follows the following:

- 1.
Pretraining Examples (${\cal D}_{\mathrm{pt}}$). For a pretraining prompt of length $m+1$ and for all $i\in[m]\cup\{q\}$, we assume
${\boldsymbol{x}}_{i}\!\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{{\sim}}\!
\mathcal{N}(0,I)$,
$\epsilon_{i}\!\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{{\sim}}\!\mathcal{N}(0,
\sigma^{2})$,
$\beta_{\mathrm{pt}}\!\sim\!\mathcal{N}(0,I)$.
- 2.
Test Time Examples (${\cal D}_{\mathrm{tt}}$).
For a test-time prompt of length $m+n+1$ and for all $i\in[m]\cup\{q\}$, we assume ${\boldsymbol{x}}_{i}\!\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{{\sim}}\!
\mathcal{N}(0,I)$,
$\epsilon_{i}\!\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{{\sim}}\!\mathcal{N}(0,
\sigma^{2})$,
$\beta_{\mathrm{tt}}\!\sim\!\mathcal{N}(0,I)$.
- 3.
Test-Time RAG Examples (${\cal D}_{\mathrm{rag}}({\boldsymbol{x}}_{q})$).
For a test-time prompt of length $m+n+1$ and for all $i\in[m+1,\dots,m+n]$, we assume ${\boldsymbol{x}}_{i}^{\mathrm{rag}}\!\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{
{\sim}}\!\mathcal{N}(0,I)$,
$\epsilon_{i}^{\mathrm{rag}}\!\sim\!\mathcal{N}(0,\sigma_{\mathrm{rag},i}^{2})$,
and the same $\beta_{\mathrm{tt}}$ as (2).

Here, we assume the isotropic Gaussian property for the input, noise and the weight vector, a common assumption made in ICL theory (Ahn et al., 2023; Gozeten et al., 2025) for simple yet meaningful analysis.

###### Assumption 1 (Gaussian Retrieval Offset) .

###### Assumption 2 (Data Assumption) .

##### Roadmap

Under these assumptions and uniform retrieval noise, we will first derive the population loss of RAG, ${\cal L}_{\mathrm{tt+rag}}({\boldsymbol{W}})$, for general ${\boldsymbol{W}}$ as in [Theorem 1](https://arxiv.org/html/2506.03100v3#Thmtheorem1), analyze its finite sample complexity under the optimal pretrained weight ${\boldsymbol{W}}^{*}$ as in [Proposition 1](https://arxiv.org/html/2506.03100v3#Thmproposition1) and derive an optimal number of RAG examples of $n^{*}$ for a given number of ICL examples $m$ as in [Proposition 2](https://arxiv.org/html/2506.03100v3#Thmproposition2). These discussions leads to our first key result. Then, under the non-uniform retrieval noise, we will prove the sample complexity under the distance-proportional noise ([Theorem 2](https://arxiv.org/html/2506.03100v3#Thmtheorem2)) and distance-weighted mixture noise ([Theorem 3](https://arxiv.org/html/2506.03100v3#Thmtheorem3)), and obtain our second key results above.

### 4.1 Uniform Retrieval Noise

###### Assumption 3 (Uniform Retrieval Noise) .

The RAG noise $\boldsymbol{\epsilon}_{\mathrm{rag}}$ shares the same Gaussian distribution with variance $\sigma^{2}_{\mathrm{rag}}$, i.e. $\forall i\in[m+1,\dots,m+n]$,
$\sigma_{\mathrm{rag},i}^{2}=\sigma_{\mathrm{rag}}^{2}$.

First, we present the assumption for uniform retrieval noise. In other words, all RAG examples are as helpful, and its improvement on the actual prediction is determined by the retrieval distance.

###### Theorem 1 (Population Loss for ICL with RAG Examples) .

Under Assumption [1](https://arxiv.org/html/2506.03100v3#Thmassumption1), [2](https://arxiv.org/html/2506.03100v3#Thmassumption2), [3](https://arxiv.org/html/2506.03100v3#Thmassumption3), the population loss of the linear self-attention predictor
$\hat{y}_{q}={\boldsymbol{x}}_{q}^{\top}\boldsymbol{W}{\boldsymbol{X}}^{\top}{
\boldsymbol{y}}$
satisfies

$$ $\mathcal{L}_{\mathrm{tt+rag}}({\boldsymbol{W}})=\underbrace{\operatorname*{ \mathbb{E}}\left(\operatorname*{\mathbb{E}}\left(\hat{y}_{q}\right)-\hat{y}_{q }\right)^{2}}_{:=\operatorname{err}_{\mathrm{variance}}({\boldsymbol{W}})}+ \underbrace{\operatorname*{\mathbb{E}}\left(\operatorname*{\mathbb{E}}\left( \hat{y}_{q}\right)-\operatorname*{\mathbb{E}}\left(y_{q}\right)\right)^{2}}_{: =\operatorname{err}_{\mathrm{bias}}({\boldsymbol{W}})}+\underbrace{\sigma^{2}} _{\text{irreducible noise}}\quad\text{, and specifically, }$ (11) $$

$$ $\displaystyle{\operatorname{err}}_{\mathrm{variance}}({\boldsymbol{W}})$ $\displaystyle=\left[m\sigma^{2}+\left(1+\delta^{2}\right)n\sigma_{\mathrm{rag} }^{2}\right]\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+n\sigma _{\mathrm{rag}}^{2}\operatorname{tr}({\boldsymbol{W}}^{2})+n\sigma_{\mathrm{ rag}}^{2}\operatorname{tr}({\boldsymbol{W}})^{2}$ $\displaystyle{\operatorname{err}}_{\mathrm{bias}}({\boldsymbol{W}})$ $\displaystyle=\beta_{\mathrm{tt}}^{\top}\left[I-(n\delta^{2}+2n+m)({ \boldsymbol{W}}+{\boldsymbol{W}}^{\top})-2n\operatorname{tr}({\boldsymbol{W}}) I+M_{4}\right]\beta_{\mathrm{tt}}$ $\displaystyle=\beta_{\mathrm{tt}}^{\top}\left[I-(n\delta^{2}+2n+m)({ \boldsymbol{W}}+{\boldsymbol{W}}^{\top})-2n\operatorname{tr}({\boldsymbol{W}}) I\right.$ $\displaystyle\quad+\left[n^{2}\left(2+\delta^{2}\right)+n\left(m+\delta^{2} \right)\right]\left({\boldsymbol{W}}^{2}+\left({\boldsymbol{W}}^{2}\right)^{ \top}\right)+2n(n+\delta^{2}){\boldsymbol{W}}{\boldsymbol{W}}^{\top}$ $\displaystyle\quad+\left[m^{2}+m+mn\left(2+2\delta^{2}\right)+n^{2}\left(2+2 \delta^{2}+\delta^{4}\right)+n\left(2\delta^{2}+\delta^{4}\right)\right]{ \boldsymbol{W}}^{\top}{\boldsymbol{W}}$ $\displaystyle\quad+\left[n^{2}\left(2+\delta^{2}\right)+n\left(m+\delta^{2} \right)\right]\left(\operatorname{tr}({\boldsymbol{W}})\left({\boldsymbol{W}}+ {\boldsymbol{W}}^{\top}\right)\right)$ $\displaystyle\left.\quad+\left[n^{2}+n\delta^{2}\right]\left(\operatorname{tr} ({\boldsymbol{W}})^{2}+\operatorname{tr}\left({\boldsymbol{W}}^{2}\right) \right)I+\left[m+n^{2}+n\left(2\delta^{2}+\delta^{4}\right)\right] \operatorname{tr}\left({\boldsymbol{W}}^{\top}{\boldsymbol{W}}\right)I\right] \beta_{\mathrm{tt}}$ $$

Here, we derive the exact bias-variance decomposition for ICL with RAG. The first line is the variance-induced error formed by a weighted sum of noise from ICL examples and RAG examples. Because of the implicit scaling of ${\boldsymbol{W}}$ as discussed in [Lemma 1](https://arxiv.org/html/2506.03100v3#Thmlemma1), the second order term in ${\boldsymbol{W}}$ will introduce an additional weight scaling of $\frac{m^{2}}{(m+n)^{2}}$ when adapting from the weight learned on $m$ size context to $m+n$ size context. Thus, larger $n$ will let ${\operatorname{err}}_{\text{variance}}({\boldsymbol{W}})\rightarrow 0$, and the convergence rate is affected by $\delta^{2}$. Larger retrieval distance leads to a slower convergence. The bias-induced error is composed of all possible monomials of ${\boldsymbol{W}}$ up to the $2$nd-order with $\operatorname{tr}$ operation. The complex dependency on $m,n,\delta^{2},d$ requires additional assumptions on ${\boldsymbol{W}}$ to further interpret. As a sanity check, when $n=0$ (ICL-only), this decomposition can exactly recover loss as in Lemma B.2 in (Gozeten et al., 2025).

As a proof sketch, we first compute ${\operatorname{err}}_{\text{variance}}({\boldsymbol{W}})=\operatorname*{
\mathbb{E}}({\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{X}}^{\top}
\boldsymbol{\epsilon})^{2}$ by splitting the calculation for ICL and RAG examples based on ${\boldsymbol{X}}$. Then, we compute ${\operatorname{err}}_{\text{bias}}({\boldsymbol{W}})=\operatorname*{\mathbb{E}
}[({\boldsymbol{x}}^{\top}_{q}(I-{\boldsymbol{W}}{\boldsymbol{X}}^{\top}{
\boldsymbol{X}})\beta_{\mathrm{tt}})^{2}]$. The main technical challenge lies in the dependency of ${\boldsymbol{X}}_{\mathrm{rag}}$ on ${\boldsymbol{x}}_{q}$, and ${\operatorname{err}}_{\text{bias}}$ has a 6th-order dependency on ${\boldsymbol{x}}_{q}$ (2 from ${\boldsymbol{x}}_{q}$ and 4 from ${\boldsymbol{X}}$). As shown in [Lemma 3](https://arxiv.org/html/2506.03100v3#Thmlemma3), $\operatorname*{\mathbb{E}}\left[{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top
}A{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}B{\boldsymbol{x}}_{q}{
\boldsymbol{x}}_{q}^{\top}\right]$ gives 15 new terms that include all the second order monomials of ${\boldsymbol{W}}$ with $\operatorname{tr}$. The calculation requires multiple careful applications of Isserlis’ theorem (Isserlis, 1918), and the full proof can be seen in [Appendix B](https://arxiv.org/html/2506.03100v3#A2). It is possible to prove this theorem for a design matrix with non-isotropic covariance, but computing the expectation of the 6th-order Gaussian monomial is more complicated.

Here, we present the finite sample bound for pretrained ${\boldsymbol{W}}^{*}$ for better interpretation.

###### Proposition 1 (Finite Sample Generalization Bound) .

Under Assumption [1](https://arxiv.org/html/2506.03100v3#Thmassumption1), [2](https://arxiv.org/html/2506.03100v3#Thmassumption2), [3](https://arxiv.org/html/2506.03100v3#Thmassumption3), if $\delta^{2}\ll 1$,

$$ ${\cal L}_{\mathrm{tt+rag}}({\boldsymbol{W}}^{*})={\cal O}\left(\sigma^{2}+ \underbrace{\frac{dm}{(m+n)^{2}}\sigma^{2}+\frac{d^{2}n}{(m+n)^{2}}\sigma_{ \mathrm{rag}}^{2}}_{{\operatorname{err}}_{\text{variance}}({\boldsymbol{W}}^{* })}+\underbrace{\|\beta_{\mathrm{tt}}\|_{2}^{2}\left[\frac{d}{m}+d^{2}\left( \frac{n}{m+n}\right)^{2}\right]}_{{\operatorname{err}}_{\text{bias}}({ \boldsymbol{W}}^{*})}\right)$ $$

$$ $\displaystyle{\operatorname{err}}_{\mathrm{variance}}({\boldsymbol{W}}^{*})$ $\displaystyle=\begin{cases}{\cal O}(\frac{d}{m}\sigma^{2}+\frac{d^{2}}{m^{2}} \sigma_{\mathrm{rag}}^{2})={\cal O}\left(\frac{1}{m}\right)&\text{ $m \rightarrow\infty$, $n$ fixed.}\\ {\cal O}(\frac{d}{n^{2}}\sigma^{2}+\frac{d^{2}}{n}\sigma_{\mathrm{rag}}^{2})={ \cal O}\left(\frac{1}{n}\right)&\text{ $n\rightarrow\infty$, $m$ fixed}\\ {\cal O}(\frac{d}{m}\sigma^{2}+\frac{d^{2}}{m}\sigma_{\mathrm{rag}}^{2})={\cal O }\left(\frac{1}{m}\right)&\text{ $m,n\rightarrow\infty$, $n=\Theta(m)$}\end{cases}$ (12) $$

$$ $\displaystyle{\operatorname{err}}_{\mathrm{bias}}({\boldsymbol{W}}^{*})$ $\displaystyle=\begin{cases}{\cal O}\left(\|\beta_{\mathrm{tt}}\|_{2}^{2}\frac{ d}{m}\right)&\text{if $m\rightarrow\infty$, $n$ is fixed}\\ {\cal O}\left(\|\beta_{\mathrm{tt}}\|_{2}^{2}d^{2}\right)=C_{1}&\text{if $n \rightarrow\infty$, $m$ is fixed}\\ {\cal O}\left(\|\beta_{\mathrm{tt}}\|_{2}^{2}\left(\frac{d}{m}+d^{2}\right) \right)=C_{2}+{\cal O}(\|\beta_{\mathrm{tt}}\|_{2}^{2}\frac{d}{m})&\text{if $m \rightarrow\infty$, $n=\Theta(m)$}\\ \end{cases}$ (13) $$

Here, we assume $\delta^{2}\ll 1$ as the test time example ${\boldsymbol{x}}_{i}$ has only a variance of $I$, and it is unrealistic to assume a higher retrieval variance than the input variance. On the limit case where $m\rightarrow\infty$ and $n$ are fixed, we observe that both variance-induced and bias-induced error decay at a rate of ${\cal O}\left(1/m\right)$, matching the results from the existing paper (Ahn et al., 2023; Zhang et al., 2024). When $n\rightarrow\infty$, the variance-induced error decays as ${\cal O}\left(1/n\right)$ matching the ${\cal O}(1/m)$ rate. However, introducing the RAG is ineffective at reducing the bias-induced error. Even when $m\rightarrow\infty$, increasing $n$ will cause a loss plateau.

This effect can be explained by the underlying adaptive ability of transformers. In an online learning setup, we could always use the mean of the queried data as the prediction. However, in the LSA setup, the pretrained ${\boldsymbol{W}}^{*}$ serves as a proxy for $\operatorname*{\mathbb{E}}^{-1}({\boldsymbol{X}}^{\top}{\boldsymbol{X}})$. In order to retain the adaptivity to the entire distribution of $\beta_{\mathrm{tt}}$, we cannot use the optimal linear classifier $({\boldsymbol{X}}^{\top}{\boldsymbol{X}})^{-1}{\boldsymbol{X}}^{\top}{
\boldsymbol{y}}$ or use the mean of the retrieved examples ad hoc. At the test stage, ${\boldsymbol{X}}_{\mathrm{rag}}$ only appears in ${\boldsymbol{X}}^{\top}{\boldsymbol{y}}$ and not in ${\boldsymbol{W}}^{*}$. The difference between ${\cal D}_{\mathrm{rag}}({\boldsymbol{x}}_{q})$ and ${\cal D}_{\text{tt}}$ directly leads to the increase of variance worsened by the increase of $n$. See full proof in [Appendix B](https://arxiv.org/html/2506.03100v3#A2). Now, a natural question is whether we can find a balance of variance and bias and obtain an optimal RAG example size $n^{*}$.

###### Proposition 2 .

Under Assumption [1](https://arxiv.org/html/2506.03100v3#Thmassumption1),[2](https://arxiv.org/html/2506.03100v3#Thmassumption2),[3](https://arxiv.org/html/2506.03100v3#Thmassumption3), $\delta^{2}\ll 1$, and reasonable choice of $\sigma^{2},\sigma_{\mathrm{rag}}^{2}$ ($\sigma^{2},\sigma_{\mathrm{rag}}^{2}\ll\|\beta_{\mathrm{tt}}\|_{2}^{2}$), the optimal $n^{*}$ that minimizes the RAG loss follows:

$$ $n^{*}={\cal O}\left(\frac{m\left(d^{2}\|\beta_{\mathrm{tt}}\|_{2}^{2}+d\sigma^ {2}-d^{2}\sigma_{\mathrm{rag}}^{2}\right)}{md^{2}\|\beta_{\mathrm{tt}}\|_{2}^{ 2}-d^{2}\sigma_{\mathrm{rag}}^{2}}\right)={\cal O}\left(\frac{d\|\beta_{ \mathrm{tt}}\|_{2}^{2}+\sigma^{2}-d\sigma_{\mathrm{rag}}^{2}}{d\|\beta_{ \mathrm{tt}}\|_{2}^{2}}\right)$ (14) $$

and the improvement on loss from picking the optimal $n^{*}$ over $n=0$ is given as:

$$ ${\cal L}_{\mathrm{tt+rag}}({\boldsymbol{W}}^{*})|_{n=0}-{\cal L}_{\mathrm{tt+ rag}}({\boldsymbol{W}}^{*})|_{n=n^{*}}={\cal O}\left(\frac{1}{m^{2}}\right)$ (15) $$

In fact, the optimal $n^{*}$ does not scale with $m$ omitting the lower-order terms. Note that for $\|\beta_{\mathrm{tt}}\|_{2}^{2}={\cal O}(1)$, $\|\beta_{\mathrm{tt}}\|_{2}^{2}$ will dominate the numerator for reasonable choices of $\sigma^{2}$ and $\sigma^{2}_{\mathrm{rag}}$. A larger ICL noise $\sigma^{2}$ leads to a larger $n^{*}$, i.e. requiring more RAG examples to compensate for the loss. A larger RAG noise $\sigma^{2}_{\mathrm{rag}}$ leads to a smaller $n^{*}$, i.e. less efficiency on RAG examples. And the improvement converges at ${\cal O}(\frac{1}{m^{2}})$, diminishing for large $m$. See the full proof in [Appendix B](https://arxiv.org/html/2506.03100v3#A2). Several empirical works also observe a performance drop when increasing the number of retrieved examples (Wang et al., 2024a; Levy et al., 2025).

###### Assumption 3 (Uniform Retrieval Noise) .

###### Theorem 1 (Population Loss for ICL with RAG Examples) .

###### Proposition 1 (Finite Sample Generalization Bound) .

###### Proposition 2 .

### 4.2 Non-Uniform Retrieval Noise

The uniform-noise setup in [Section 4.1](https://arxiv.org/html/2506.03100v3#S4.SS1) relies on a clean retrieval pool, so we could keep the variance $\sigma_{\mathrm{rag}}^{2}$ fixed. In open-domain retrieval, this assumption could collapse: many retrieved examples could contain no answer or even a wrong answer. Empirically, people have observed that passages that are closer to the query vector $\boldsymbol{x}_{q}$ are more likely (Yang and Seo, 2020; Yoran et al., 2023; Lewis et al., 2020b) to contain the correct label. We want to theoretically investigate if the following hypothesis still holds:

Closer to query ${\boldsymbol{x}}_{q}$ $\implies$ more likely to contain correct answer.

#### 4.2.1 Distance-Proportional Noise (DPN)

We first investigate the scenario where the retrieval noise is proportional to the retrieval distance. Since the ICL analysis only applies to the mean-squared error loss, we study the effect of RAG under DPN on the correctness of the predictions.

###### Assumption 4 (Distance-Proportional Noise) .

There exists a constant $\gamma_{1}>0$ such that, for every retrieved sample $i$,
$\sigma_{\mathrm{rag},i}^{2}=\gamma_{1}\sigma^{2}\delta_{i}^{2},$
i.e. the RAG noise variance grows linearly with the variance $\delta_{i}^{2}$ that governs the retrieval distance.

Under the new data assumption, we denote the corresponding RAG loss, bias-induced error, and variance-induced error for ${\boldsymbol{W}}$ to be $\hat{{\cal L}}_{\mathrm{tt+rag}}({\boldsymbol{W}})$, $\hat{{\operatorname{err}}}_{\mathrm{bias}}({\boldsymbol{W}})$, and $\hat{{\operatorname{err}}}_{\mathrm{variance}}({\boldsymbol{W}})$.

###### Theorem 2 (Finite Sample RAG Generalization Bound under DPN) .

Under Assumption [1](https://arxiv.org/html/2506.03100v3#Thmassumption1), [2](https://arxiv.org/html/2506.03100v3#Thmassumption2), [4](https://arxiv.org/html/2506.03100v3#Thmassumption4), the population loss is given as:

$$ $\hat{{\operatorname{err}}}_{\mathrm{variance}}({\boldsymbol{W}})=m\sigma^{2} \operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+\sum_{i=1}^{n} \gamma_{1}\delta_{i}^{2}[(1+\delta_{i}^{2})\operatorname{tr}({\boldsymbol{W}}^ {\top}{\boldsymbol{W}})+\operatorname{tr}({\boldsymbol{W}}^{2})+\operatorname{ tr}({\boldsymbol{W}})^{2}]$ $$

If the variance of the retrieval distance follows power law, i.e. $\exists\gamma_{2}>0,q\geq 0$ s.t. $\delta_{i}^{2}=\gamma_{2}i^{q}$, then

$$ $\displaystyle\hat{{\operatorname{err}}}_{\mathrm{bias}}({\boldsymbol{W}}^{*})$ $\displaystyle={\cal O}\left({\operatorname{err}}_{\text{bias}}({\boldsymbol{W} }^{*})+\|\beta_{\mathrm{tt}}\|_{2}^{2}\left[\frac{dn^{2q+1}+n^{2q+2}}{(m+n)^{2 }}\right]\right)$ (16) $$

and

$$ $\displaystyle\hat{{\operatorname{err}}}_{\mathrm{variance}}({\boldsymbol{W}}^{ *})$ $\displaystyle={\cal O}\left(\frac{dm\sigma^{2}+d(n^{2q+1})\sigma^{2}}{(m+n)^{2 }}\right)=\begin{cases}{\cal O}\left(dn^{2q-1}\sigma^{2}\right)&\text{ if $n \rightarrow\infty$, $q\leq 1/2$ }\\ \text{diverges}&\text{ if $n\rightarrow\infty$, $q>1/2$}\\ \end{cases}$ (17) $$

Here, we derive the sample complexity under DPN. A second order dependency on $\delta_{i}^{2}$ shows up in both the variance-induced and bias-induced error (exact form seen in [Appendix B](https://arxiv.org/html/2506.03100v3#A2)). Thus, the $\delta_{i}^{2}$-involved constant will dominate the other constants. Specifically, it even leads to divergence for $q>1/2$ for the variance-induced error and $q>0$ for the bias-induced error.

###### Assumption 4 (Distance-Proportional Noise) .

###### Theorem 2 (Finite Sample RAG Generalization Bound under DPN) .

#### 4.2.2 Distance-Weighted Mixture Noise

In this section, we discuss the scenario where further RAG examples are less likely to contain the correct answers. We use a pair of large and small noises to model the correct/incorrect examples.

###### Assumption 5 (Distance-Weighted Mixture Noise) .

We assume that the RAG noise is formed by a mixture of small and large noise:

$$ $y({\boldsymbol{x}}_{\mathrm{rag}})=\begin{cases}f({\boldsymbol{x}}_{\mathrm{ rag},i})+\epsilon_{s}&\text{w.p. $p_{i}$}\\ f({\boldsymbol{x}}_{\mathrm{rag},i})+\epsilon_{l}&\text{w.p. $1-p_{i}$}\end{cases}$ $$

where $\epsilon_{s}\sim{\cal N}(0,c_{s}\sigma^{2})$ corresponds to the small noise and $\epsilon_{l}\sim{\cal N}(0,c_{l}\sigma^{2})$ corresponds to the large noise, with $c_{l}\geq c_{s}\geq 0$. The probability of sampling small noise $p_{i}$ follows an inverse power law of the variance of the retrieval distance, i.e. $p_{i}=(1+\delta_{i}^{2})^{-\tilde{q}},\,\tilde{q}\geq 0$.

Here, we choose the sampling probability (of small noise) $p_{i}$ to follow a polynomial decay and the constant 1 here is to ensure $p_{i}=0$ when $\delta_{i}^{2}=0$. Under the new data assumption, we denote the corresponding RAG loss, bias-induced error, and variance-induced error for ${\boldsymbol{W}}$ to be $\tilde{{\cal L}}_{\mathrm{tt+rag}}({\boldsymbol{W}})$, $\tilde{{\operatorname{err}}}_{\mathrm{bias}}({\boldsymbol{W}})$, and $\tilde{{\operatorname{err}}}_{\mathrm{variance}}({\boldsymbol{W}})$.

###### Theorem 3 (Finite Sample RAG Bound under Distance-Weighted Mixture Noise) .

Under Assumption [1](https://arxiv.org/html/2506.03100v3#Thmassumption1), [2](https://arxiv.org/html/2506.03100v3#Thmassumption2), [5](https://arxiv.org/html/2506.03100v3#Thmassumption5), then $\tilde{{\operatorname{err}}}_{\text{bias}}({\boldsymbol{W}})=\hat{{
\operatorname{err}}}_{\text{bias}}({\boldsymbol{W}})$, and

$$ $\hat{{\operatorname{err}}}_{\mathrm{variance}}({\boldsymbol{W}})=m\sigma^{2} \operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+\sum_{i=1}^{n}\left (p_{i}\sigma_{s}^{2}+(1-p_{i})\sigma_{l}^{2}\right)[(1+\delta_{i}^{2}) \operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+\operatorname{tr}({ \boldsymbol{W}}^{2})+\operatorname{tr}({\boldsymbol{W}})^{2}]$ $$

If the variance of the retrieval distance follows power law, i.e. $\exists\gamma_{2}>0,q\geq 0$ s.t. $\delta_{i}^{2}=\gamma_{2}i^{q}$, then:

$$ $\displaystyle\tilde{{\operatorname{err}}}_{\mathrm{variance}}({\boldsymbol{W}} ^{*})$ $\displaystyle=\begin{cases}{\cal O}\left(c_{l}dn^{q-1}\sigma^{2}-(c_{l}-c_{s}) \sigma^{2}dn^{q-1-q\tilde{q}}\right)&\text{ if $n\rightarrow\infty$, $q\leq 1$ }\\ \text{diverges}&\text{ if $n\rightarrow\infty$, $q>1$}\\ \end{cases}$ (18) $$

The bias-induced error here is the same as in DPN, since we assume a polynomial dependency for $\delta_{i}^{2}$ on $i$ in both setting and the bias-induced error is independent of the variance of noise. Even though the variance of small/large noise are bounded, the dependency on the retrieval distance leads to the divergence at large $q$ ($q>1$). The large prediction noise will dominate the variance-induced error, but a larger gap between large and small noise ($c_{l}-c_{s}$) can mitigate the error by a ratio of ${\cal O}(n^{-q\tilde{q}})$. That is, the smaller $q$ and $\tilde{q}$ are, the lower the error.

We note that the uniform noise scenario can also admit the mixture noise model by taking a constant $p_{i},\;\forall i$, resulting in a form similar to the standard uniform retrieval noise in [Proposition 1](https://arxiv.org/html/2506.03100v3#Thmproposition1).

###### Assumption 5 (Distance-Weighted Mixture Noise) .

###### Theorem 3 (Finite Sample RAG Bound under Distance-Weighted Mixture Noise) .

## 5 Experiments

We investigate the effect of RAG focusing on the following questions: (Q1) Whether RAG data outperform randomly sampled in-context examples? (Q2) What are the impacts of the RAG examples from training data and RAG passages from external corpora? (Q3) With a fixed budget of example numbers, what is the effect of varying the ratio between the two types of RAG data?
Our experiments provide the following findings: (A1) RAG data lead to better performance than in-context ones under different data budgets. (A2) Interestingly, the first few RAG training examples significantly improve performance, but later ones are harmful, because the first few are highly relevant but later ones are noise rather than signal.
In contrast, RAG passages from external corpora can slowly but monotonically improve the performance, because external corpora are large enough to provide noisy but still relevant data. These are captured by different noise models in our theory. (A3) The performance is not monotonic with the ratio, and the sweet spot depends on the data/model.

##### Setup

For Natural Questions (NQ), the retrieval index is constructed from the December 2018 Wikipedia dump. For TriviaQA, we use the December 2021 version. To accommodate hardware limitations, we randomly subsample 10% of the full index for both datasets. This reduces retrieval cost and memory usage, allowing all experiments to be conducted on a single NVIDIA A100 or L40 GPU.

We use representative models ATLAS Izacard et al. (2022) and RAVEN Huang et al. (2024) on two standard open-domain question answering benchmarks Natural Questions (NQ) Kwiatkowski et al. (2019) and TriviaQA Joshi et al. (2017).
For evaluation, the context consists of $m$ in-context examples, and $n$ RAG data points (including $n_{1}$ RAG examples from the training data and $n_{2}$ RAG passages from external corpora like Wikipedia, so $n=n_{1}+n_{2}$). We choose different $m,n_{1},n_{2}$’s for our study purpose and report the standard exact match (EM) accuracy on 1000 random samples from the test set.

##### RAG v.s. In-Context

For a budget $c$, we compare using RAG only ($m=0,n_{1}=n_{2}=c/2$) and in-context examples only ($m=c,n=0$). The results in Figure [1](https://arxiv.org/html/2506.03100v3#S5.F1) show that RAG consistently outperforms in-context examples, as RAG provides query-relevant data with more signals to address the query, consistent with our analysis.

Figure: Figure 1: We compare performance between the RAG-only ($c=m$) versus in-context-only methods ($c=n_{1}+n_{2},n_{1}=n_{2}$), where $c$ is the total number of data, $n_{1}$ refers to retrieved examples and $n_{2}$ to passages.
Refer to caption: x1.png

Figure: Figure 2: We compare the performance of RAG using examples ($c=n_{1}$) versus passages ($c=n_{2}$).
Refer to caption: x3.png

Figure: (a) ATLAS Performance as a function of $n_{1}/c$ under different data points $c$ on NQ.
Refer to caption: x5.png

##### RAG Examples v.s. RAG External Passages

Next, we compare using RAG examples from training data only ($m=0,n_{1}=c,n_{2}=0$) and RAG passages from external corpora only ($m=0,n_{1}=0,n_{2}=c$). The results in Figure [2](https://arxiv.org/html/2506.03100v3#S5.F2) show interesting patterns. For RAG examples only, with more examples, the performance first significantly improves but later drops. This suggests that the first few examples are highly relevant but later ones contain more noise than signal. In contrast, for RAG passages only, the performance increases more slowly but steadily for larger budgets. This suggests the passages retrieved are noisy but still have relevant signals. This aligns with our noise modeling.
When $n_{1}$ is small ($\leq 20$ for NQ and $\leq 10$ for TriviaQA), RAG examples resemble *uniform noise* due to the relevance of retrieved examples. As $n_{1}$ increases, $n_{1}$ introduces more irrelevant or conflicting examples (i.e., *non-uniform noise*). On the other hand, $n_{2}$ resembles a *uniform noise* regime as the retrieval pool is broad with relevant data but also noisy.

When the retrieval budget is small, retrieval from training examples yield higher accuracy than from passages, even though both operate in the uniform-noise regime. This discrepancy follows from the mixture-noise effects: a passage judged relevant may still lack any answer-bearing text, raising its effective noise level relative to examples.
Furthermore, the significant drop for the retrieval from examples as opposed to retrieval from passages can be explained by the size difference for the training data and passages pool (i.e. Wikipedia). Since the passages provide a denser coverage of the semantic space, more passages will remain relevant as opposed to examples. In all, our theory covers both practical data types and matches the empirical results.

##### Ratio between RAG Examples and Passages

The different noise properties of the two kinds of RAG data imply that we should find a proper ratio between them when the total budget $c$ is fixed. Figure [3](https://arxiv.org/html/2506.03100v3#S5.F3) in the appendix shows that as the ratio $n_{1}/c$ increases, the performance initially improves—benefiting from signal information—but eventually declines as low-quality examples dominate the context. This again supports our theoretical view of signal versus noise in the retrieved data. The results demonstrate that performance initially improves as more signal (examples) is added, but eventually declines due to increasing noise from irrelevant or low-quality examples. This supports the theoretical perspective of balancing signal and noise in retrieval-augmented inputs.

## 6 Conclusion and Limitations

We model RAG as query-dependent noisy in-context learning, derive the first finite-sample error bounds for linear regression that isolate the contributions of retrieval signal and noise, and extend those bounds to different noise regimes and test-time training. Experiments on Natural Questions, TriviaQA with RAVEN, and ATLAS corroborated our theoretical analysis.

Regarding limitations, our bounds focus on the linear setting, opening avenues for future studies on nonlinear methods like kernels and neural networks. While our framework accounts for common RAG noise models, new models may be needed for other types of RAG data. A further direction is to combine RAG with test-time training, studying how on-the-fly adaptation affects both theoretical guarantees and empirical performance. Our experiments feature representative models and datasets, but future research can explore newer retrievers, LLMs like Qwen 3 and Llama 4, and more advanced RAG applications.

## 7 Acknowledgment

This work is partially supported by the National Science Foundation (NSF) under Grant CCF-2046710. I also thank my colleague Haoyue Bai for insightful discussions.

## References

- Ahn et al. [2023]
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra.
Transformers learn to implement preconditioned gradient descent for
in-context learning.
*Advances in Neural Information Processing Systems*,
36:45614–45650, 2023.
- Asai et al. [2023]
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi.
Self-RAG: Learning to retrieve, generate, and critique through
self-reflection.
*arXiv preprint arXiv:2310.11511*, 2023.
- Borgeaud et al. [2021]
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza
Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau,
Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick,
Roman Ring, T. W. Hennigan, Saffron Huang, Lorenzo Maggiore, Chris Jones,
Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals,
Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and L. Sifre.
Improving language models by retrieving from trillions of tokens.
In *International Conference on Machine Learning*, 2021.
- Brown et al. [2020]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al.
Language models are few-shot learners.
*Advances in neural information processing systems*,
33:1877–1901, 2020.
- Chen et al. [2017]
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes.
Reading wikipedia to answer open-domain questions.
*arXiv preprint arXiv:1704.00051*, 2017.
- Chen et al. [2024]
Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu.
Bge m3-embedding: Multi-lingual, multi-functionality,
multi-granularity text embeddings through self-knowledge distillation.
In *Annual Meeting of the Association for Computational
Linguistics*, 2024.
- Cheng et al. [2025]
Chen Cheng, Xinzhi Yu, Haodong Wen, Jingsong Sun, Guanzhang Yue, Yihao Zhang,
and Zeming Wei.
Exploring the robustness of in-context learning with noisy labels.
In *ICASSP 2025-2025 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP)*, pages 1–5. IEEE, 2025.
- Cheng et al. [2023]
Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing
Wang, Hao Sun, Furu Wei, Denvy Deng, and Qi Zhang.
Uprise: Universal prompt retrieval for improving zero-shot
evaluation.
*arXiv preprint arXiv:2303.08518*, 2023.
- Cover and Hart [1967]
Thomas Cover and Peter Hart.
Nearest neighbor pattern classification.
*IEEE transactions on information theory*, 13(1):21–27, 1967.
- Dong et al. [2022]
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun,
Jingjing Xu, and Zhifang Sui.
A survey for in-context learning.
*arXiv preprint arXiv:2301.00234*, 2022.
- Dong et al. [2024]
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia,
Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui.
A survey on in-context learning.
In *Proceedings of the 2024 Conference on Empirical Methods in
Natural Language Processing*, pages 1107–1128, Miami, Florida, USA, November
2024. Association for Computational Linguistics.
- Garg et al. [2022]
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant.
What can transformers learn in-context? a case study of simple
function classes.
*Advances in Neural Information Processing Systems*,
35:30583–30598, 2022.
- Gozeten et al. [2025]
Halil Alperen Gozeten, M Emrullah Ildiz, Xuechen Zhang, Mahdi Soltanolkotabi,
Marco Mondelli, and Samet Oymak.
Test-time training provably improves transformers as in-context
learners.
*arXiv preprint arXiv:2503.11842*, 2025.
- Huang et al. [2024]
Jie Huang, Wei Ping, Peng Xu, Mohammad Shoeybi, Kevin Chen-Chuan Chang, and
Bryan Catanzaro.
Raven: In-context learning with retrieval-augmented encoder-decoder
language models, 2024.
URL [https://arxiv.org/abs/2308.07922](https://arxiv.org/abs/2308.07922).
- Huang et al. [2023]
Rongjie Huang, Jia-Bin Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li,
Zhenhui Ye, Jinglin Liu, Xiaoyue Yin, and Zhou Zhao.
Make-an-audio: Text-to-audio generation with prompt-enhanced
diffusion models.
*ArXiv*, abs/2301.12661, 2023.
- Isserlis [1918]
Leon Isserlis.
On a formula for the product-moment coefficient of any order of a
normal frequency distribution in any number of variables.
*Biometrika*, 12(1/2):134–139, 1918.
- Izacard and Grave [2020]
Gautier Izacard and Edouard Grave.
Leveraging passage retrieval with generative models for open domain
question answering.
*ArXiv*, abs/2007.01282, 2020.
- Izacard et al. [2022]
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni,
Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard
Grave.
Atlas: Few-shot learning with retrieval augmented language models,
2022.
URL [https://arxiv.org/abs/2208.03299](https://arxiv.org/abs/2208.03299).
- Joshi et al. [2017]
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer.
TriviaQA: A large scale distantly supervised challenge dataset
for reading comprehension.
In Regina Barzilay and Min-Yen Kan, editors, *Proceedings of the
55th Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers)*, pages 1601–1611, Vancouver, Canada, July 2017. Association
for Computational Linguistics.
doi: 10.18653/v1/P17-1147.
URL [https://aclanthology.org/P17-1147/](https://aclanthology.org/P17-1147/).
- Kwiatkowski et al. [2019]
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur
Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin,
Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang,
Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.
Natural questions: A benchmark for question answering research.
*Transactions of the Association for Computational Linguistics*,
7:452–466, 2019.
doi: 10.1162/tacl_a_00276.
URL [https://aclanthology.org/Q19-1026/](https://aclanthology.org/Q19-1026/).
- Levy et al. [2024]
Mosh Levy, Alon Jacoby, and Yoav Goldberg.
Same task, more tokens: the impact of input length on the reasoning
performance of large language models.
*arXiv preprint arXiv:2402.14848*, 2024.
- Levy et al. [2025]
Shahar Levy, Nir Mazor, Lihi Shalmon, Michael Hassid, and Gabriel Stanovsky.
More documents, same length: Isolating the challenge of multiple
documents in rag.
*arXiv preprint arXiv:2503.04388*, 2025.
- Lewis et al. [2020a]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir
Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim
Rocktäschel, Sebastian Riedel, and Douwe Kiela.
Retrieval-augmented generation for knowledge-intensive nlp tasks.
In *Proceedings of the 34th International Conference on Neural
Information Processing Systems*, NIPS ’20, Red Hook, NY, USA,
2020a. Curran Associates Inc.
ISBN 9781713829546.
- Lewis et al. [2020b]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir
Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim
Rocktäschel, et al.
Retrieval-augmented generation for knowledge-intensive nlp tasks.
*Advances in neural information processing systems*,
33:9459–9474, 2020b.
- Li et al. [2023]
Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao.
Making large language models a better foundation for dense retrieval.
*ArXiv*, abs/2312.15503, 2023.
- Li et al. [2022]
Junlong Li, Jinyuan Wang, Zhuosheng Zhang, and Hai Zhao.
Self-prompting large language models for zero-shot open-domain qa.
*arXiv preprint arXiv:2212.08635*, 2022.
- Li and Qiu [2023]
Xiaonan Li and Xipeng Qiu.
Mot: Pre-thinking and recalling enable chatgpt to self-improve with
memory-of-thoughts.
*CoRR*, 2023.
- Li et al. [2024]
Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya
Poria, and Lidong Bing.
Chain-of-knowledge: Grounding large language models via dynamic
knowledge adapting over heterogeneous sources.
In *The Twelfth International Conference on Learning
Representations*, 2024.
- Liu et al. [2021]
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu
Chen.
What makes good in-context examples for gpt-$3$?
*arXiv preprint arXiv:2101.06804*, 2021.
- Luo et al. [2024]
Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi.
In-context learning with retrieved demonstrations for language
models: A survey.
*arXiv preprint arXiv:2401.11624*, 2024.
- Lyu et al. [2022]
Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannaneh Hajishirzi.
Z-icl: Zero-shot in-context learning with pseudo-demonstrations.
*arXiv preprint arXiv:2212.09865*, 2022.
- Meng et al. [2024]
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih
Yavuz.
Sfr-embedding-mistral:enhance text retrieval with transfer learning.
Salesforce AI Research Blog, 2024.
URL
[https://blog.salesforceairesearch.com/sfr-embedded-mistral/](https://blog.salesforceairesearch.com/sfr-embedded-mistral/).
- Min et al. [2021]
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi.
Metaicl: Learning to learn in context.
*arXiv preprint arXiv:2110.15943*, 2021.
- Petersen et al. [2008]
Kaare Brandt Petersen, Michael Syskind Pedersen, et al.
The matrix cookbook.
*Technical University of Denmark*, 7(15):510, 2008.
- Ram et al. [2023]
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin
Leyton-Brown, and Yoav Shoham.
In-context retrieval-augmented language models.
*Transactions of the Association for Computational Linguistics*,
11:1316–1331, 2023.
- Ramos et al. [2022]
Rita Parada Ramos, Bruno Martins, Desmond Elliott, and Yova Kementchedjhieva.
Smallcap: Lightweight image captioning prompted with retrieval
augmentation.
*2023 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR)*, pages 2840–2849, 2022.
- Sarto et al. [2022]
Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara.
Retrieval-augmented transformer for image captioning.
*Proceedings of the 19th International Conference on
Content-based Multimedia Indexing*, 2022.
- Shi et al. [2022]
Peng Shi, Rui Zhang, He Bai, and Jimmy Lin.
Xricl: Cross-lingual retrieval-augmented in-context learning for
cross-lingual text-to-sql semantic parsing.
*arXiv preprint arXiv:2210.13693*, 2022.
- Wang et al. [2024a]
Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu,
Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al.
Leave no document behind: Benchmarking long-context llms with
extended multi-doc qa.
*arXiv preprint arXiv:2406.17419*, 2024a.
- Wang et al. [2024b]
Qixun Wang, Yifei Wang, Yisen Wang, and Xianghua Ying.
Can in-context learning really generalize to out-of-distribution
tasks?
*arXiv preprint arXiv:2410.09695*, 2024b.
- Xiang et al. [2024]
Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek
Mittal.
Certifiably robust rag against retrieval corruption.
*arXiv preprint arXiv:2405.15556*, 2024.
- Xie et al. [2021]
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma.
An explanation of in-context learning as implicit bayesian inference.
*arXiv preprint arXiv:2111.02080*, 2021.
- Xu et al. [2024]
Fangyuan Xu, Weijia Shi, and Eunsol Choi.
RECOMP: Improving retrieval-augmented LMs with context
compression and selective augmentation.
In *The Twelfth International Conference on Learning
Representations*, 2024.
- Yang and Seo [2020]
Sohee Yang and Minjoon Seo.
Is retriever merely an approximator of reader?
*arXiv preprint arXiv:2010.10999*, 2020.
- Ye et al. [2023]
Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong.
Compositional exemplars for in-context learning.
In *International Conference on Machine Learning*, pages
39818–39833. PMLR, 2023.
- Yoran et al. [2023]
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant.
Making retrieval-augmented language models robust to irrelevant
context.
*arXiv preprint arXiv:2310.01558*, 2023.
- Zhang et al. [2024]
Ruiqi Zhang, Spencer Frei, and Peter L Bartlett.
Trained transformers learn linear models in-context.
*Journal of Machine Learning Research*, 25(49):1–55, 2024.
- Zhang et al. [2022]
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola.
Automatic chain of thought prompting in large language models.
*arXiv preprint arXiv:2210.03493*, 2022.
- Zhao et al. [2024a]
Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng
Fu, Ling Yang, Wentao Zhang, Jie Jiang, and Bin Cui.
Retrieval-augmented generation for ai-generated content: A survey.
*arXiv preprint arXiv:2402.19473*, 2024a.
- Zhao et al. [2024b]
Yufeng Zhao, Yoshihiro Sakai, and Naoya Inoue.
Noisyicl: A little noise in model parameters calibrates in-context
learning.
*arXiv preprint arXiv:2402.05515*, 2024b.

## Appendix A Technical Preliminaries

##### Additional Notations

For two integer indices $i$ and $j$, we denote $\delta_{ij}=\begin{cases}1&\text{ if $i=j$}\\
0&\text{ if $i\neq j$}\end{cases}$ as Kronecker delta.

###### Lemma 1 (Adapt ${\boldsymbol{W}}$ to Different Context Size) .

Suppose $\bar{{\boldsymbol{W}}}$ is the weight with context length $m$, then the induced ${\boldsymbol{W}}$ when evaluating on context of length $m^{\prime}$ is:

$$ ${\boldsymbol{W}}=\frac{m}{m^{\prime}}\bar{{\boldsymbol{W}}}$ $$

###### Proof.

We note that $\bar{{\boldsymbol{W}}}$ is the un-normalized weight, i.e. scaling with the inverse context size $1/m$. Only the normalized weight is preserved when applying to a sentence with a different context length.

Then, the prediction is given as:

$$ $\displaystyle\hat{y}$ $\displaystyle:={\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{X}}^{ \top}{\boldsymbol{y}}$ $\displaystyle={\boldsymbol{x}}_{q}^{\top}\frac{1}{m^{\prime}}{\boldsymbol{W}}_ {\text{Normalized}}{\boldsymbol{X}}^{\top}{\boldsymbol{y}}$ $\displaystyle={\boldsymbol{x}}_{q}^{\top}\underbrace{\frac{1}{m^{\prime}}m\bar {{\boldsymbol{W}}}}_{={\boldsymbol{W}}}{\boldsymbol{X}}^{\top}{\boldsymbol{y}}$ $$

Thus,

$$ ${\boldsymbol{W}}=\frac{m}{m^{\prime}}\bar{{\boldsymbol{W}}}$ $$

∎

###### Lemma 2 (Mixed 4th-Order Moment of Gaussian) .

Suppose ${\boldsymbol{x}}\sim{\cal N}(0,I),{\boldsymbol{r}}\sim{\cal N}(0,\delta^{2}I)$, then

- 1.
$\displaystyle\operatorname*{\mathbb{E}}[{\boldsymbol{r}}{\boldsymbol{r}}^{\top
}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}
}{\boldsymbol{r}}{\boldsymbol{r}}^{\top}]$
$\displaystyle=2\delta^{4}{\boldsymbol{W}}^{\top}{\boldsymbol{W}}+\delta^{4}
\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})I$
(19)
- 2.
$\displaystyle\operatorname*{\mathbb{E}}[{\boldsymbol{r}}{\boldsymbol{x}}^{\top
}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}
}{\boldsymbol{x}}{\boldsymbol{r}}^{\top}]=\left(\operatorname{tr}\left({
\boldsymbol{W}}^{2}\right)+\operatorname{tr}\left({\boldsymbol{W}}^{\top}{
\boldsymbol{W}}\right)+\operatorname{tr}\left({\boldsymbol{W}}\right)^{2}
\right)\delta^{2}I$
(20)
- 3.
$\displaystyle\operatorname*{\mathbb{E}}[{\boldsymbol{x}}{\boldsymbol{r}}^{\top
}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}
}{\boldsymbol{r}}{\boldsymbol{x}}^{\top}]=2\delta^{2}{\boldsymbol{W}}{
\boldsymbol{W}}^{\top}+\delta^{2}\operatorname{tr}({\boldsymbol{W}}^{\top}{
\boldsymbol{W}})I$
(21)
- 4.
$\operatorname*{\mathbb{E}}[{\boldsymbol{r}}{\boldsymbol{x}}^{\top}{\boldsymbol
{W}}^{\top}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}}{\boldsymbol
{r}}{\boldsymbol{x}}^{\top}]=\delta^{2}\left({\boldsymbol{W}}^{\top}{
\boldsymbol{W}}+{\boldsymbol{W}}^{\top}{\boldsymbol{W}}^{\top}+{\boldsymbol{W}
}^{\top}\operatorname{tr}({\boldsymbol{W}})\right)$
(22)
- 5.
$\operatorname*{\mathbb{E}}[{\boldsymbol{r}}{\boldsymbol{r}}^{\top}{\boldsymbol
{W}}^{\top}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}}{\boldsymbol
{x}}{\boldsymbol{x}}^{\top}]=\delta^{2}\left({\boldsymbol{W}}^{\top}{
\boldsymbol{W}}+{\boldsymbol{W}}^{\top}{\boldsymbol{W}}^{\top}+{\boldsymbol{W}
}^{\top}\operatorname{tr}({\boldsymbol{W}})\right)$
(23)

###### Proof.

- 1.
We have
$\displaystyle\operatorname*{\mathbb{E}}_{{\boldsymbol{x}},{\boldsymbol{r}}}[{
\boldsymbol{r}}{\boldsymbol{r}}^{\top}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}{
\boldsymbol{x}}^{\top}{\boldsymbol{W}}{\boldsymbol{r}}{\boldsymbol{r}}^{\top}]$
$\displaystyle=\operatorname*{\mathbb{E}}_{\boldsymbol{r}}[{\boldsymbol{r}}{
\boldsymbol{r}}^{\top}{\boldsymbol{W}}^{\top}{\boldsymbol{W}}{\boldsymbol{r}}{
\boldsymbol{r}}^{\top}]$
(24)
$\displaystyle=2\delta^{2}I{\boldsymbol{W}}^{\top}{\boldsymbol{W}}\delta^{2}I+
\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}}\delta^{2}I)\delta^{2}I$
$\displaystyle=2\delta^{4}{\boldsymbol{W}}^{\top}{\boldsymbol{W}}+\delta^{4}
\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})I$
$\displaystyle=2\delta^{4}{\boldsymbol{W}}^{\top}{\boldsymbol{W}}+\delta^{4}
\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})I$
where the first step follows from [Equation 32](https://arxiv.org/html/2506.03100v3#A1.E32).
- 2.
$\displaystyle\operatorname*{\mathbb{E}}_{{\boldsymbol{x}},{\boldsymbol{r}}}[{
\boldsymbol{r}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}{
\boldsymbol{x}}^{\top}{\boldsymbol{W}}{\boldsymbol{x}}{\boldsymbol{r}}^{\top}]$
$\displaystyle=\operatorname*{\mathbb{E}}_{\boldsymbol{r}}\left[{\boldsymbol{r}
}\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}}\left[{\boldsymbol{x}}^{\top}{
\boldsymbol{W}}^{\top}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}}{
\boldsymbol{x}}\right]{\boldsymbol{r}}^{\top}\right]$
(25)
$\displaystyle=\left(\operatorname{tr}\left({\boldsymbol{W}}^{2}\right)+
\operatorname{tr}\left({\boldsymbol{W}}^{\top}{\boldsymbol{W}}\right)+
\operatorname{tr}\left({\boldsymbol{W}}\right)^{2}\right)\delta^{2}I$
where the first step follows from [Equation 34](https://arxiv.org/html/2506.03100v3#A1.E34).
- 3.
$\displaystyle\operatorname*{\mathbb{E}}[{\boldsymbol{x}}{\boldsymbol{r}}^{\top
}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}
}{\boldsymbol{r}}{\boldsymbol{x}}^{\top}]$
$\displaystyle=\operatorname*{\mathbb{E}}\left[{\boldsymbol{x}}\operatorname*{
\mathbb{E}}\left[\operatorname{tr}\left({\boldsymbol{r}}^{\top}{\boldsymbol{W}
}^{\top}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}}{\boldsymbol{r}
}\right)\right]{\boldsymbol{x}}^{\top}\right]$
(26)
$\displaystyle=\operatorname*{\mathbb{E}}\left[{\boldsymbol{x}}\operatorname*{
\mathbb{E}}\left[\operatorname{tr}\left({\boldsymbol{r}}{\boldsymbol{r}}^{\top
}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}
}\right)\right]{\boldsymbol{x}}^{\top}\right]$
$\displaystyle=\delta^{2}\operatorname*{\mathbb{E}}{\boldsymbol{x}}
\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{x}}{\boldsymbol{x}}^{
\top}{\boldsymbol{W}}){\boldsymbol{x}}^{\top}$
$\displaystyle=\delta^{2}\operatorname*{\mathbb{E}}{\boldsymbol{x}}
\operatorname{tr}({\boldsymbol{x}}^{\top}{\boldsymbol{W}}{\boldsymbol{W}}^{
\top}{\boldsymbol{x}}){\boldsymbol{x}}^{\top}$
$\displaystyle=\delta^{2}\operatorname*{\mathbb{E}}{\boldsymbol{x}}{\boldsymbol
{x}}^{\top}{\boldsymbol{W}}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}{\boldsymbol
{x}}^{\top}$
$\displaystyle=2\delta^{2}{\boldsymbol{W}}{\boldsymbol{W}}^{\top}+\delta^{2}
\operatorname{tr}({\boldsymbol{W}}{\boldsymbol{W}}^{\top})I$
$\displaystyle=2\delta^{2}{\boldsymbol{W}}{\boldsymbol{W}}^{\top}+\delta^{2}
\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})I$
where the first three steps follow from the cyclic property of trace and the last step follows from [Equation 32](https://arxiv.org/html/2506.03100v3#A1.E32).
- 4.
$\displaystyle\operatorname*{\mathbb{E}}[{\boldsymbol{r}}{\boldsymbol{x}}^{\top
}{\boldsymbol{W}}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}}{
\boldsymbol{r}}{\boldsymbol{x}}^{\top}]$
$\displaystyle=\operatorname*{\mathbb{E}}\left[{\boldsymbol{r}}({\boldsymbol{x}
}^{\top}{\boldsymbol{W}}{\boldsymbol{r}})^{\top}{\boldsymbol{x}}^{\top}{
\boldsymbol{W}}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}\right]$
$\displaystyle=\operatorname*{\mathbb{E}}\left[{\boldsymbol{r}}{\boldsymbol{r}}
^{\top}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{
\boldsymbol{W}}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}\right]$
$\displaystyle=\delta^{2}\operatorname*{\mathbb{E}}\left[{\boldsymbol{W}}^{\top
}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}}{\boldsymbol{x}}{
\boldsymbol{x}}^{\top}\right]$
$\displaystyle=\delta^{2}{\boldsymbol{W}}^{\top}\left({\boldsymbol{W}}+{
\boldsymbol{W}}^{\top}+\operatorname{tr}({\boldsymbol{W}})I\right)$
$\displaystyle=\delta^{2}\left({\boldsymbol{W}}^{\top}{\boldsymbol{W}}+{
\boldsymbol{W}}^{\top}{\boldsymbol{W}}^{\top}+{\boldsymbol{W}}^{\top}
\operatorname{tr}({\boldsymbol{W}})\right)$
where the first step follows from ${\boldsymbol{x}}^{\top}{\boldsymbol{W}}{\boldsymbol{r}}$ being scalar, and the third step follows from [Equation 32](https://arxiv.org/html/2506.03100v3#A1.E32).
- 5.
$\displaystyle\operatorname*{\mathbb{E}}[{\boldsymbol{r}}{\boldsymbol{r}}^{\top
}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}
}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}]$
$\displaystyle=\delta^{2}{\boldsymbol{W}}^{\top}\left({\boldsymbol{W}}+{
\boldsymbol{W}}^{\top}+\operatorname{tr}\left({\boldsymbol{W}}\right)\right)$
(27)
$\displaystyle=\delta^{2}\left({\boldsymbol{W}}^{\top}{\boldsymbol{W}}+{
\boldsymbol{W}}^{\top}{\boldsymbol{W}}^{\top}+{\boldsymbol{W}}^{\top}
\operatorname{tr}({\boldsymbol{W}})\right)$
It follows from the application of [Equation 32](https://arxiv.org/html/2506.03100v3#A1.E32).

∎

###### Lemma 3 (Expectation of 6th-Order Gaussian Monomial) .

If ${\boldsymbol{x}}\sim{\cal N}(0,I)$, then

$$ $\displaystyle\operatorname*{\mathbb{E}}[{\boldsymbol{x}}{\boldsymbol{x}}^{\top }A{\boldsymbol{x}}{\boldsymbol{x}}^{\top}B{\boldsymbol{x}}{\boldsymbol{x}}^{ \top}]$ $\displaystyle=AB+AB^{\top}+A^{\top}B+A^{\top}B^{\top}+B^{\top}A+B^{\top}A^{ \top}+BA+BA^{\top}$ (28) $\displaystyle\quad+\operatorname{tr}(B)A+\operatorname{tr}(B)A^{\top}+ \operatorname{tr}(A)B+\operatorname{tr}(A)B^{\top}$ $\displaystyle\quad+\operatorname{tr}(A)\operatorname{tr}(B)I+\operatorname{tr} \left(AB^{\top}\right)I+\operatorname{tr}(AB)I$ $$

$$ $\displaystyle\operatorname*{\mathbb{E}}[{\boldsymbol{x}}{\boldsymbol{x}}^{\top }{\boldsymbol{W}}^{\top}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W} }{\boldsymbol{x}}{\boldsymbol{x}}^{\top}]$ $\displaystyle=\operatorname*{\mathbb{E}}[{\boldsymbol{x}}{\boldsymbol{x}}^{ \top}{\boldsymbol{W}}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}}{ \boldsymbol{x}}{\boldsymbol{x}}^{\top}]$ (29) $\displaystyle=2\left({\boldsymbol{W}}^{2}+{\boldsymbol{W}}^{\top}{\boldsymbol{ W}}^{\top}+{\boldsymbol{W}}^{\top}{\boldsymbol{W}}+{\boldsymbol{W}}{ \boldsymbol{W}}^{\top}+\operatorname{tr}\left({\boldsymbol{W}}\right){ \boldsymbol{W}}+\operatorname{tr}\left({\boldsymbol{W}}\right){\boldsymbol{W}} ^{\top}\right)$ $\displaystyle+\operatorname{tr}\left({\boldsymbol{W}}\right)^{2}I+ \operatorname{tr}\left({\boldsymbol{W}}^{2}\right)I+\operatorname{tr}\left({ \boldsymbol{W}}^{\top}{\boldsymbol{W}}\right)I$ $$

###### Proof.

Let $T:=\operatorname*{\mathbb{E}}[{\boldsymbol{x}}{\boldsymbol{x}}^{\top}A{
\boldsymbol{x}}{\boldsymbol{x}}^{\top}B{\boldsymbol{x}}{\boldsymbol{x}}^{\top}]$. Then, let’s consider one scalar entry:

$$ $T_{ij}=\operatorname*{\mathbb{E}}\left[\sum_{k,\ell,m,n}x_{i}x_{k}A_{k\ell}x_{ \ell}x_{m}B_{mn}x_{n}x_{j}\right]=\sum_{k,\ell,m,n}A_{k\ell}B_{mn}\cdot \operatorname*{\mathbb{E}}\left[x_{i}x_{k}x_{\ell}x_{m}x_{n}x_{j}\right]$ (30) $$

We now need to compute the 6th-order central moment of standard normal variables. This can be computed using the Isserlis’ Theorem [Isserlis, 1918]:

$$ $\operatorname*{\mathbb{E}}[x_{1}\cdots x_{s}]=\sum_{p\in P_{s}^{2}}\prod_{(i,j )\in p}\operatorname*{\mathbb{E}}[x_{i}x_{j}]$ (31) $$

where $P_{s}^{2}$ stands for all distinct ways of partitioning $\{1,\dots,s\}$ into pairs ${i,j}$ (perfect matching), and the product is over the pairs contained in $p$.

We note that the number of perfect matching for $s$ examples is given as:

$$ $\#\text{perfect matching}=\frac{s!}{2^{s/2}(s/2)!}$ $$

where $2^{s/2}$ is for ignoring the ordering inside pairs and $(s/2)!$ is for ignoring the ordering between pairs.

We note that there are $\frac{6!}{2^{3}\cdot 3!}=15$ distinct partitions for the $6$-th order product of Gaussian random variable. Suppose $(x_{a},x_{b}),(x_{c},x_{d}),(x_{e},x_{f})$ is a valid pairing, then:

$$ $\operatorname*{\mathbb{E}}[x_{a}x_{b}]\operatorname*{\mathbb{E}}[x_{c}x_{d}] \operatorname*{\mathbb{E}}[x_{e}x_{f}]=\begin{cases}1&\text{if }a=b,c=d,e=f\\ 0&\text{else}\end{cases}=\delta_{ab}\cdot\delta_{cd}\cdot\delta_{ef}$ $$

where $\delta_{ij}:=\mathbbm{1}[i=j]$ stands for the Kronecker delta.

Here, we will discuss the result for all 15 distinct pairings:

- 1.
$(i,k)(l,m)(n,j)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{m}A_{im}B_{mj}=A_{i\cdot}B_{\cdot j}=(AB)_{ij}$
- 2.
$(i,k)(l,n)(m,j)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{m}A_{im}B_{jm}=A_{i\cdot}B_{j\cdot}=(AB^{\top
})_{ij}$
- 3.
$(i,k)(l,j)(m,n)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{m}A_{ij}B_{mm}=\operatorname{tr}(B)A_{ij}$
- 4.
$(i,l)(k,m)(n,j)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{m}A_{mi}B_{mj}=A_{\cdot i}B_{\cdot j}=(A^{
\top}B)_{ij}$
- 5.
$(i,l)(k,n)(m,j)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{k}A_{ki}B_{jk}=A_{\cdot i}B_{j\cdot}=(A^{\top
}B^{\top})_{i,j}$
- 6.
$(i,l)(k,j)(m,n)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{m}A_{ji}B_{mm}=(A^{\top})_{ij}\operatorname{
tr}(B)$
- 7.
$(i,m)(k,l)(n,j)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{k}A_{kk}B_{ij}=\operatorname{tr}(A)B_{ij}$
- 8.
$(i,m)(k,n)(l,j)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{k}A_{kj}B_{ik}=A_{\cdot j}B_{i\cdot}=(BA)_{ij}$
- 9.
$(i,m)(k,j)(l,n)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{l}A_{jl}B_{il}=A_{j\cdot}B_{i\cdot}=(BA^{\top
})_{ij}$
- 10.
$(i,n)(k,l)(m,j)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{k}A_{kk}B_{ji}=\operatorname{tr}(A)(B^{\top})
_{ij}$
- 11.
$(i,n)(k,m)(l,j)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{m}A_{mj}B_{mi}=A_{\cdot j}B_{\cdot i}=(B^{
\top}A)_{ij}$
- 12.
$(i,n)(k,j)(l,m)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{m}A_{jm}B_{mi}=A_{j\cdot}B_{\cdot i}=(B^{\top
}A^{\top})_{ij}$
- 13.
$(i,j)(k,l)(m,n)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{k,m}A_{kk}B_{mm}=\operatorname{tr}(A)
\operatorname{tr}(B)\delta_{ij}$
- 14.
$(i,j)(k,m)(l,n)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{k,l}A_{kl}B_{kl}=\operatorname{tr}(AB^{\top})
\delta_{ij}$
- 15.
$(i,j)(k,n)(l,m)$
$\sum_{k,l,m,n}A_{kl}B_{mn}=\sum_{m,k}A_{km}B_{mk}=\operatorname{tr}(AB)\delta_
{ij}$

Summing up all of these 15 terms together, we obtain Eq. ([28](https://arxiv.org/html/2506.03100v3#A1.E28)). Then, we plug in $A={\boldsymbol{W}},B={\boldsymbol{W}}^{\top}$, we obtain Eq. ([29](https://arxiv.org/html/2506.03100v3#A1.E29)).
∎

###### Lemma 4 (4th-Order Gaussian Monomial) .

Let ${\boldsymbol{x}},{\boldsymbol{x}}_{1},\dots,{\boldsymbol{x}}_{m}\sim{\cal N}(0
,I)$ and ${\boldsymbol{X}}=[{\boldsymbol{x}}_{1}^{\top};\dots;{\boldsymbol{x}}_{m}^{\top}]$. Then, we have

$$ $\displaystyle\operatorname*{\mathbb{E}}{\boldsymbol{x}}{\boldsymbol{x}}^{\top} {\boldsymbol{W}}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}$ $\displaystyle={\boldsymbol{W}}+{\boldsymbol{W}}^{\top}+\operatorname{tr}({ \boldsymbol{W}})I$ (32) $\displaystyle=2{\boldsymbol{W}}+\operatorname{tr}({\boldsymbol{W}})I\quad\text {if ${\boldsymbol{W}}$ is symmetric}$ $$

and

$$ $\displaystyle\operatorname*{\mathbb{E}}{\boldsymbol{X}}^{\top}{\boldsymbol{X}} {\boldsymbol{W}}{\boldsymbol{X}}^{\top}{\boldsymbol{X}}$ $\displaystyle=m^{2}{\boldsymbol{W}}+m{\boldsymbol{W}}^{\top}+m\operatorname{tr }({\boldsymbol{W}})I$ (33) $\displaystyle=m(m+1){\boldsymbol{W}}+m\operatorname{tr}({\boldsymbol{W}})I \quad\text{if ${\boldsymbol{W}}$ is symmetric}$ $$

$$ $\displaystyle\operatorname*{\mathbb{E}}{\boldsymbol{x}}^{\top}A{\boldsymbol{x} }{\boldsymbol{x}}^{\top}B{\boldsymbol{x}}=\operatorname{tr}\left(A\left(B+B^{ \top}\right)\right)+\operatorname{tr}(A)\operatorname{tr}(B)$ (34) $$

If $A={\boldsymbol{W}}^{\top},B={\boldsymbol{W}}$, then

$$ $\displaystyle\operatorname*{\mathbb{E}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}} ^{\top}{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}}{\boldsymbol{x}}$ $\displaystyle=\operatorname*{\mathbb{E}}{\boldsymbol{x}}^{\top}{\boldsymbol{W} }{\boldsymbol{x}}{\boldsymbol{x}}^{\top}{\boldsymbol{W}}{\boldsymbol{x}}= \operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+\operatorname{tr}({ \boldsymbol{W}}^{2})+\operatorname{tr}({\boldsymbol{W}})^{2}$ (35) $$

###### Proof.

[Equation 32](https://arxiv.org/html/2506.03100v3#A1.E32) follows from section 8.2.4 of [Petersen et al., 2008] by plugging in mean 0 and variance $I$.

$$ $\displaystyle\operatorname*{\mathbb{E}}{\boldsymbol{X}}^{\top}{\boldsymbol{X}} {\boldsymbol{W}}{\boldsymbol{X}}^{\top}{\boldsymbol{X}}$ $\displaystyle=\sum_{i}{\boldsymbol{x}}_{i}{\boldsymbol{x}}_{i}^{\top}{ \boldsymbol{W}}{\boldsymbol{x}}_{i}{\boldsymbol{x}}_{i}^{\top}+\sum_{i\neq j}{ \boldsymbol{x}}_{i}{\boldsymbol{x}}_{i}^{\top}{\boldsymbol{W}}{\boldsymbol{x}} _{j}{\boldsymbol{x}}_{j}^{\top}$ (36) $\displaystyle=m\left({\boldsymbol{W}}+{\boldsymbol{W}}^{\top}+\operatorname{tr }({\boldsymbol{W}})I\right)+m(m-1){\boldsymbol{W}}$ $\displaystyle=m^{2}{\boldsymbol{W}}+m{\boldsymbol{W}}^{\top}+m\operatorname{tr }({\boldsymbol{W}})I$ $\displaystyle=m(m+1){\boldsymbol{W}}+m\operatorname{tr}({\boldsymbol{W}})I$ $$

where the second step follows from plugging in [Equation 32](https://arxiv.org/html/2506.03100v3#A1.E32).

[Equation 34](https://arxiv.org/html/2506.03100v3#A1.E34) follows from section 8.2.4 of [Petersen et al., 2008] by plugging in mean 0 and variance $I$.

∎

###### Lemma 1 (Adapt 𝑾 𝑾 {\boldsymbol{W}} bold_italic_W to Different Context Size) .

###### Proof.

###### Lemma 2 (Mixed 4th-Order Moment of Gaussian) .

###### Proof.

###### Lemma 3 (Expectation of 6th-Order Gaussian Monomial) .

###### Proof.

###### Lemma 4 (4th-Order Gaussian Monomial) .

###### Proof.

## Appendix B Additional Proof for RAG

Here, we provide an overview of the organization of the proof. First, we consider the uniform retrieval noise scenario, and compute the population loss for generic ${\boldsymbol{W}}$ in [Theorem 1](https://arxiv.org/html/2506.03100v3#Thmtheorem1). Then, we plug in the special case ${\boldsymbol{W}}^{*}$ (isotropic pretrained weight), and provide a closed-form loss in [Proposition 3](https://arxiv.org/html/2506.03100v3#Thmproposition3). Then, we analyze its finite sample complexity in [Proposition 1](https://arxiv.org/html/2506.03100v3#Thmproposition1) and the optimal RAG examples in relation to ICL examples in [Proposition 2](https://arxiv.org/html/2506.03100v3#Thmproposition2).

Later on, we provide an finite sample complexity analysis for non-uniform retrieval noise, [Theorem 2](https://arxiv.org/html/2506.03100v3#Thmtheorem2) for Distance Proportional Noise, and [Theorem 3](https://arxiv.org/html/2506.03100v3#Thmtheorem3) for Distance-Weighted Mixture Noise.

### B.1 Uniform Retrieval Noise

###### Theorem (Restatement of Theorem 1 ) .

Under Assumption [1](https://arxiv.org/html/2506.03100v3#Thmassumption1), [2](https://arxiv.org/html/2506.03100v3#Thmassumption2), [3](https://arxiv.org/html/2506.03100v3#Thmassumption3), the population loss of the linear self-attention predictor
$\hat{y}_{q}={\boldsymbol{x}}_{q}^{\top}\boldsymbol{W}{\boldsymbol{X}}^{\top}{
\boldsymbol{y}}$
satisfies

$$ $\mathcal{L}_{\text{tt+rag}}({\boldsymbol{W}})=\underbrace{\operatorname*{ \mathbb{E}}\left(\operatorname*{\mathbb{E}}\left(\hat{y}_{q}\right)-\hat{y}_{q }\right)^{2}}_{:=\operatorname{err}_{\text{variance}}({\boldsymbol{W}})}+ \underbrace{\operatorname*{\mathbb{E}}\left(\operatorname*{\mathbb{E}}\left( \hat{y}_{q}\right)-\operatorname*{\mathbb{E}}\left(y_{q}\right)\right)^{2}}_{: =\operatorname{err}_{\text{bias}}({\boldsymbol{W}})}+\underbrace{\sigma^{2}}_{ \text{irreducible noise}}$ (37) $$

Specifically,

$$ $\displaystyle{\operatorname{err}}_{\text{variance}}({\boldsymbol{W}})$ $\displaystyle=\left[m\sigma^{2}+\left(1+\delta^{2}\right)n\sigma_{\text{rag}}^ {2}\right]\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+n\sigma_{ \text{rag}}^{2}\operatorname{tr}({\boldsymbol{W}}^{2})+n\sigma_{\text{rag}}^{2 }\operatorname{tr}({\boldsymbol{W}})^{2}$ (38) $\displaystyle{\operatorname{err}}_{\text{bias}}({\boldsymbol{W}})$ $\displaystyle=\beta_{\mathrm{tt}}^{\top}\left[I-(n\delta^{2}+2n+m)({ \boldsymbol{W}}+{\boldsymbol{W}}^{\top})-2n\operatorname{tr}({\boldsymbol{W}}) I+M_{4}\right]\beta_{\mathrm{tt}}$ $\displaystyle=\beta_{\mathrm{tt}}^{\top}\left[I-(n\delta^{2}+2n+m)({ \boldsymbol{W}}+{\boldsymbol{W}}^{\top})-2n\operatorname{tr}({\boldsymbol{W}}) I\right.$ $\displaystyle\quad+\underbrace{\left[n^{2}\left(2+\delta^{2}\right)+n\left(m+ \delta^{2}\right)\right]}_{:=c_{1}}\left({\boldsymbol{W}}^{2}+\left({ \boldsymbol{W}}^{2}\right)^{\top}\right)+\underbrace{2n(n+\delta^{2})}_{:=c_{2 }}{\boldsymbol{W}}{\boldsymbol{W}}^{\top}$ $\displaystyle\quad+\underbrace{\left[m^{2}+m+mn\left(2+2\delta^{2}\right)+n^{2 }\left(2+2\delta^{2}+\delta^{4}\right)+n\left(2\delta^{2}+\delta^{4}\right) \right]}_{:=c_{3}}{\boldsymbol{W}}^{\top}{\boldsymbol{W}}$ $\displaystyle\quad+\underbrace{\left[n^{2}\left(2+\delta^{2}\right)+n\left(m+ \delta^{2}\right)\right]}_{:=c_{4},\;c_{4}=c_{1}}\left(\operatorname{tr}({ \boldsymbol{W}})\left({\boldsymbol{W}}+{\boldsymbol{W}}^{\top}\right)\right)$ $\displaystyle\left.\quad+\underbrace{\left[n^{2}+n\delta^{2}\right]}_{:=c_{5}} \left(\operatorname{tr}({\boldsymbol{W}})^{2}+\operatorname{tr}\left({ \boldsymbol{W}}^{2}\right)\right)I+\underbrace{\left[m+n^{2}+n\left(2\delta^{2 }+\delta^{4}\right)\right]}_{:=c_{6}}\operatorname{tr}\left({\boldsymbol{W}}^{ \top}{\boldsymbol{W}}\right)I\right]\beta_{\mathrm{tt}}$ $$

###### Proof.

For computational convenience, I will define the following quantities for Gram matrix: ${\boldsymbol{G}}_{0}={\boldsymbol{X}}_{\mathrm{icl}}^{\top}{\boldsymbol{X}}_{
\mathrm{icl}}$, ${\boldsymbol{G}}_{i}:=({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i})({\boldsymbol
{x}}_{q}+{\boldsymbol{r}}_{i})^{\top}$, and ${\boldsymbol{G}}:={\boldsymbol{G}}_{0}+\sum_{i\in[n]}{\boldsymbol{G}}_{i}$.

We write down the error explicitly:

$$ $\displaystyle y_{q}-{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{X} }^{\top}{\boldsymbol{y}}$ $\displaystyle={\boldsymbol{x}}_{q}^{\top}\beta_{\mathrm{tt}}+\epsilon_{q}-{ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{X}}^{\top}{\boldsymbol{ X}}\beta_{\mathrm{tt}}-{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol {X}}^{\top}\boldsymbol{\epsilon}$ (39) $\displaystyle={\boldsymbol{x}}_{q}^{\top}\left(\beta_{\mathrm{tt}}-{ \boldsymbol{W}}{\boldsymbol{G}}\beta_{\mathrm{tt}}\right)-{\boldsymbol{x}}_{q} ^{\top}{\boldsymbol{W}}{\boldsymbol{X}}^{\top}\boldsymbol{\epsilon}+\epsilon_{q}$ $\displaystyle={\boldsymbol{x}}_{q}^{\top}\left(I-{\boldsymbol{W}}{\boldsymbol{ G}}\right)\beta_{\mathrm{tt}}-{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{X}}^{\top}\boldsymbol{\epsilon}+\epsilon_{q}$ $$

Therefore, the population loss is equal to:

$$ $\displaystyle{\cal L}_{\text{tt+rag}}({\boldsymbol{W}})$ $\displaystyle=\operatorname*{\mathbb{E}}_{({\boldsymbol{x}}_{q},y_{q}),({ \boldsymbol{X}},{\boldsymbol{y}}),\boldsymbol{\epsilon}}\left[\left({ \boldsymbol{x}}_{q}^{\top}\left(I-{\boldsymbol{W}}{\boldsymbol{G}}\right)\beta _{\mathrm{tt}}-{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{X}}^{ \top}\boldsymbol{\epsilon}\right)^{2}\right]+\sigma^{2}$ $$

We note that both $\boldsymbol{\epsilon}_{\mathrm{icl}}$ and $\boldsymbol{\epsilon}_{\mathrm{rag}}$ are independent of ${\boldsymbol{x}}_{q},{\boldsymbol{X}}$ (including ${\boldsymbol{r}}$), and $\operatorname*{\mathbb{E}}[\boldsymbol{\epsilon}]=0$.

$$ $\operatorname*{\mathbb{E}}_{\boldsymbol{\epsilon}}\left[-2\left({\boldsymbol{x }}_{q}^{\top}\left(I-{\boldsymbol{W}}{\boldsymbol{G}}\right)\beta_{\mathrm{tt} }\right)\left({\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{X}}^{ \top}\boldsymbol{\epsilon}\right)\right]=0$ $$

And therefore, we have the following loss decomposition:

$$ ${\cal L}_{\text{tt+rag}}({\boldsymbol{W}})=\operatorname*{\mathbb{E}}_{{ \boldsymbol{x}}_{q},{\boldsymbol{X}},\boldsymbol{\epsilon}}\left[\left({ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{X}}^{\top}\boldsymbol{ \epsilon}\right)^{2}\right]+\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{ \boldsymbol{X}}}\left[\left({\boldsymbol{x}}_{q}^{\top}\left(I-{\boldsymbol{W} }{\boldsymbol{G}}\right)\beta_{\mathrm{tt}}\right)^{2}\right]+\sigma^{2}$ (40) $$

Then, we compute the mean of the prediction and the label:

$$ $\operatorname*{\mathbb{E}}_{\epsilon_{q}}y_{q}=\operatorname*{\mathbb{E}}_{ \epsilon_{q}}\left({\boldsymbol{x}}_{q}^{\top}\beta_{\mathrm{tt}}+\epsilon_{q} \right)={\boldsymbol{x}}_{q}^{\top}\beta_{\mathrm{tt}}$ $$

$$ $\displaystyle\operatorname*{\mathbb{E}}_{\boldsymbol{\epsilon}}\hat{y}_{q}$ $\displaystyle=\operatorname*{\mathbb{E}}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{X}}^{\top}{\boldsymbol{y}}$ $\displaystyle=\operatorname*{\mathbb{E}}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{X}}^{\top}({\boldsymbol{X}}\beta_{\mathrm{tt}}+ \boldsymbol{\epsilon})$ $\displaystyle=\operatorname*{\mathbb{E}}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{G}}\beta_{\mathrm{tt}}$ $$

And further, we have

$$ $\displaystyle\operatorname*{\mathbb{E}}_{\epsilon_{q}}\left(y_{q}- \operatorname*{\mathbb{E}}_{\epsilon_{q}}y_{q}\right)^{2}$ $\displaystyle=\operatorname*{\mathbb{E}}_{\epsilon_{q}}\left({\boldsymbol{x}}_ {q}^{\top}\beta_{\mathrm{tt}}+\epsilon_{q}-{\boldsymbol{x}}_{q}^{\top}\beta_{ \mathrm{tt}}\right)^{2}=\operatorname*{\mathbb{E}}_{\epsilon_{q}}\epsilon_{q}^ {2}=\sigma^{2}$ (41) $\displaystyle\left(\hat{y}_{q}-\operatorname*{\mathbb{E}}_{\boldsymbol{ \epsilon}}\hat{y}_{q}\right)^{2}$ $\displaystyle=\left({\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{X} }^{\top}({\boldsymbol{X}}\beta_{\mathrm{tt}}+\boldsymbol{\epsilon})-{ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{X}}^{\top}{\boldsymbol{ X}}\beta_{\mathrm{tt}}\right)^{2}=\left({\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{X}}^{\top}\boldsymbol{\epsilon}\right)^{2}$ $\displaystyle\left(\operatorname*{\mathbb{E}}_{\epsilon_{q}}(y_{q})- \operatorname*{\mathbb{E}}_{\boldsymbol{\epsilon}}\hat{y}_{q}\right)^{2}$ $\displaystyle=\left({\boldsymbol{x}}_{q}^{\top}\beta_{\mathrm{tt}}-{ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}}\beta_{\mathrm{tt}} \right)^{2}=\left({\boldsymbol{x}}_{q}^{\top}(I-{\boldsymbol{W}}{\boldsymbol{G }})\beta_{\mathrm{tt}}\right)^{2}$ $$

If we plug [Equation 41](https://arxiv.org/html/2506.03100v3#A2.E41) into the loss decomposition [Equation 40](https://arxiv.org/html/2506.03100v3#A2.E40), we have

$$ $\displaystyle{\cal L}_{\text{tt+rag}}({\boldsymbol{W}})$ $\displaystyle=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X} },\boldsymbol{\epsilon}}\left[\left({\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W} }{\boldsymbol{X}}^{\top}\boldsymbol{\epsilon}\right)^{2}\right]+\operatorname* {\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X}}}\left[\left({\boldsymbol{x }}_{q}^{\top}\left(I-{\boldsymbol{W}}{\boldsymbol{G}}\right)\beta_{\mathrm{tt} }\right)^{2}\right]+\sigma^{2}$ (42) $\displaystyle=\underbrace{\operatorname*{\mathbb{E}}\left(\operatorname*{ \mathbb{E}}_{\boldsymbol{\epsilon}}\left(\hat{y}_{q}\right)-\hat{y}_{q}\right) ^{2}}_{:=\operatorname{err}_{\text{variance}}({\boldsymbol{W}})}+\underbrace{ \operatorname*{\mathbb{E}}\left(\operatorname*{\mathbb{E}}_{\epsilon_{q}}\left (\hat{y}_{q}\right)-\operatorname*{\mathbb{E}}_{\boldsymbol{\epsilon}}\left(y_ {q}\right)\right)^{2}}_{:=\operatorname{err}_{\text{bias}}({\boldsymbol{W}})}+ \underbrace{\operatorname*{\mathbb{E}}\left(y_{q}-\operatorname*{\mathbb{E}}_{ \epsilon_{q}}y_{q}\right)^{2}}_{\begin{subarray}{c}=\sigma^{2}\\ \text{(irreducible noise)}\end{subarray}}$ $$

and we can obtain the bias-variance tradeoff as given in [Equation 37](https://arxiv.org/html/2506.03100v3#A2.E37).

###### Theorem (Restatement of Theorem 1 ) .

###### Proof.

##### Compute 𝔼 𝒙 q , 𝑿 , 𝒓 , ϵ [ ( 𝒙 q ⊤ ⁢ 𝑾 ⁢ 𝑿 ⊤ ⁢ ϵ ) 2 ] subscript 𝔼 subscript 𝒙 𝑞 𝑿 𝒓 bold-italic-ϵ superscript superscript subscript 𝒙 𝑞 top 𝑾 superscript 𝑿 top bold-italic-ϵ 2 \operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X}},{\boldsymbol%
{r}},\boldsymbol{\epsilon}}\left[\left({\boldsymbol{x}}_{q}^{\top}{\boldsymbol%
{W}}{\boldsymbol{X}}^{\top}\boldsymbol{\epsilon}\right)^{2}\right] blackboard_E start_POSTSUBSCRIPT bold_italic_x start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT , bold_italic_X , bold_italic_r , bold_italic_ϵ end_POSTSUBSCRIPT [ ( bold_italic_x start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⊤ end_POSTSUPERSCRIPT bold_italic_W bold_italic_X start_POSTSUPERSCRIPT ⊤ end_POSTSUPERSCRIPT bold_italic_ϵ ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ]

First, we let

$$ $z:={\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{X}}^{\top} \boldsymbol{\epsilon}=\sum_{i=1}^{m+n}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{ W}}{\boldsymbol{x}}_{i}\cdot\epsilon_{i}$ $$

Then,

$$ $z^{2}=\sum_{i,j=1}^{m+n}({\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{x}}_{i})({\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{x }}_{j})\epsilon_{i}\epsilon_{j}=\sum_{i,j=1}^{m+n}({\boldsymbol{x}}_{i}^{\top} {\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q})({\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{x}}_{j})\epsilon_{i}\epsilon_{j}$ $$

Taking expectation:

$$ $\operatorname*{\mathbb{E}}_{\boldsymbol{\epsilon}}[z^{2}]=\sum_{i,j=1}^{m+n}({ \boldsymbol{x}}_{i}^{\top}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q})({ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{x}}_{j})\cdot \operatorname*{\mathbb{E}}[\epsilon_{i}\epsilon_{j}]$ $$

Because the noise terms are independent and zero-mean, we have:

$$ $\operatorname*{\mathbb{E}}[\epsilon_{i}\epsilon_{j}]=\begin{cases}\sigma^{2},& i=j\leq m\\ \sigma^{2}_{\text{rag}},&i=j>m\\ 0,&i\neq j\end{cases}$ $$

So only the diagonal terms survive:

$$ $\operatorname*{\mathbb{E}}[z^{2}]=\sum_{i=1}^{m}\sigma^{2}\cdot\operatorname*{ \mathbb{E}}\left[({\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{x}}_ {i})^{2}\right]+\sum_{i=m+1}^{m+n}\sigma^{2}_{\text{rag}}\cdot\operatorname*{ \mathbb{E}}\left[({\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}({\boldsymbol{x}} _{q}+{\boldsymbol{r}}_{i-m}))^{2}\right]$ $$

- •
ICL Term: Since ${\boldsymbol{x}}_{q},{\boldsymbol{x}}_{i}\sim{\cal N}(0,I)$ and are independent,
$\displaystyle\operatorname*{\mathbb{E}}[({\boldsymbol{x}}_{q}^{\top}{
\boldsymbol{W}}{\boldsymbol{x}}_{i})^{2}]$
$\displaystyle=\operatorname*{\mathbb{E}}\left[{\boldsymbol{x}}_{i}^{\top}{
\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}^{\top}_{q}{
\boldsymbol{W}}{\boldsymbol{x}}_{i}\right]$
(43)
$\displaystyle=\operatorname*{\mathbb{E}}\left[\operatorname{tr}\left({
\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}^{\top}_{q}{
\boldsymbol{W}}{\boldsymbol{x}}_{i}{\boldsymbol{x}}_{i}^{\top}\right)\right]$
$\displaystyle=\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})$
where the first step follows from the cyclic property of trace, the last step follows from the symmetry of ${\boldsymbol{W}}$.
$\Rightarrow\quad\text{ICL contribution}=m\cdot\sigma^{2}\cdot\operatorname{tr}
({\boldsymbol{W}}^{\top}{\boldsymbol{W}})$
(44)
- •
RAG Term:
Each row in RAG has the form ${\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i}$, so:
${\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}({\boldsymbol{x}}_{q}+{\boldsymbol{
r}}_{i})={\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{x}}_{q}+{
\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{r}}_{i}$
Then, we plug in [Equation 34](https://arxiv.org/html/2506.03100v3#A1.E34) into the RAG term:
$\displaystyle\operatorname*{\mathbb{E}}\left[\left({\boldsymbol{x}}_{q}^{\top}
{\boldsymbol{W}}({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i})\right)^{2}\right]$
$\displaystyle=\operatorname*{\mathbb{E}}[({\boldsymbol{x}}_{q}^{\top}{
\boldsymbol{W}}{\boldsymbol{x}}_{q})^{2}]+\operatorname*{\mathbb{E}}[({
\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{r}}_{i})^{2}]+2
\operatorname*{\mathbb{E}}[{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{
\boldsymbol{x}}_{q}\cdot{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{
\boldsymbol{r}}_{i}]$
(45)
$\displaystyle=\operatorname*{\mathbb{E}}[({\boldsymbol{x}}_{q}^{\top}{
\boldsymbol{W}}{\boldsymbol{x}}_{q})^{2}]+\operatorname*{\mathbb{E}}[({
\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{r}}_{i})^{2}]$
$\displaystyle=\operatorname*{\mathbb{E}}[({\boldsymbol{x}}_{q}^{\top}{
\boldsymbol{W}}{\boldsymbol{x}}_{q})^{2}]+\delta^{2}\cdot\operatorname{tr}({
\boldsymbol{W}}^{\top}{\boldsymbol{W}})$
$\displaystyle=\left[\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})
+\operatorname{tr}({\boldsymbol{W}}^{2})+\operatorname{tr}({\boldsymbol{W}})^{
2}\right]+\delta^{2}\cdot\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol
{W}})$
$\displaystyle=\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+
\operatorname{tr}({\boldsymbol{W}}^{2})+\delta^{2}\operatorname{tr}({
\boldsymbol{W}}^{\top}{\boldsymbol{W}})+\operatorname{tr}({\boldsymbol{W}})^{2}$
where the second step follows from $\operatorname*{\mathbb{E}}[{\boldsymbol{r}}_{i}]=0$, the third step follows from the cyclic property of trace.
$\Rightarrow\quad\text{RAG contribution}=n\cdot\sigma^{2}_{\text{rag}}\cdot
\left[(1+\delta^{2})\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})
+\operatorname{tr}({\boldsymbol{W}}^{2})+\operatorname{tr}({\boldsymbol{W}})^{
2}\right]$

Thus, we can combine the two terms above and obtain the following:

$$ $\displaystyle\operatorname*{\mathbb{E}}\left[\left({\boldsymbol{x}}^{\top}_{q} {\boldsymbol{W}}{\boldsymbol{X}}^{\top}\boldsymbol{\epsilon}\right)^{2}\right]$ $\displaystyle=\left[m\sigma^{2}+\left(1+\delta^{2}\right)n\sigma_{\text{rag}}^ {2}\right]\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+n\sigma_{ \text{rag}}^{2}\operatorname{tr}({\boldsymbol{W}}^{2})+n\sigma_{\text{rag}}^{2 }\operatorname{tr}({\boldsymbol{W}})^{2}$ (46) $$

##### Compute 𝔼 𝒙 q , 𝑿 [ ( 𝒙 q ⊤ ⁢ ( I − 𝑾 ⁢ 𝑮 ) ⁢ β tt ) 2 ] subscript 𝔼 subscript 𝒙 𝑞 𝑿 superscript subscript superscript 𝒙 top 𝑞 𝐼 𝑾 𝑮 subscript 𝛽 tt 2 \operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X}}}\left[({%
\boldsymbol{x}}^{\top}_{q}\left(I-{\boldsymbol{W}}{\boldsymbol{G}}\right)\beta%
_{\mathrm{tt}})^{2}\right] blackboard_E start_POSTSUBSCRIPT bold_italic_x start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT , bold_italic_X end_POSTSUBSCRIPT [ ( bold_italic_x start_POSTSUPERSCRIPT ⊤ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT ( italic_I - bold_italic_W bold_italic_G ) italic_β start_POSTSUBSCRIPT roman_tt end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ]

First, we can expand the expectation and decompose the inner terms into 4 terms:

$$ $\displaystyle\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X}} }\left[\left(I-{\boldsymbol{W}}{\boldsymbol{G}}\right)^{\top}{\boldsymbol{x}}_ {q}{\boldsymbol{x}}_{q}^{\top}\left(I-{\boldsymbol{W}}{\boldsymbol{G}}\right)\right]$ $\displaystyle=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X} }}\left(I-{\boldsymbol{G}}{\boldsymbol{W}}^{\top}\right){\boldsymbol{x}}_{q}{ \boldsymbol{x}}_{q}^{\top}(I-{\boldsymbol{W}}{\boldsymbol{G}})$ (47) $\displaystyle=\underbrace{\operatorname*{\mathbb{E}}{\boldsymbol{x}}_{q}{ \boldsymbol{x}}_{q}^{\top}}_{:=M_{1}}-\underbrace{\operatorname*{\mathbb{E}}{ \boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}} }_{:=M_{2}}-\underbrace{\operatorname*{\mathbb{E}}{\boldsymbol{G}}{\boldsymbol {W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}}_{:=M_{3}}+ \underbrace{\operatorname*{\mathbb{E}}{\boldsymbol{G}}{\boldsymbol{W}}^{\top}{ \boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}} }_{:=M_{4}}$ $$

We denote the four pieces $M_{1},M_{2},M_{3},M_{4}$ in order.
First, we note that:

$$ $M_{1}=\operatorname*{\mathbb{E}}\left[{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q} ^{\top}\right]=I$ $$

Then, we expand out the terms in $M_{2}$:

$$ $\displaystyle\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{r}} }{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}}$ $\displaystyle=\left(\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{ \boldsymbol{r}}}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}\right){ \boldsymbol{W}}{\boldsymbol{G}}_{0}+\operatorname*{\mathbb{E}}_{{\boldsymbol{x }}_{q},{\boldsymbol{r}}}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}\sum_{i=1}^{n}({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i})({ \boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i})^{\top}$ (48) $\displaystyle={\boldsymbol{W}}{\boldsymbol{G}}_{0}+\operatorname*{\mathbb{E}}_ {{\boldsymbol{x}}_{q},{\boldsymbol{r}}}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q }^{\top}{\boldsymbol{W}}\sum_{i=1}^{n}({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{ i})({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i})^{\top}$ $\displaystyle={\boldsymbol{W}}{\boldsymbol{G}}_{0}+\operatorname*{\mathbb{E}}_ {{\boldsymbol{x}}_{q},{\boldsymbol{r}}}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q }^{\top}{\boldsymbol{W}}\sum_{i=1}^{n}({\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q }^{\top}+{\boldsymbol{r}}_{i}{\boldsymbol{r}}_{i}^{\top})$ $\displaystyle={\boldsymbol{W}}{\boldsymbol{G}}_{0}+\operatorname*{\mathbb{E}}_ {{\boldsymbol{x}}_{q},{\boldsymbol{r}}}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q }^{\top}{\boldsymbol{W}}\sum_{i=1}^{n}({\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q }^{\top}+\delta^{2}I)$ $\displaystyle={\boldsymbol{W}}{\boldsymbol{G}}_{0}+n({\boldsymbol{W}}+{ \boldsymbol{W}}^{\top}+\operatorname{tr}({\boldsymbol{W}})I)+n\delta^{2}{ \boldsymbol{W}}$ $\displaystyle={\boldsymbol{W}}{\boldsymbol{G}}_{0}+n(1+\delta^{2}){\boldsymbol {W}}+n{\boldsymbol{W}}^{\top}+n\operatorname{tr}({\boldsymbol{W}})I$ $$

where the first step follows from the independence between ${\boldsymbol{X}}$ and ${\boldsymbol{x}}_{q}$, the second step follows from $\operatorname*{\mathbb{E}}{\boldsymbol{r}}_{i}=0,\;\forall i\in[n]$, the third step follows from the expectation of ${\boldsymbol{r}}_{i}{\boldsymbol{r}}_{i}^{\top}=\delta^{2}I$, and the last step follows from [Equation 32](https://arxiv.org/html/2506.03100v3#A1.E32). Then,

$$ $\displaystyle M_{2}$ $\displaystyle={\boldsymbol{W}}{\boldsymbol{G}}_{0}+n(1+\delta^{2}){\boldsymbol {W}}+n{\boldsymbol{W}}^{\top}+n\operatorname{tr}({\boldsymbol{W}})I$ (49) $\displaystyle=(n\delta^{2}+n+m){\boldsymbol{W}}+n{\boldsymbol{W}}^{\top}+n \operatorname{tr}({\boldsymbol{W}})I$ $$

Similarly, $M_{3}=M_{2}^{\top}=(n\delta^{2}+n+m){\boldsymbol{W}}^{\top}+n{\boldsymbol{W}}+
n\operatorname{tr}({\boldsymbol{W}})I$. Now, we perform similar expansion for $M_{4}$:

$$ $\displaystyle M_{4}$ $\displaystyle=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X} }}[{\boldsymbol{G}}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}} _{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}}]$ (50) $\displaystyle=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X} }}\left[\left({\boldsymbol{G}}_{0}+\sum_{i\in[n]}{\boldsymbol{G}}_{i}\right){ \boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}\left({\boldsymbol{G}}_{0}+\sum_{i\in[n]}{\boldsymbol{G}}_{i} \right)\right]$ $\displaystyle=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X} }}\left[{\boldsymbol{G}}_{0}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}}_{0}+{\boldsymbol{G} }_{0}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}\sum_{i\in[n]}{\boldsymbol{G}}_{i}+\sum_{i\in[n]}{\boldsymbol{G }}_{i}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{G}}_{0}\right.$ $\displaystyle\quad+\left.\sum_{i\in n}{\boldsymbol{G}}_{i}{\boldsymbol{W}}^{ \top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{G}}_{i}+\sum_{i,j\in n,i\neq j}{\boldsymbol{G}}_{i}{\boldsymbol{W} }^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{G}}_{j}\right]$ $\displaystyle=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X} }}\left[{\boldsymbol{G}}_{0}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}}_{0}+n\underbrace{{ \boldsymbol{G}}_{0}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}} _{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}}_{i}}_{i\in[n]}+n\underbrace{{ \boldsymbol{G}}_{i}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}} _{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}}_{0}}_{i\in[n]}\right.$ $\displaystyle\left.\quad+n\underbrace{{\boldsymbol{G}}_{i}{\boldsymbol{W}}^{ \top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{G}}_{i}}_{i\in[n]}+n(n-1)\underbrace{{\boldsymbol{G}}_{i}{ \boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{G}}_{j}}_{i,j\in[n],i\neq j}\right]$ $$

First, we can compute that:

$$ $\displaystyle M_{41}:=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{ \boldsymbol{X}}}[{\boldsymbol{G}}_{0}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{ q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}}_{0}]$ $\displaystyle=\operatorname*{\mathbb{E}}_{{\boldsymbol{X}}}[{\boldsymbol{G}}_{ 0}{\boldsymbol{W}}^{\top}{\boldsymbol{W}}{\boldsymbol{G}}_{0}]$ (51) $\displaystyle=m(m+1){\boldsymbol{W}}^{\top}{\boldsymbol{W}}+m\operatorname{tr} ({\boldsymbol{W}}^{\top}{\boldsymbol{W}})I$ $$

where the last line follows from [Equation 33](https://arxiv.org/html/2506.03100v3#A1.E33) and symmetry of ${\boldsymbol{W}}^{\top}{\boldsymbol{W}}$. Then, $\forall i\in[n]$, we have:

$$ $\displaystyle M_{42}:=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{ \boldsymbol{X}}}{\boldsymbol{G}}_{0}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q }{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}}_{i}$ $\displaystyle=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X} }}{\boldsymbol{G}}_{0}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{ x}}_{q}^{\top}{\boldsymbol{W}}\left({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i} \right)\left({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i}\right)^{\top}$ (52) $\displaystyle=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X} }}{\boldsymbol{G}}_{0}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{ x}}_{q}^{\top}{\boldsymbol{W}}\left({\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{ \top}+{\boldsymbol{r}}_{i}{\boldsymbol{r}}_{i}^{\top}\right)$ $\displaystyle=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X} }}{\boldsymbol{G}}_{0}{\boldsymbol{W}}^{\top}\left({\boldsymbol{W}}+{ \boldsymbol{W}}^{\top}+\operatorname{tr}({\boldsymbol{W}})+{\boldsymbol{W}} \delta^{2}\right)$ $\displaystyle=m\left({\boldsymbol{W}}^{\top}{\boldsymbol{W}}+{\boldsymbol{W}}^ {\top}{\boldsymbol{W}}^{\top}+\operatorname{tr}({\boldsymbol{W}}){\boldsymbol{ W}}^{\top}+\delta^{2}{\boldsymbol{W}}^{\top}{\boldsymbol{W}}\right)$ $\displaystyle=m\left((1+\delta^{2}){\boldsymbol{W}}^{\top}{\boldsymbol{W}}+{ \boldsymbol{W}}^{\top}{\boldsymbol{W}}^{\top}+\operatorname{tr}({\boldsymbol{W }}){\boldsymbol{W}}^{\top}\right)$ $$

where the first steps follows from $\operatorname*{\mathbb{E}}[{\boldsymbol{r}}_{i}]=0$, the second step follows from [Equation 32](https://arxiv.org/html/2506.03100v3#A1.E32).

Moreover, we note that $\forall i\in[n]$:

$$ $\displaystyle M_{43}:=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{ \boldsymbol{X}},{\boldsymbol{r}}_{i}}{\boldsymbol{G}}_{i}{\boldsymbol{W}}^{ \top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{G}}_{i}$ $\displaystyle=({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i})({\boldsymbol{x}}_{q} +{\boldsymbol{r}}_{i})^{\top}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}({\boldsymbol{x}}_{q}+{\boldsymbol{r }}_{i})({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i})^{\top}$ (53) $\displaystyle=({\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}+{\boldsymbol{r} }_{i}{\boldsymbol{x}}_{q}^{\top}+{\boldsymbol{x}}_{q}{\boldsymbol{r}}_{i}^{ \top}+{\boldsymbol{r}}_{i}{\boldsymbol{r}}_{i}^{\top}){\boldsymbol{W}}^{\top}{ \boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}({\boldsymbol{x} }_{q}{\boldsymbol{x}}_{q}^{\top}+{\boldsymbol{r}}_{i}{\boldsymbol{x}}_{q}^{ \top}+{\boldsymbol{x}}_{q}{\boldsymbol{r}}_{i}^{\top}+{\boldsymbol{r}}_{i}{ \boldsymbol{r}}_{i}^{\top})$ $\displaystyle=\underbrace{{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}}_{\text{0 order in ${\boldsymbol{r}}_{i}$}}+\underbrace{{\boldsymbol{r}}_{i}{\boldsymbol{r}}_ {i}^{\top}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{ \top}{\boldsymbol{W}}{\boldsymbol{r}}_{i}{\boldsymbol{r}}_{i}^{\top}}_{\text{4 th-order in ${\boldsymbol{r}}_{i}$}}$ $\displaystyle+\underbrace{({\boldsymbol{r}}_{i}{\boldsymbol{x}}_{q}^{\top}+{ \boldsymbol{x}}_{q}{\boldsymbol{r}}_{i}^{\top}){\boldsymbol{W}}^{\top}{ \boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}({\boldsymbol{r} }_{i}{\boldsymbol{x}}_{q}^{\top}+{\boldsymbol{x}}_{q}{\boldsymbol{r}}_{i}^{ \top})}_{\text{2nd-order in ${\boldsymbol{r}}_{i}$}}$ $\displaystyle+\underbrace{{\boldsymbol{r}}_{i}{\boldsymbol{r}}_{i}^{\top}{ \boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}+{\boldsymbol{x} }_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{r}}_{i}{\boldsymbol{r}} _{i}^{\top}}_{\text{2nd-order in ${\boldsymbol{r}}_{i}$}}$ $$

It worth noting that given Gaussian vector ${\boldsymbol{r}}_{i}$, then its monomial of odd order has 0 expectation according to Isserlis’ Theorem  [Isserlis, 1918]. And we can thus obtain the third line by keeping only the even order monomials of ${\boldsymbol{r}}_{i}$.

By adding up [Lemma 3](https://arxiv.org/html/2506.03100v3#Thmlemma3) and all the terms above, we obtain that:

$$ $\displaystyle\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X}} ,{\boldsymbol{r}}_{i}}{\boldsymbol{G}}_{i}{\boldsymbol{W}}^{\top}{\boldsymbol{ x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}}_{i}$ (54) $\displaystyle=\left.\begin{aligned} &2\left({\boldsymbol{W}}^{2}+({\boldsymbol {W}}^{2})^{\top}+{\boldsymbol{W}}^{\top}{\boldsymbol{W}}+{\boldsymbol{W}}{ \boldsymbol{W}}^{\top}+\operatorname{tr}\left({\boldsymbol{W}}\right)({ \boldsymbol{W}}+{\boldsymbol{W}}^{\top})\right)\\ &+\operatorname{tr}\left({\boldsymbol{W}}\right)^{2}I+\operatorname{tr}\left({ \boldsymbol{W}}^{2}\right)I+\operatorname{tr}\left({\boldsymbol{W}}^{\top}{ \boldsymbol{W}}\right)I\\ \end{aligned}\right\}\text{0th-order in ${\boldsymbol{r}}_{i}$, \lx@cref{ creftype~refnum}{lem:6order_gaussian}}$ $\displaystyle\quad+\underbrace{2\delta^{4}{\boldsymbol{W}}^{\top}{\boldsymbol{ W}}+\delta^{4}\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})I}_{ \text{4th-order in ${\boldsymbol{r}}_{i}$, \lx@cref{creftype~refnum}{eq:M4_4or der_term}}}$ $\displaystyle\quad+\underbrace{\delta^{2}\left[\operatorname{tr}\left({ \boldsymbol{W}}\right)\left({\boldsymbol{W}}^{\top}+{\boldsymbol{W}}\right)+{ \boldsymbol{W}}^{2}+({\boldsymbol{W}}^{2})^{\top}+2{\boldsymbol{W}}^{\top}{ \boldsymbol{W}}\right]}_{\text{\lx@cref{creftype~refnum}{eq:M4_2order_term4} and its transpose}}$ $\displaystyle\quad+\underbrace{\left(\operatorname{tr}\left({\boldsymbol{W}}^{ 2}\right)+\operatorname{tr}\left({\boldsymbol{W}}^{\top}{\boldsymbol{W}}\right )+\operatorname{tr}\left({\boldsymbol{W}}\right)^{2}\right)\delta^{2}I}_{\text {\lx@cref{creftype~refnum}{eq:M4_2order_term1}}}$ $\displaystyle\quad+\underbrace{2\delta^{2}{\boldsymbol{W}}{\boldsymbol{W}}^{ \top}+\delta^{2}\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})I}_{ \text{\lx@cref{creftype~refnum}{eq:M4_2order_term2}}}$ $\displaystyle\quad+\underbrace{\delta^{2}\left[\operatorname{tr}\left({ \boldsymbol{W}}\right)\left({\boldsymbol{W}}^{\top}+{\boldsymbol{W}}\right)+{ \boldsymbol{W}}^{2}+({\boldsymbol{W}}^{2})^{\top}+2{\boldsymbol{W}}^{\top}{ \boldsymbol{W}}\right]}_{\text{\lx@cref{creftype~refnum}{eq:M4_2order_term3} and its transpose}}$ $\displaystyle=(2+2\delta^{2})\left[\operatorname{tr}\left({\boldsymbol{W}} \right)\left({\boldsymbol{W}}^{\top}+{\boldsymbol{W}}\right)+{\boldsymbol{W}}^ {2}+({\boldsymbol{W}}^{2})^{\top}\right]$ $\displaystyle\quad+(2+4\delta^{2}){\boldsymbol{W}}^{\top}{\boldsymbol{W}}+2{ \boldsymbol{W}}{\boldsymbol{W}}^{\top}$ $\displaystyle\quad+(1+\delta^{2})\left[\operatorname{tr}\left({\boldsymbol{W}} \right)^{2}I+\operatorname{tr}\left({\boldsymbol{W}}^{2}\right)I+\operatorname {tr}\left({\boldsymbol{W}}^{\top}{\boldsymbol{W}}\right)I\right]$ $\displaystyle\quad+2\delta^{4}{\boldsymbol{W}}^{\top}{\boldsymbol{W}}+\delta^{ 4}\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})I+2\delta^{2}{ \boldsymbol{W}}{\boldsymbol{W}}^{\top}+\delta^{2}\operatorname{tr}({ \boldsymbol{W}}^{\top}{\boldsymbol{W}})I$ $\displaystyle=(2+2\delta^{2})\left[\operatorname{tr}\left({\boldsymbol{W}} \right)\left({\boldsymbol{W}}^{\top}+{\boldsymbol{W}}\right)+{\boldsymbol{W}}^ {2}+({\boldsymbol{W}}^{2})^{\top}\right]$ $\displaystyle\quad+(2+4\delta^{2}+2\delta^{4}){\boldsymbol{W}}^{\top}{ \boldsymbol{W}}+(2+2\delta^{2}){\boldsymbol{W}}{\boldsymbol{W}}^{\top}$ $\displaystyle\quad+(1+\delta^{2})\left(\operatorname{tr}\left({\boldsymbol{W}} \right)^{2}+\operatorname{tr}\left({\boldsymbol{W}}^{2}\right)\right)I+(1+2 \delta^{2}+\delta^{4})\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W} })I$ $$

Also, we expand the cross-term out for $\forall i,j\in[n],i\neq j$:

$$ $\displaystyle M_{44}:=\operatorname*{\mathbb{E}}{\boldsymbol{G}}_{i}{ \boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{G}}_{j}$ $\displaystyle=\operatorname*{\mathbb{E}}({\boldsymbol{x}}_{q}+{\boldsymbol{r}} _{i})({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i})^{\top}{\boldsymbol{W}}^{\top} {\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}({\boldsymbol{x }}_{q}+{\boldsymbol{r}}_{j})({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{j})^{\top}$ (55) $\displaystyle=\left({\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}+{ \boldsymbol{r}}_{i}{\boldsymbol{r}}_{i}^{\top}\right){\boldsymbol{W}}^{\top}{ \boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}\left({ \boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}+{\boldsymbol{r}}_{j}{ \boldsymbol{r}}_{j}^{\top}\right)$ $\displaystyle={\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}^ {\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}+{\boldsymbol{r}}_{i}{ \boldsymbol{r}}_{i}^{\top}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{x}}_{q}{\boldsymbol{x}} _{q}^{\top}$ $\displaystyle\quad+{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol {W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{r}}_{j}{\boldsymbol{r}}_{j}^{\top}+{\boldsymbol{r}}_{i}{ \boldsymbol{r}}_{i}^{\top}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{r}}_{j}{\boldsymbol{r}} _{j}^{\top}$ $\displaystyle=2\left({\boldsymbol{W}}^{2}+\left({\boldsymbol{W}}^{2}\right)^{ \top}+{\boldsymbol{W}}^{\top}{\boldsymbol{W}}+{\boldsymbol{W}}{\boldsymbol{W}} ^{\top}+\operatorname{tr}\left({\boldsymbol{W}}\right){\boldsymbol{W}}+ \operatorname{tr}\left({\boldsymbol{W}}\right){\boldsymbol{W}}^{\top}\right)$ $\displaystyle\quad+\operatorname{tr}\left({\boldsymbol{W}}\right)^{2}I+ \operatorname{tr}\left({\boldsymbol{W}}^{2}\right)I+\operatorname{tr}\left({ \boldsymbol{W}}^{\top}{\boldsymbol{W}}\right)I$ $\displaystyle\quad+\delta^{2}\left({\boldsymbol{W}}^{2}+({\boldsymbol{W}}^{2}) ^{\top}+2{\boldsymbol{W}}^{\top}{\boldsymbol{W}}\right)+\operatorname{tr}({ \boldsymbol{W}})({\boldsymbol{W}}+{\boldsymbol{W}}^{\top})+\delta^{4}{ \boldsymbol{W}}^{\top}{\boldsymbol{W}}$ $\displaystyle=(2+\delta^{2})\left({\boldsymbol{W}}^{2}+\left({\boldsymbol{W}}^ {2}\right)^{\top}+\operatorname{tr}\left({\boldsymbol{W}}\right){\boldsymbol{W }}+\operatorname{tr}\left({\boldsymbol{W}}\right){\boldsymbol{W}}^{\top}\right)$ $\displaystyle\quad+(2+2\delta^{2}){\boldsymbol{W}}^{\top}{\boldsymbol{W}}+2{ \boldsymbol{W}}{\boldsymbol{W}}^{\top}$ $\displaystyle\quad+\operatorname{tr}\left({\boldsymbol{W}}\right)^{2}I+ \operatorname{tr}\left({\boldsymbol{W}}^{2}\right)I+\operatorname{tr}\left({ \boldsymbol{W}}^{\top}{\boldsymbol{W}}\right)I+\delta^{4}{\boldsymbol{W}}^{ \top}{\boldsymbol{W}}$ $\displaystyle=(2+\delta^{2})\left({\boldsymbol{W}}^{2}+\left({\boldsymbol{W}}^ {2}\right)^{\top}+\operatorname{tr}\left({\boldsymbol{W}}\right){\boldsymbol{W }}+\operatorname{tr}\left({\boldsymbol{W}}\right){\boldsymbol{W}}^{\top}\right)$ $\displaystyle\quad+(2+2\delta^{2}+\delta^{4}){\boldsymbol{W}}^{\top}{ \boldsymbol{W}}+2{\boldsymbol{W}}{\boldsymbol{W}}^{\top}$ $\displaystyle\quad+\operatorname{tr}\left({\boldsymbol{W}}\right)^{2}I+ \operatorname{tr}\left({\boldsymbol{W}}^{2}\right)I+\operatorname{tr}\left({ \boldsymbol{W}}^{\top}{\boldsymbol{W}}\right)I$ $$

where the first step follows from the independence of ${\boldsymbol{x}}_{q},{\boldsymbol{r}}_{i},{\boldsymbol{r}}_{j}$, and the second step follows from applying [Lemma 3](https://arxiv.org/html/2506.03100v3#Thmlemma3) and [Equation 32](https://arxiv.org/html/2506.03100v3#A1.E32).

Combining the above terms together, we have

$$ $\displaystyle M_{4}$ $\displaystyle=M_{41}+n(M_{42}+M_{42}^{\top})+nM_{43}+n(n-1)M_{44}$ (56) $\displaystyle=m(m+1){\boldsymbol{W}}^{\top}{\boldsymbol{W}}+m\operatorname{tr} ({\boldsymbol{W}}^{\top}{\boldsymbol{W}})I+mn\left(\left(2+2\delta^{2}\right){ \boldsymbol{W}}^{\top}{\boldsymbol{W}}+{\boldsymbol{W}}^{2}+({\boldsymbol{W}}^ {2})^{\top}+\operatorname{tr}\left({\boldsymbol{W}}\right)\left({\boldsymbol{W }}+{\boldsymbol{W}}^{\top}\right)\right)$ $\displaystyle\quad+nM_{43}+n(n-1)M_{44}$ $\displaystyle=2n\left(2n+\delta^{2}\right){\boldsymbol{W}}^{2}+2n\left(n+ \delta^{2}\right){\boldsymbol{W}}{\boldsymbol{W}}^{\top}$ $\displaystyle\quad+\left[m^{2}+m+(4+2\delta^{2})mn+n^{2}\left(2+4\delta^{2}+ \delta^{4}\right)+n\left(2\delta^{2}+\delta^{4}\right)\right]{\boldsymbol{W}}^ {\top}{\boldsymbol{W}}$ $\displaystyle\quad+\left[n^{2}\left(2+\delta^{2}\right)+n\left(m+\delta^{2} \right)\right]\operatorname{tr}({\boldsymbol{W}})\left({\boldsymbol{W}}+{ \boldsymbol{W}}^{\top}\right)$ $\displaystyle\quad+\left(n^{2}+n\delta^{2}\right)\left(\operatorname{tr}({ \boldsymbol{W}})^{2}+\operatorname{tr}({\boldsymbol{W}}^{2})\right)I+\left[m+n ^{2}+n\left(2\delta^{2}+\delta^{4}\right)\right]\operatorname{tr}\left({ \boldsymbol{W}}^{\top}{\boldsymbol{W}}\right)I$ $\displaystyle=\left[n^{2}\left(2+\delta^{2}\right)+n\left(m+\delta^{2}\right) \right]\left({\boldsymbol{W}}^{2}+\left({\boldsymbol{W}}^{2}\right)^{\top}+ \operatorname{tr}({\boldsymbol{W}})\left({\boldsymbol{W}}+{\boldsymbol{W}}^{ \top}\right)\right)$ $\displaystyle\quad+\left[2n^{2}+2n\delta^{2}\right]{\boldsymbol{W}}{ \boldsymbol{W}}^{\top}$ $\displaystyle\quad+{\left[m^{2}+m+mn\left(2+2\delta^{2}\right)+n\left(2\delta^ {2}+\delta^{4}\right)+n^{2}\left(2+2\delta^{2}+\delta^{4}\right)\right]{ \boldsymbol{W}}^{\top}{\boldsymbol{W}}}$ $\displaystyle\quad+\left[n^{2}+n\delta^{2}\right]\left(\operatorname{tr}({ \boldsymbol{W}})^{2}+\operatorname{tr}\left({\boldsymbol{W}}^{2}\right)\right)I$ $\displaystyle\quad+\left[m+n^{2}+n\left(2\delta^{2}+\delta^{4}\right)\right] \operatorname{tr}\left({\boldsymbol{W}}^{\top}{\boldsymbol{W}}\right)I$ $$

In summary, combining all terms together, we have:

$$ ${\cal L}({\boldsymbol{W}}):={\operatorname{err}}_{\text{variance}}+{ \operatorname{err}}_{\text{bias}}+\sigma^{2}$ $$

where the irreducible variance is $\sigma^{2}$, and the reducible variance (variance of ICL + RAG) is

$$ $\text{Variance of ICL}+\text{Variance of RAG}=\left[m\sigma^{2}+\left(1+\delta ^{2}\right)n\sigma_{\text{rag}}^{2}\right]\operatorname{tr}({\boldsymbol{W}}^{ \top}{\boldsymbol{W}})+n\sigma_{\text{rag}}^{2}\operatorname{tr}({\boldsymbol{ W}}^{2})+n\sigma_{\text{rag}}^{2}\operatorname{tr}({\boldsymbol{W}})^{2}$ $$

And the err from the bias term is given as:

$$ $\displaystyle{\operatorname{err}}_{\text{bias}}$ $\displaystyle=\beta_{\mathrm{tt}}^{\top}[M_{1}-M_{2}-M_{3}+M_{4}]\beta_{ \mathrm{tt}}$ $\displaystyle=\beta_{\mathrm{tt}}^{\top}\left[I-(n\delta^{2}+2n+m)({ \boldsymbol{W}}+{\boldsymbol{W}}^{\top})-2n\operatorname{tr}({\boldsymbol{W}}) I+M_{4}\right]\beta_{\mathrm{tt}}$ $\displaystyle=\beta_{\mathrm{tt}}^{\top}\left[I-(n\delta^{2}+2n+m)({ \boldsymbol{W}}+{\boldsymbol{W}}^{\top})-2n\operatorname{tr}({\boldsymbol{W}}) I\right.$ $\displaystyle\quad+\left[n^{2}\left(2+\delta^{2}\right)+n\left(m+\delta^{2} \right)\right]\left({\boldsymbol{W}}^{2}+\left({\boldsymbol{W}}^{2}\right)^{ \top}\right)+2n(n+\delta^{2}){\boldsymbol{W}}{\boldsymbol{W}}^{\top}$ $\displaystyle\quad+{\left[m^{2}+m+mn\left(2+2\delta^{2}\right)+n^{2}\left(2+2 \delta^{2}+\delta^{4}\right)+n\left(2\delta^{2}+\delta^{4}\right)\right]{ \boldsymbol{W}}^{\top}{\boldsymbol{W}}}$ $\displaystyle\quad+\left[n^{2}\left(2+\delta^{2}\right)+n\left(m+\delta^{2} \right)\right]\left(\operatorname{tr}({\boldsymbol{W}})\left({\boldsymbol{W}}+ {\boldsymbol{W}}^{\top}\right)\right)$ $\displaystyle\left.\quad+\left[n^{2}+n\delta^{2}\right]\left(\operatorname{tr} ({\boldsymbol{W}})^{2}+\operatorname{tr}\left({\boldsymbol{W}}^{2}\right) \right)I+\left[m+n^{2}+n\left(2\delta^{2}+\delta^{4}\right)\right] \operatorname{tr}\left({\boldsymbol{W}}^{\top}{\boldsymbol{W}}\right)I\right] \beta_{\mathrm{tt}}$ $$

∎
The previous theorem gives the exact form the RAG population with general ${\boldsymbol{W}}$. In the following proposition, we will compute the population under special ${\boldsymbol{W}}$ in order to obtain a more fine-grained complexity analysis.

###### Proposition 3 (RAG Population loss under isotropic setting) .

Assuming ${\boldsymbol{W}}^{*}=\frac{m}{(m+d+1)(m+n)}I$. Then, the population loss are given as:

$$ $\displaystyle{\cal L}_{\text{tt+rag}}({\boldsymbol{W}}^{*})$ $\displaystyle={\operatorname{err}}_{\text{variance}}({\boldsymbol{W}}^{*})+{ \operatorname{err}}_{\text{bias}}({\boldsymbol{W}}^{*})+\sigma^{2}$ $\displaystyle{\operatorname{err}}_{\text{variance}}({\boldsymbol{W}}^{*})$ $\displaystyle=\frac{m^{3}d}{[(m+d+1)(m+n)]^{2}}\sigma^{2}+\frac{dm^{2}n(2+ \delta^{2}+d)}{[(m+d+1)(m+n)]^{2}}\sigma_{\mathrm{rag}}^{2}$ $\displaystyle{\operatorname{err}}_{\text{bias}}({\boldsymbol{W}}^{*})$ $\displaystyle=\left\|\beta_{TT}\right\|_{2}^{2}\left[1-\frac{2m}{(m+d+1)(m+n)} \left(n\delta^{2}+2n+m+nd\right)+\frac{P(m,n,d,\delta)m^{2}}{(m+d+1)^{2}(m+n)^ {2}}\right]$ $$

where

$$ $\displaystyle P(m,n,d,\delta)=$ $\displaystyle 6n^{2}+4n\delta^{2}+m^{2}+m+\left(4+2\delta^{2}\right)mn$ $\displaystyle\quad+n^{2}\left(2+4\delta^{2}+\delta^{4}\right)+n\left(2\delta^{ 2}+\delta^{4}\right)+2dn^{2}\left(2+\delta^{2}\right)+2dn\left(m+\delta^{2}\right)$ $\displaystyle\quad+d(d+1)\left(n^{2}+n\delta^{2}\right)+dm+dn^{2}+dn\left(2 \delta^{2}+\delta^{4}\right)$ $$

###### Proof.

Plugging in the value of ${\boldsymbol{W}}^{*}$, we first compute the error from input variance.

$$ $\displaystyle\operatorname{tr}(({\boldsymbol{W}}^{*})^{2})$ $\displaystyle=\frac{dm^{2}}{(m+d+1)^{2}(m+n)^{2}}$ $\displaystyle\operatorname{tr}({\boldsymbol{W}}^{*})$ $\displaystyle=\frac{dm}{(m+d+1)(m+n)}$ $$

$$ $\displaystyle{\operatorname{err}}_{\text{variance}}({\boldsymbol{W}}^{*})$ $\displaystyle=\left[m\sigma^{2}+\left(1+\delta^{2}\right)n\sigma_{rag}^{2} \right]\operatorname{tr}\left({\boldsymbol{W}}^{\top}{\boldsymbol{W}}\right)+n \sigma_{rag}^{2}\operatorname{tr}\left({\boldsymbol{W}}^{2}\right)+n\sigma_{ rag}^{2}\operatorname{tr}({\boldsymbol{W}})^{2}$ $\displaystyle=\frac{dm^{2}[m\sigma^{2}+(1+\delta^{2})n\sigma_{\text{rag}}^{2}] }{(m+d+1)^{2}(m+n)^{2}}+n\sigma_{\text{rag}}^{2}\frac{dm^{2}}{(m+d+1)^{2}(m+n) ^{2}}+n\sigma_{\text{rag}}^{2}\frac{d^{2}m^{2}}{(m+d+1)^{2}(m+n)^{2}}$ $\displaystyle=\frac{m^{3}d}{[(m+d+1)(m+n)]^{2}}\sigma^{2}+\frac{dm^{2}n(2+ \delta^{2}+d)}{[(m+d+1)(m+n)]^{2}}\sigma_{\mathrm{rag}}^{2}$ $$

Then, we proceed to plug in the value and compute the error from the estimation bias.

$$ $\displaystyle{\operatorname{err}}_{\text{bias}}({\boldsymbol{W}}^{*})$ $\displaystyle=\|\beta_{\mathrm{tt}}\|_{2}^{2}\left[1-\frac{2m(n\delta^{2}+2n+m )}{(m+n)(m+d+1)}-\frac{2ndm}{(m+n)(m+d+1)}+\frac{m^{2}}{(m+d+1)^{2}(m+n)^{2}} \underbrace{(\dots)}_{P(m,n,d,\delta)}\right]$ $\displaystyle=\left\|\beta_{TT}\right\|_{2}^{2}\left[1-\frac{2m}{(m+d+1)(m+n)} \left(n\delta^{2}+2n+m+nd\right)+\frac{P(m,n,d,\delta)m^{2}}{(m+d+1)^{2}(m+n)^ {2}}\right]$ $$

where

$$ $\displaystyle P(m,n,d,\delta)$ $\displaystyle=(2c_{1}+c_{2}+c_{3})+2dc_{4}+(d^{2}+d)c_{5}+dc_{6}$ $\displaystyle=2\left(n^{2}\left(2+\delta^{2}\right)+n\left(m+\delta^{2}\right) \right)+2n(n+\delta^{2})$ $\displaystyle\quad+\left[m^{2}+m+mn\left(2+2\delta^{2}\right)+n^{2}\left(2+2 \delta^{2}+\delta^{4}\right)+n\left(2\delta^{2}+\delta^{4}\right)\right]$ $\displaystyle\quad+2d[n^{2}(2+\delta^{2})+n(m+\delta^{2})]+(d^{2}+d)(n^{2}+n \delta^{2})+d[m+n^{2}+n(2\delta^{2}+\delta^{4})]$ $\displaystyle=6n^{2}+4n\delta^{2}+m^{2}+m+\left(4+2\delta^{2}\right)mn$ $\displaystyle\quad+n^{2}\left(2+4\delta^{2}+\delta^{4}\right)+n\left(2\delta^{ 2}+\delta^{4}\right)+2dn^{2}\left(2+\delta^{2}\right)+2dn\left(m+\delta^{2}\right)$ $\displaystyle\quad+d(d+1)\left(n^{2}+n\delta^{2}\right)+dm+dn^{2}+dn\left(2 \delta^{2}+\delta^{4}\right)$ $$

∎

###### Proposition 3 (RAG Population loss under isotropic setting) .

###### Proof.

#### B.1.1 Finite Sample Complexity of RAG

###### Proposition (Restatement of Proposition 1 ) .

Under Assumption [1](https://arxiv.org/html/2506.03100v3#Thmassumption1), [2](https://arxiv.org/html/2506.03100v3#Thmassumption2), [3](https://arxiv.org/html/2506.03100v3#Thmassumption3), if $\delta^{2}\ll 1$,

$$ ${\cal L}_{\text{tt+rag}}({\boldsymbol{W}}^{*})={\cal O}\left(\sigma^{2}+ \underbrace{\frac{dm}{(m+n)^{2}}\sigma^{2}+\frac{d^{2}n}{(m+n)^{2}}\sigma_{ \text{rag}}^{2}}_{{\operatorname{err}}_{\text{variance}}({\boldsymbol{W}}^{*}) }+\underbrace{\|\beta_{\mathrm{tt}}\|_{2}^{2}\left[\frac{d}{m}+d^{2}\left( \frac{n}{m+n}\right)^{2}\right]}_{{\operatorname{err}}_{\text{bias}}({ \boldsymbol{W}}^{*})}\right)$ $$

$$ $\displaystyle{\operatorname{err}}_{\text{variance}}({\boldsymbol{W}}^{*})$ $\displaystyle=\begin{cases}{\cal O}(\frac{d}{m}\sigma^{2}+\frac{d^{2}}{m^{2}} \sigma_{\text{rag}}^{2})={\cal O}\left(\frac{1}{m}\right)&\text{ $m\rightarrow \infty$, $n$ fixed.}\\ {\cal O}(\frac{d}{n^{2}}\sigma^{2}+\frac{d^{2}}{n}\sigma_{\text{rag}}^{2})={ \cal O}\left(\frac{1}{n}\right)&\text{ $n\rightarrow\infty$, $m$ fixed}\\ {\cal O}(\frac{d}{m}\sigma^{2}+\frac{d^{2}}{m}\sigma_{\text{rag}}^{2})={\cal O }\left(\frac{1}{m}\right)&\text{ $m,n\rightarrow\infty$, $n=\Theta(m)$}\end{cases}$ (57) $$

$$ $\displaystyle{\operatorname{err}}_{\text{bias}}({\boldsymbol{W}}^{*})$ $\displaystyle=\begin{cases}{\cal O}\left(\|\beta_{\mathrm{tt}}\|_{2}^{2}\frac{ d}{m}\right)&\text{if $m\rightarrow\infty$, $n$ is fixed}\\ {\cal O}\left(\|\beta_{\mathrm{tt}}\|_{2}^{2}d^{2}\right)=C_{1}&\text{if $n \rightarrow\infty$, $m$ is fixed}\\ {\cal O}\left(\|\beta_{\mathrm{tt}}\|_{2}^{2}\left(\frac{d}{m}+d^{2}\right) \right)=C_{2}+{\cal O}(\|\beta_{\mathrm{tt}}\|_{2}^{2}\frac{d}{m})&\text{if $m \rightarrow\infty$, $n=\Theta(m)$}\\ \end{cases}$ (58) $$

###### Proof.

We will bound the variance-induced error and the bias-induced error separately.

###### Proposition (Restatement of Proposition 1 ) .

###### Proof.

##### Variance-Induced Error

First, we try to bound ${\operatorname{err}}_{\text{variance}}({\boldsymbol{W}}^{*})$:

$$ $\displaystyle{\operatorname{err}}_{\text{variance}}({\boldsymbol{W}}^{*})$ $\displaystyle=\frac{dm^{3}}{(m+d+1)^{2}(m+n)^{2}}\sigma^{2}+\frac{dm^{2}n(2+ \delta^{2}+d)}{(m+d+1)^{2}(m+n)^{2}}\sigma_{\text{rag}}^{2}$ (59) $\displaystyle\leq\frac{dm^{3}}{m^{2}(m+n)^{2}}\sigma^{2}+\frac{dm^{2}n(d+ \delta^{2}+2)}{m^{2}(m+n)^{2}}\sigma^{2}_{\text{rag}}$ $\displaystyle=\frac{dm}{(m+n)^{2}}\sigma^{2}+\frac{d(2+\delta^{2}+d)n}{(m+n)^{ 2}}\sigma_{\text{rag}}^{2}$ $\displaystyle={\cal O}\left(\frac{dm}{(m+n)^{2}}\sigma^{2}+\frac{d^{2}n}{(m+n) ^{2}}\sigma_{\text{rag}}^{2}\right)$ $\displaystyle=\begin{cases}{\cal O}(\frac{d}{m}\sigma^{2}+\frac{d^{2}}{m^{2}} \sigma_{\text{rag}}^{2})={\cal O}\left(\frac{1}{m}\right)&\text{ $m\rightarrow \infty$, $n$ fixed.}\\ {\cal O}(\frac{d}{n^{2}}\sigma^{2}+\frac{d^{2}}{n}\sigma_{\text{rag}}^{2})={ \cal O}\left(\frac{1}{n}\right)&\text{ $n\rightarrow\infty$, $m$ fixed}\\ {\cal O}(\frac{d}{m}\sigma^{2}+\frac{d^{2}}{m}\sigma_{\text{rag}}^{2})={\cal O }\left(\frac{1}{m}\right)&\text{ $m,n\rightarrow\infty$, $n=\Theta(m)$}\end{cases}$ $$

where the second line follows from $(m+d+1)\geq m$ and the fourth line follows from the fact that $\delta^{2}$ is small relative to $m,n,d$.

##### Bias-Induced Error

We will expand out the term

$$ $\displaystyle{\operatorname{err}}_{\text{bias}}({\boldsymbol{W}}^{*})$ $\displaystyle=\|\beta_{\mathrm{tt}}\|_{2}^{2}\frac{Q(m,n;d,\delta^{2})}{(m+d+1 )^{2}(m+n)^{2}}$ (60) $$

where

$$ $\displaystyle Q(m,n;d,\delta^{2})$ $\displaystyle:=(m+n)^{2}(m+d+1)^{2}-2m(m+n)(m+d+1)(n\delta^{2}+2n+m+nd)+m^{2}P (m,n,d,\delta^{2})$ (61) $\displaystyle=(d+1)m^{3}+\underbrace{(d^{2}+2d\delta^{2}+4d+\delta^{4}+2\delta ^{2}+5)}_{:=\kappa_{22}}m^{2}n^{2}$ $\displaystyle\quad+\underbrace{(d^{2}\delta^{2}-2d^{2}+d\delta^{4}+3d\delta^{2 }-4d+\delta^{4}+4\delta^{2}-2)}_{:=\kappa_{21}}m^{2}n$ $\displaystyle\quad\underbrace{-\left(2d^{2}+2d\delta^{2}+4d+2\delta^{2}+2 \right)}_{:=\kappa_{12}}mn^{2}+(d^{2}+2d+1)(m+n)^{2}$ $\displaystyle=(d+1)m^{3}+\kappa_{22}m^{2}n^{2}+|\kappa_{21}|m^{2}n+\text{lower -order terms}$ $\displaystyle\leq(d+1)m^{3}+\kappa_{22}m^{2}n^{2}+|\kappa_{21}|m^{2}n+(d+1)^{2 }(m+n)^{2}$ $$

where the last line follows from $\kappa_{12}<0$.

Note that we assume $\delta^{2}\ll 1$. Now, we can bound each of the term in $Q$ divided individually:

- •
Cubic term:
$\frac{(d+1)m^{3}}{m^{2}(m+n)^{2}}=\frac{d+1}{m}\left(\frac{m}{m+n}\right)^{2}
\leq\frac{d+1}{m}$
(62)
- •
Skew-cubic term:
$\frac{\left|\kappa_{21}\right|m^{2}n}{m^{2}(m+n)^{2}}=\left|\kappa_{21}\right|
\frac{n}{(m+n)^{2}}\leq\left|\kappa_{21}\right|\frac{n}{(m+n)^{2}}$
(63)
- •
Quartic term:
$\frac{\kappa_{22}m^{2}n^{2}}{m^{2}(m+n)^{2}}=\kappa_{22}\left(\frac{n}{m+n}
\right)^{2}$
(64)
- •
last term:
$(d+1)^{2}(m+n)^{2}\frac{1}{m^{2}(m+n)^{2}}=\frac{d^{2}}{m^{2}}$

Combining [Equation 61](https://arxiv.org/html/2506.03100v3#A2.E61), [Equation 62](https://arxiv.org/html/2506.03100v3#A2.E62), [Equation 63](https://arxiv.org/html/2506.03100v3#A2.E63), [Equation 64](https://arxiv.org/html/2506.03100v3#A2.E64), we can obtain that

$$ $\displaystyle{\operatorname{err}}_{\text{bias}}({\boldsymbol{W}}^{*})$ $\displaystyle={\cal O}\left(\|\beta_{\mathrm{tt}}\|_{2}^{2}\left[\frac{dm}{(m+ n)^{2}}+d^{2}\frac{n^{2}}{(m+n)^{2}}+\frac{d^{2}}{m^{2}}\right]\right)$ (65) $\displaystyle=\begin{cases}{\cal O}\left(\|\beta_{\mathrm{tt}}\|_{2}^{2}\frac{ d}{m}\right)&\text{if $m\rightarrow\infty$, $n$ is fixed}\\ {\cal O}\left(\|\beta_{\mathrm{tt}}\|_{2}^{2}d^{2}\right)=C_{1}&\text{if $n \rightarrow\infty$, $m$ is fixed}\\ {\cal O}\left(\|\beta_{\mathrm{tt}}\|_{2}^{2}\left(\frac{d}{m}+d^{2}\right) \right)=C_{2}+{\cal O}(\|\beta_{\mathrm{tt}}\|_{2}^{2}\frac{d}{m})&\text{if $m \rightarrow\infty$, $n=\Theta(m)$}\\ \end{cases}$ $$

where the third step follows from plugging in the highest order monomial of $d$ from $\kappa_{21},\kappa_{22}$.

∎

#### B.1.2 Optimality of Number of RAG Examples

###### Proposition (Restatement of Proposition 2 ) .

Under Under Assumption [1](https://arxiv.org/html/2506.03100v3#Thmassumption1),[2](https://arxiv.org/html/2506.03100v3#Thmassumption2),[3](https://arxiv.org/html/2506.03100v3#Thmassumption3), $\delta^{2}\ll 1$, and reasonable choice of $\sigma^{2},\sigma_{\text{rag}}^{2}$ ($\sigma^{2},\sigma_{\text{rag}}^{2}\ll\|\beta_{\mathrm{tt}}\|_{2}^{2}$), the optimal $n^{*}$ that minimizes the RAG loss follows:

$$ $n^{*}={\cal O}\left(\frac{m\left(d^{2}\|\beta_{\mathrm{tt}}\|_{2}^{2}+d\sigma^ {2}-d^{2}\sigma_{\text{rag}}^{2}\right)}{md^{2}\|\beta_{\mathrm{tt}}\|_{2}^{2} -d^{2}\sigma_{\text{rag}}^{2}}\right)={\cal O}\left(\frac{d\|\beta_{\mathrm{tt }}\|_{2}^{2}+\sigma^{2}-d\sigma_{\mathrm{rag}}^{2}}{d\|\beta_{\mathrm{tt}}\|_{ 2}^{2}}\right)$ (66) $$

and the improvement on loss from picking the optimal $n^{*}$ over $n=0$ is given as:

$$ ${\cal L}_{\text{tt+rag}}({\boldsymbol{W}}^{*})|_{n=0}-{\cal L}_{\text{tt+rag}} ({\boldsymbol{W}}^{*})|_{n=n^{*}}={\cal O}\left(\frac{1}{m^{2}}\right)$ (67) $$

###### Proof.

First, we define several constants that can lead to a cleaner calculation. Let $\omega_{1}:=d$, $\omega_{2}:=d^{2}$. Then,

$$ $\displaystyle{\operatorname{err}}_{\text{variance}}({\boldsymbol{W}}^{*})$ $\displaystyle=\frac{dm^{3}}{(m+d+1)^{2}(m+n)^{2}}\sigma^{2}+\frac{dm^{2}n(2+ \delta^{2}+d)}{(m+d+1)^{2}(m+n)^{2}}\sigma_{\text{rag}}^{2}$ $\displaystyle\approx\frac{m^{3}}{(m+d+1)^{2}(m+n)^{2}}\omega_{1}\sigma^{2}+ \frac{m^{2}n}{(m+d+1)^{2}(m+n)^{2}}\omega_{2}\sigma_{\text{rag}}^{2}$ $$

where the last line follows from $\delta^{2}\ll 1$. Let $Q(m,n,d,\delta^{2}):=\frac{{\operatorname{err}}_{\text{bias}}({\boldsymbol{W}}
^{*})(m+d+1)^{2}(m+n)^{2}}{\|\beta_{\mathrm{tt}}\|_{2}^{2}}$ as in [Equation 61](https://arxiv.org/html/2506.03100v3#A2.E61). Then,

$$ $\displaystyle Q(m,n;d,\delta^{2})$ $\displaystyle=(m+n)^{2}(m+d+1)^{2}-2m(m+n)(m+d+1)(n\delta^{2}+2n+m+nd)+m^{2}P( m,n,d,\delta^{2})$ (68) $\displaystyle=(d+1)m^{3}+(d^{2}+2d\delta^{2}+4d+\delta^{4}+2\delta^{2}+5)m^{2} n^{2}$ $\displaystyle\quad+(d^{2}\delta^{2}-2d^{2}+d\delta^{4}+3d\delta^{2}-4d+\delta^ {4}+4\delta^{2}-2)m^{2}n$ $\displaystyle\quad-\left(2d^{2}+2d\delta^{2}+4d+2\delta^{2}+2\right)mn^{2}+(d^ {2}+2d+1)(m^{2}+n^{2})$ $\displaystyle\approx\underbrace{d}_{:=\tau_{30}}m^{3}+\underbrace{d^{2}}_{\tau _{22}}m^{2}n^{2}\underbrace{-2d^{2}}_{:=\tau_{21}}m^{2}n\underbrace{-2d^{2}}_{ :=\tau_{12}}mn^{2}+\underbrace{d^{2}}_{:=\tau_{2}}(m^{2}+n^{2})$ $\displaystyle=\tau_{30}m^{3}+\tau_{22}m^{2}n^{2}+\tau_{21}m^{2}n+\tau_{12}mn^{ 2}+\tau_{2}(m^{2}+n^{2})$ $$

Now, we want to find the optimal $n^{*}$ w.r.t. ${\cal L}_{\text{tt+rag}}$. That is, we want to minimize

$$ $\left[m^{3}\omega_{1}\sigma^{2}+m^{2}n\omega_{2}\sigma_{\text{rag}}^{2}+\| \beta_{\mathrm{tt}}\|_{2}^{2}\left(\tau_{30}m^{3}+\tau_{22}m^{2}n^{2}+\tau_{21 }m^{2}n+\tau_{12}mn^{2}+\tau_{2}\left(m^{2}+n^{2}\right)\right)\right]\frac{1} {(m+n)^{2}(m+d+1)^{2}}$ (69) $$

where all $\tau,\omega$ are positive except that $\tau_{12}$ is negative. First, we take out the terms that does not depend on $n$, and we equivalently minimize

$$ $L(n):=\left[m^{3}\omega_{1}\sigma^{2}+m^{2}n\omega_{2}\sigma_{\text{rag}}^{2}+ \|\beta_{\mathrm{tt}}\|_{2}^{2}\left(\tau_{30}m^{3}+\tau_{22}m^{2}n^{2}+\tau_{ 21}m^{2}n+\tau_{12}mn^{2}+\tau_{2}\left(m^{2}+n^{2}\right)\right)\right]\frac{ 1}{(m+n)^{2}}$ $$

Let

$$ $\displaystyle A=m^{3}\omega_{1}\sigma^{2}+\|\beta_{\mathrm{tt}}\|^{2}\tau_{30} m^{3}+\|\beta_{\mathrm{tt}}\|^{2}\tau_{2}m^{2},$ (70) $\displaystyle B=m^{2}\left(\omega_{2}\sigma_{\mathrm{rag}}^{2}+\|\beta_{ \mathrm{tt}}\|^{2}\tau_{21}\right),$ $\displaystyle C=\|\beta_{\mathrm{tt}}\|^{2}\left(\tau_{22}m^{2}+\tau_{12}m+ \tau_{2}\right).$ $$

Then,

$$ $L(n)=(A+Bn+Cn^{2})/(m+n)^{2}$ $$

Then, by the rule for derivative of quotient,

$$ $\displaystyle\frac{\partial\left({\cal L}_{\text{tt+rag}}({\boldsymbol{W}}^{*} )\right)}{\partial n}$ $\displaystyle=\frac{(B+2Cn)(m+n)^{2}-2(m+n)\left(A+Bn+Cn^{2}\right)}{(m+n)^{4}}$ $\displaystyle=\frac{(B+2Cn)(m+n)-2\left(A+Bn+Cn^{2}\right)}{(m+n)^{3}}$ $\displaystyle=\frac{Bm+Bn+2Cmn+2Cn^{2}-2A-2Bn-2Cn^{2}}{(m+n)^{3}}$ $\displaystyle=\frac{Bm-Bn+2Cmn-2A}{(m+n)^{3}}$ $$

Set the derivative to be zero, we have

$$ $Bm-Bn+2Cmn-2A=0$ $$

and

$$ $\displaystyle n^{\star}$ $\displaystyle=\frac{Bm-2A}{B-2Cm}$ $\displaystyle=\frac{m(m^{2}\left(\omega_{2}\sigma_{\mathrm{rag}}^{2}+\|\beta_{ \mathrm{tt}}\|^{2}\tau_{21}\right))-2(m^{3}\omega_{1}\sigma^{2}+\|\beta_{ \mathrm{tt}}\|^{2}\tau_{30}m^{3}+\|\beta_{\mathrm{tt}}\|^{2}\tau_{2}m^{2})}{(m ^{2}\left(\omega_{2}\sigma_{\mathrm{rag}}^{2}+\|\beta_{\mathrm{tt}}\|^{2}\tau_ {21}\right))-2m(\left\|\beta_{\mathrm{tt}}\right\|^{2}\left(\tau_{22}m^{2}+ \tau_{12}m+\tau_{2}\right))}$ $\displaystyle=\frac{m\left(2\|\beta_{\mathrm{tt}}\|_{2}^{2}dm+2\|\beta_{ \mathrm{tt}}\|_{2}^{2}d+2\|\beta_{\mathrm{tt}}\|_{2}^{2}m-dm\sigma_{\text{rag} }^{2}+2m\sigma^{2}\right)}{d\left(2\|\beta_{\mathrm{tt}}\|_{2}^{2}m^{2}-2\| \beta_{\mathrm{tt}}\|_{2}^{2}m+2\|\beta_{\mathrm{tt}}\|_{2}^{2}-m\sigma_{\text {rag}}^{2}\right)}$ $\displaystyle\leq\frac{md\left(2\|\beta_{\mathrm{tt}}\|_{2}^{2}dm-dm\sigma_{ \text{rag}}^{2}+2m\sigma^{2}\right)}{d^{2}\left(2\|\beta_{\mathrm{tt}}\|_{2}^{ 2}m^{2}-2\|\beta_{\mathrm{tt}}\|_{2}^{2}m+2\|\beta_{\mathrm{tt}}\|_{2}^{2}-m \sigma_{\text{rag}}^{2}\right)}$ $\displaystyle={\cal O}\left(\frac{md\left(2\|\beta_{\mathrm{tt}}\|_{2}^{2}dm- dm\sigma_{\text{rag}}^{2}+2m\sigma^{2}\right)}{d^{2}\left(2\|\beta_{\mathrm{tt }}\|_{2}^{2}m^{2}-m\sigma_{\text{rag}}^{2}\right)}\right)$ $\displaystyle={\cal O}\left(\frac{m\left(d^{2}\|\beta_{\mathrm{tt}}\|_{2}^{2}+ d\sigma^{2}-d^{2}\sigma_{\text{rag}}^{2}\right)}{md^{2}\|\beta_{\mathrm{tt}}\| _{2}^{2}-d^{2}\sigma_{\text{rag}}^{2}}\right)$ $\displaystyle={\cal O}\left(\frac{d\|\beta_{\mathrm{tt}}\|_{2}^{2}+\sigma^{2}- d\sigma_{\mathrm{rag}}^{2}}{d\|\beta_{\mathrm{tt}}\|_{2}^{2}}\right)$ $$

where the third step follows from upper bounding the numerator, and the fourth step follows from lower bounding the denominator.

###### Proposition (Restatement of Proposition 2 ) .

###### Proof.

##### n ∗ superscript 𝑛 n^{*} italic_n start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT as Global Minimizer

Now, we will show that the stationary point is the global minimizer. The second order derivative is give as:

$$ $\displaystyle\frac{\partial\left({\cal L}_{\text{tt+rag}}({\boldsymbol{W}}^{*} )\right)}{\partial n}$ $\displaystyle=\frac{2\left(Cm^{2}-2Cmn-2Bm+Bn+3A\right)}{(m+n)^{4}}$ (71) $$

Plug in $Bm-Bn^{*}+2Cmn^{*}-2A=0$, we have

$$ $\displaystyle\frac{\partial\left({\cal L}_{\text{tt+rag}}({\boldsymbol{W}}^{*} )\right)}{\partial n}|_{n=n^{*}}$ $\displaystyle=\frac{2\left(Cm^{2}-A\right)}{(m-n)(m+n)^{3}}\geq 0$ (72) $$

Since $n^{*}={\cal O}(1)$, we have $m>n^{*}$ for large $m$. Also, we have $Cm^{2}>A$ for large $m$, thus we have $\frac{\partial\left({\cal L}_{\text{tt+rag}}({\boldsymbol{W}}^{*})\right)}{
\partial n}|_{n=n^{*}}\geq 0$, and $n^{*}$ is the local minimum.
Now, we check the first order derivative of $n\geq n^{*}$,

$$ $\displaystyle Bm-Bn+2Cmn-2A$ $\displaystyle=Bm-Bn+2Cmn-2A-(Bm-Bn^{*}+2Cmn^{*}-2A)$ $\displaystyle=-B(n-n^{*})+2Cm(n-n^{*})\geq 0$ $$

where it follows from $B\leq 0,C\geq 0$. Similarly, we can show that $Bm-Bn+2Cmn-2A\leq 0,\quad\forall n\leq n^{*}$. Thus, we have $n^{*}$ to be the global minimum of the loss.

##### Improvement from n ∗ superscript 𝑛 n^{*} italic_n start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT

Here, we plug in $n=n^{*}$ and $n=0$ into [Equation 69](https://arxiv.org/html/2506.03100v3#A2.E69). We have

$$ $\displaystyle{\cal L}_{\text{tt+rag}}|_{n=n^{*}}({\boldsymbol{W}}^{*})$ $\displaystyle=\frac{A+Bn^{*}+C(n^{*})^{2}}{(m+n^{*})^{2}(m+d+1)^{2}}$ (73) $\displaystyle{\cal L}_{\text{tt+rag}}|_{n=0}({\boldsymbol{W}}^{*})$ $\displaystyle=\frac{A}{m^{2}(m+d+1)^{2}}$ $$

Then, the improvement is give as

$$ $\displaystyle{\cal L}_{\text{tt+rag}}|_{n=0}({\boldsymbol{W}}^{*})-{\cal L}_{ \text{tt+rag}}|_{n=n^{*}}({\boldsymbol{W}}^{*})$ $\displaystyle=\frac{A(m+n^{*})^{2}-m^{2}(A+Bn^{*}+C(n^{*})^{2})}{m^{2}(m+n^{*} )^{2}(m+d+1)^{2}}$ (74) $\displaystyle=\frac{(n^{*})^{2}(2Cm-B)}{2m^{2}\left(m+n^{*}\right)(m+d+1)^{2}}$ $\displaystyle={\cal O}\left(\frac{Cm}{m^{5}}\right)$ $\displaystyle={\cal O}\left(\frac{m^{2}d^{2}\|\beta_{\mathrm{tt}}\|_{2}^{2}m}{ m^{5}}\right)$ $\displaystyle={\cal O}\left(\frac{1}{m^{2}}\right)$ $$

where the second step step follows from $Bm-Bn^{*}+2Cmn^{*}-2A=0$ and the third step follows from $n^{*}={\cal O}(1)$, and the four step follows from $B\leq 0$ and $|B|={\cal O}(C)$. It finishes the proof.
∎

### B.2 Non-Uniform Retrieval Noise

Now, we proceed to the proof for non-uniform retrieval noise.

#### B.2.1 Distance-Proportional Noise

###### Theorem (Restatement of Theorem 2 ) .

Under Assumption [1](https://arxiv.org/html/2506.03100v3#Thmassumption1), [2](https://arxiv.org/html/2506.03100v3#Thmassumption2), [4](https://arxiv.org/html/2506.03100v3#Thmassumption4), the population loss is given as:

$$ $\hat{{\operatorname{err}}}_{\text{variance}}({\boldsymbol{W}})=m\sigma^{2} \operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+\sum_{i=1}^{n} \gamma_{1}\delta_{i}^{2}[(1+\delta_{i}^{2})\operatorname{tr}({\boldsymbol{W}}^ {\top}{\boldsymbol{W}})+\operatorname{tr}({\boldsymbol{W}}^{2})+\operatorname{ tr}({\boldsymbol{W}})^{2}]$ $$

If the variance of the retrieval distance follows power law, i.e. $\exists\gamma_{2}>0,q\geq 0$ s.t. $\delta_{i}^{2}=\gamma_{2}i^{q}$, then

$$ $\displaystyle\hat{{\operatorname{err}}}_{\text{bias}}({\boldsymbol{W}}^{*})$ $\displaystyle={\cal O}\left({\operatorname{err}}_{\text{bias}}({\boldsymbol{W} }^{*})+\|\beta_{\mathrm{tt}}\|_{2}^{2}\left[\frac{dn^{2q+1}+n^{2q+2}}{(m+n)^{2 }}\right]\right)$ (75) $$

and

$$ $\displaystyle\hat{{\operatorname{err}}}_{\text{variance}}({\boldsymbol{W}}^{*})$ $\displaystyle={\cal O}\left(\frac{dm\sigma^{2}+d(n^{2q+1})\sigma^{2}}{(m+n)^{2 }}\right)=\begin{cases}{\cal O}\left(dn^{2q-1}\sigma^{2}\right)&\text{ if $n \rightarrow\infty$, $q\leq 1/2$ }\\ \text{diverges}&\text{ if $n\rightarrow\infty$, $q>1/2$}\\ \end{cases}$ (76) $$

###### Proof.

We first write down the error explicitly similar to [Equation 39](https://arxiv.org/html/2506.03100v3#A2.E39).

$$ $\displaystyle y_{q}-{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{X} }^{\top}{\boldsymbol{y}}={\boldsymbol{x}}_{q}^{\top}\left(I-{\boldsymbol{W}}{ \boldsymbol{G}}\right)\beta_{\mathrm{tt}}-{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{X}}^{\top}\boldsymbol{\epsilon}+\epsilon_{q}$ $$

And we can break down the population loss as

$$ $\hat{{\cal L}}_{\text{tt+rag}}({\boldsymbol{W}})=\operatorname*{\mathbb{E}}_{( {\boldsymbol{x}}_{q},y_{q}),({\boldsymbol{X}},{\boldsymbol{y}}),\boldsymbol{ \epsilon},{\boldsymbol{r}}}\left({\boldsymbol{x}}_{q}^{\top}\left(I-{ \boldsymbol{W}}{\boldsymbol{G}}\right)\beta_{\mathrm{tt}}\right)^{2}+\left({ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{X}}^{\top}\boldsymbol{ \epsilon}\right)^{2}+\sigma^{2}$ (77) $$

###### Theorem (Restatement of Theorem 2 ) .

###### Proof.

##### Variance-Induced Error

$$ $\displaystyle\hat{{\operatorname{err}}}_{\text{variance}}({\boldsymbol{W}})$ $\displaystyle=\operatorname*{\mathbb{E}}({\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{X}}^{\top}\boldsymbol{\epsilon})^{2}$ (78) $\displaystyle=\sum_{i,j=1}^{m+n}({\boldsymbol{x}}_{i}^{\top}{\boldsymbol{W}}^{ \top}{\boldsymbol{x}}_{q})({\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{x}}_{j})\operatorname*{\mathbb{E}}(\epsilon_{i}\epsilon_{j})$ $$

Because the noise are independent and zero-mean, we have

$$ $\operatorname*{\mathbb{E}}[\epsilon_{j}\epsilon_{j}]=\begin{cases}\sigma^{2},& i=j\leq m\\ \sigma^{2}_{\text{rag},i},&i=j>m\\ 0,&i\neq j\end{cases}$ $$

Then,

$$ $\text{LHS}=\sum_{i=1}^{m}\sigma^{2}\operatorname*{\mathbb{E}}[({\boldsymbol{x} }_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{x}}_{i})^{2}]+\sum_{i=m+1}^{m+n} \sigma^{2}_{\text{rag},i-m}\cdot\operatorname*{\mathbb{E}}[{\boldsymbol{x}}_{q }^{\top}{\boldsymbol{W}}({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i-m})^{2}]$ $$

Thus, the ICL contribution remains the same as [Theorem 1](https://arxiv.org/html/2506.03100v3#Thmtheorem1), i.e.

$$ $\sum_{i=1}^{m}\sigma^{2}\operatorname*{\mathbb{E}}[({\boldsymbol{x}}_{q}^{\top }{\boldsymbol{W}}{\boldsymbol{x}}_{i})^{2}]=m\sigma^{2}\operatorname{tr}({ \boldsymbol{W}}^{\top}{\boldsymbol{W}})$ $$

To compute the RAG contribution, we evaluate the formula similar to [Equation 45](https://arxiv.org/html/2506.03100v3#A2.E45).

$$ $\displaystyle\operatorname*{\mathbb{E}}\left[\left({\boldsymbol{x}}_{q}^{\top} {\boldsymbol{W}}({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i})\right)^{2}\right]$ $\displaystyle=\operatorname*{\mathbb{E}}[({\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{x}}_{q})^{2}]+\operatorname*{\mathbb{E}}[({ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{r}}_{i})^{2}]+2 \operatorname*{\mathbb{E}}[{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{x}}_{q}\cdot{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{r}}_{i}]$ (79) $\displaystyle=\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+ \operatorname{tr}({\boldsymbol{W}}^{2})+\delta^{2}_{i}\operatorname{tr}({ \boldsymbol{W}}^{\top}{\boldsymbol{W}})+\operatorname{tr}({\boldsymbol{W}})^{2}$ $$

And thus, the RAG error contribution is

$$ $\sum_{i=m+1}^{m+n}\sigma^{2}_{\text{rag},i-m}\cdot\operatorname*{\mathbb{E}}[{ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}({\boldsymbol{x}}_{q}+{\boldsymbol{r }}_{i-m})^{2}]=\sum_{i=1}^{n}\sigma_{\text{rag},i}^{2}[(1+\delta_{i}^{2}) \operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+\operatorname{tr}({ \boldsymbol{W}}^{2})+\operatorname{tr}({\boldsymbol{W}})^{2}]$ $$

Plug in $\sigma_{\text{rag},i}^{2}=\gamma_{1}\delta_{i}^{2}$, and combining all terms together, we have

$$ $\hat{{\operatorname{err}}}_{\text{variance}}({\boldsymbol{W}})=m\sigma^{2} \operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+\sum_{i=1}^{n} \gamma_{1}\delta_{i}^{2}[(1+\delta_{i}^{2})\operatorname{tr}({\boldsymbol{W}}^ {\top}{\boldsymbol{W}})+\operatorname{tr}({\boldsymbol{W}}^{2})+\operatorname{ tr}({\boldsymbol{W}})^{2}]$ $$

Now, if we further assume $\delta_{i}^{2}=\gamma_{2}i^{q}$, and plug in the value of

$$ $\displaystyle\hat{{\operatorname{err}}}_{\text{variance}}({\boldsymbol{W}}^{*})$ $\displaystyle=m\sigma^{2}\operatorname{tr}\left(({\boldsymbol{W}}^{*})^{\top}{ \boldsymbol{W}}^{*}\right)+\sum_{i=1}^{n}\gamma_{1}\gamma_{2}i^{q}[(1+\gamma_{ 2}i^{q})\operatorname{tr}\left(({\boldsymbol{W}}^{*})^{\top}{\boldsymbol{W}}^{ *}\right)+\operatorname{tr}\left(({\boldsymbol{W}}^{*})^{2}\right)+ \operatorname{tr}({\boldsymbol{W}}^{*})^{2}]$ $\displaystyle=\frac{m^{2}}{(m+d+1)^{2}(m+n)^{2}}\left[dm\sigma^{2}+\gamma_{1} \gamma_{2}\left[\left(2d+d^{2}\right)\sum_{i=1}^{n}i^{q}\sigma^{2}+d\gamma_{2} \sum_{i=1}^{n}i^{2q}\sigma^{2}\right]\right]$ $\displaystyle=\frac{m^{2}}{(m+d+1)^{2}(m+n)^{2}}\left[dm\sigma^{2}+\gamma_{1} \gamma_{2}\sigma^{2}\left[\left(2d+d^{2}\right){\cal O}\left(\frac{n^{q+1}}{q+ 1}+\frac{n^{q}}{2}\right)+d\gamma_{2}{\cal O}\left(\frac{n^{2q+1}}{2q+1}+\frac {n^{2q}}{2}\right)\right]\right]$ $\displaystyle={\cal O}\left(\frac{dm\sigma^{2}+d(n^{2q+1})}{(m+n)^{2}}\right)$ $\displaystyle=\begin{cases}{\cal O}\left(dn^{2q-1}\sigma^{2}\right)&\text{ if $n\rightarrow\infty$, $q<1/2$ }\\ {\cal O}\left(d\sigma^{2}\right)&\text{ if $n\rightarrow\infty$, $q=1/2$}\\ \text{diverges}&\text{ if $n\rightarrow\infty$, $q>1/2$}\\ \end{cases}$ $$

where the second step follows from the Euler–Maclaurin expansion of the power sum.

##### Bias-Induced Error

From [Equation 56](https://arxiv.org/html/2506.03100v3#A2.E56), we note that

$$ ${\operatorname{err}}_{\text{bias}}({\boldsymbol{W}})=\beta_{\mathrm{tt}}^{\top }\left[M_{1}-M_{2}-M_{3}+M_{41}+\sum_{i=1}^{n}(M_{42}+M_{42}^{\top})+\sum_{i=1 }^{n}M_{43}+\sum_{i\neq j,i,j\in[n]}M_{44}\right]\beta_{\mathrm{tt}}$ $$

Specifically,

$$ $\displaystyle\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X}} }\left[\left(I-{\boldsymbol{W}}{\boldsymbol{G}}\right)^{\top}{\boldsymbol{x}}_ {q}{\boldsymbol{x}}_{q}^{\top}\left(I-{\boldsymbol{W}}{\boldsymbol{G}}\right)\right]$ $\displaystyle=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X} }}\left(I-{\boldsymbol{G}}{\boldsymbol{W}}^{\top}\right){\boldsymbol{x}}_{q}{ \boldsymbol{x}}_{q}^{\top}(I-{\boldsymbol{W}}{\boldsymbol{G}})$ (80) $\displaystyle=\underbrace{\operatorname*{\mathbb{E}}{\boldsymbol{x}}_{q}{ \boldsymbol{x}}_{q}^{\top}}_{:=M_{1}}-\underbrace{\operatorname*{\mathbb{E}}{ \boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}} }_{:=M_{2}}-\underbrace{\operatorname*{\mathbb{E}}{\boldsymbol{G}}{\boldsymbol {W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}}_{:=M_{3}}+ \underbrace{\operatorname*{\mathbb{E}}{\boldsymbol{G}}{\boldsymbol{W}}^{\top}{ \boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}} }_{:=M_{4}}$ $$

To avoid the repeated computation, we will highlight the calculation that involves $\delta_{i}$, omit some calculation steps given in the standard case and discuss its bound after allowing for non-uniform offset. We will only compute $\delta^{2}_{i}$-involving term and use $\dots$ to denote the rest terms, since we assume $\delta^{2}\ll 1$ in proving [Theorem 1](https://arxiv.org/html/2506.03100v3#Thmtheorem1). The final bound will be given as

$$ $\hat{{\operatorname{err}}}_{\text{bias}}({\boldsymbol{W}}^{*})={\operatorname{ err}}_{\text{bias}}({\boldsymbol{W}}^{*})+\delta^{2}\text{-involved terms}$ $$

$M_{1}=\operatorname*{\mathbb{E}}\left[{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}
^{\top}\right]=I$ and remains the same. Let $s_{\delta}:=\sum_{i}\delta_{i}^{2},S_{\delta}:=\sum_{i}(\delta_{i}^{2})^{2}$.

Then, we expand out the terms in $M_{2}$:

$$ $\displaystyle M_{2}=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{ \boldsymbol{r}}}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W} }{\boldsymbol{G}}$ $\displaystyle=\left(\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{ \boldsymbol{r}}}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}\right){ \boldsymbol{W}}{\boldsymbol{G}}_{0}+\operatorname*{\mathbb{E}}_{{\boldsymbol{x }}_{q},{\boldsymbol{r}}}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}\sum_{i=1}^{n}({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i})({ \boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i})^{\top}$ (81) $\displaystyle={\boldsymbol{W}}{\boldsymbol{G}}_{0}+\operatorname*{\mathbb{E}}_ {{\boldsymbol{x}}_{q},{\boldsymbol{r}}}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q }^{\top}{\boldsymbol{W}}\sum_{i=1}^{n}({\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q }^{\top}+{\boldsymbol{r}}_{i}{\boldsymbol{r}}_{i}^{\top})$ $\displaystyle={\boldsymbol{W}}{\boldsymbol{G}}_{0}+\operatorname*{\mathbb{E}}_ {{\boldsymbol{x}}_{q},{\boldsymbol{r}}}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q }^{\top}{\boldsymbol{W}}\sum_{i=1}^{n}({\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q }^{\top}+\delta^{2}_{i}I)$ $\displaystyle=\dots+s_{\delta}{\boldsymbol{W}}$ $$

Similarly, $M_{3}=M_{2}^{\top}=\dots+s_{\delta}{\boldsymbol{W}}^{\top}$. Now, we perform similar expansion for $M_{4}$.

First, we note that $M_{41}=\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{\boldsymbol{X}}}[{
\boldsymbol{G}}_{0}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}
_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}}_{0}]$ is independent of $\delta_{i}^{2}$.

$$ $\displaystyle\sum_{i\in[n]}M_{42}$ $\displaystyle:=\sum_{i\in[n]}\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q}, {\boldsymbol{X}}}{\boldsymbol{G}}_{0}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{ q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{G}}_{i}$ (82) $\displaystyle=\sum_{i\in[n]}\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{ \boldsymbol{X}}}{\boldsymbol{G}}_{0}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q }{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}\left({\boldsymbol{x}}_{q}+{ \boldsymbol{r}}_{i}\right)\left({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i} \right)^{\top}$ $\displaystyle=\sum_{i\in[n]}\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{ \boldsymbol{X}}}{\boldsymbol{G}}_{0}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q }{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}\left({\boldsymbol{x}}_{q}{ \boldsymbol{x}}_{q}^{\top}+{\boldsymbol{r}}_{i}{\boldsymbol{r}}_{i}^{\top}\right)$ $\displaystyle=\sum_{i\in[n]}\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q},{ \boldsymbol{X}}}{\boldsymbol{G}}_{0}{\boldsymbol{W}}^{\top}\left({\boldsymbol{ W}}+{\boldsymbol{W}}^{\top}+\operatorname{tr}({\boldsymbol{W}})+{\boldsymbol{W }}\delta^{2}\right)$ $\displaystyle=\sum_{i\in[n]}m\left({\boldsymbol{W}}^{\top}{\boldsymbol{W}}+{ \boldsymbol{W}}^{\top}{\boldsymbol{W}}^{\top}+\operatorname{tr}({\boldsymbol{W }}){\boldsymbol{W}}^{\top}+\delta^{2}{\boldsymbol{W}}^{\top}{\boldsymbol{W}}\right)$ $\displaystyle=\dots+ms_{\delta}{\boldsymbol{W}}^{\top}{\boldsymbol{W}}$ $$

Following the derivation of the 6th-order and 4th-order moments as in [Lemma 3](https://arxiv.org/html/2506.03100v3#Thmlemma3) and [Lemma 2](https://arxiv.org/html/2506.03100v3#Thmlemma2), we have

$$ $\displaystyle\sum_{i\in[n]}M_{43}$ $\displaystyle:=\sum_{i\in[n]}\operatorname*{\mathbb{E}}_{{\boldsymbol{x}}_{q}, {\boldsymbol{X}},{\boldsymbol{r}}_{i}}{\boldsymbol{G}}_{i}{\boldsymbol{W}}^{ \top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{G}}_{i}$ (83) $\displaystyle=\left.\begin{aligned} &2\left({\boldsymbol{W}}^{2}+({\boldsymbol {W}}^{2})^{\top}+{\boldsymbol{W}}^{\top}{\boldsymbol{W}}+{\boldsymbol{W}}{ \boldsymbol{W}}^{\top}+\operatorname{tr}\left({\boldsymbol{W}}\right)({ \boldsymbol{W}}+{\boldsymbol{W}}^{\top})\right)\\ &+\operatorname{tr}\left({\boldsymbol{W}}\right)^{2}I+\operatorname{tr}\left({ \boldsymbol{W}}^{2}\right)I+\operatorname{tr}\left({\boldsymbol{W}}^{\top}{ \boldsymbol{W}}\right)I\\ \end{aligned}\right\}\text{0th-order in ${\boldsymbol{r}}_{i}$, \lx@cref{ creftype~refnum}{lem:6order_gaussian}}$ $\displaystyle\quad+\underbrace{2\delta_{i}^{4}{\boldsymbol{W}}^{\top}{ \boldsymbol{W}}+\delta_{i}^{4}\operatorname{tr}({\boldsymbol{W}}^{\top}{ \boldsymbol{W}})I}_{\text{4th-order in ${\boldsymbol{r}}_{i}$, \lx@cref{ creftype~refnum}{eq:M4_4order_term}}}$ $\displaystyle\quad+\underbrace{\delta_{i}^{2}\left[\operatorname{tr}\left({ \boldsymbol{W}}\right)\left({\boldsymbol{W}}^{\top}+{\boldsymbol{W}}\right)+{ \boldsymbol{W}}^{2}+({\boldsymbol{W}}^{2})^{\top}+2{\boldsymbol{W}}^{\top}{ \boldsymbol{W}}\right]}_{\text{\lx@cref{creftype~refnum}{eq:M4_2order_term4} and its transpose}}$ $\displaystyle\quad+\underbrace{\left(\operatorname{tr}\left({\boldsymbol{W}}^{ 2}\right)+\operatorname{tr}\left({\boldsymbol{W}}^{\top}{\boldsymbol{W}}\right )+\operatorname{tr}\left({\boldsymbol{W}}\right)^{2}\right)\delta_{i}^{2}I}_{ \text{\lx@cref{creftype~refnum}{eq:M4_2order_term1}}}$ $\displaystyle\quad+\underbrace{2\delta_{i}^{2}{\boldsymbol{W}}{\boldsymbol{W}} ^{\top}+\delta_{i}^{2}\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W} })I}_{\text{\lx@cref{creftype~refnum}{eq:M4_2order_term2}}}$ $\displaystyle\quad+\underbrace{\delta_{i}^{2}\left[\operatorname{tr}\left({ \boldsymbol{W}}\right)\left({\boldsymbol{W}}^{\top}+{\boldsymbol{W}}\right)+{ \boldsymbol{W}}^{2}+({\boldsymbol{W}}^{2})^{\top}+2{\boldsymbol{W}}^{\top}{ \boldsymbol{W}}\right]}_{\text{\lx@cref{creftype~refnum}{eq:M4_2order_term3} and its transpose}}$ $\displaystyle=\sum_{i\in[n]}(2+2\delta_{i}^{2})\left[\operatorname{tr}\left({ \boldsymbol{W}}\right)\left({\boldsymbol{W}}^{\top}+{\boldsymbol{W}}\right)+{ \boldsymbol{W}}^{2}+({\boldsymbol{W}}^{2})^{\top}\right]$ $\displaystyle\quad+\sum_{i\in[n]}(2+4\delta_{i}^{2}){\boldsymbol{W}}^{\top}{ \boldsymbol{W}}+\sum_{i\in[n]}2{\boldsymbol{W}}{\boldsymbol{W}}^{\top}$ $\displaystyle\quad+\sum_{i\in[n]}(1+\delta_{i}^{2})\left[\operatorname{tr} \left({\boldsymbol{W}}\right)^{2}I+\operatorname{tr}\left({\boldsymbol{W}}^{2} \right)I+\operatorname{tr}\left({\boldsymbol{W}}^{\top}{\boldsymbol{W}}\right) I\right]$ $\displaystyle\quad+\sum_{i\in[n]}\left(2\delta_{i}^{4}{\boldsymbol{W}}^{\top}{ \boldsymbol{W}}+\delta_{i}^{4}\operatorname{tr}({\boldsymbol{W}}^{\top}{ \boldsymbol{W}})I+2\delta_{i}^{2}{\boldsymbol{W}}{\boldsymbol{W}}^{\top}+ \delta_{i}^{2}\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})I\right)$ $\displaystyle=\dots+2s_{\delta}\left[\operatorname{tr}\left({\boldsymbol{W}} \right)\left({\boldsymbol{W}}^{\top}+{\boldsymbol{W}}\right)+{\boldsymbol{W}}^ {2}+({\boldsymbol{W}}^{2})^{\top}\right]$ $\displaystyle\quad+(4s_{\delta}+2S_{\delta}){\boldsymbol{W}}^{\top}{ \boldsymbol{W}}+2s_{\delta}{\boldsymbol{W}}{\boldsymbol{W}}^{\top}$ $\displaystyle\quad+s_{\delta}\left(\operatorname{tr}\left({\boldsymbol{W}} \right)^{2}+\operatorname{tr}\left({\boldsymbol{W}}^{2}\right)\right)I+(2s_{ \delta}+S_{\delta})\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})I$ $$

Also, we expand the cross-term out for $\forall i,j\in[n],i\neq j$:

$$ $\displaystyle\sum_{i\neq j}M_{44}$ $\displaystyle:=\sum_{i\neq j}\operatorname*{\mathbb{E}}{\boldsymbol{G}}_{i}{ \boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{G}}_{j}$ (84) $\displaystyle=\sum_{i\neq j}\left({\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{ \top}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top}+{\boldsymbol{r} }_{i}{\boldsymbol{r}}_{i}^{\top}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{x}}_{q}{\boldsymbol{x}} _{q}^{\top}\right)$ $\displaystyle\quad+\sum_{i\neq j}\left({\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q }^{\top}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{\boldsymbol{x}}_{q}^{\top} {\boldsymbol{W}}{\boldsymbol{r}}_{j}{\boldsymbol{r}}_{j}^{\top}+{\boldsymbol{r }}_{i}{\boldsymbol{r}}_{i}^{\top}{\boldsymbol{W}}^{\top}{\boldsymbol{x}}_{q}{ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{r}}_{j}{\boldsymbol{r}} _{j}^{\top}\right)$ $\displaystyle=\dots+\sum_{i\neq j}\delta^{2}_{i}\left({\boldsymbol{W}}^{2}+{ \boldsymbol{W}}^{\top}{\boldsymbol{W}}+\operatorname{tr}\left({\boldsymbol{W}} \right){\boldsymbol{W}}\right)$ $\displaystyle\quad+\sum_{i\neq j}\delta^{2}_{i}\left(\left({\boldsymbol{W}}^{2 }\right)^{\top}+{\boldsymbol{W}}^{\top}{\boldsymbol{W}}+\operatorname{tr}\left ({\boldsymbol{W}}\right){\boldsymbol{W}}^{\top}\right)$ $\displaystyle\quad+\sum_{i\neq j}\delta^{2}_{i}\delta_{j}^{2}{\boldsymbol{W}}^ {\top}{\boldsymbol{W}}$ $$

In the non-uniform noise scenario, 4th-order term in $\delta_{i}$ will dominate the 2nd-order term in $\delta_{i}$. Thus, we will plug $\delta_{i}^{2}=\gamma_{2}i^{q},{\boldsymbol{W}}^{*}=\frac{m}{(m+d+1)(m+n)}$ into ${\operatorname{err}}_{\text{bias}}$:

$$ $\displaystyle{\operatorname{err}}_{\text{bias}}({\boldsymbol{W}}^{*})$ $\displaystyle={\operatorname{err}}_{\text{bias}}({\boldsymbol{W}}^{*})+{\cal O }\left(\beta_{\mathrm{tt}}^{\top}\left[2\sum_{i}^{n}(\delta_{i}^{2})^{2}({ \boldsymbol{W}}^{*})^{\top}{\boldsymbol{W}}^{*}+\sum_{i\neq j,i\in[n],j\in[n]} \delta_{i}^{2}\delta_{j}^{2}({\boldsymbol{W}}^{*})^{\top}{\boldsymbol{W}}^{*}+ \sum_{i}^{n}(\delta_{i}^{2})^{2}\operatorname{tr}(({\boldsymbol{W}}^{*})^{\top }{\boldsymbol{W}})I\right]\beta_{\mathrm{tt}}\right)$ $\displaystyle={\operatorname{err}}_{\text{bias}}({\boldsymbol{W}}^{*})+{\cal O }\left(\beta_{\mathrm{tt}}^{\top}\left[dn^{2q+1}({\boldsymbol{W}}^{*})^{\top}{ \boldsymbol{W}}^{*}+n^{2q+2}({\boldsymbol{W}}^{*})^{\top}{\boldsymbol{W}}^{*} \right]\beta_{\mathrm{tt}}\right)$ $\displaystyle={\operatorname{err}}_{\text{bias}}({\boldsymbol{W}}^{*})+{\cal O }\left(\beta_{\mathrm{tt}}^{\top}\left[\frac{dn^{2q+1}+n^{2q+2}}{(m+n)^{2}} \right]\beta_{\mathrm{tt}}\right)$ $$

It finishes the proof.
∎

#### B.2.2 Distance-Weighted Probabilistic Noise

###### Theorem (Restatement of Theorem 3 ) .

Under Assumption [1](https://arxiv.org/html/2506.03100v3#Thmassumption1), [2](https://arxiv.org/html/2506.03100v3#Thmassumption2), [5](https://arxiv.org/html/2506.03100v3#Thmassumption5), then $\tilde{{\operatorname{err}}}_{\text{bias}}({\boldsymbol{W}})=\hat{{
\operatorname{err}}}_{\text{bias}}({\boldsymbol{W}})$, and

$$ $\hat{{\operatorname{err}}}_{\text{variance}}({\boldsymbol{W}})=m\sigma^{2} \operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+\sum_{i=1}^{n}\left (p_{i}\sigma_{s}^{2}+(1-p_{i})\sigma_{l}^{2}\right)[(1+\delta_{i}^{2}) \operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+\operatorname{tr}({ \boldsymbol{W}}^{2})+\operatorname{tr}({\boldsymbol{W}})^{2}]$ $$

If the variance of the retrieval distance follows power law, i.e. $\exists\gamma_{2}>0,q\geq 0$ s.t. $\delta_{i}^{2}=\gamma_{2}i^{q}$, then:

$$ $\displaystyle\tilde{{\operatorname{err}}}_{\text{variance}}({\boldsymbol{W}}^{ *})$ $\displaystyle=\begin{cases}{\cal O}\left(c_{l}dn^{q-1}\sigma^{2}-(c_{l}-c_{s}) \sigma^{2}dn^{q-1-q\tilde{q}}\right)&\text{ if $n\rightarrow\infty$, $q\leq 1$ }\\ \text{diverges}&\text{ if $n\rightarrow\infty$, $q>1$}\\ \end{cases}$ (85) $$

###### Proof.

First, we note that $\tilde{{\operatorname{err}}}_{\text{bias}}({\boldsymbol{W}})=\hat{{
\operatorname{err}}}_{\text{bias}}({\boldsymbol{W}})$, since both are independent of $\sigma_{\text{rag}}^{2}$ and depend on the same set of $\forall i,\;\delta_{i}^{2}$.

We write down error explicitly similar to [Equation 39](https://arxiv.org/html/2506.03100v3#A2.E39) and break down the population loss as:

$$ $\tilde{{\cal L}}_{\text{tt+rag}}({\boldsymbol{W}})=\operatorname*{\mathbb{E}}_ {({\boldsymbol{x}}_{q},y_{q}),({\boldsymbol{X}},{\boldsymbol{y}}),\boldsymbol{ \epsilon},{\boldsymbol{r}}}\left({\boldsymbol{x}}_{q}^{\top}\left(I-{ \boldsymbol{W}}{\boldsymbol{G}}\right)\beta_{\mathrm{tt}}\right)^{2}+\left({ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{X}}^{\top}\boldsymbol{ \epsilon}\right)^{2}+\sigma^{2}$ (86) $$

We note that $\tilde{{\operatorname{err}}}_{\text{bias}}({\boldsymbol{W}})={\operatorname{
err}}_{\text{bias}}({\boldsymbol{W}})$, since the error from bias does not depend on the sample complexity.

$$ $\displaystyle\tilde{{\operatorname{err}}}_{\text{variance}}({\boldsymbol{W}})$ $\displaystyle=\operatorname*{\mathbb{E}}({\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{X}}^{\top}\boldsymbol{\epsilon})^{2}$ (87) $\displaystyle=\sum_{i,j=1}^{m+n}({\boldsymbol{x}}_{i}^{\top}{\boldsymbol{W}}^{ \top}{\boldsymbol{x}}_{q})({\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{x}}_{j})\operatorname*{\mathbb{E}}(\epsilon_{i}\epsilon_{j})$ $$

Because the noise are independent and zero-mean, we have

$$ $\operatorname*{\mathbb{E}}[\epsilon_{j}\epsilon_{j}]=\begin{cases}\sigma^{2},& i=j\leq m\\ \sigma^{2}_{s},&i=j>m,\text{ w.p. }p\\ \sigma^{2}_{l},&i=j>m,\text{ w.p. }1-p\\ 0,&i\neq j\end{cases}$ $$

Thus, the ICL contribution remains the same as [Theorem 1](https://arxiv.org/html/2506.03100v3#Thmtheorem1), i.e.

$$ $\sum_{i=1}^{m}\sigma^{2}\operatorname*{\mathbb{E}}[({\boldsymbol{x}}_{q}^{\top }{\boldsymbol{W}}{\boldsymbol{x}}_{i})^{2}]=m\sigma^{2}\operatorname{tr}({ \boldsymbol{W}}^{\top}{\boldsymbol{W}})$ $$

To compute the RAG contribution, we evaluate the formula similar to [Equation 45](https://arxiv.org/html/2506.03100v3#A2.E45).

$$ $\displaystyle\operatorname*{\mathbb{E}}\left[\left({\boldsymbol{x}}_{q}^{\top} {\boldsymbol{W}}({\boldsymbol{x}}_{q}+{\boldsymbol{r}}_{i})\right)^{2}\right]$ $\displaystyle=\operatorname*{\mathbb{E}}[({\boldsymbol{x}}_{q}^{\top}{ \boldsymbol{W}}{\boldsymbol{x}}_{q})^{2}]+\operatorname*{\mathbb{E}}[({ \boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{\boldsymbol{r}}_{i})^{2}]+2 \operatorname*{\mathbb{E}}[{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{x}}_{q}\cdot{\boldsymbol{x}}_{q}^{\top}{\boldsymbol{W}}{ \boldsymbol{r}}_{i}]$ (88) $\displaystyle=\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+ \operatorname{tr}({\boldsymbol{W}}^{2})+\delta^{2}_{i}\operatorname{tr}({ \boldsymbol{W}}^{\top}{\boldsymbol{W}})+\operatorname{tr}({\boldsymbol{W}})^{2}$ $$

And thus, the RAG error contribution is

$$ $\sum_{i=1}^{n}\left(p_{i}\sigma_{s}^{2}+(1-p_{i})\sigma_{l}^{2}\right)[(1+ \delta_{i}^{2})\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+ \operatorname{tr}({\boldsymbol{W}}^{2})+\operatorname{tr}({\boldsymbol{W}})^{2}]$ $$

Plug in $\sigma_{\text{rag},i}^{2}=\gamma_{1}\delta_{i}^{2}$, and combining all terms together, we have

$$ $\hat{{\operatorname{err}}}_{\text{variance}}({\boldsymbol{W}})=m\sigma^{2} \operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+\sum_{i=1}^{n}\left (p_{i}\sigma_{s}^{2}+(1-p_{i})\sigma_{l}^{2}\right)[(1+\delta_{i}^{2}) \operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+\operatorname{tr}({ \boldsymbol{W}}^{2})+\operatorname{tr}({\boldsymbol{W}})^{2}]$ $$

Now we further assume $p_{i}=(1+\delta_{i}^{2})^{-\tilde{q}},\;\tilde{q}\geq 0$, and plug in the value of ${\boldsymbol{W}}^{*}$. Let $B:=\frac{m^{2}}{(m+d+1)^{2}(m+n)^{2}}$,

$$ $\displaystyle\tilde{{\operatorname{err}}}_{\text{variance}}({\boldsymbol{W}}^{ *})$ $\displaystyle=m\sigma^{2}\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol {W}})+\sum_{i=1}^{n}\left(p_{i}\sigma_{s}^{2}+(1-p_{i})\sigma_{l}^{2}\right)[( 1+\delta_{i}^{2})\operatorname{tr}({\boldsymbol{W}}^{\top}{\boldsymbol{W}})+ \operatorname{tr}({\boldsymbol{W}}^{2})+\operatorname{tr}({\boldsymbol{W}})^{2}]$ $\displaystyle=B\left[dm\sigma^{2}+\sum_{i=1}^{n}\left(c_{l}\sigma^{2}-(1+ \delta_{i}^{2})^{-\tilde{q}}(c_{l}-c_{s})\sigma^{2}\right)\left[(1+\delta_{i}^ {2})\cdot d+d+d^{2}\right]\right]$ $\displaystyle\approx B\left[dm\sigma^{2}+c_{l}\sigma^{2}\sum_{i=1}^{n}\left(d \delta_{i}^{2}+d^{2}\right)-(c_{l}-c_{s})\sigma^{2}\sum_{i=1}^{n}\left(d(1+ \delta_{i}^{2})^{1-\tilde{q}}+d^{2}(1+\delta_{i}^{2})^{-\tilde{q}}\right)\right]$ $\displaystyle\approx B\left[dm\sigma^{2}+c_{l}\sigma^{2}\sum_{i=1}^{n}d\delta_ {i}^{2}-(c_{l}-c_{s})\sigma^{2}\sum_{i=1}^{n}d(1+\delta_{i}^{2})^{1-\tilde{q}}\right]$ $\displaystyle\approx\begin{cases}B\left[dm\sigma^{2}+c_{l}\sigma^{2}dn^{q+1}-( c_{l}-c_{s})\sigma^{2}d\log(n)\right]&\text{ if $\tilde{q}=1+1/q$}\\ B\left[dm\sigma^{2}+c_{l}\sigma^{2}dn^{q+1}-(c_{l}-c_{s})\sigma^{2}dn^{1+q-q \tilde{q}}\right]&\text{ else}\\ \end{cases}$ $$

where the second line follows from omitting the lower order term.

If $\tilde{q}=1+1/q$, we note that the middle term will dominate the error. And combining all cases, we could obtain

$$ $\displaystyle\tilde{{\operatorname{err}}}_{\text{variance}}({\boldsymbol{W}}^{ *})$ $\displaystyle=\begin{cases}{\cal O}\left(c_{l}dn^{q-1}\sigma^{2}-(c_{l}-c_{s}) \sigma^{2}dn^{q-1-q\tilde{q}}\right)&\text{ if $n\rightarrow\infty$, $q\leq 1$ }\\ \text{diverges}&\text{ if $n\rightarrow\infty$, $q>1$}\\ {\cal O}\left(c_{l}dn^{q-1}\sigma^{2}+(c_{l}-c_{s})d^{2}\frac{\log n}{n^{2}} \sigma^{2}\right)&\text{ if $n\rightarrow\infty$, $\tilde{q}=1+1/q$}\\ \end{cases}$ $$

∎

###### Theorem (Restatement of Theorem 3 ) .

###### Proof.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. I'm curious if you have any thoughts on how your theoretical framework might extend to more complex, non-linear models like Transformers. Do you think the fundamental trade-offs you identified would still hold?
soundness: 3
strengths: 1. It's great to see a paper that finally puts some solid theory behind RAG, which has been a mostly empirical field until now.

2. The idea of modeling RAG as noisy ICL is quite clever and provides a unified lens to understand its connection to standard in-context learning.

3. The experimental results do a good job of backing up the theoretical claims, especially in showing how performance can drop when you add too many retrieved examples.
summary: This paper presents a theoretical framework that views RAG as a form of noisy ICL. The authors derive the first finite-sample generalization bounds for RAG in a linear regression setting, analyzing how the number and quality of retrieved examples impact model performance. They introduce two noise models—uniform and non-uniform—to capture different retrieval scenarios (e.g., retrieving from a generic corpus vs. a labeled training set). Their theoretical findings suggest that while RAG can initially reduce variance-induced error, there's a ceiling to its effectiveness, and adding too many noisy examples can hurt performance. These theoretical insights are then backed up by experiments on common QA datasets like Natural Questions and TriviaQA, where the results align well with their proposed models.
weaknesses: 1. The analysis is limited to a linear regression setting, which feels a bit disconnected from the complex, non-linear reality of modern language models.

2. The paper doesn't touch on how RAG fine-tuning might change the dynamics, which is a pretty common way people use RAG in practice.

3. While the noise models are a good start, they might be too simple to capture all the different ways retrieval can be "noisy" in the real world.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: How sensitive are your bounds and the optimal 𝑛* to deviations from Gaussian linear assumptions, and to adding RAG fine-tuning or context compression?
soundness: 2
strengths: The most tangible contribution is a finite-sample, closed-form risk analysis that cleanly separates variance and bias effects of adding retrieved examples. Partition-depth-like roles are played by the number and distance of retrieved items, producing an optimal 𝑛* with diminishing returns O(1/m²). This provides decision-relevant guidance for when to stop retrieving and how to trade off close versus far items.
summary: This paper (1) formalizes RAG as noisy in-context learning for in-context linear regression with a single-layer LSA predictor, (2) derives finite-sample generalization bounds and an exact bias–variance decomposition under query-dependent retrieval, and (3) analyzes both uniform and non-uniform noise regimes to explain when retrieved context helps, plateaus, or harms.
weaknesses: 1. The theory rests on stylized assumptions, e.g., Gaussian linear data, LSA proxy, no RAG finetuning, and power-law retrieval distance distributions. These assumptions enable tractability, and the conclusions should be read as qualitative guidance.
2. The empirical section is still preliminary in scope, and does not benchmark against strong retrieval policies or compression strategies, such as error–time–memory Pareto. Consequently, claims about optimal budgets and mixing ratios remain suggestive rather than decisive for deployment.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: Can the authors provide an online diagnostic that separates variance-driven gains from bias-driven plateaus to indicate when to stop adding documents. 

How robust are the conclusions when the retrieval-offset distribution deviates from Gaussian due to ANN index structures or filtered negative mining.
soundness: 2
strengths: * The paper offers a rigorous, unified view by casting RAG as noisy ICL and deriving finite-sample bounds plus an explicit bias–variance split.

* The query-dependent retrieval offset and the two noise regimes (uniform and distance-dependent) align well with practical retrieval behavior.

* The theory isolates why extra retrieval beyond a point stops helping by showing variance reduction without bias reduction under uniform noise.
summary: The paper models RAG as noisy in-context learning and presents the first finite-sample generalization bounds with an exact bias–variance decomposition for in-context linear regression, revealing an intrinsic performance ceiling. The framework recovers vanilla ICL and standard RAG as limiting cases and formalizes query-dependent retrieval via Gaussian offsets around the query. Under uniform retrieval noise, variance error decreases with more retrieved examples while bias error does not, producing diminishing returns and a plateau.
weaknesses: * The theoretical results rely on a single-layer linear self-attention model with Gaussian inputs and isotropic noises, which may limit transfer to modern nonlinear transformer stacks.

* The evaluation metric in theory is MSE while the experiments report EM on QA, and the paper does not quantify how this metric mismatch affects conclusions. 


* The Gaussian retrieval-offset assumption simplifies real retrieval distributions and ignores indexing heuristics and filtering used in practice

The experimental scope covers two QA datasets and two model families without sensitivity to retriever choice, retrieval-pool size, or fitted exponents q and \hat_q

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses
soundness: 3
strengths: 1. Important Problem: The paper identifies and systematically investigates an important problem.

2. Clear Organization: The paper is well-structured, with clear problem formulation, and visualizations.

3. Solid theoretical background: The proposed framework for RAG in in-context linear regression appears to be built upon a solid theoretical foundation.

4. Comprehensive Experimental Evaluation: Extensive and detailed experimental results provide clear evidence and detailed analysis.
summary: The paper systematically investigates generalization bound for retrieval-augmented generation (RAG) in in-context linear regression and derive an exact bias-variance tradeoff. 
The proposed framework recovers classical in-context learning (ICL) and standard RAG as limiting cases by viewing the retrieved texts as query-dependent noisy in-context examples.
The paper presents an interesting finding: an intrinsic ceiling on generalization error exists in RAG, as opposed to ICL.
Experiments across two downstream datasets valiate the sample efficiency of ICL and RAG.
weaknesses: 1. The paper assumes that all RAG or ICL examples follow independent Gaussian distributions. However, some studies have shown that ICL or RAG use similarity-based retrievers to select demonstrations [1]. It would be useful for the authors to discuss the scenario where RAG or ICL examples are retrieved based on similarity-based retrieval methods.

2. The paper only considers a single-layer linear self-attention (LSA) model. It would be useful to extend the proposed framework to nonlinear or multi-layer self-attention architectures.

3. The paper validate the proposed framework only on two simple QA datasets. The authors should add more complex downstream tasks.

4. The author needs to correct the page margins of the whole paper. 

5. Missing the analysis of different model sizes and architectures. 

6. Missing the analysis of different prompt length. 

7. Missing the details of used dataset, such as dataset sizes, prompts.

[1] What makes good in-context examples for GPT-3


Decision: Reject