SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hengyu Fu*
2100010881@stu.pku.edu.cn
Tianyu Guo*
tianyu_guo@berkeley.edu
Yu Bai
Salesforce AI Research
Song Mei
Attention layers—which map a sequence of inputs to a sequence of outputs—are core building blocks of the Transformer architecture which has achieved significant breakthroughs in modern artificial intelligence. This paper presents a rigorous theoretical study on the learning and generalization of a single multi-head attention layer, with activation function replaced by ReLU, which have recently shown comparable performance with the original Softmax activation. We consider the random feature setting where the attention layer has a large number of heads, with randomly sampled frozen query and key matrices, and trainable value matrices. We show that such a random-feature attention layer can express a broad class of target functions that are permutation invariant to the key vectors. We further provide quantitative excess risk bounds for learning these target functions from finite samples, using random feature attention with finitely many heads.
Our results feature several implications unique to the attention structure compared with existing random features theory for neural networks, such as (1) Advantages over standard fully connected random-feature models; (2) Concrete and natural classes of functions that can be learned efficiently by a random-feature attention layer. Additionally, we show that the sampling distribution of the query-key matrix (the product of the query and key matrix) matters—A biased Gaussian random matrix results in better sample complexities over the standard zero-mean counterpart for learning certain natural target functions. Experiments on simulated data corroborate our theoretical findings and further illustrate the interplay between the sample size and the complexity of the target function.
The transformer architecture [86] has achieved remarkable recent successes in many areas of artificial intelligence (AI) such as vision, language, speech, graph processing, reinforcement learning, and more recently general AI capabilities [29-31, 19, 73, 101, 22, 77, 65, 20]. A central building block in transformers is the attention layers [10]—sequence-to-sequence mappings that allow each token within the input sequence to "attend to" other tokens that are most relevant to the present token, and produce outputs based on those tokens. Attention layers implement this mechanism in a compact way that allows them to handle sequences of arbitrary length using a fixed set of parameters, a crucial reason behind their success in handling long input sequences.
Despite its wide applicability, the theoretical properties of attention layers are less well understood. While multi-layer attention networks (transformers) have been shown to be universal approximators for certain classes of functions, such as equivariant sequence-to-sequence functions [102], their results only focus on the expressive power and do not account for learning from finite samples. Another line of work derives generalization bounds for learning with multi-layer transformers in terms of the number of layers, heads, and weight norms [91, 35], yet the results are either instantiated on specific target functions such as sparse boolean functions [35], or generic but arguably elusive function classes such as Turing machines [91]. Understandings about the more basic building block—a single attention layer—remain largely open. This is in stark contrast with the situation for fully connected neural networks, where there is by now a decent understanding of the learning and generalization in the important basic case of two-layer neural networks on generic and natural function classes (e.g., [8, 51, 7, 40] and many other results along the line). This motivates the following open question:
What function classes can be learned by a single attention layer with benign sample complexities?
This work makes progress on this problem by studying the learning and generalization with a single attention layer in the random feature setting [74, 76, 28, 100], in which the query and key matrices are frozen at their random initialization, and the value matrices remain to be learnable parameters. Motivated by the attention structure in practical architectures, we consider attention layers that take in a single query token  $\mathbf{x}_0\in \mathbb{R}^d$  and  $N$  key tokens  $\{\mathbf{x}_i\}_{i\in [N]}$  as the input, and produce a scalar-valued output—A simplified setting capturing the essence (the interaction between the query and keys) of attention models. We study the sample complexity of learning certain target functions (of  $\mathbf{x}_{0:N}$ ) using an attention layer with a large but finite number of heads, and finitely many samples.
- We show that a Random Feature Attention layer (the RFA model) with a sufficiently large number of heads can express a broad class of target functions that are averages over a generic function of two tokens, which are in particular permutation invariant with respect to the key tokens (Section 3.1). We give several natural examples of target functions in this class (Section 3.3) with concrete bounds on the number of heads and weight norms.
- We derive an  $\widetilde{\mathcal{O}} (\sqrt{B(f_{\star}) / n})$  excess risk bound for learning with the RFA model with sufficiently many heads, where  $B(f_{\star})$  is an inherent complexity measure of the target function  $f_{\star}$  and  $n$  is the sample size (Section 3.2). When instantiated on the aforementioned examples, the bounds only depend on the input dimension and not the number of key tokens, improving over a naive two-layer random feature neural network model (RFMLP). Such improvement is expected due to the permutation invariance structure of target functions, aligning with the attention mechanism.
- Towards moving beyond standard random feature settings, we study a biased RFA model where the query-key matrices (product of transposed query matrices and key matrices) are drawn from a distribution with a non-zero mean (more precisely the identity matrix as the mean), motivated by a similar observation on learned attention layers in practice. We show that this model achieves provably superior sample complexity than the standard zero-mean RFA for learning certain functions of the correlations between the query token and key tokens (Section 4).
- Experiments on simulated data verify our theoretical findings in realistic settings of learning from finite samples using a RFA layer with a mild number of heads, and characterize the interplay between the complexity of the target function and the sample size (Section 5).
Transformers The Transformer architecture, initially proposed by [86], brought about a revolutionary change in natural language processing and has been widely adopted in large language models such as GPT and BERT [72, 29, 19]. At the core of transformers lie the attention layers, which were originally introduced as neural network modules for machine translation tasks [10, 49, 68].
A recent line of work investigated the capabilities of transformers by viewing transformers to be function approximators [102], computational models [91, 70, 99, 15, 56], or algorithms [82], and using transformers to perform synthetic reasoning tasks [103]. Among these works, the closest to our work is [102], which shows that multi-layer transformers can approximate any permutation-equivariant sequence-to-sequence function, and any function if positional encodings are added. Our
paper instead uses a single attention layer to approximate sequence-to-scalar functions and focuses on the generalization property with quantitative bounds.
In terms of generalization properties of transformers, [35] analyzed the generalization bound of a single attention network through the Rademacher complexity and showed that a single self-attention head could efficiently represent a sparse function of the input sequence. Besides, several works studied the sample complexities of vision transformers [48, 50], and prompt-tuning using attention [67] with special target function classes. Our paper also studies the generalization bound of a single attention network, but from the different perspective of kernel methods, and for a more general class of target functions. The kernel limit of transformers was derived in [43, 98], which shows that multi-head attention architectures behave as Gaussian processes as the number of heads tends to infinity. However, they do not study the representation power of the limiting kernel.
Besides approximation and generalization capabilities, recent work also studied the limitations [42, 14], internal working mechanisms [36, 81, 94, 64], and in-context learning capabilities [19, 96, 37, 88, 3, 26, 41, 52] of Transformer models.
Theory of random features and neural tangent kernels A recent line of work [28, 51, 33, 32, 6, 7, 104, 66, 25] studied the training dynamics of overparametrized neural networks under certain random initialization, and showed that it converges to a kernel estimator, which corresponds to the "neural tangent kernel" (NTK). These works suggested that one could use kernel or random-feature models [74] to study the properties of deep neural networks.
For NTK of MLPs and their corresponding random-feature models, there is a vast number of literature that studies their approximation power [12, 71, 8], as well as their generalization properties [13, 21, 93, 92, 54, 55, 76, 79, 100, 57, 60, 40, 39]. More recently, a line of work studies the NTK beyond MLPs, including convolution networks [53, 17, 59, 62, 18, 16], residual networks [45, 83, 4], graph networks [97, 47], and transformers [43, 98].
Although the kernel approach is a powerful tool for studying neural networks, it received criticism since it does not capture the feature learning of neural networks. Going beyond the kernel regime, a series of works used the mean field method to establish the evolution of the network parameters via a Wasserstein gradient flow [58, 9, 24, 78]. Several other mechanisms have been proven to obtain superior results over the NTK, including the Quadratic NTK [5, 11, 23, 63], regularization [90], Neural Tangent Hierarchy [34, 44], representation learning [27], and staircase-like mechanisms [1, 2].
We consider a sequence of  $N + 1$  input tokens  $\mathbf{x}_{0:N} = (\mathbf{x}_0, \{\mathbf{x}_i\}_{i \in [N]}) \in \mathcal{X} = (\mathbb{R}^d)^{N + 1}$ , where each  $\{\mathbf{x}_i\}_{i \in [N]} \subseteq \mathbb{R}^d$  represents a sequence of key vectors, and  $\mathbf{x}_0$  represents the query vector. This model simplifies standard self-attention, which maps  $N$  input tokens to  $N$  output tokens, where the output  $i$  only uses input  $i$  as the query token and all of  $[N]$  as key tokens. Results obtained in this model can be directly mapped back to full self-attention, by simply concatenating  $N$  outputs generated by our model with  $\mathbf{x}_0$  ranging over  $\{\mathbf{x}_i\}_{i \in [N]}$ . In addition, throughout the paper, we consider scalar-valued attention models, which take the sequence  $\mathbf{x}_{0:N}$  as input and give a scalar output in  $\mathbb{R}$ .
Attention layer We consider a scalar-valued,  $M$ -head, multiplicative attention layer that takes  $\mathbf{x}_{0:N} = (\mathbf{x}_0,\{\mathbf{x}_i\}_{i\in [N]})$  as the input. The attention layer first applies affine (linear with bias) transformations to the input vectors to obtain {query, keys, values} at each head  $m\in [M]$ :
$$
\mathbf {q} _ {m, 0} = \mathbf {Q} _ {m} [ \mathbf {x} _ {0}; 1 ] =: \mathbf {Q} _ {m} \widetilde {\mathbf {x}} _ {0} \in \mathbb {R} ^ {d}, \quad \mathbf {k} _ {m, i} = \mathbf {K} _ {m} [ \mathbf {x} _ {i}; 1 ] =: \mathbf {K} _ {m} \widetilde {\mathbf {x}} _ {i} \in \mathbb {R} ^ {d}, \tag {1}
$$
$$
v _ {m, i} = \mathbf {v} _ {m} ^ {\top} [ \mathbf {x} _ {i}; 1 ] = \mathbf {v} _ {m} ^ {\top} \widetilde {\mathbf {x}} _ {i} \in \mathbb {R}, i \in [ N ],
$$
where  $\mathbf{Q}_m, \mathbf{K}_m \in \mathbb{R}^{(d + 1) \times d}$ ,  $\mathbf{v}_m \in \mathbb{R}^{d + 1}$  are the parameters of the attention layer, and  $\widetilde{\mathbf{x}}_i \coloneqq [\mathbf{x}_i; 1]$  for a more compact display. Then, it computes the output value by an attention mechanism
$$
f \left(\mathbf {x} _ {0: N}\right) = \sum_ {m = 1} ^ {M} \frac {1}{N} \sum_ {i = 1} ^ {N} f _ {m, i} \left(\mathbf {x} _ {0}, \mathbf {x} _ {i}\right), \quad f _ {m, i} \left(\mathbf {x} _ {0}, \mathbf {x} _ {i}\right) = \sigma \left(\left\langle \mathbf {q} _ {m, 0}, \mathbf {k} _ {m, i} \right\rangle\right) \cdot v _ {m, i} \in \mathbb {R}. \tag {2}
$$
Above,  $\sigma : \mathbb{R} \to \mathbb{R}$  is an activation function applied entry-wisely to each attention score  $\langle \mathbf{q}_{m,0}, \mathbf{k}_{m,i} \rangle$ . We choose  $\sigma$  to be the ReLU activation  $\sigma(t) = \max \{t, 0\}$  throughout this paper. Notice that
this choice of the attention non-linearity is different from standard transformers [86] with softmax-attention. We remark that we choose to study the (normalized) ReLU attention for theoretical convenience, and this replacement does not change the essence of the attention mechanism. Such a choice is also recently explored in the literatures such as [80] and [95], which show that transformers with ReLU-attention perform as well as standard softmax-attention transformers in certain NLP and CV tasks. Moreover, our results can extend to other activation functions such as the exponential activation  $\sigma(t) = \exp(t)$ , which is more similar to the standard Softmax activation. We refer to Section B.5 for a short discussion.
Simplifying the expression, we reparametrize the attention layer (1) and (2) using parameters  $\{(\mathbf{W}_m,\mathbf{v}_m)\}_{m\in [M]}\subseteq \mathbb{R}^{(d + 1)\times (d + 1)}\times \mathbb{R}^{d + 1}$ :
$$
f _ {i, m} \left(\mathbf {x} _ {0: N}\right) = \sigma \left(\widetilde {\mathbf {x}} _ {0} ^ {\top} \mathbf {Q} _ {m} \mathbf {K} _ {m} \widetilde {\mathbf {x}} _ {i}\right) \cdot \left\langle \mathbf {v} _ {m}, \widetilde {\mathbf {x}} _ {i} \right\rangle = \sigma \left(\left\langle \mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \right\rangle\right) \cdot \left\langle \mathbf {v} _ {m}, \widetilde {\mathbf {x}} _ {i} \right\rangle . \tag {3}
$$
For technical convenience, we assume all input tokens have unit norm throughout the rest of the paper:  $\| \mathbf{x}_i\| _2\equiv 1$  so that  $\| \widetilde{\mathbf{x}}_i\| _2\equiv \sqrt{2}$ , for all  $i\in \{0\} \cup [N]$ .
Random-feature attention models We consider a random-feature version $^2$  of the multiplicative attention mechanism (3), where the weight matrices  $\{\mathbf{W}_m\}_{m\in [M]}$  have i.i.d. Gaussian entries $^3$ :
$$
\left(\mathbf {W} _ {m}\right) _ {i j} \sim_ {\mathrm {i i d}} \mathsf {N} (0, 1 / 4), \quad (m, i, j) \in [ M ] \times [ d + 1 ] ^ {2}. \tag {4}
$$
The variance is chosen to be  $1/4$  without loss of generality: this choice of variance is such that  $\langle \mathbf{W}_m, \widetilde{\mathbf{x}}_0 \widetilde{\mathbf{x}}_i^\top \rangle \sim \mathsf{N}(0, 1)$  has a unit variance. The weight matrices  $\{\mathbf{W}_m\}_{m \in [M]}$  are then held to be fixed during the entire learning process, whereas the value vectors  $\{\mathbf{v}_m\}_{m \in [M]}$  are the learnable parameters. The random-feature attention model with input  $\mathbf{x}_{0:N}$  is thus given by
$$
f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N}; \mathbf {V}\right) = \sum_ {m = 1} ^ {M} \frac {1}{N} \sum_ {i = 1} ^ {N} \sigma \left(\left\langle \mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \right\rangle\right) \left\langle \mathbf {v} _ {m}, \widetilde {\mathbf {x}} _ {i} \right\rangle . \tag {5}
$$
Notice that random-feature attention model is linear in the parameter  $\mathbf{V}$ , so training this model with a convex loss function gives a convex optimization problem.
Additional notation For any  $\mathbf{x} \in \mathbb{R}^{d_1}$  and  $\mathbf{y} \in \mathbb{R}^{d_2}$ , let  $\mathbf{x} \otimes \mathbf{y} \in \mathbb{R}^{d_1 \times d_2}$  denote their tensor product (outer product), and  $\mathbf{x}^{\otimes n} \coloneqq \mathbf{x} \otimes \dots \otimes \mathbf{x}$  denote the  $n$ -fold self tensor product of  $\mathbf{x}$ . For a tensor  $\mathbf{A}$ , we use  $\| \mathbf{A} \|_{\mathrm{Fr}}$  to denote its Frobenius norm. For a function  $f: \mathcal{X} \to \mathbb{R}$ , we use  $\| f \|_{\infty}$  to denote its  $L^{\infty}$  norm. We use  $\mathcal{O}(\cdot)$  (resp.  $\Theta(\cdot)$ ) for standard Big-O (resp. Big-Theta) relations. We use  $\widetilde{\mathcal{O}}(\cdot)$  for hiding the multiplicative terms that are logarithmic in problem parameters, including  $(M, d, n, N, \delta^{-1})$ . We use  $\mathrm{Poly}(p)$  to denote a polynomial of  $p$  that is less than  $p^C$  for some universal constant  $0 < C < \infty$ .
3 Learning with random-feature attention models
In this section, we study the expressivity and generalization of random-feature attention models. We will consider a broad class of target functions that can be well approximated and is efficiently learnable by random-feature attention models.
3.1 Expressivity of random-feature attention
Consider a broad class of permutation invariant $^4$  target functions  $f_{\star}:\mathcal{X}\to \mathbb{R}$  that takes form
$$
f _ {\star} \left(\mathbf {x} _ {0: N}\right) = \frac {1}{N} \sum_ {i = 1} ^ {N} F \left(\mathbf {x} _ {0}, \mathbf {x} _ {i}\right). \tag {6}
$$
2A different and closely related model is the Neural Tangent Kernel [46, 32], which is however similar in essence to the random feature model in terms of the sample complexity of learning, e.g. [40].
Another feasible choice for (4) is to sample the key matrix and the query matrix separately with independent Gaussian entries, which however will produce a mean-zero product matrix similar to (4) in many aspects.
4 A function  $f(\mathbf{x}_0, \mathbf{x}_1, \dots, \mathbf{x}_N)$  is permutation invariant (with respect to  $\mathbf{x}_{1:N}$ ) if  $f(\mathbf{x}_0, \mathbf{x}_1, \dots, \mathbf{x}_N) = f(\mathbf{x}_0, \mathbf{x}_{\sigma(1)}, \dots, \mathbf{x}_{\sigma(N)})$  for any permutation  $\sigma : [N] \to [N]$ . We consider permutation invariant target functions since attention layers can only fit these functions due to the structure of attention models.
Assume that there exists symmetric tensors  $\{\mathbf{f}_{rs} \in \mathbb{R}^{d^{r + s}}\}_{r,s \geq 0}$  such that  $F: \mathbb{R}^{2d} \to \mathbb{R}$  admits representation
$$
F \left(\mathbf {x} _ {0}, \mathbf {x} _ {i}\right) = \sum_ {r, s \geq 0} ^ {\infty} \left\langle \mathbf {x} _ {0} ^ {\otimes r} \otimes \mathbf {x} _ {i} ^ {\otimes s}, \mathbf {f} _ {r s} \right\rangle . \tag {7}
$$
Note that such an expression allows  $F(\mathbf{x}_0, \mathbf{x}_i)$  to be any general nonlinear function that admits convergent Taylor expansions. In particular, any polynomials of  $[\mathbf{x}_0, \mathbf{x}_i]$  (e.g.,  $\beta^\top \mathbf{x}_0$ ,  $\beta^\top \mathbf{x}_i$ , and  $\langle \mathbf{x}_0, \mathbf{S} \mathbf{x}_i \rangle$  for some  $\beta \in \mathbb{R}^d$  and  $\mathbf{S} \in \mathbb{R}^{d^2}$ ) are within this function class. We will discuss more specific target functions in Section 3.3.
Theorem 1 (Expressivity of RFA model). Suppose function  $f_{\star} : \mathcal{X} \to \mathbb{R}$  takes form (6). Then for any input distribution  $P$  on  $\mathcal{X}$ , with probability at least  $1 - \delta$  (over  $\{\mathbf{W}_m\}_{m \in [M]}$  sampled from (4)), there exists an  $M$ -head RFA model (5) with coefficients  $\mathbf{V} = \{\mathbf{v}_m\}_{m \in [M]} \subseteq \mathbb{R}^{d + 1}$  that approximates  $f_{\star}$  in  $L^2(P)$  up to error
$$
\mathbb {E} _ {\mathbf {x} _ {0: N} \sim P} \left[ \left(f _ {\star} \left(\mathbf {x} _ {0: N}\right) - f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N}; \mathbf {V}\right)\right) ^ {2} \right] \leq \mathcal {O} \left(\frac {\left(d ^ {2} + \log M\right) B \left(f _ {\star}\right) \delta^ {- 1}}{M}\right). \tag {8}
$$
In addition, the norms of the weight of this random-feature attention model are bounded as
$$
\sum_ {m = 1} ^ {M} \| \mathbf {v} _ {m} \| _ {2} \leq \mathcal {O} \left(\sqrt {B (f _ {\star})} + \sqrt {\frac {B (f _ {\star}) \delta^ {- 1}}{M}}\right), \quad \sum_ {m = 1} ^ {M} \| \mathbf {v} _ {m} \| _ {2} ^ {2} \leq \mathcal {O} \left(\frac {B (f _ {\star}) \delta^ {- 1}}{M}\right). \tag {9}
$$
Here  $B(f_{\star})$  is a complexity measure of  $f_{\star}$  defined as
$$
B \left(f _ {\star}\right) = \sum_ {k = 0} ^ {\infty} C _ {k} \sum_ {\max  \{r, s \} = k} \| \mathbf {f} _ {r s} \| _ {\mathbf {F} r} ^ {2}, \quad C _ {k} = k ^ {4. 5} 4 ^ {k} \vee 1. \tag {10}
$$
In case where  $f_{\star}$  admits multiple representations of the form (7),  $B(f_{\star})$  is the infimum of the right-hand-side over all such representations.
The proof of Theorem 1 is contained in Appendix B.1. Our proof relies on standard analyses of infinite-width random feature model with ReLU-Gaussian kernel, combined with a sampling argument to obtain approximation with finite-width.
This theorem is applicable to general functions with a finite  $B(f_{\star})$  norm. The  $4^{k}$  scaling of  $C_k$  in the summand of equation (10) seemingly confines the target function class to those with exponentially fast decaying  $\| \mathbf{f}_{rs}\|_{\mathrm{Fr}}$ , which suggests a relatively narrow target function class. However, as we will demonstrate in the forthcoming examples, this class includes a diverse range of functions.
3.2 Generalization and sample complexity of learning
Given  $n$  samples  $\{\mathbf{x}_{0:N}^{(j)},y_j\}_{j\in [n]}\sim_{\mathrm{iid}}\mathsf{P}$ , where  $\mathbf{x}_{0:N}^{(j)} = \{\mathbf{x}_i^{(j)}\}_{0\leq i\leq N}$  is the  $j$ -th token sequence with length  $N + 1$ , and  $y_{j}$  is the label corresponding to the  $i$ -th token sequence. Assume that we are given a loss function  $\ell (\widehat{y},y)$  that is 1-Lipschitz in  $\widehat{y}$ , and  $\ell (0,y)\leq 1$  for any  $y$ . The population risk is then given by  $L_{D}(f) = \mathbb{E}_{(\mathbf{x}_{0:N},y)\sim \mathsf{P}}[\ell (f(\mathbf{x}_{0:N}),y)]$ . We consider the empirical risk minimization (ERM) over the RFA model (5),
$$
\widehat {\mathbf {V}} = \arg \min  _ {\mathbf {V} \in \mathcal {V} _ {M}} \widehat {L} _ {D} \left(f _ {M} ^ {\mathbf {W}} (\cdot ; \mathbf {V})\right), \quad \widehat {L} _ {D} (f) = \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f \left(\mathbf {x} _ {0: N} ^ {(j)}, y _ {j}\right), \right. \tag {11}
$$
where the constrained class  $\nu_{M}$  is given by
$$
\mathcal {V} _ {M} = \left\{\mathbf {V} = \left\{\mathbf {v} _ {m} \right\} _ {m = 1} ^ {M}: \sum_ {m = 1} ^ {M} \| \mathbf {v} _ {m} \| _ {2} \leq K _ {1}, \sum_ {m = 1} ^ {M} \| \mathbf {v} _ {m} \| _ {2} ^ {2} \leq K _ {2} / M \right\}, \tag {12}
$$
with  $K_{1}$  and  $K_{2}$  being two constants. Theorem 2 below provides the excess risk bound for the empirical risk minimizer.
Theorem 2. Assume  $M > \delta^{-1}$  and  $n > \log(dM)$ . Let  $f_{\star}$  be the minimizer of the population risk  $L_{D}(f)$  within the target function class (6) (7). Let  $\widehat{f}_{M}^{\mathbf{W}} = f_{M}^{\mathbf{W}}(\cdot; \widehat{\mathbf{V}})$  be the empirical risk minimizer given by (11), where in (12) we choose  $K_{1} = C\sqrt{B(f_{\star})}$  and  $K_{2} = CB(f_{\star})\delta^{-1}$ , with  $C$  being a constant. Then for any joint distribution  $\mathsf{P}$ , with probability at least  $1 - \delta$  over  $\{\mathbf{W}_m\}_{m \in [M]}$  sampled according to (4) and  $\{(\mathbf{x}_{0:N}^{(j)}, y_j)\}_{j \in [n]} \sim_{iid} \mathsf{P}$ , the excess risk is bounded by
$$
L _ {D} \left(\widehat {f} _ {M} ^ {\mathbf {W}}\right) - L _ {D} \left(f _ {\star}\right) \leq \widetilde {\mathcal {O}} \left(\sqrt {B \left(f _ {\star}\right)} \left[ \sqrt {\frac {1}{n}} + \sqrt {\frac {d ^ {2} \delta^ {- 1}}{M}} \right]\right). \tag {13}
$$