SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition B.5 (Restatement of Example 3). For  $f_{1,\star} = \frac{1}{N}\sum_{i = 1}^{N}\langle \mathbf{x}_0,\mathbf{x}_i\rangle^p$ , the excess risk bound of RFA (by Theorem 2) and RFMLP scale as
For  $f_{2,\star} = \frac{1}{N}\sum_{i = 1}^{N}\cos (\langle \mathbf{x}_0,\mathbf{x}_i\rangle)\left\langle \mathbf{x}_i^{\otimes p},\mathbf{G}\right\rangle$  with  $\| \mathbf{G}\|_{\mathrm{Fr}} = 1$ . Then the excess risk bound of RFA and RFMLP scale as
$$
\mathrm {R F A}: \widetilde {\mathcal {O}} \Big (\operatorname {P o l y} (p d) \sqrt {e ^ {4 \sqrt {d}} 4 ^ {p} / n} \Big), \quad \mathrm {R F M L P}: \widetilde {\mathcal {O}} \Big (\operatorname {P o l y} (p N d) \sqrt {e ^ {2 (N + 2) d} (N + 2) ^ {p} / n} \Big).
$$
Proof of Proposition B.5. For  $f_{1,\star}$ , direct calculation gives the value of  $B(f_{1,\star})$  and the excess risk follows. For  $f_{2,\star}$ , using the Taylor expansion of  $\cos(z)$ , we get
$$
f _ {2, \star} = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {k = 0} ^ {\infty} (- 1) ^ {k} \frac {\left\langle \mathbf {x} _ {0} , \mathbf {x} _ {i} \right\rangle^ {2 k}}{(2 k) !} \left\langle \mathbf {x} _ {i} ^ {\otimes p}, \mathbf {G} \right\rangle = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {k = 0} ^ {\infty} \left\langle \left(\mathbf {x} _ {0} \mathbf {x} _ {i} ^ {\top}\right) ^ {\otimes 2 k} \otimes \mathbf {x} _ {i} ^ {\otimes p}, \frac {(- 1) ^ {k}}{(2 k) !} \mathbf {I} _ {d} ^ {\otimes 2 k} \otimes \mathbf {G} \right\rangle .
$$
$$
B \left(f _ {2, \star}\right) = 4 ^ {p} \operatorname {P o l y} (p) \sum_ {k = 0} ^ {\infty} (2 k) ^ {4. 5} 4 ^ {2 k} \frac {d ^ {2 k}}{\left((2 k) !\right) ^ {2}} = 4 ^ {p} \operatorname {P o l y} (p d) \Theta \left(\sum_ {k = 0} ^ {\infty} \frac {(4 d) ^ {2 k}}{((2 k) !) ^ {2}}\right).
$$
Note that for any  $z > 0$
$$
\sum_ {k = 0} ^ {\infty} \frac {(z) ^ {2 k}}{((2 k) !) ^ {2}} \leq \left[ \sum_ {k = 0} ^ {\infty} \frac {(\sqrt {z}) ^ {2 k}}{(2 k) !} \right] ^ {2} = \Theta \left(e ^ {2 \sqrt {z}}\right). \tag {62}
$$
Plug  $z = 4d$  into (62) for  $B(f_{2,\star})$  gives the excess risk for RFA model. As for  $B_{\mathrm{MLP}}(f_{2,\star})$ , we have
$$
B _ {\mathrm {M L P}} (f _ {2, \star}) = \operatorname {P o l y} (p) \sum_ {k = 0} ^ {\infty} (N + 2) ^ {p} (4 k) ^ {4. 5} (N + 2) ^ {4 k} \frac {d ^ {2 k}}{((2 k) !) ^ {2}} = 4 ^ {p} \operatorname {P o l y} (p d N) \Theta \Big (\sum_ {k = 0} ^ {\infty} \frac {((N + 2) \sqrt {d}) ^ {4 k}}{((2 k) !) ^ {2}} \Big).
$$
Using similar argument, we get that  $B_{\mathrm{MLP}}(f_{2,\star})$  is bounded by  $\Theta (\mathrm{Poly}(pdN)4^p\exp (2(N + 2)\sqrt{d}))$ . Using Theorem 2 and Proposition B.2, we can get the excess risk bound. This proves Proposition B.5.
B.5 Random Feature Attention with Exponential Activation
We give a brief discussion on our results with activation function replaced by the exponential function  $\sigma'(t) = \exp(t)$ . The random feature attention model with exponential activation (we call it ERFA) is given as follows.
$$
f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N}; \mathbf {V}\right) = \sum_ {m = 1} ^ {M} \frac {1}{N} \sum_ {i = 1} ^ {N} \sigma^ {\prime} \left(\left\langle \mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \right\rangle\right) \left\langle \mathbf {v} _ {m}, \widetilde {\mathbf {x}} _ {i} \right\rangle . \tag {63}
$$
We consider deriving the explicit form of the Kernel  $K_{\mathrm{ERFA}}(\mathbf{x}_{0:N}, \mathbf{x}_{0:N}^{\prime})$  associated with ERFA model
$$
K _ {\text {E R F A}} \left(\mathbf {x} _ {0: N}, \mathbf {x} _ {0: N} ^ {\prime}\right) = \frac {1}{N ^ {2}} \mathbb {E} _ {\mathbf {W}} \left[ \sum_ {i, j = 1} ^ {N} \exp \left(\left\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} + \widetilde {\mathbf {x}} _ {0} ^ {\prime} \left(\widetilde {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {\top} \right\rangle\right) \left\langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \right\rangle \right] \tag {64}
$$
For any  $i,j\in [N]$ , let  $u_{i,j} = \left\langle \widetilde{\mathbf{x}}_0\widetilde{\mathbf{x}}_i^\top ,\widetilde{\mathbf{x}}_0'(\widetilde{\mathbf{x}}_j')^\top \right\rangle /4$  For a single component in (64), we have
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {W} \sim \mathbf {N} (0, 1 / 4)} \left[ \exp (\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} + \widetilde {\mathbf {x}} _ {0} ^ {\prime} (\widetilde {\mathbf {x}} _ {j} ^ {\prime}) ^ {\top} \rangle) \right] \langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \rangle \\ = \exp \left(\frac {1}{8} \left\| \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} + \widetilde {\mathbf {x}} _ {0} ^ {\prime} \left(\widetilde {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {\top} \right\| _ {\mathcal {F}} ^ {2}\right) \langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \rangle \\ = \exp \left(1 + u _ {i, j}\right) \langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \rangle \\ = e \left(\sum_ {\ell = 0} ^ {\infty} \frac {1}{\ell !} u _ {i, j} ^ {\ell}\right) \langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \rangle \\ \end{array}
$$
$$
\begin{array}{l} = \sum_ {\ell = 0} ^ {\infty} d _ {\ell} \left\langle \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}, \widetilde {\mathbf {x}} _ {0} ^ {\prime} \left(\widetilde {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {\top} \right\rangle^ {\ell} 4 ^ {- \ell} \left\langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \right\rangle \\ = \sum_ {\ell = 0} ^ {\infty} d _ {\ell} \left\langle 2 ^ {- \ell} \left(\widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}\right) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {i}, 2 ^ {- \ell} \left(\widetilde {\mathbf {x}} _ {0} ^ {\prime} \left(\widetilde {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {\top}\right) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {j} ^ {\prime} \right\rangle . \\ \end{array}
$$
Here  $d_{\ell} = e / \ell!$  for any  $\ell \geq 0$ . Therefore, the kernel can be expressed as:
$$
K _ {\text {E R F A}} \left(\mathbf {x} _ {0: N}, \mathbf {x} _ {0: N} ^ {\prime}\right) = \sum_ {\ell = 0} ^ {\infty} \left\langle \frac {\sqrt {d _ {\ell}}}{N} \sum_ {i = 1} ^ {N} 2 ^ {- \ell} \left(\widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}\right) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {i}, \frac {\sqrt {d _ {\ell}}}{N} \sum_ {j = 1} ^ {N} 2 ^ {- \ell} \left(\widetilde {\mathbf {x}} _ {0} ^ {\prime} \left(\widetilde {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {\top}\right) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {j} ^ {\prime} \right\rangle .
$$
Thus, for any target function  $f_{\star}$  that in the form of (6), we can redefine its complexity measure as
$$
B \left(f _ {\star}\right) = \sum_ {k = 0} ^ {\infty} C _ {k} \sum_ {\max  \{r, s \} = k} \| \mathbf {f} _ {r s} \| _ {\mathcal {F r}} ^ {2}, \quad C _ {k} = k! 4 ^ {k} k ^ {2} \vee 1 \tag {65}
$$
and obtain the corresponding results of Theorem 1 and Theorem 2 by repeating the proof of these two theorems.
We consider the empirical risk minimizer over the BRFA model (15),
$$
\widehat {\mathbf {V}} = \arg \min  _ {\mathbf {V} \in \mathcal {V} _ {M}} \widehat {L} _ {D} \left(f _ {M} ^ {\mathbf {W}, \mathbf {W} _ {0}} (\cdot ; \mathbf {V})\right), \quad \widehat {L} _ {D} (f) = \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f \left(\mathbf {x} _ {0: N} ^ {(j)}\right), y _ {j}\right), \tag {66}
$$
where the constrained class  $\mathcal{V}_M$  gives (12), copied here for reader's convenience,
$$
\mathcal {V} _ {M} = \left\{\mathbf {V} = \left\{\mathbf {v} _ {m} \right\} _ {m = 1} ^ {M}: \sum_ {m = 1} ^ {M} \| \mathbf {v} _ {m} \| _ {2} \leq K _ {1}, \sum_ {m = 1} ^ {M} \| \mathbf {v} _ {m} \| _ {2} ^ {2} \leq K _ {2} / M \right\}. \tag {67}
$$
Denote  $\mathcal{G}$  to be the set of all functions in the function class (16), i.e.,
$$
\mathcal {G} = \left\{g _ {\star} = \frac {1}{N} \sum_ {i = 1} ^ {N} F \left(\left\langle \mathbf {x} _ {0}, \mathbf {x} _ {i} \right\rangle\right) G \left(\mathbf {x} _ {0}, \mathbf {x} _ {i}\right): F (t) = \sum_ {k = 0} ^ {\infty} a _ {k} t ^ {k}, G = \left\langle \widetilde {\mathbf {x}} _ {i} ^ {\otimes 3} \otimes \widetilde {\mathbf {x}} _ {0} ^ {\otimes 2}, \mathbf {A} _ {\star} \right\rangle \right\}. \tag {68}
$$
We restate Theorem 3 in Theorem C.1 with detailed assumptions.
Theorem C.1 (Restatement of Theorem 3). Assume  $M > \delta^{-1}$ ,  $n > \log(dM)$  and let  $L \in \mathbb{Z}_{\geq 1}$ . Let  $g_{\star} = \arg \min_{g \in \mathcal{G}} L_D(g)$  be the population risk minimizer within the target function class  $\mathcal{G}$  (68). Take  $K_1 = C \sqrt{B(g_{\star}, L)}$  and  $K_2 = CB(g_{\star}, L) \delta^{-1}$  in (67), with  $C$  being a constant. Let  $\widehat{f}_M^{\mathbf{W}, \mathbf{W}_0} = f_M^{\mathbf{W}, \mathbf{W}_0}(\cdot; \widehat{\mathbf{V}})$  be empirical risk minimizer given by (66). Then for any joint distribution  $\mathsf{P}$ , with probability at least  $1 - \delta$  over  $\{\mathbf{W}_m\}_{m \in [M]}$  sampled according to (4) and  $\{(x_{0:N}^{(j)}, y_j)\}_{j \in [n]} \sim_{iid} \mathsf{P}$ , the excess risk is bounded by
$$
\begin{array}{l} L _ {D} \left(\widehat {f} _ {M} ^ {\mathbf {W}, \mathbf {W} _ {0}}\right) - L _ {D} (g _ {\star}) \\ \lesssim \inf  _ {L} \left\{\sqrt {B (g _ {\star} , L)} \left[ \sqrt {\frac {\log (d M) \log (n N M) + \log (\delta^ {- 1})}{n}} + \sqrt {\frac {(d ^ {2} + \log M) \delta^ {- 1}}{M}} \right] + \varepsilon_ {L} \| g _ {\star} \| _ {\infty} \right\}, \\ \end{array}
$$
$$
B \left(g _ {\star}, L\right) = \left\| \mathbf {A} _ {\star} \right\| _ {\mathrm {F r}} ^ {2} \cdot \left(\sum_ {k = 0} ^ {\infty} \left| a _ {k} \right| \cdot C _ {k}\right) ^ {2}, \quad \text {w i t h} C _ {k} = (2 L + k) ^ {(k + 3) / 2} 8 ^ {L + k / 2}. \tag {69}
$$
C.1 Auxiliary results for the proof of Theorem 3
In this section, we give some auxiliary results used in the proof of Theorem 3. The proof will be given in Section C.2. We define the biased transformer with infinite width (informally corresponding to Eq. (15) with  $M \to \infty$ ), given by
$$
f _ {\mathbf {v}} ^ {\mathbf {W} _ {0}} \left(\mathbf {x} _ {0: N}\right) = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} _ {\mathbf {W}} \left[ \sigma \left(\left\langle \mathbf {W} + \mathbf {W} _ {0}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \right\rangle\right) \left\langle \mathbf {v} (\mathbf {W}), \widetilde {\mathbf {x}} _ {i} \right\rangle \right] \tag {70}
$$
$$
= \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} _ {\mathbf {W}} \left[ \sigma \left(\left\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \right\rangle + h _ {i}\right) \left\langle \mathbf {v} (\mathbf {W}), \widetilde {\mathbf {x}} _ {i} \right\rangle \right],
$$
where  $h_i \coloneqq \langle \mathbf{W}_0, \widetilde{\mathbf{x}}_0 \widetilde{\mathbf{x}}_i^\top \rangle$  and  $\mathbf{W}$  is sampled according to (4). Here we set  $\mathbf{W}_0 = [\mathbf{I}_{d \times d}, \mathbf{0}_{d \times 1}; \mathbf{0}_{1 \times d}, 0]$ . Then  $h_i = \langle \mathbf{x}_0, \mathbf{x}_i \rangle$ .
We consider a class of target functions  $\widetilde{g}_{\star}:\mathcal{X}\to \mathbb{R}$  that takes form
$$
\widetilde {g} _ {\star} \left(\mathbf {x} _ {0: N}\right) = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {\ell = 0} ^ {\infty} \left\langle u _ {\ell} \left(\widetilde {\mathbf {x}} _ {0}, \widetilde {\mathbf {x}} _ {i}\right), \mathbf {D} _ {\ell} \right\rangle , \tag {71}
$$
for some coefficients  $\{\mathbf{D}_{\ell}\in \mathbb{R}^{(d + 1)^{2\ell +1}}\}_{\ell \geq 0}$  and
$$
u _ {\ell} (\widetilde {\mathbf {x}} _ {0}, \widetilde {\mathbf {x}} _ {i}) = \left\{ \begin{array}{c l} [ h _ {i} \Phi (h _ {i}) + \phi (h _ {i}) ] \widetilde {\mathbf {x}} _ {i} & (\ell = 0), \\ \Phi (h _ {i}) \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \otimes \widetilde {\mathbf {x}} _ {i} & (\ell = 1), \\ \phi (h _ {i}) \mathrm {H e} _ {\ell - 2} (h _ {i}) (\widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {i} & (\ell \geq 2), \end{array} \right.
$$
where  $\phi$  and  $\Phi$  are the PDF and CDF of the standard Gaussian random variable, respectively. Lemma C.1 below provides a counterpart of Lemma B.2 for the BRFA model and the target function (71).
Lemma C.1. Any function  $\widetilde{g}_{\star}$  of form (71) can be expressed exactly as an infinite-head random feature attention model (70)
$$
\widetilde {g} _ {\star} \left(\mathbf {x} _ {0: N}\right) = f _ {\mathbf {v}} ^ {\mathbf {W} _ {0}} \left(\mathbf {x} _ {0: N}\right) = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} _ {\mathbf {W}} \left[ \sigma \left(\left\langle \mathbf {W} + \mathbf {W} _ {0}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \right\rangle\right) \left\langle \mathbf {v} (\mathbf {W}), \widetilde {\mathbf {x}} _ {i} \right\rangle \right], \tag {72}
$$
where the coefficients  $\mathbf{v}(\cdot)$  satisfy
$$
\mathbb {E} _ {\mathbf {W}} \left[ \| \mathbf {v} (\mathbf {W}) \| _ {2} ^ {2} \right] \leq \sum_ {\ell \geq 0} 4 ^ {\ell} \ell ! \| \mathbf {D} _ {\ell} \| _ {\mathrm {F r}} ^ {2}.
$$
Given Lemma C.1, we can get a counterpart of Theorem 1 for the BRFA model as Proposition C.1 below.
Proposition C.1 (Counterpart of Theorem 1 for BRFA). Suppose function  $\widetilde{g}_{\star}:\mathcal{X}\to \mathbb{R}$  takes form (71). Then for any input distribution  $P$  on  $\mathcal{X}$ , with probability at least  $1 - \delta$  (over  $\{\mathbf{W}_m\}_{m\in [M]}$  sampled from (4)), there exists an  $M$ -head BRFA model (15) with coefficients  $\mathbf{V} = \{\mathbf{v}_m\}_{m\in [M]}\subseteq \mathbb{R}^{d + 1}$  that approximates  $\widetilde{g}_{\star}$  in  $L^2 (P)$  up to error
$$
\mathbb {E} _ {\mathbf {x} _ {0: N} \sim P} \left[ \left(\widetilde {g} _ {\star} \left(\mathbf {x} _ {0: N}\right) - f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N}; \mathbf {V}\right)\right) ^ {2} \right] \leq \mathcal {O} \left(\frac {\left(d ^ {2} + \log M\right) B \left(\widetilde {g} _ {\star}\right) \delta^ {- 1}}{M}\right). \tag {73}
$$
$$
\sum_ {m = 1} ^ {M} \| \mathbf {v} _ {m} \| _ {2} \leq \mathcal {O} \left(\sqrt {B (\widetilde {g} _ {\star})} + \sqrt {\frac {B (\widetilde {g} _ {\star}) \delta^ {- 1}}{M}}\right), \quad \sum_ {m = 1} ^ {M} \| \mathbf {v} _ {m} \| _ {2} ^ {2} \leq \mathcal {O} \left(\frac {B (\widetilde {g} _ {\star}) \delta^ {- 1}}{M}\right). \tag {74}
$$
Here  $B(\widetilde{g}_{\star})$  is defined alternatively as
$$
B \left(\widetilde {g} _ {\star}\right) = \sum_ {k = 0} ^ {\infty} C _ {k} \| \mathbf {D} _ {k} \| _ {\mathrm {F r}} ^ {2}, \quad \text {w i t h} \quad C _ {k} = k! 4 ^ {k} \vee 1. \tag {75}
$$
Furthermore, we could approximate the target function in the form (16) to any precision by a function that takes form (72), which is discussed in Lemma C.2.
Lemma C.2. For any target function  $g_{\star}$  in the form of (16), and for any precision  $\varepsilon_{\ell} \coloneqq \frac{1}{2^{\ell + 1}(\ell + 1)!}$ , there exists a function  $f_{\mathbf{v}}^{\mathbf{W}_0}$  in the form of (72) such that
$$
\left\| g _ {\star} - f _ {\mathbf {v}} ^ {\mathbf {W} _ {0}} \right\| _ {\infty} \leq \| g _ {\star} \| _ {\infty} \varepsilon_ {\ell},
$$
$$
\mathbb {E} _ {\mathbf {W}} \left[ \| \mathbf {v} (\mathbf {W}) \| _ {2} ^ {2} \right] \leq \left(\sum_ {p = 0} ^ {\infty} a _ {p} 8 ^ {\ell + \frac {p + 3}{2}} (2 \ell + p) ^ {\frac {p + 3}{2}}\right) ^ {2} \| \mathbf {A} _ {\star} \| _ {\mathbf {F r}} ^ {2},
$$
where the tensor  $\mathbf{A}_{\star} \in \mathbb{R}^{d^5}$  parameterizes  $g_{\star}$ .
C.1.1 Proof of auxiliary lemmas
The function (72) can be interpreted as a linear function of the feature map
$$
\Psi (\mathbf {x} _ {0: N}) = \frac {1}{N} \sum_ {i = 1} ^ {N} \sigma \big (\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \rangle + h _ {i} \big) \widetilde {\mathbf {x}} _ {i}.
$$
So the kernel w.r.t. the feature map takes the form that
$$
\begin{array}{l} K (\mathbf {x} _ {0: N}, \mathbf {x} _ {0: N} ^ {\prime}) \\ = \mathbb {E} _ {\mathbf {W}} [ \langle \Psi (\mathbf {x} _ {0: N}), \Psi (\mathbf {x} _ {0: N} ^ {\prime}) \rangle ] \\ = \frac {1}{N ^ {2}} \sum_ {1 \leq i, j \leq N} \mathbb {E} _ {\mathbf {W}} \left[ \sigma \left(\left\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \right\rangle + h _ {i}\right) \sigma \left(\left\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} ^ {\prime} \widetilde {\mathbf {x}} _ {j} ^ {\prime \top} \right\rangle + h _ {j} ^ {\prime}\right) \right] \left\langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \right\rangle , \\ \end{array}
$$
where  $h_i = \langle \mathbf{x}_0, \mathbf{x}_i \rangle$  and  $h_j' = \langle \mathbf{x}_0', \mathbf{x}_j' \rangle$ . Similar to the proof of Lemma B.2, we also consider a single component of the equation above first, which has the form of
$$
\mathbb {E} _ {\mathbf {W}} \left[ \sigma \left(\left\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \right\rangle + h _ {i}\right) \sigma \left(\left\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} ^ {\prime} \widetilde {\mathbf {x}} _ {j} ^ {\prime \top} \right\rangle + h _ {j} ^ {\prime}\right) \right] \langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \rangle . \tag {76}
$$
We expand  $\sigma\left(\langle \mathbf{W}, \widetilde{\mathbf{x}}_0 \widetilde{\mathbf{x}}_i^\top \rangle + h_i\right)$  by Hermite polynomials in the space  $L^2(\mathbb{R}, \phi)$  using Lemma A.3,
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {W}} \left[ \sigma \left(\left\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \right\rangle + h _ {i}\right) \operatorname {H e} _ {\ell} \left(\left\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \right\rangle\right) \right] \\ = \mathbb {E} _ {z \sim \mathrm {N} (0, 1)} [ \sigma (z + h _ {i}) \mathrm {H e} _ {\ell} (z) ] \\ = \left\{ \begin{array}{c c} h _ {i} \Phi (h _ {i}) + \phi (h _ {i}) & (\ell = 0), \\ \Phi (h _ {i}) & (\ell = 1), \\ (- 1) ^ {\ell} \phi (h _ {i}) \mathrm {H e} _ {\ell - 2} (h _ {i}) & (\ell \geq 2). \end{array} \right. \\ \end{array}
$$
$$
\begin{array}{l} \sigma \left(\left\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \right\rangle + h _ {i}\right) = h _ {i} \Phi (h _ {i}) + \phi (h _ {i}) + \Phi (h _ {i}) \mathrm {H e} _ {1} \left(\left\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \right\rangle\right) \\ + \sum_ {\ell = 2} ^ {\infty} \frac {(- 1) ^ {\ell}}{\ell !} \phi (h _ {i}) \mathrm {H e} _ {\ell - 2} (h _ {i}) \mathrm {H e} _ {\ell} \big (\big <   \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \big > \big). \\ \end{array}
$$
Using Lemma A.1, we obtain the expansion:
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {W}} \left[ \sigma \left(\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \rangle + h _ {i}\right) \sigma \left(\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} ^ {\prime} \widetilde {\mathbf {x}} _ {j} ^ {\top} \rangle + h _ {j} ^ {\prime}\right) \right] \langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \rangle \\ = \left[ h _ {i} \Phi \left(h _ {i}\right) + \phi \left(h _ {i}\right) \right] \left[ h _ {j} ^ {\prime} \Phi \left(h _ {j} ^ {\prime}\right) + \phi \left(h _ {j} ^ {\prime}\right) \right] \langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \rangle \\ + 2 ^ {- 2} \Phi (h _ {i}) \Phi \left(h _ {j} ^ {\prime}\right) \left\langle \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}, \widetilde {\mathbf {x}} _ {0} ^ {\prime} \widetilde {\mathbf {x}} _ {j} ^ {\top} \right\rangle \left\langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \right\rangle \\ + \sum_ {\ell = 2} ^ {\infty} \left\{\frac {1}{\ell !} 2 ^ {- 2 \ell} \phi (h _ {i}) \phi \left(h _ {j} ^ {\prime}\right) \mathrm {H e} _ {\ell - 2} (h _ {i}) \mathrm {H e} _ {\ell - 2} \left(h _ {j} ^ {\prime}\right) \left\langle \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}, \widetilde {\mathbf {x}} _ {0} ^ {\prime} \widetilde {\mathbf {x}} _ {j} ^ {\prime \top} \right\rangle^ {\ell} \right\} \left\langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \right\rangle \\ = \sum_ {\ell = 0} ^ {\infty} \left\langle \varphi_ {\ell} \left(\widetilde {\mathbf {x}} _ {0}, \widetilde {\mathbf {x}} _ {i}\right), \varphi_ {\ell} \left(\widetilde {\mathbf {x}} _ {0} ^ {\prime}, \widetilde {\mathbf {x}} _ {j} ^ {\prime}\right) \right\rangle , \\ \end{array}
$$
$$
\varphi_ {\ell} (\widetilde {\mathbf {x}} _ {0}, \widetilde {\mathbf {x}} _ {i}) = \sqrt {\frac {1}{\ell !}} 2 ^ {- \ell} u _ {\ell} (\widetilde {\mathbf {x}} _ {0}, \widetilde {\mathbf {x}} _ {i}) = \left\{ \begin{array}{c c} [ h _ {i} \Phi (h _ {i}) + \phi (h _ {i}) ] \widetilde {\mathbf {x}} _ {i} & (\ell = 0), \\ 2 ^ {- 1} \Phi (h _ {i}) \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \otimes \widetilde {\mathbf {x}} _ {i} & (\ell = 1), \\ \sqrt {\frac {1}{\ell !}} 2 ^ {- \ell} \phi (h _ {i}) \mathrm {H e} _ {\ell - 2} (h _ {i}) (\widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {i} & (\ell \geq 2). \end{array} \right.
$$
Then by taking summation with respect to  $i$  and  $j$ , we derive the expansion of the kernel:
$$
K (\mathbf {x} _ {0: N}, \mathbf {x} _ {0: N} ^ {\prime}) = \sum_ {\ell = 0} ^ {\infty} \left\langle \frac {1}{N} \sum_ {i = 1} ^ {N} \varphi_ {\ell} (\mathbf {x} _ {0}, \mathbf {x} _ {i}), \frac {1}{N} \sum_ {j = 1} ^ {N} \varphi_ {\ell} (\mathbf {x} _ {0} ^ {\prime}, \mathbf {x} _ {j} ^ {\prime}) \right\rangle .
$$
Then for the target function that takes the form in (71), we have the RKHS norm of  $\widetilde{g}_{\star}$  bounded by:
$$
\| \widetilde {g} _ {\star} \| _ {\mathcal {H} _ {K}} ^ {2} \leq \sum_ {\ell = 0} ^ {\infty} \left(2 ^ {\ell} \sqrt {\ell !}\right) ^ {2} \| \mathbf {D} _ {\ell} \| _ {\mathrm {F r}} ^ {2} = \sum_ {\ell = 0} ^ {\infty} 4 ^ {\ell} \ell ! \| \mathbf {D} _ {\ell} \| _ {\mathrm {F r}} ^ {2}
$$
by the feature equivalence property (23). Thus, there exists coefficients  $\mathbf{v}(\mathbf{W})$  such that
$$
f _ {\mathbf {v}} ^ {\mathbf {W} _ {0}} \left(\mathbf {x} _ {0: N}\right) = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} _ {\mathbf {W}} \left[ \sigma \left(\left\langle \mathbf {W} + \mathbf {W} _ {0}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \right\rangle\right) \left\langle \mathbf {v} (\mathbf {W}), \widetilde {\mathbf {x}} _ {i} \right\rangle \right]
$$
$$
\mathbb {E} _ {\mathbf {W}} \left[ \| \mathbf {v} (\mathbf {W}) \| _ {2} ^ {2} \right] \leq \sum_ {\ell \geq 0} 4 ^ {\ell} \ell ! \| \mathbf {D} _ {\ell} \| _ {\mathrm {F r}} ^ {2},
$$
which proves Lemma C.1.
Note that Lemma B.1 is also applicable to the model BRFA. Combining it with Lemma C.1 proves Proposition C.1.
Firstly, we consider approximating  $x^p$  using the function class  $\{\phi(x)\mathrm{He}_\ell(x)\}_{n \geq 0}$ , which is equivalent to approximating  $x^p\phi^{-1}(x)$  using Hermite polynomials. We take  $p = 0$  first and compute the Hermite expansion of the  $2\ell$ -th-order term in the Taylor expansion of  $\phi^{-1}(x) = e^{x^2/2}$ , i.e.  $\psi_\ell(x) = \sum_{n=0}^\ell \frac{x^{2n}}{2^n n!}$ . Using Lemma A.2, we have that: