SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \psi_ {\ell} (x) = \sum_ {n = 0} ^ {\ell} \frac {x ^ {2 n}}{2 ^ {n} n !} = \sum_ {n = 0} ^ {\ell} \frac {(2 n) !}{2 ^ {n} n !} \sum_ {m = 0} ^ {n} \frac {\mathrm {H e} _ {2 n - 2 m} (x)}{2 ^ {m} m ! (2 n - 2 m) !} \\ = \sum_ {k = 0} ^ {\ell} \mathrm {H e} _ {2 k} (x) \sum_ {m = 0} ^ {\ell - k} \frac {(2 m + 2 k) !}{2 ^ {m + k} (m + k) !} \cdot \frac {1}{2 ^ {m} m ! (2 k) !} \\ =: \sum_ {k = 0} ^ {\ell} c _ {\ell , k} \mathrm {H e} _ {2 k} (x), \\ \end{array}
$$
where  $c_{\ell,k} = \frac{2^k}{(2k)!} \sum_{m=0}^{\ell-k} \frac{(2m+2k)!}{2^{2m+2k}(m+k)!m!}$ . Similarly, for any  $p \geq 0$ , define  $p_0 = \left\lfloor \frac{p}{2} \right\rfloor$  with  $r = p - 2p_0$ . We have that
$$
\begin{array}{l} \psi_ {\ell} (x) x ^ {p} = \sum_ {n = 0} ^ {\ell} \frac {x ^ {2 n + 2 p _ {0} + r}}{2 ^ {n} n !} = \sum_ {n = 0} ^ {\ell} \frac {(2 n + 2 p _ {0} + r) !}{2 ^ {n} n !} \sum_ {m = 0} ^ {n + p _ {0}} \frac {\mathrm {H e} _ {2 n + 2 p _ {0} + r - 2 m} (x)}{2 ^ {m} m ! (2 n + 2 p _ {0} + r - 2 m) !} \\ = \sum_ {k = 0} ^ {\ell + p _ {0}} \mathrm {H e} _ {2 k + r} (x) \sum_ {m = \max  \left(p _ {0} - k, 0\right)} ^ {\ell + p _ {0} - k} \frac {(2 m + 2 k + r) !}{2 ^ {m + k - p _ {0}} (m + k - p _ {0}) !} \cdot \frac {1}{2 ^ {m} m ! (2 k + r) !} \\ =: \sum_ {k = 0} ^ {\ell + p _ {0}} c _ {\ell , k, p _ {0}} \mathrm {H e} _ {2 k + r} (x), \\ \end{array}
$$
where  $c_{\ell,k,p_0} = \sum_{m=\max(p_0-k,0)}^{\ell+p_0-k} \frac{(2m+2k+r)!}{2^{m+k-p_0}(m+k-p_0)!} \cdot \frac{1}{2^mm!(2k+r)!}$ . Then we can bound  $c_{\ell,k,p_0}$  as follows:
$$
\begin{array}{l} c _ {\ell , k, p _ {0}} = \sum_ {m = \max  (p _ {0} - k, 0)} ^ {\ell + p _ {0} - k} \frac {(2 m + 2 k + r) !}{2 ^ {m + k - p _ {0}} (m + k - p _ {0}) !} \cdot \frac {1}{2 ^ {m} m ! (2 k + r) !} \\ \leq \sum_ {m = 0} ^ {\ell + p _ {0} - k} 2 ^ {p _ {0}} \frac {(m + k) !}{(m + k - p _ {0}) !} \frac {(2 m + 2 k + r) !}{2 ^ {m + k} (m + k) !} \cdot \frac {1}{2 ^ {m} m ! (2 k + r) !} \\ \end{array}
$$
$$
\leq 2 ^ {p _ {0}} (\ell + p _ {0}) ^ {p _ {0}} (2 l + p) ^ {r} \sum_ {m = 0} ^ {\ell + p _ {0} - k} \frac {(2 m + 2 k) !}{2 ^ {m + k} (m + k) !} \cdot \frac {1}{2 ^ {m} m ! (2 k) !} \leq (2 \ell + p) ^ {p _ {0} + r} c _ {\ell + p _ {0}, k}.
$$
Next, we give an upper bound of  $c_{\ell,k}$ :
$$
\begin{array}{l} c _ {\ell , k} = \frac {2 ^ {k}}{(2 k) !} \sum_ {m = 0} ^ {\ell - k} \frac {(2 m + 2 k) !}{2 ^ {2 m + 2 k} (m + k) ! m !} \\ = \frac {2 ^ {k}}{(2 k) !} \sum_ {m = 0} ^ {\ell - k} \frac {(2 m + 2 k) !}{2 ^ {2 m + 2 k} (m + k) ! (m + k) !} \cdot \frac {(m + k) !}{m !} \\ \leq \frac {2 ^ {k}}{(2 k) !} \sum_ {m = 0} ^ {\ell - k} \sqrt {\frac {1}{m + k}} \cdot \frac {(m + k) !}{m !} \\ = \frac {2 ^ {k}}{(2 k) !} \sum_ {m = 0} ^ {\ell - k} \sqrt {m + k} \cdot \frac {(m + k - 1) !}{m !} \\ \leq \frac {2 ^ {k}}{(2 k) !} \sqrt {\ell} \sum_ {m = 0} ^ {\ell - k} \frac {(m + k - 1) !}{m !} \\ = \frac {2 ^ {k}}{(2 k) !} \sqrt {\ell} \frac {\ell !}{k (\ell - k) !}. \\ \end{array}
$$
Here we use the inequality that
$$
\sqrt {\frac {2}{\pi (2 n + 1)}} \leq \frac {(2 n) !}{2 ^ {2 n} n ! n !} \leq \sqrt {\frac {1}{2 n}}, \forall n \geq 1.
$$
and we will use it again in the following proof. Since the function
$$
\frac {1}{N} \phi (\langle \mathbf {x} _ {0}, \mathbf {x} _ {i} \rangle) \psi_ {\ell} (\langle \mathbf {x} _ {0}, \mathbf {x} _ {i} \rangle) \sum_ {i = 1} ^ {N} \langle \mathbf {x} _ {0}, \mathbf {x} _ {i} \rangle^ {p} \left\langle \widetilde {\mathbf {x}} _ {i} ^ {\otimes 3} \otimes \widetilde {\mathbf {x}} _ {0} ^ {\otimes 2}, \mathbf {A} _ {\star} \right\rangle
$$
$$
\frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {k = 0} ^ {\ell + p _ {0}} \left\langle u _ {2 k + r + 2} \left(\widetilde {\mathbf {x}} _ {0}, \widetilde {\mathbf {x}} _ {i}\right), c _ {\ell , k, p _ {0}} \mathbf {A} _ {\star} \otimes \mathbf {e} ^ {\otimes (4 k + 2 r)} \right\rangle ,
$$
where  $\mathbf{e} = \mathrm{diag}(0,0,\dots,1)$ . By Lemma C.1, there exists  $\mathbf{v}_{p,\ell}(\mathbf{W})$  s.t.
$$
f _ {\mathbf {v} _ {p, \ell}} ^ {\mathbf {W} _ {0}} (\mathbf {x} _ {0: N}) = \phi (\langle \mathbf {x} _ {0}, \mathbf {x} _ {i} \rangle) \psi_ {\ell} (\langle \mathbf {x} _ {0}, \mathbf {x} _ {i} \rangle) \frac {1}{N} \sum_ {i = 1} ^ {N} \left\langle \mathbf {x} _ {0}, \mathbf {x} _ {i} \right\rangle^ {p} \left\langle \widetilde {\mathbf {x}} _ {i} ^ {\otimes 3} \otimes \widetilde {\mathbf {x}} _ {0} ^ {\otimes 2}, \mathbf {A} _ {\star} \right\rangle
$$
$$
\mathbb {E} _ {\mathbf {W}} \left[ \| \mathbf {v} _ {p, \ell} (\mathbf {W}) \| _ {2} ^ {2} \right] \leq \| \mathbf {A} _ {\star} \| _ {\mathrm {F r}} ^ {2} \sum_ {k = 0} ^ {\ell + p _ {0}} 4 ^ {2 k + r + 2} (2 k + r + 2)! c _ {\ell , k, p _ {0}} ^ {2}.
$$
$$
\begin{array}{l} \sum_ {k = 0} ^ {\ell + p _ {0}} 4 ^ {2 k + r + 2} (2 k + r + 2)! c _ {\ell , k, p _ {0}} ^ {2} \\ \leq \sum_ {k = 0} ^ {\ell + p _ {0}} 4 ^ {2 k + r + 2} (2 k + r + 2)! \left[ (2 \ell + p) ^ {p _ {0} + r} c _ {\ell + p _ {0}, k} \right] ^ {2} \\ \leq (2 \ell + p) ^ {2 p _ {0} + 2 r} \sum_ {k = 0} ^ {\ell + p _ {0}} 4 ^ {2 k + r + 2} (2 k + r + 2)! \left[ \frac {2 ^ {k}}{(2 k) !} \sqrt {\ell + p _ {0}} \frac {(\ell + p _ {0}) !}{k (\ell + p _ {0} - k) !} \right] ^ {2} \\ \end{array}
$$
$$
\leq 4 (2 \ell + p) ^ {2 p _ {0} + 2 r} (2 \ell + 2 p _ {0} + r + 2) \sum_ {k = 0} ^ {\ell_ {p}} 4 ^ {2 k + 2} (2 k + 2)! \left[ \frac {2 ^ {k}}{(2 k) !} \sqrt {\ell_ {p}} \frac {(\ell_ {p}) !}{k (\ell_ {p}) !} \right] ^ {2},
$$
where  $\ell_p = \ell + p_0$ , and by the inequality that
$$
\begin{array}{l} \sum_ {k = 0} ^ {\ell_ {p}} 4 ^ {2 k + 2} (2 k + 2)! \Bigg [ \frac {2 ^ {k}}{(2 k) !} \sqrt {\ell_ {p}} \frac {(\ell_ {p}) !}{k (\ell_ {p}) !} \Bigg ] ^ {2} \\ = \sum_ {k = 0} ^ {\ell_ {p}} 2 ^ {4 k + 4} \ell_ {p} \frac {(2 k + 2) (2 k + 1)}{k ^ {2}} \frac {2 ^ {2 k}}{(2 k) !} \left[ \frac {\ell_ {p} !}{(\ell_ {p} - k) !} \right] ^ {2} \\ \leq \sum_ {k = 0} ^ {\ell_ {p}} 2 ^ {4 k + 4} \ell_ {p} \frac {(2 k + 2) (2 k + 1)}{k ^ {2}} \sqrt {\frac {\pi}{2} (2 k + 1)} \left[ \frac {\ell_ {p} !}{k ! (\ell_ {p} - k) !} \right] ^ {2} \\ \leq 8 \times 2 ^ {4 \ell_ {p} + 4} \ell_ {p} ^ {\frac {3}{2}} \sum_ {k = 0} ^ {\ell} \left[ \frac {\ell_ {p} !}{k ! (\ell_ {p} - k) !} \right] ^ {2} \\ = 4 \sqrt {2} \times 2 ^ {4 \ell_ {p} + 4} \ell_ {p} ^ {\frac {3}{2}} \left( \begin{array}{c} 2 \ell_ {p} \\ \ell_ {p} \end{array} \right) \\ \leq 4 \sqrt {2} \times 2 ^ {4} \ell_ {p} \cdot (6 4) ^ {\ell_ {p}}, \\ \end{array}
$$
we obtain an upper bound:
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {W}} \left[ \| \mathbf {v} _ {p, \ell} (\mathbf {W}) \| _ {2} ^ {2} \right] \leq (2 \ell + p) ^ {2 p _ {0} + 2 r} \| \mathbf {A} _ {\star} \| _ {\mathrm {F r}} ^ {2} \sum_ {k = 0} ^ {\ell + p _ {0}} 4 ^ {2 k + r + 2} (2 k + r + 2)! c _ {\ell + p _ {0}, k} ^ {2} \\ \leq \| \mathbf {A} _ {\star} \| _ {\mathrm {F r}} ^ {2} \times 4 (2 \ell + p) ^ {2 p _ {0} + 2 r} (2 \ell + 2 p _ {0} + r + 2) \times 4 \sqrt {2} \times 2 ^ {4} \ell_ {p} \cdot (6 4) ^ {\ell_ {p}} \\ \leq \left\| \mathbf {A} _ {\star} \right\| _ {\mathrm {F r}} ^ {2} (2 \ell + p) ^ {p + r + 1} 8 ^ {2 \ell + 2 p _ {0} + 3}. \\ \end{array}
$$
Finally, we consider the target function (16)
$$
g _ {\star} (\mathbf {x} _ {0: N}) = \frac {1}{N} \sum_ {i = 1} ^ {N} F (\langle \mathbf {x} _ {0}, \mathbf {x} _ {i} \rangle) G (\mathbf {x} _ {0}, \mathbf {x} _ {i}), \quad F (t) = \sum_ {p = 0} ^ {\infty} a _ {p} t ^ {p}, \quad G (\mathbf {x} _ {0}, \mathbf {x} _ {i}) = \left\langle \widetilde {\mathbf {x}} _ {i} ^ {\otimes 3} \otimes \widetilde {\mathbf {x}} _ {0} ^ {\otimes 2}, \mathbf {A} _ {\star} \right\rangle .
$$
By approximating  $\frac{1}{N}\sum_{i = 1}^{N}a_{p}\langle \mathbf{x}_{0},\mathbf{x}_{i}\rangle^{p}G(\mathbf{x}_{0},\mathbf{x}_{i})$  separately and adding the approximation functions together, we obtain a  $\mathbf{v}_{\ell}$  s.t.  $f_{\mathbf{V}_{\ell}}^{\mathbf{W}_0}(\mathbf{x}_{0:N}) = \phi (\langle \mathbf{x}_0,\mathbf{x}_i\rangle)\psi_\ell (\langle \mathbf{x}_0,\mathbf{x}_i\rangle)g_\star (\mathbf{x}_{0:N})$  and
$$
\mathbb {E} _ {\mathbf {W}} \Big [ \| \mathbf {v} _ {p, \ell} (\mathbf {W}) \| _ {2} ^ {2} \Big ] \leq \left(\sum_ {p = 0} ^ {\infty} a _ {p} 8 ^ {\ell + p / 2 + 3 / 2} (2 \ell + p) ^ {\frac {p + 3}{2}}\right) ^ {2} \| \mathbf {A} _ {\star} \| _ {\mathsf {F r}} ^ {2} = 8 ^ {3} B (g _ {\star}, L)
$$
and  $\left\| g_{\star} - f_{\mathbf{v}_{\ell}}^{\mathbf{W}_0}\right\|_{\infty} = \left\| (\phi \psi_{\ell} - 1)g_{\star}\right\|_{\infty}\leq \left\| \phi \psi_{\ell} - 1\right\|_{\infty}\left\| g_{\star}\right\|_{\infty}\leq \frac{e}{2^{\ell + 1}(\ell + 1)!}\left\| g_{\star}\right\|_{\infty} = \left\| g_{\star}\right\|_{\infty}\varepsilon_{\ell}$ . This finishes the proof of Lemma C.2.
Proof. For any  $L > 0$ , using Lemma C.2, we can find a function  $f_{\mathbf{v}L}^{\mathbf{W}_0}$  such that  $\| g_\star - f_{\mathbf{v}L}^{\mathbf{W}_0} \|_\infty \leq \| g_\star \|_\infty \varepsilon_L$  with  $B(f_{\mathbf{v}L}^{\mathbf{W}_0}) \leq 8^3 B(g_\star, L)$ . Follow the same manner as the proof of Theorem 2. We can get that
$$
\begin{array}{l} L _ {D} \left(\widehat {f} _ {M} ^ {\mathbf {W}, \mathbf {W} _ {0}}\right) - L _ {D} \left(f _ {\mathbf {v} _ {L}} ^ {\mathbf {W} _ {0}}\right) \\ \lesssim \sqrt {B (g _ {\star} , L)} \left[ \sqrt {\frac {\log (d M) \log (n N M) + \log (\delta^ {- 1})}{n}} + \sqrt {\frac {(d ^ {2} + \log M) \delta^ {- 1}}{M}} \right]. \\ \end{array}
$$
$$
\begin{array}{l} L _ {D} \left(\widehat {f} _ {M} ^ {\mathbf {W}, \mathbf {W} _ {0}}\right) - L _ {D} \left(g _ {\star}\right) \\ = L _ {D} \left(\widehat {f} _ {M} ^ {\mathbf {W}, \mathbf {W} _ {0}}\right) - L _ {D} \left(f _ {\mathbf {v} _ {L}} ^ {\mathbf {W} _ {0}}\right) + L _ {D} \left(f _ {\mathbf {v} _ {L}} ^ {\mathbf {W} _ {0}}\right) - L _ {D} \left(g _ {\star}\right) \\ \lesssim \sqrt {B (g _ {\star} , L)} \left[ \sqrt {\frac {\log (d M) \log (n N M) + \log (\delta^ {- 1})}{n}} + \sqrt {\frac {(d ^ {2} + \log M) \delta^ {- 1}}{M}} \right] + \varepsilon_ {L} \| g _ {\star} \| _ {\infty}. \tag {77} \\ \end{array}
$$
Taking infimum over  $L$  proves Theorem 3.
C.3 Proof of Examples in Section 4
Proposition C.2 (Restatement of Example 4). Consider the target function
$$
g _ {\star} = \frac {1}{N} \sum_ {i = 1} ^ {N} \langle \mathbf {x} _ {i} ^ {\otimes 3} \otimes \mathbf {x} _ {0} ^ {\otimes 2}, \mathbf {A} \rangle .
$$
It has norm bound  $B(g_{\star}, L) = \|\mathbf{A}\|_{\mathbb{F}_r}^2 L^{38^{2L}}$ . So for any  $\eta > 0$ , if we take  $n \geq \exp(\exp(\Theta(1/\eta)))$ ,  $L = \Theta((1 + \log \log n)^{-1} \log n)$ , and  $M = \Theta(d^2 n)$ , the excess risk will scale as  $\widetilde{\mathcal{O}}(\sqrt{\|\mathbf{A}\|_{\mathbb{F}_r}^2 / n^{1-\eta}})$ .
Proof of Proposition C.2. The value of  $B(g_{\star}, L)$  can be obtained by definition and by direct calculation. As for the second part of the proposition, take  $L = r \log n$ , where  $r > 0$  is a parameter that is to be chosen to minimize the excess risk. Eq. (17) becomes
$$
\widetilde {\mathcal {O}} \left(\| \mathbf {A} \| _ {\mathrm {F r}} r ^ {3 / 2} n ^ {3 r - 1 / 2} + n ^ {- r \log (2 / e) - r \log r - r \log \log n}\right), \tag {78}
$$
where  $\tilde{\mathcal{O}}$  hides all the logarithm factors and constants of  $n, d$ , and  $M$ . To minimize the excess risk, we need to make the two terms in (78) have the same scale. So we set  $3r - 1/2 = -r\log(2/e) - r\log r - r\log\log n$ . Denoting the solution as  $r_{\star}$ , we get that
$$
r _ {\star} = \frac {1 / 2 - r _ {\star} \log r _ {\star}}{\log (2 e ^ {2}) + \log \log n}.
$$
So this gives  $r_{\star} < (1/2 + 1/e)(\log(2e^2) + \log \log n)^{-1}$ . Assume  $r_{\star} < 1$  (otherwise  $3r_{\star} > 1/2$  and the excess risk is meaningless), then  $r_{\star} > (2\log(2e^2) + 2\log \log n)^{-1}$ . Therefore, we get that  $r_{\star} = C(1 + \log \log n)^{-1}$ , with  $C$  being a constant. As a result, when choosing  $L = r_{\star}\log n = \Theta((1 + \log \log n)^{-1}\log n)$ , the excess risk Eq. (78) scales as  $\widetilde{\mathcal{O}}(\|\mathbf{A}\|_{\mathrm{Fr}}n^{3r_{\star} - \frac{1}{2}}) = \widetilde{\mathcal{O}}(\|\mathbf{A}\|_{\mathrm{Fr}}n^{C/(1 + \log \log n) - \frac{1}{2}})$ . As a result, let  $n > \exp(\exp(C/\eta - 1))$  where  $C$  is a constant, we get an excess risk that scales as  $\widetilde{\mathcal{O}}(\sqrt{\|\mathbf{A}\|_{\mathrm{Fr}}^2/n^{1-\eta}})$ . This finishes the proof of Proposition C.2.
Proposition C.3 (Restatement of Example 5). Consider the target function
$$
g _ {\star} = \frac {1}{N} \sum_ {i = 1} ^ {N} \left\langle \mathbf {x} _ {0}, \mathbf {x} _ {i} \right\rangle^ {p} \left\langle \boldsymbol {\beta}, \mathbf {x} _ {i} \right\rangle , \quad \boldsymbol {\beta} \in \mathbb {S} ^ {d - 1}.
$$
It has  $B(g_{\star}, L) = (2L + p)^{p + 3}8^{2L + p}$ . For any  $\eta > 0$ , choosing the same parameters  $(n, L, M)$  as Example 4, the excess risk bound scales as  $\tilde{\mathcal{O}}(\sqrt{(\log n + p)^{(p + 3)}8^p / n^{1 - \eta}})$ .
Furthermore, to reach an accuracy of 0.01, the BRFA model requires  $n_{\star} = \widetilde{\mathcal{O}}((8p + 48)^{p+3})$ , whereas the RFA model requires  $n_{\star} = \widetilde{\mathcal{O}}((4d)^p)$ .
Proof of Proposition C.3. The value of  $B(g_{\star},L)$  can be obtained by direct calculation, and we use the same method as the proof of Proposition C.2 to get the  $\widetilde{\mathcal{O}} (\sqrt{1 / n^{1 - \eta}})$  excess risk. To reach an accuracy of 0.01, we can set  $L = 3$  , note that  $\varepsilon_L\| g_\star \|_\infty <  0.006$  . Therefore we just need to choose  $n_\star$  s.t.  $n > \widetilde{\mathcal{O}} ((6 + p)^{p + 3}8^{6 + p})$  , and this gives  $n_\star = \widetilde{\mathcal{O}} ((8p + 48)^{p + 3})$  for the BRFA model. Direct calculation using Theorem 2 gives the value of  $n_\star$  for the model RFA. This finishes the proof of Proposition C.3.
Proposition C.4 (Restatement of Example 6). Consider the target function that has the form
$$
g _ {\star} = \frac {1}{N} \sum_ {i = 1} ^ {N} \cos (\langle \mathbf {x} _ {0}, \mathbf {x} _ {i} \rangle) \left\langle \mathbf {x} _ {i} ^ {\otimes 3}, \mathbf {G} \right\rangle .
$$
It has  $B(g_{\star}, L) = \Theta((8e)^{2L})$ . For any  $\eta > 0$ , choosing the same parameters as Example 4, the excess risk bound scales as  $\widetilde{\mathcal{O}}(\sqrt{1 / n^{1 - \eta}})$ .
Furthermore, to reach an accuracy of 0.01. The BRFA model requires  $n_{\star} = \widetilde{\mathcal{O}}(1)$ , whereas the RFA model requires  $n_{\star} = \widetilde{\mathcal{O}}(\mathrm{Poly}(d)\exp(\sqrt{d}))$ .
Proof of Proposition C.4. We use the expansion of the cos function,
$$
g _ {\star} = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {k = 0} ^ {\infty} \frac {(- 1) ^ {k}}{(2 k) !} \left\langle \mathbf {x} _ {0}, \mathbf {x} _ {i} \right\rangle^ {2 k} \left\langle \mathbf {x} _ {i} ^ {\otimes 3}, \mathbf {G} \right\rangle .
$$
Then use the Eq. (17), we get that
$$
B (g _ {\star}, L) = \sum_ {k = 0} ^ {\infty} \frac {(2 L + 2 k) ^ {k + 3 / 2} 8 ^ {L + k}}{(2 k) !} \leq \sum_ {k} \frac {(2 k + 3) ^ {k + 3 / 2} 8 ^ {k}}{(2 k) !} (8 e) ^ {L} = \Theta ((8 e) ^ {L}).
$$
This gives the formula for  $B(g_{\star}, L)$ . To reach an accuracy of 0.01, we set  $L = 3$ . Then  $B(g_{\star}, L) = \Theta(1)$ . So BRFA needs  $n_{\star} = \widetilde{\mathcal{O}}(1)$ . And Theorem 2 and Example 3 show that RFA model would require  $n_{\star} = \widetilde{\mathcal{O}}(\mathrm{Poly}(d)\exp(\sqrt{d}))$ . This finishes the proof of Proposition C.4.
In addition to Section 5, we perform further simulations to examine our theory upon the approximation power of RFA (5), BRFA (15), and RFMLP (14). Besides the target function (21), we consider two additional target functions of the form
$$
f _ {3, p} \left(\mathbf {x} _ {0: N}\right) = \langle \boldsymbol {\beta}, \mathbf {x} _ {0} \rangle^ {p}, \quad p \in \mathbb {N}, \quad \boldsymbol {\beta} \in \mathbb {S} ^ {d - 1}, \tag {79}
$$
$$
f _ {4, \gamma} (\mathbf {x} _ {0: N}) = \frac {1}{N} \sum_ {i = 1} ^ {N} \left\langle \mathbf {x} _ {0}, \mathbf {S} \mathbf {x} _ {i} \right\rangle^ {3} \left\langle \boldsymbol {\beta}, \mathbf {x} _ {i} \right\rangle , \quad \boldsymbol {\beta} \in \mathbb {S} ^ {d - 1}, \quad \mathbf {S} = \boldsymbol {Z} + \gamma I _ {d}. \tag {80}
$$
The target function (79) is a specific instance of Example 1, and the target function (80) is a specific instance of Example 3. In (80), we sample  $Z_{ij} \sim_{\mathrm{id}} \mathsf{N}(0,1 / d)$  for  $(i,j) \in [d]^2$  and vary  $\gamma$  in the experiment. Other experimental settings are the same as in Section 5.
Figure 3 demonstrates the effect of sequence length  $N$  on the performance of three RF models when fitting the target function  $f_{3,2}$  (79), which solely depends on  $x_0$ . We observe that a larger sequence length  $N$  leads to a larger separation of the test error between RFMLP and the other two random-feature attention models. This result aligns with our sample complexity analysis detailed in Example 1, where the sample complexity bound of RFMLP for learning average of functions of  $\mathbf{x}_i$  is found to be  $\mathcal{O}((N / 4)^p)$  times greater than that of RFA.
Figure 4 demonstrates the performance comparison between RFA and BRFA under different choices of the token dimension  $d$ . As we can see from the left panel of Figure 4, in the case of  $d = 4$ , RFA outperforms BRFA for large sample size, which may result from that BRFA has slower convergence rate with respect to the sample size  $n$  as we state in Example 5. In the middle and the right panel (i.e., when  $d$  is larger), BRFA exceeds RFA, and the largest separation lies in the case of  $d = 32$ . This result is consistent with our analyses that BRFA saves a Poly(d) factor in the sample complexity for  $d \gg p$ .
Figure 5 demonstrates that BRFA has no advantage in approximating the target function  $f_{4,\gamma}$  for  $\gamma = 0$  (left panel; i.e.,  $(S_{ij}\sim_{\mathrm{iid}}\mathsf{N}(0,1 / d)$ , for  $(i,j)\in [d]^2$ ). However, as  $\gamma$  increases, BRFA outperforms RFA and their separation increases with a larger  $\gamma$  (middle and right panel). Notice that  $\lim_{\gamma \to \infty}\langle \mathbf{x}_0,\mathbf{S}\mathbf{x}_i\rangle /\gamma = \langle \mathbf{x}_0,\mathbf{x}_i\rangle$ . So this result also conforms to our analysis that BRFA is adept at approximating functions of correlations as in Example 5.
In conclusion, RFA and BRFA have similar performance in fitting functions without correlation structure, such as (20) and (79), and RFA may behave even better in some cases. However, BRFA is presumably more powerful than RFA in approximating functions of correlations.
D.1 Weight matrices in BERT
As noted in Section 4, we plot the query-key matrices (weight matrices) of the BERT-Base model [29] and show that many query-key matrices are diagonally dominated. The BERT-Base model has 12 attention layers with 12 heads in each layer. Denote the query matrix in the  $i$ -th head of  $j$ -th layer as  $Q_{ij} \in \mathbb{R}^{768 \times 64}$  and the key matrix as  $K_{ij} \in \mathbb{R}^{768 \times 64}$ . We compute  $W_{ij} = \sqrt{768} \cdot Q_{ij} K_{ij}^{\top}$  for  $i, j$  from 1 to 12, and then take the absolute value for each entry of the weight matrices  $W_{ij}$ . Figure 1
<sup>6</sup>Downloaded from https://huggingface.co/bert-base-uncased.
Figure 3: Test error of three RF models for learning  $f_{3,2}$  (79). We fix  $d = 16$  while varying  $N = 4$  (left), 8 (middle), and 16 (right). The other settings are the same as in Figure 2.
Figure 4: Test error of three RF models for learning  $f_{2,3}$  (21). We fix  $N = 16$  while vary  $d = 8$  (left), 16 (middle), and 32 (right). For a fair comparison among RFA, BRFA, and RFMLP, the number of heads of RFMLP is taken to be  $M_{\mathrm{RFMLP}} = 9000$  (left), 17000 (middle), and 33000 (right). The other settings are the same as in Figure 2.
Figure 5: Test error of three RF models for learning  $f_{4,\gamma}$  (80). We choose  $\gamma = 0$  (left), 2 (middle), and 8 (right). The other settings are the same as in Figure 2.
shows the heat maps of weight matrices of the 2nd, 5th, 8th, and 11th layers, where all matrices are clipped to the top-left  $32 \times 32$  block. As we can see, a lot of weight matrices are diagonally dominated. We remark that a very recent and concurrent paper [84] observed similar phenomena for the ViT-Tiny model. They further show that diagonally dominated weight initialization of self-attention layers allows training transformers faster and obtaining higher final accuracies on CIFAR-10 and ImageNet datasets.