SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[82] D. A. Tarzanagh, Y. Li, X. Zhang, and S. Oymak. Max-Margin Token Selection in Attention Mechanism, Sept. 2023. URL http://arxiv.org/abs/2306.13596. arXiv:2306.13596 [cs, math].
[83] T. Tirer, J. Bruna, and R. Giryes. Kernel-based smoothness analysis of residual networks. In Mathematical and Scientific Machine Learning, pages 921–954. PMLR, 2022.
[84] A. Trockman and J. Z. Kolter. Mimetic initialization of self-attention layers. arXiv preprint arXiv:2305.09828, 2023.
[85] J. A. Tropp et al. An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning, 8(1-2):1-230, 2015.
[86] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[87] R. Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[88] J. von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vlademyrov. Transformers learn in-context by gradient descent. arXiv preprint arXiv:2212.07677, 2022.
[89] M. J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019.
[90] C. Wei, J. D. Lee, Q. Liu, and T. Ma. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. Advances in Neural Information Processing Systems, 32, 2019.
[91] C. Wei, Y. Chen, and T. Ma. Statistically meaningful approximation: a case study on approximating Turing machines with transformers. Advances in Neural Information Processing Systems, 35:12071-12083, 2022.
[92] E. Weinan, C. Ma, and L. Wu. Barron spaces and the compositional function spaces for neural network models. arXiv preprint arXiv:1906.08039, 2019.
[93] E. Weinan, C. Ma, and L. Wu. A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics. Sci. China Math, 2019.
[94] G. Weiss, Y. Goldberg, and E. Yahav. Thinking like transformers. In International Conference on Machine Learning, pages 11080-11090. PMLR, 2021.
[95] M. Wortsman, J. Lee, J. Gilmer, and S. Kornblith. Replacing softmax with relu in vision transformers, 2023.
[96] S. M. Xie, A. Raghunathan, P. Liang, and T. Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080, 2021.
[97] K. Xu, M. Zhang, J. Li, S. S. Du, K.-i. Kawarabayashi, and S. Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. arXiv preprint arXiv:2009.11848, 2020.
[98] G. Yang. Tensor programs ii: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006.14548, 2020.
[99] S. Yao, B. Peng, C. Papadimitriou, and K. Narasimhan. Self-attention networks can process bounded hierarchical languages. arXiv preprint arXiv:2105.11115, 2021.
[100] G. Yehudai and O. Shamir. On the power and limitations of random features for understanding neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[101] C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877-28888, 2021.
[102] C. Yun, S. Bhojanapalli, A. S. Rawat, S. J. Reddi, and S. Kumar. Are transformers universal approximators of sequence-to-sequence functions? arXiv preprint arXiv:1912.10077, 2019.
[103] Y. Zhang, A. Backurs, S. Bubeck, R. Eldan, S. Gunasekar, and T. Wagner. Unveiling transformers with lego: a synthetic reasoning task. arXiv preprint arXiv:2206.04301, 2022.
[104] D. Zou, Y. Cao, D. Zhou, and Q. Gu. Gradient descent optimizes over-parameterized deep relu networks. Machine learning, 109:467-492, 2020.
A.1 Basics on Hermite Polynomials
In this section, we briefly review Hermite polynomials and their properties. Let  $\mathrm{He}_n$  be the probabilists' Hermite polynomial of degree  $n$ :
$$
\mathrm {H e} _ {n} (x) := (- 1) ^ {n} e ^ {\frac {x ^ {2}}{2}} \frac {d ^ {n}}{d x ^ {n}} e ^ {- \frac {x ^ {2}}{2}}.
$$
Here are some basic properties of Hermite polynomials  $\{\mathrm{He}_n\}_{n\geq 0}$ :
- Let  $(x,y) \sim \mathsf{N}(\mathbf{0}, [1, \rho; \rho, 1])$ . Then  $\mathbb{E}[\mathrm{He}_m(x)\mathrm{He}_n(y)] = n! \rho^n 1_{m=n}$  for  $m, n \geq 0$ .
-  $\mathrm{He}_n(-x) = (-1)^n\mathrm{He}_n(x)$ ;
-  $\mathrm{He}_{n+1}(x) = x\mathrm{He}_n(x) - \mathrm{He}_n'(x)$ .
-  $\mathrm{He}_n'(x) = n\mathrm{He}_{n - 1}(x)$ .
It can be shown that Hermite polynomials are a complete orthogonal basis of the following Hilbert space with  $\phi(x) = (2\pi)^{-1/2} e^{-x^2/2}$ :
$$
L ^ {2} (\mathbb {R}, \phi (x)) := \left\{f: \mathbb {R} \to \mathbb {R}: \int_ {- \infty} ^ {\infty} f (x) ^ {2} \phi (x) \mathrm {d} x <   \infty \right\}.
$$
For two functions  $f, g: \mathbb{R} \to \mathbb{R}$ , we define their inner product  $\langle f, g \rangle$  as:
$$
\langle f, g \rangle := \mathbb {E} _ {x \sim \mathrm {N} (0, 1)} [ f (x) g (x) ] = \int_ {- \infty} ^ {\infty} f (x) g (x) \phi (x) \mathrm {d} x.
$$
For any function  $f \in L^{2}(\mathbb{R},\phi (x))$ , we can derive its Hermite expansion:
$$
f (x) = \sum_ {n = 0} ^ {\infty} \frac {a _ {n}}{n !} \mathrm {H e} _ {n} (x),
$$
where  $a_{n} = \langle f, \mathrm{He}_{n} \rangle = \int_{-\infty}^{\infty} f(x) \mathrm{He}_{n}(x) \phi(x) \, \mathrm{d}x$ . Then for another function  $g(x) = \sum_{n=0}^{\infty} \frac{b_{n}}{n!} \mathrm{He}_{n}(x)$ , the inner product of  $f$  and  $g$  gives:
$$
\langle f, g \rangle = \sum_ {n = 0} ^ {\infty} \frac {a _ {n} b _ {n}}{n !}.
$$
Here are some formulae for Hermite expansion of certain functions.
Lemma A.1 (See e.g., [38]). Let  $f, g \in L^{2}(\mathbb{R}, \phi(x))$ . Then, for any unit vectors  $u, v \in \mathbb{R}^{d}$ , we have that
$$
\mathbb {E} _ {x \sim \mathcal {N} (0, \mathbf {I} _ {d})} \big [ f (u ^ {\top} x) g (v ^ {\top} x) \big ] = \sum_ {n = 0} ^ {\infty} \frac {a _ {n} b _ {n}}{n !} \langle u, v \rangle^ {n},
$$
where  $f(x) = \sum_{n=0}^{\infty} \frac{a_n}{n!} \mathrm{He}_n(x)$  and  $g(x) = \sum_{n=0}^{\infty} \frac{b_n}{n!} \mathrm{He}_n(x)$ .
Lemma A.2 (Inverse explicit expression [69]). Hermite expansion of  $x^n$  gives:
$$
x ^ {n} = n! \sum_ {m = 0} ^ {\lfloor \frac {n}{2} \rfloor} \frac {1}{2 ^ {m} m ! (n - 2 m) !} \mathrm {H e} _ {n - 2 m} (x).
$$
Lemma A.3. Let  $\sigma_c(x) \coloneqq \sigma(x + c)$  be the shifted ReLU function. Then the Hermite expansion of  $\sigma_c$  gives:
$$
\sigma_ {c} (x) = c \Phi (c) + \phi (c) + \Phi (c) \mathrm {H e} _ {1} (x) + \sum_ {n = 2} ^ {\infty} \frac {(- 1) ^ {n}}{n !} \phi (c) \mathrm {H e} _ {n - 2} (c) \mathrm {H e} _ {n} (x),
$$
where  $\phi (x),\Phi (x)$  are the PDF and CDF of standard Gaussian.
Proof of Lemma A.3. Denote  $a_{n,i} \coloneqq \int_{-c}^{\infty}(x + c)^{i}\mathrm{He}_{n}(x)\phi (x)dx$ . It is easy to obtain that
$$
a _ {0, 0} = \Phi (c), \quad a _ {0, 1} = c \Phi (c) + \phi (c), \quad a _ {1, 0} = \phi (c), \quad \text {a n d} \quad a _ {1, 1} = \Phi (c).
$$
Then using the two different formulae for  $a_{n,0}$
$$
\begin{array}{l} a _ {n, 0} = \int_ {- c} ^ {\infty} \operatorname {H e} _ {n} (x) \phi (x) \mathrm {d} x = - \frac {\operatorname {H e} _ {n + 1} (- c)}{n + 1} \phi (- c) + \int_ {- c} ^ {\infty} \frac {\operatorname {H e} _ {n + 1} (x)}{n + 1} x \phi (x) \mathrm {d} x \\ = - \frac {\mathrm {H e} _ {n + 1} (- c)}{n + 1} \phi (- c) + \frac {1}{n + 1} a _ {n + 1, 1} - \frac {c}{n + 1} a _ {n + 1, 0}, \quad n \geq 0, \quad \text {a n d} \\ \end{array}
$$
$$
\begin{array}{l} a _ {n, 0} = \int_ {- c} ^ {\infty} (x \mathrm {H e} _ {n - 1} (x) - (n - 1) \mathrm {H e} _ {n - 2} (x)) \phi (x) \mathrm {d} x \\ = a _ {n - 1, 1} - c a _ {n - 1, 0} - (n - 1) a _ {n - 2, 0}, \quad n \geq 2, \\ \end{array}
$$
$$
\begin{array}{l} a _ {n, 1} = (- 1) ^ {n} \mathrm {H e} _ {n} (c) \phi (c) + c a _ {n - 1, 1} + (n - c ^ {2}) a _ {n - 1, 0} - (n - 1) c a _ {n - 2, 0}, \quad \text {a n d} \\ a _ {n, 0} = a _ {n - 1, 1} - c a _ {n - 1, 0} - (n - 1) a _ {n - 2, 0}, \quad n \geq 2. \\ \end{array}
$$
Then it is easy to prove by induction that
$$
\begin{array}{l} a _ {n, 1} = (- 1) ^ {n} \mathrm {H e} _ {n - 2} (c) \phi (c), \quad n \geq 2, \\ a _ {n, 0} = (- 1) ^ {n - 1} \mathrm {H e} _ {n - 1} (c) \phi (c), \quad n \geq 1. \\ \end{array}
$$
A.2 Basics on Random Features
In this section, we give some basic properties of the random feature model considered in our work, which can be seen as an extension of the standard random feature model (e.g. of Rahimi and Recht [74, 75]) to the vector-valued case.
Given a functional  $\sigma(x; w): \mathcal{X} \times \mathcal{W} \to \mathbb{R}^d$ . Denote  $\mu$  as a probability measure on  $\mathcal{W}$ . We define the (infinite-width) random feature model as:
$$
\mathcal {F} = \{f: f (x) = \langle \mathbf {v}, \boldsymbol {\sigma} (x; \cdot) \rangle_ {\mathcal {H} _ {\mathcal {W}}}, \quad \mathbf {v} \in \mathcal {H} _ {\mathcal {W}} \}, \tag {22}
$$
where  $\mathcal{H}_{\mathcal{W}} = \{\mathbf{v}(w):\int_{\mathcal{W}}\mathbf{v}(w)^{\top}\mathbf{v}(w)\mu (\mathrm{d}w) < \infty \}$  is a Hilbert space with norm  $\| \mathbf{v}\|_{\mathcal{H}_w}^2 = \int_{\mathcal{W}}\mathbf{v}(w)^{\top}\mathbf{v}(w)\mu (\mathrm{d}w)$  and inner product  $\langle \mathbf{v},\mathbf{u}\rangle_{\mathcal{H}_w} = \int_{\mathcal{W}}\mathbf{v}(w)^{\top}\mathbf{u}(w)\mu (\mathrm{d}w)$ . Besides, we endow  $\mathcal{F}$  with a norm  $\| \cdot \|_{\mathcal{F}}$  and the corresponding inner product  $\langle \cdot ,\cdot \rangle_{\mathcal{F}}$  defined as:
$$
\| f \| _ {\mathcal {F}} = \inf _ {f = \langle \mathbf {v}, \pmb {\sigma} (\cdot) \rangle_ {\mathcal {H} _ {\mathcal {W}}}} \| \mathbf {v} \| _ {\mathcal {H} _ {\mathcal {W}}}, \quad \langle f, g \rangle_ {\mathcal {F}} = \frac {\| f + g \| _ {\mathcal {F}} ^ {2} - \| f - g \| _ {\mathcal {F}} ^ {2}}{4}.
$$
We further define the corresponding reproducing kernel  $K:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ , s.t.
$$
K (x, y) = \int_ {\mathcal {W}} \boldsymbol {\sigma} (x; w) ^ {\top} \boldsymbol {\sigma} (y; w) \mu (\mathrm {d} w),
$$
which is positive definite. Define the RKHS induced by this kernel as  $\mathcal{H}_K$  with corresponding norm  $\| \cdot \|_{\mathcal{H}_K}$  and the inner product  $\langle \cdot ,\cdot \rangle_{\mathcal{H}_K}$ . Then we have the following proposition according to [61]:
Proposition A.1. Given the above definition of  $\mathcal{F}$  and  $\mathcal{H}_K$ , we have that  $(\mathcal{F},\| \cdot \|_{\mathcal{F}}) = (\mathcal{H}_K,\| \cdot \|_{\mathcal{H}_K})$
More generally [11], for any feature map  $\phi : \mathcal{X} \to \mathcal{H}$  (where  $\mathcal{H}$  is a Hilbert space) that induces the kernel  $K$ , i.e.,  $K(x, y) = \langle \phi(x), \phi(y) \rangle_{\mathcal{H}}$ , we have for any function  $f$  that
$$
\| f \| _ {\mathcal {H} _ {K}} = \inf  _ {f = \langle \mathbf {u}, \phi (\cdot) \rangle_ {\mathcal {H}}} \| \mathbf {u} \| _ {\mathcal {H}}, \tag {23}
$$
which shows the equivalence among different feature maps that generate the same kernel.
A.3 Concentration inequalities
Definition A.1 (Sub-Gaussian and Sub-Exponential random variables [87]). For a random variable  $X$ , its sub-gaussian norm, denoted  $\| X\|_{\psi_2}$ , is defined as
$$
\left\| X \right\| _ {\psi_ {2}} = \inf  \left\{t > 0: \mathbb {E} \exp \left(X ^ {2} / t ^ {2}\right) \leq 2 \right\}.
$$
If  $\sigma \equiv \| X\|_{\psi_2} < \infty$ , we say that  $X$  is  $\sigma$ -sub-Gaussian.
For a random variable  $X$ , its sub-exponential norm, denoted  $\| X\|_{\psi_1}$ , is defined as
$$
\left\| X \right\| _ {\psi_ {1}} = \inf  \left\{t > 0: \mathbb {E} \exp (| X | / t) \leq 2 \right\}.
$$
If  $\sigma \equiv \| X\|_{\psi_1} < \infty$ , we say that  $X$  is  $\sigma$ -sub-exponential.
Theorem A.1 (Gaussian concentration inequality (e.g., [89])). Let  $(X_{1},\ldots ,X_{n})$  be a vector of i.i.d. standard Gaussian variables, and let  $f:\mathbb{R}^n\to \mathbb{R}$  be L-Lipschitz with respect to the Euclidean norm. Then the random variable  $f(X) - \mathbb{E}[f(X)]$  is  $L$ -sub-Gaussian, and hence
$$
\mathbb {P} \big (| f (X) - \mathbb {E} [ f (X) ] | \geq t \big) \leq 2 e ^ {- \frac {t ^ {2}}{2 L ^ {2}}} \quad f o r a l l t \geq 0.
$$
Theorem A.2 (Bounded difference inequality (e.g., [89])). Consider a function  $f(X): \mathbb{R}^n \to \mathbb{R}$ . Assume that for any  $X = (X_1, \ldots, X_n)$  and  $X^{i','} = (X_1, \ldots, X_i', \ldots, X_n)$ , we have difference bound  $|f(X) - f(\bar{X}^{i','})| \leq L_i$ . We further assume that the random vector  $X = (X_1, X_2, \ldots, X_n)$  has independent components. Then
$$
\mathbb {P} \big (| f (X) - \mathbb {E} [ f (X) ] | \geq t \big) \leq 2 e ^ {- \frac {2 t ^ {2}}{\sum_ {k = 1} ^ {n} L _ {k} ^ {2}}} \quad f o r a l l t \geq 0.
$$
Theorem A.3 (Matrix Bernstein Inequality (e.g., [85])). Consider a sequence  $\{S_k\}_{k \in [n]}$  of independent random matrices with common dimension  $d_1 \times d_2$ . Assume that
$$
\mathbb {E} \boldsymbol {S} _ {k} = \mathbf {0} \quad a n d \| \boldsymbol {S} _ {k} \| _ {\mathrm {o p}} \leq L \text {a l m o s t s u r e l y , f o r e a c h i n d e x k}.
$$
Introduce the random matrix
$$
Z = \sum_ {k = 1} ^ {n} S _ {k}.
$$
Let  $v(Z)$  be the matrix variance statistic of the sum
$$
\begin{array}{l} v (\boldsymbol {Z}) = \max  \left\{\left\| \mathbb {E} \left(\boldsymbol {Z} \boldsymbol {Z} ^ {\top}\right) \right\| _ {\mathrm {o p}}, \left\| \mathbb {E} \left(\boldsymbol {Z} ^ {\top} \boldsymbol {Z}\right) \right\| _ {\mathrm {o p}} \right\} \\ = \max  \left\{\left\| \sum_ {k = 1} ^ {n} \mathbb {E} \left(\boldsymbol {S} _ {k} \boldsymbol {S} _ {k} ^ {\top}\right) \right\| _ {\text {o p}}, \left\| \sum_ {k = 1} ^ {n} \mathbb {E} \left(\boldsymbol {S} _ {k} ^ {\top} \boldsymbol {S} _ {k}\right) \right\| _ {\text {o p}} \right\}. \\ \end{array}
$$
$$
\mathbb {E} \| \boldsymbol {Z} \| _ {\mathrm {o p}} \leq \sqrt {2 v (\boldsymbol {Z}) \log \left(d _ {1} + d _ {2}\right)} + \frac {1}{3} L \log \left(d _ {1} + d _ {2}\right),
$$
and for all  $t \geq 0$ ,
$$
\mathbb {P} \left(\| \boldsymbol {Z} \| _ {\mathrm {o p}} \geq t\right) \leq \left(d _ {1} + d _ {2}\right) \exp \left(\frac {- t ^ {2} / 2}{v (\boldsymbol {Z}) + L t / 3}\right).
$$
Theorem A.4 (Ledoux-Talagrand contraction inequality (e.g., [89])). Let  $\{\xi_i\}_{i\in [n]}\sim_{\mathrm{id}}\mathsf{Unif}(\{\pm 1\})$  be independent Rademacher random variables. For any set  $\mathbb{T}\subset \mathbb{R}^n$  and any family of  $L$ -Lipschitz functions  $\{\phi_j\}_{j\in [n]}$  with  $\phi_i(0) = 0$ , we have
$$
\mathbb {E} \left[ \sup  _ {\theta \in \mathbb {T}} \left| \sum_ {i = 1} ^ {n} \xi_ {i} \phi_ {i} \left(\theta_ {i}\right) \right| \right] \leq 2 L \cdot \mathbb {E} \left[ \sup  _ {\theta \in \mathbb {T}} \left| \sum_ {i = 1} ^ {n} \xi_ {i} \theta_ {i} \right| \right]. \tag {24}
$$
Throughout this section, we use the notation  $\lesssim$  to hide a universal constant  $C$ . Also, we use  $\overline{\sigma}(\mathbf{W}, \mathbf{X})$  to denote a function of  $\mathbf{W} \in \mathbb{R}^{(d+1) \times (d+1)}$  and  $\mathbf{X} \in \{\overline{\mathbf{X}} \in \mathbb{R}^{(d+1) \times (d+1)} : \| \overline{\mathbf{X}} \|_{\mathrm{Fr}}^2 \leq 4\} \equiv \mathcal{D}$  that satisfies the following properties:
1. For any  $\mathbf{X} \in \mathcal{D}$ , we have that  $\overline{\sigma}(\mathbf{W}, \mathbf{X})$  is  $L_{1}$ -Lipschitz with respect to  $\mathbf{W}$ .
2. For any  $\mathbf{X}\in \mathcal{D}$ , the expectation  $\mathbb{E}_{\mathbf{W}}[\overline{\sigma} (\mathbf{W},\mathbf{X})]\leq L_2$
3. For any  $\mathbf{X}\in \mathcal{D}$ , we have  $\overline{\sigma} (\mathbf{0},\mathbf{X})\leq L_3$
Here  $L_{1}$ ,  $L_{2}$ , and  $L_{3}$  are universal constants. Let  $\{\mathbf{W}_m\}_{m\in [M]}$  be sampled from Eq. (4). Let  $\mathbf{v}:\mathbb{R}^{(d + 1)\times (d + 1)}\to \mathbb{R}$  with  $\mathbb{E}_{\mathbf{W}}[\| \mathbf{v}(\mathbf{W})\| _2^2 ]\leq R^2$ . Consider a random feature model associated with  $\overline{\sigma}$  (hereafter, we will refer to it as RF model)
$$
f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N}; \mathbf {V}\right) = \sum_ {m = 1} ^ {M} \frac {1}{N} \sum_ {i = 1} ^ {N} \bar {\sigma} \left(\mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}\right) \left\langle \mathbf {v} _ {m}, \widetilde {\mathbf {x}} _ {i} \right\rangle , \tag {25}
$$
as well as the infinite-width version of the random feature model,
$$
f _ {\mathbf {v}} \left(\mathbf {x} _ {0: N}\right) = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} _ {\mathbf {W}} \left[ \bar {\sigma} \left(\mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}\right) \langle \mathbf {v} (\mathbf {W}), \widetilde {\mathbf {x}} _ {i} \rangle \right]. \tag {26}
$$
Note that both model RFA and BRFA correspond to special choices of  $\overline{\sigma}$ . Thus, all lemmas and theorems in this section are applicable to both model RFA and model BRFA.
To prove Theorem 1, we first state two auxiliary lemmas, Lemma B.1 and B.2.
Lemma B.1 (From infinite-width RF model to finite-width RF model). Consider  $f_{\mathbf{v}}$  that takes the form as Eq. (26), with  $\mathbb{E}_{\mathbf{W}}[\| \mathbf{v}(\mathbf{W})\| _2^2 ]\leq R^2$ . Let  $\mathbf{x}_{0:N}\sim P$ . Define the  $\| \cdot \|_{L^2 (P)}^2$  norm by
$$
\| g \| _ {L ^ {2} (P)} ^ {2} = \int g (\mathbf {x} _ {0: N}) ^ {2} P (\mathrm {d} \mathbf {x} _ {0: N}).
$$
Then with probability at least  $1 - \delta$ , there exists a sequence of vectors  $\{\mathbf{v}_m\}_{m=1}^M \subseteq \mathbb{R}^{d+1}$  and constant universal  $C < \infty$  that only depends on  $L_1, L_2$ , and  $L_3$  s.t.
$$
\left\| f _ {\mathbf {v}} - \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {m = 1} ^ {M} \bar {\sigma} \left(\mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}\right) \left\langle \mathbf {v} _ {m}, \widetilde {\mathbf {x}} _ {i} \right\rangle \right\| _ {L ^ {2} (P)} ^ {2} \leq \frac {C \left(d ^ {2} + \log M\right) R ^ {2} \delta^ {- 1}}{M} \tag {27}
$$
$$
\sum_ {m = 1} ^ {M} \left\| \mathbf {v} _ {m} \right\| _ {2} \leq \sqrt {2} R + \sqrt {\frac {C R ^ {2} \delta^ {- 1}}{M}} \quad a n d \quad \sum_ {m = 1} ^ {M} \left\| \mathbf {v} _ {m} \right\| _ {2} ^ {2} \leq \frac {C R ^ {2} \delta^ {- 1}}{M}. \tag {28}
$$
Lemma B.2. Under the setting of Theorem 1, let  $f_{\star}$  be a target function of form (6) and (7). Then there exists an infinite-width RFA model (26) with  $\mathbf{v} : \mathbb{R}^{(d + 1)\times (d + 1)} \to \mathbb{R}$  such that
$$
f _ {\star} = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} _ {\mathbf {W}} \left[ \bar {\sigma} \left(\mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}\right) \langle \mathbf {v} (\mathbf {W}), \widetilde {\mathbf {x}} _ {i} \rangle \right], \tag {29}
$$
$$
\mathbb {E} _ {\mathbf {W}} \left[ \| \mathbf {v} (\mathbf {W}) \| _ {2} ^ {2} \right] \leq B (f _ {\star}),
$$
where  $B(f_{\star})$  is as defined in (10).
The proofs of Lemma B.1 and B.2 are given in Section B.1.1. Now we assume these two lemmas hold, and use them to prove Theorem 1.
Proof of Theorem 1. For any function that takes form (6) and (7), by Lemma B.2, it admits representation (29) with  $\mathbb{E}_{\mathbf{W}}[\| \mathbf{v}(\mathbf{W})\| _2^2 ]\leq B(f_\star)$ . Then by Lemma B.1, since RFA model is a special case of the RF model, there exists  $\{\mathbf{v}_m\}_{m\in [M]}$  such that Eq. (27) and Eq. (28) hold with probability larger than  $1 - \delta$ . This proves Theorem 1.
B.1.1 Proof of auxiliary lemmas
Step 1. Proof of Eq. (27). To prove Eq. (27), we use a truncation argument.
Fix a  $R_{\mathbf{W}} > 0$  which we will choose its value later in the proof. Recall that we have  $\mathbf{W}_1, \ldots, \mathbf{W}_m$  i.i.d. with  $\mathbf{W}_{m,i,j} \sim_{\mathrm{id}} \mathsf{N}(0,1/4)$ . Define  $\mathbf{v}_m = \mathbf{v}(\mathbf{W}_m) 1\{\| \mathbf{W}_m \|_{\mathrm{Fr}} \leq R_{\mathbf{W}} / 2\} / M$  for  $m = 1, \ldots, M$ . Consider the truncated infinite-width random feature model  $f_{\mathbf{v}}^{R_{\mathbf{W}}} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{E}_{\mathbf{W}}[\overline{\sigma}(\mathbf{W}, \widetilde{\mathbf{x}}_0 \widetilde{\mathbf{x}}_i^\top) \langle \mathbf{v}(\mathbf{W}), \widetilde{\mathbf{x}}_i \rangle 1\{\| \mathbf{W} \|_{\mathrm{Fr}} \leq R_{\mathbf{W}} / 2\}]$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {W}} \left[ \left\| f _ {\mathbf {v}} ^ {R \mathbf {w}} - \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {m = 1} ^ {M} \overline {{\sigma}} (\mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}) \left\langle \mathbf {v} _ {m}, \widetilde {\mathbf {x}} _ {i} \right\rangle \right\| _ {L ^ {2} (P)} ^ {2} \right] \\ \leq \mathbb {E} _ {\mathbf {X}} \Big (\mathbb {E} _ {\mathbf {W}} \left[ \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \overline {{\sigma}} (\mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}) \widetilde {\mathbf {x}} _ {i} \right\| _ {2} ^ {2} \left\| \mathbf {v} (\mathbf {W}) \right\| _ {2} ^ {2} \mathbf {1} \left\{\| \mathbf {W} \| _ {\mathrm {F r}} \leq R _ {\mathbf {W}} / 2 \right\} \right] \Big) / M \\ \leq \mathbb {E} _ {\mathbf {X}} \left(\mathbb {E} _ {\mathbf {W}} \left[ \frac {2}{N} \sum_ {i = 1} ^ {N} \| \widetilde {\mathbf {x}} _ {i} \| _ {2} ^ {2} \left(L _ {1} ^ {2} \| \mathbf {W} \| _ {2} ^ {2} + L _ {3} ^ {2}\right) \| \mathbf {v} (\mathbf {W}) \| _ {2} ^ {2} \mathbf {1} \left\{\| \mathbf {W} \| _ {\mathbf {F r}} \leq R _ {\mathbf {W}} / 2 \right\} \right]\right) / M \\ \leq \frac {\widetilde {C} R _ {\mathbf {W}} ^ {2} R ^ {2}}{M}, \\ \end{array}
$$
where  $\widetilde{C_1}$  only depends on  $L_{1}, L_{2}$ , and  $L_{3}$ . Then using Markov's inequality,
$$
\left\| f _ {\mathbf {v}} ^ {R \mathbf {w}} - \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {m = 1} ^ {M} \overline {{\sigma}} (\mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}) \left\langle \mathbf {v} _ {m}, \widetilde {\mathbf {x}} _ {i} \right\rangle \right\| _ {L ^ {2} (P)} ^ {2} \leq \frac {3 \widetilde {C} R _ {\mathbf {W}} ^ {2} R ^ {2}}{\delta M}
$$
holds with probability at least  $1 - \delta /3$ . Next for the difference between  $f_{\mathbf{v}}^{R\mathbf{w}}$  and  $f_{\mathbf{v}}$ , we have
$$
\begin{array}{l} \left. \right.\left\| f _ {\mathbf {v}} - f _ {\mathbf {v}} ^ {R \mathbf {w}} \right\| _ {L ^ {2} (P)} ^ {2} \leq \mathbb {E} _ {\mathbf {X}} \left[\left(\mathbb {E} _ {\mathbf {W}} \left[ \frac {1}{N} \sum_ {i = 1} ^ {N} \overline {{\sigma}} (\mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}) \langle \mathbf {v}, \widetilde {\mathbf {x}} _ {i} \rangle \mathbf {1} \left\{\| \mathbf {W} \| _ {\mathsf {F r}} \geq R _ {\mathbf {W}} / 2 \right\}\right]\right) ^ {2} \right] \\ \leq \mathbb {E} _ {\mathbf {X}} \Big [ \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} _ {\mathbf {W}} \Big [ \langle \mathbf {v}, \widetilde {\mathbf {x}} _ {i} \rangle^ {2} \Big ] \sqrt {\mathbb {E} _ {\mathbf {W}} [ \overline {{\sigma}} (\mathbf {W} , \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}) ^ {4} ]} \sqrt {\mathbb {P} (\| \mathbf {W} \| _ {\mathsf {F r}} \geq R _ {\mathbf {W}} / 2)} \Big ] \\ \leq \widetilde {C _ {2}} R ^ {2} \mathbb {P} \left(\| \mathbf {W} \| _ {\mathrm {F r}} > R _ {\mathbf {W}} / 2\right) ^ {\frac {1}{2}}. \\ \end{array}
$$
Here  $\widetilde{C_2}$  is a constant that only depends on  $L_{1}$  and  $L_{2}$ . By concentration of functions of Gaussian random vectors (Theorem A.1),  $\| \overline{\sigma} (\mathbf{W},\widetilde{\mathbf{x}}_0\widetilde{\mathbf{x}}_i^\top)\|_{\psi_2}\leq L_1 + L_2$  for any  $i$ . So in the last inequality, we used the bound  $(\mathbb{E}_{\mathbf{W}}[\overline{\sigma} (\mathbf{W},\widetilde{\mathbf{x}}_0\widetilde{\mathbf{x}}_i^\top)^4 ])^{1 / 2}$  by  $\Theta ((L_1 + L_2)^2)$ . To bound  $\mathbb{P}(\| \mathbf{W}\|_{\mathrm{Fr}} > R_{\mathbf{W}} / 2)$ , we use concentration of functions of Gaussian random vectors (Theorem A.1) again, and get that
$$
\mathbb {P} \big (\| \mathbf {W} \| _ {\mathrm {F r}} - \mathbb {E} (\| \mathbf {W} \| _ {\mathrm {F r}}) \geq t / 2 \big) \leq \exp \big (- t ^ {2} / 2 \big).
$$