SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This work shows that transformers can perform complex in-context learning procedures with strong in-context algorithm selection capabilities, by both explicit theoretical constructions and experiments. We believe our work opens up many exciting directions, such as (1) more mechanisms for in-context algorithm selection; (2) Bayes-optimal ICL on other problems by either the post-ICL validation mechanism or new approaches; (3) understanding the internal workings of transformers performing in-context algorithm selection; (4) other mechanisms for implementing complex ICL procedures beyond in-context algorithm selection; (5) further statistical analyses, e.g. of pretraining. Besides, this work focuses on the transformer architecture; alternative sequence-to-sequence architectures (such as RNNs) are beyond our scope but would be interesting directions for future work.
The authors would like to thank Tengyu Ma and Jason D. Lee for the many insightful discussions. S. Mei is supported in part by NSF DMS-2210827 and NSF CCF-2315725.
[1] A. Agarwal, S. Negahban, and M. J. Wainwright. Fast global convergence rates of gradient methods for high-dimensional statistical recovery. Advances in Neural Information Processing Systems, 23, 2010.
[2] E. Akyurek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661, 2022.
[3] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[4] F. Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research, 18(1):629-681, 2017.
[5] Y. Bai, M. Chen, P. Zhou, T. Zhao, J. Lee, S. Kakade, H. Wang, and C. Xiong. How important is the train-validation split in meta-learning? In International Conference on Machine Learning, pages 543-553. PMLR, 2021.
[6] Y. Bai, S. Mei, H. Wang, and C. Xiong. Don't just blame over-parametrization for overconfidence: Theoretical analysis of calibration in binary classification. In International Conference on Machine Learning, pages 566-576. PMLR, 2021.
[7] J. Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149-198, 2000.
[8] A. Beck and M. Teboulle. Gradient-based algorithms with applications to signal recovery. Convex optimization in signal processing and communications, pages 42-88, 2009.
[9] S. Bengio, Y. Bengio, J. Cloutier, and J. Gescei. On the optimization of a synaptic learning rule. In Optimality in Biological and Artificial Networks?, pages 281-303. Routledge, 2013.
[10] S. Bhattamishra, K. Ahuja, and N. Goyal. On the ability and limitations of transformers to recognize formal languages. arXiv preprint arXiv:2009.11264, 2020.
[11] S. Bhattamishra, A. Patel, and N. Goyal. On the computational power of transformers and its implications in sequence modeling. arXiv preprint arXiv:2006.09286, 2020.
[12] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[13] S. Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning, 8(3-4):231-357, 2015.
[14] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
[15] S. Chan, A. Santoro, A. Lampinen, J. Wang, A. Singh, P. Richemond, J. McClelland, and F. Hill. Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 35:18878-18891, 2022.
[16] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
[36] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9):5149-5169, 2021.
[37] J. Hron, Y. Bahri, J. Sohl-Dickstein, and R. Novak. Infinite attention: Nngp and ntk for deep attention networks. In International Conference on Machine Learning, pages 4376-4386. PMLR, 2020.
[38] D. Hsu, S. M. Kakade, and T. Zhang. Random design analysis of ridge regression. In Conference on learning theory, pages 9-1. JMLR Workshop and Conference Proceedings, 2012.
[39] S. Jelassi, M. E. Sander, and Y. Li. Vision transformers provably learn spatial structure. arXiv preprint arXiv:2210.09221, 2022.
[40] K. Ji, J. D. Lee, Y. Liang, and H. V. Poor. Convergence of meta-learning with task-specific adaptation over partial parameters. Advances in Neural Information Processing Systems, 33: 11490-11500, 2020.
[41] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. vZidek, A. Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589, 2021.
[42] S. M. Kakade, V. Kanade, O. Shamir, and A. Kalai. Efficient learning of generalized linear and single index models with isotonic regression. Advances in Neural Information Processing Systems, 24, 2011.
[43] M. Khodak, M.-F. F. Balcan, and A. S. Talwalkar. Adaptive gradient-based meta-learning methods. Advances in Neural Information Processing Systems, 32, 2019.
[44] L. Kirsch and J. Schmidhuber. Meta learning backpropagation and improving it. Advances in Neural Information Processing Systems, 34:14122-14134, 2021.
[45] L. Kirsch, J. Harrison, J. Sohl-Dickstein, and L. Metz. General-purpose in-context learning by meta-learning transformers. arXiv preprint arXiv:2212.04458, 2022.
[46] K. Li and J. Malik. Learning to optimize. arXiv preprint arXiv:1606.01885, 2016.
[47] Y. Li, M. E. Ildiz, D. Papailiopoulos, and S. Oymak. Transformers as algorithms: Generalization and implicit model selection in in-context learning. arXiv preprint arXiv:2301.07067, 2023.
[48] B. Liu, J. T. Ash, S. Goel, A. Krishnamurthy, and C. Zhang. Transformers learn shortcuts to automata. arXiv preprint arXiv:2210.10749, 2022.
[49] J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, and W. Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804, 2021.
[50] Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786, 2021.
[51] A. Madani, B. McCann, N. Naik, N. S. Keskar, N. Anand, R. R. Eguchi, P.-S. Huang, and R. Socher. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.03497, 2020.
[52] A. Maurer, M. Pontil, and B. Romera-Paredes. The benefit of multitask representation learning. Journal of Machine Learning Research, 17(81):1-32, 2016.
[53] P. McCullagh. Generalized linear models. Routledge, 2019.
[54] S. Mei, Y. Bai, and A. Montanari. The landscape of empirical risk for nonconvex losses. The Annals of Statistics, 46(6A):2747-2774, 2018.
[55] S. Min, M. Lewis, H. Hajishirzi, and L. Zettlemoyer. Noisy channel language model prompting for few-shot text classification. arXiv preprint arXiv:2108.04106, 2021.
[56] S. Min, M. Lewis, L. Zettlemoyer, and H. Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943, 2021.
[57] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022.
[58] N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel. A simple neural attentive meta-learner. arXiv preprint arXiv:1707.03141, 2017.
[59] S. Müller, N. Hollmann, S. P. Arango, J. Grabocka, and F. Hutter. Transformers can do bayesian inference. arXiv preprint arXiv:2112.10510, 2021.
[60] T. Nagler. Statistical foundations of prior-data fitted networks. arXiv preprint arXiv:2305.11097, 2023.
[61] D. K. Naik and R. J. Mammone. Meta-neural networks that learn by learning. In [Proceedings 1992] IJCNN International Joint Conference on Neural Networks, volume 1, pages 437-442. IEEE, 1992.
[62] S. N. Negahban, P. Ravikumar, M. J. Wainwright, and B. Yu. A unified framework for high-dimensional analysis of m-estimators with decomposable regularizers. 2012.
[63] Y. Nesterov. Lectures on convex optimization, volume 137. Springer, 2018.
[64] C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022.
[65] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[66] N. Parikh, S. Boyd, et al. Proximal algorithms. Foundations and trends® in Optimization, 1 (3):127-239, 2014.
[67] J. Pérez, J. Marinković, and P. Barceló. On the Turing completeness of modern neural network architectures. arXiv preprint arXiv:1901.03429, 2019.
[68] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[69] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[70] A. Raventos, M. Paul, F. Chen, and S. Ganguli. The effects of pretraining task diversity on in-context learning of ridge regression. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
[71] S. Ravi and H. Larochelle. Optimization as a model for few-shot learning. In International conference on learning representations, 2017.
[72] Y. Razeghi, R. L. Logan IV, M. Gardner, and S. Singh. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:2202.07206, 2022.
[73] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022.
[74] O. Rubin, J. Herzig, and J. Berant. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633, 2021.
[95] S. Yao, B. Peng, C. Papadimitriou, and K. Narasimhan. Self-attention networks can process bounded hierarchical languages. arXiv preprint arXiv:2105.11115, 2021.
[96] C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877-28888, 2021.
[97] C. Yun, S. Bhojanapalli, A. S. Rawat, S. J. Reddi, and S. Kumar. Are transformers universal approximators of sequence-to-sequence functions? arXiv preprint arXiv:1912.10077, 2019.
[98] Y. Zhang, A. Backurs, S. Bubeck, R. Eldan, S. Gunasekar, and T. Wagner. Unveiling transformers with lego: a synthetic reasoning task. arXiv preprint arXiv:2206.04301, 2022.
[99] Y. Zhang, B. Liu, Q. Cai, L. Wang, and Z. Wang. An analysis of attention via the lens of exchangeability and latent variable models. arXiv preprint arXiv:2212.14852, 2022.
[100] Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697-12706. PMLR, 2021.
[101] X. Zuo, Z. Chen, H. Yao, Y. Cao, and Q. Gu. Understanding train-validation split in meta-learning with neural networks. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=JVlyfHEEm0k.
In-context learning The in-context learning (ICL) capability of large language models (LLMs) has gained significant attention since demonstrated on GPT-3 Brown et al. [12]. A number of subsequent empirical studies have contributed to a better understanding of the capabilities and limitations of ICL in LLM systems, which include but are not limited to [49, 55, 56, 50, 100, 74, 72, 28, 45, 91]. For an overview of ICL, see the survey by Dong et al. [24] which highlights some key findings and advancements in this direction.
A line of recent work investigates why and how LLMs perform ICL [94, 31, 86, 2, 18, 32, 47, 70]. In particular, Xie et al. [94] propose a Bayesian inference framework explaining how ICL works despite formatting differences between training and inference distributions. Garg et al. [31] show empirically that transformers could be trained from scratch to perform ICL of linear models, sparse linear models, two-layer neural networks, and decision trees. Li et al. [47] analyze the generalization error of trained ICL transformers from a stability viewpoint. They also experimentally show that transformers could perform "in-context model selection" (conceptually similar to in-context algorithm selection considered in this work) in specific tasks and presented related theoretical hypotheses. However, they do not provide concrete mechanisms or constructions for in-context model selection. A recent work [99] shows that pretrained transformers can perform Bayesian inference in latent variable models, which may also be interpreted as a mechanism for ICL. Our experimental findings extend these results by unveiling and demonstrating the in-context algorithm selection capabilities of transformers.
Closely related to our theoretical results are [86, 2, 18, 32], which show (among many things) that transformers can perform ICL by simulating gradient descent. However, these results do not provide quantitative error bounds for simulating multi-step gradient descent, and only handle linear regression models or their simple variants. Among these works, Akyurek et al. [2] showed that transformers can implement learning algorithms for linear models based on gradient descent and closed-form ridge regression; it also presented preliminary evidence that learned transformers perform ICL similar to Bayes-optimal ridge regression. Our work builds upon and substantially extends this line of work by (1) providing a more efficient construction for in-context gradient descent; (2) providing an end-to-end theory with additional results for pretraining and statistical power; (3) analyzing a broader spectrum of ICL algorithms, including least squares, ridge regression, Lasso, convex risk minimization for generalized linear models, and gradient descent on two-layer neural networks; and (4) constructing more complex ICL procedures using in-context algorithm selection.
When in-context data are generated from a prior, the Bayes risk is a theoretical lower bound for the risk of any possible ICL algorithm, including transformers. Xie et al. [94], Akyurek et al. [2] observe
that learned transformers behave closely to the Bayes predictor on a variety of tasks such as hidden Markov models [94] and noisy linear regression with a fixed noise level [2, 47]. Using the in-context algorithm selection mechanism (more precisely the post-ICL validation mechanism), we show that transformers can perform nearly-Bayes optimal ICL in noisy linear models with mixed noise levels (a strictly more challenging task than considered in [2, 47]), with both concrete theoretical guarantees (Section 4.1.1) and empirical evidence (Figure 2 & 3b). Complementary to these works, a line of work on "prior-data fitted networks" [59, 60, 35] also empirically demonstrates the Bayesian optimality of transformers in various settings. Our expressivity results support these empirical findings and are applicable beyond the Bayesian setting, e.g. for providing frequentist in-context prediction guarantees for transformers.
Transformers and its theory The transformer architecture, introduced by [84], has revolutionized natural language processing and been adopted in most of the recently developed large language models such as BERT and GPT [68, 21, 12]. Broaderly, transformers have demonstrated remarkable performance in many other fields of artificial intelligence such as computer vision, speech, graph processing, reinforcement learning, and biological applications [23, 25, 51, 69, 96, 16, 41, 73, 65, 14]. Towards a better theoretical understanding, recent work has studied the capabilities [97, 67, 37, 95, 11, 98, 48], limitations [33, 10], and internal workings [28, 79, 92, 27, 64] of transformers.
We remark that the transformer architecture used in our theoretical constructions differs from the standard one by replacing the softmax activation (in the attention layers) with a (normalized) ReLU function. Transformers with ReLU activations is experimentally studied in the recent work of Shen et al. [78], who find that they perform as well as the standard softmax activation in many NLP tasks.
Meta-learning Training models (such as transformers) to perform ICL can be viewed as an approach for the broader problem of learning-to-learn or meta-learning [77, 61, 81]. A number of other approaches has been studied extensively for this problem, including (and not limited to) training a meta-learner on how to update the parameters of a downstream learner [9, 46], learning parameter initializations that quickly adapt to downstream tasks [29, 71], learning latent embeddings that allow for effective similarity search [80]. Most relevant to the ICL setting are approaches that directly take as input examples from a downstream task and a query input and produce the corresponding output [34, 58, 75, 44]. For a comprehensive overview, see the survey [36].
Theoretical aspects of meta-learning have received significant recent interest [7, 52, 26, 83, 19, 30, 43, 40, 88, 20, 5, 76, 17, 101]. In particular, [52, 26, 83] analyzed the benefit of multi-task learning through a representation learning perspective, and [88, 20, 5, 76, 101] studied the statistical properties of learning the parameter initialization for downstream tasks.
Techniques We build on various existing techniques from the statistics and learning theory literature to establish our approximation and generalization guarantees for transformers. For the approximation component, we rely on a technical result of Bach [4] on the approximation power of ReLU networks. We use this result to show that transformers can approximate gradient descent (GD) on a broad range of loss functions, substantially extending the results of [86, 2, 18] who primarily consider the square loss. The recent work of Giannou et al. [32] also approximates GD with general loss functions by transformers, though using a different technique of forcing the softmax activations to act as sigmoids. Our analyses of Lasso and generalized linear models build on [87, 62, 1, 54]. Our generalization bound for transformers (used in our pretraining results) build on a chaining argument [87].
B Technical tools
Additional notation for proofs We say a random variable  $X$  is  $\sigma^2$ -sub-Gaussian (or  $\mathrm{SG}(\sigma)$  interchangeably) if  $\mathbb{E}[\exp(X^2/\sigma^2)] \leq 2$ . A random vector  $\mathbf{x} \in \mathbb{R}^d$  is  $\sigma^2$ -sub-Gaussian if  $\langle \mathbf{v}, \mathbf{x} \rangle$  is  $\sigma^2$ -sub-Gaussian for all  $\|\mathbf{v}\|_2 = 1$ . A random variable  $X$  is  $K$ -sub-Exponential (or  $\mathrm{SE}(K)$  interchangeably) if  $\mathbb{E}[\exp(|X|/K)] \leq 2$ .
B.1 Concentration inequalities
Lemma B.1. Let  $\beta \sim \mathsf{N}(\mathbf{0},\mathbf{I}_d / d)$ . Then we have
$$
\mathbb {P} \left(\| \boldsymbol {\beta} \| _ {2} ^ {2} \geq (1 + \delta) ^ {2}\right) \leq e ^ {- d \delta^ {2} / 2}.
$$
Lemma B.2 (Theorem 6.1 of [87]). Let  $X = [X_{ij}] \in \mathbb{R}^{n \times d}$  be a Gaussian random matrix with  $X_{ij} \sim \mathsf{N}(0,1)$ . Let  $\sigma_{\min}(X)$  and  $\sigma_{\min}(X)$  be the minimum and maximum singular value of  $X$ , respectively. Then we have
$$
\mathbb {P} \Big (\sigma_ {\max } (X) / \sqrt {n} \geq 1 + \sqrt {d / n} + \delta \Big) \leq e ^ {- n \delta^ {2} / 2},
$$
$$
\mathbb {P} \Big (\sigma_ {\min} (X) / \sqrt {n} \leq 1 - \sqrt {d / n} - \delta \Big) \leq e ^ {- n \delta^ {2} / 2}.
$$
The following lemma is a standard result of covariance concentration, see e.g. [85, Theorem 4.6.1].
Lemma B.3. Suppose that  $\mathbf{x}_1, \dots, \mathbf{x}_N$  are independent  $d$ -dimensional  $K$ -sub-Gaussian random vectors. Then as long as  $N \geq C_0 d$ , with probability at least  $1 - \exp(-N / C_0)$  we have
$$
\left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top} \right\| _ {\mathrm {o p}} \leq 8 K ^ {2},
$$
where  $C_0$  is a universal constant.
Lemma B.4. For random matrix  $\mathbf{X} = [x_{ij}] \in \mathbb{R}^{N \times d}$  with  $x_{ij} \stackrel{\mathrm{id}}{\sim} \mathsf{N}(0,1)$  and  $\varepsilon = [\varepsilon_i] \in \mathbb{R}^N$  with  $\varepsilon_i \stackrel{\mathrm{id}}{\sim} \mathsf{N}(0, \sigma^2)$ , it holds that
$$
\mathbb {P} \Big (\left\| \mathbf {X} ^ {\top} \boldsymbol {\varepsilon} \right\| _ {\infty} \geq \sqrt {8 N \sigma^ {2} \log (2 d / \delta)} \Big) \leq \delta + \exp (- N / 2).
$$
Proof. We consider  $\mathbf{u}_j \coloneqq [x_{ij}]_i \in \mathbb{R}^N$ , then  $\left\| \mathbf{X}^\top \boldsymbol{\varepsilon} \right\|_\infty = \max_{i \in [d]} |\langle \mathbf{u}_j, \boldsymbol{\varepsilon} \rangle|$ . Notice that the random variables  $\langle \mathbf{u}_1, \boldsymbol{\varepsilon} \rangle, \dots, \langle \mathbf{u}_d, \boldsymbol{\varepsilon} \rangle$  are independent  $\mathsf{N}(0, \| \boldsymbol{\varepsilon} \|_2^2)$ , and hence
$$
\mathbb {P} \left(\max _ {i \in [ d ]} | \langle \mathbf {u} _ {j}, \pmb {\varepsilon} \rangle | \geq t \Bigg | \pmb {\varepsilon}\right) \leq 2 d \exp \left(- \frac {t ^ {2}}{2 \| \pmb {\varepsilon} \| _ {2} ^ {2}}\right).
$$
Further, by Lemma B.1,  $\mathbb{P}(\| \varepsilon \|_2 \geq 2\sigma \sqrt{N}) \leq \exp(-N/2)$ . Taking  $t = \sqrt{8N\sigma^2\log(2d/\delta)}$  completes the proof.
B.2 Approximation theory
For any signed measure  $\mu$  over a space  $\mathcal{W}$ , let  $\mathrm{TV}(\mu) := \int_{\mathcal{W}} |d\mu(\mathbf{w})| \in [0, \infty]$  denote its total measure. Recall  $\sigma(\cdot) = \mathrm{ReLU}(\cdot)$  is the standard relu activation, and  $\mathsf{B}_{\infty}^{k}(R) = [-R, R]^{k}$  denotes the standard  $\ell_{\infty}$  ball in  $\mathbb{R}^k$  with radius  $R > 0$ .
Definition B.1 (Sufficiently smooth  $k$ -variable function). We say a function  $g: \mathbb{R}^k \to \mathbb{R}$  is  $(R, C_\ell)$ -smooth, if for  $s = \lceil (k - 1)/2 \rceil + 2$ ,  $g$  is a  $C^s$  function on  $\mathsf{B}_{\infty}^k(R)$ , and
$$
\sup  _ {\mathbf {z} \in \mathsf {B} _ {\infty} ^ {k} (R)} \left\| \nabla^ {i} g (\mathbf {z}) \right\| _ {\infty} = \sup  _ {\mathbf {z} \in \mathsf {B} _ {\infty} ^ {k} (R)} \max  _ {j _ {1}, \dots , j _ {i} \in [ k ]} | \partial_ {x _ {j _ {1}} \dots x _ {j _ {i}}} g (\mathbf {x}) | \leq L _ {i}
$$
for all  $i\in \{0,1,\ldots ,s\}$ , with  $\max_{0\leq i\leq s}L_iR^i\leq C_\ell$
The following result for expressing smooth functions as a random feature model with relu activation is adapted from Bach [4, Proposition 5].
Lemma B.5 (Expressing sufficiently smooth functions by relu random features). Suppose function  $g: \mathcal{R}^k \to \mathbb{R}$  is  $(R, C_\ell)$  smooth. Then there exists a signed measure  $\mu$  over  $\mathcal{W} = \{\mathbf{w} \in \mathbb{R}^{k+1}: \| \mathbf{w} \|_1 = 1\}$  such that
$$
g (\mathbf {x}) = \int_ {\mathcal {W}} \frac {1}{R} \sigma (\mathbf {w} ^ {\top} [ \mathbf {x}; R ]) d \mu (\mathbf {w}), \qquad \forall \mathbf {x} \in \mathcal {X}
$$
and  $\mathrm{TV}(\mu) \leq C(k)C_{\ell}$ , where  $C(k) < \infty$  is a constant that only depends on  $k$ .
Lemma B.6 (Uniform finite-neuron approximation). Let  $\mathcal{X}$  be a space equipped with a distance function  $d_{\mathcal{X}}(\cdot, \cdot): \mathcal{X} \times \mathcal{X} \to \mathbb{R}_{\geq 0}$ . Suppose function  $g: \mathcal{X} \to \mathbb{R}$  is given by
$$
g (\mathbf {x}) = \int_ {\mathcal {W}} \phi (\mathbf {x}; \mathbf {w}) d \mu (\mathbf {w}),
$$