Title: Transformers are Minimax Optimal Nonparametric In-Context Learners
Venue: NeurIPS 2024
Accepted: yes
Keywords: in-context learning

transformers

nonparametric regression

minimax optimality

meta-learning
Abstract: In-context learning (ICL) of large language models has proven to be a surprisingly effective method of learning a new task from only a few demonstrative examples. In this paper, we shed light on the efficacy of ICL from the viewpoint of statistical learning theory. We develop approximation and generalization error analyses for a transformer model composed of a deep neural network and one linear attention layer, pretrained on nonparametric regression tasks sampled from general function spaces including the Besov space and piecewise $\gamma$-smooth class. In particular, we show that sufficiently trained transformers can achieve -- and even improve upon -- the minimax optimal estimation risk in context by encoding the most relevant basis representations during pretraining. Our analysis extends to high-dimensional or sequential data and distinguishes the \emph{pretraining} and \emph{in-context} generalization gaps, establishing upper and lower bounds w.r.t. both the number of tasks and in-context examples. These findings shed light on the effectiveness of few-shot prompting and the roles of task diversity and representation learning for ICL.

Transformers are Minimax Optimal Nonparametric In-Context Learners

Abstract

In-context learning (ICL) of large language models has proven to be a surprisingly effective method of learning a new task from only a few demonstrative examples. In this paper, we study the efficacy of ICL from the viewpoint of statistical learning theory. We develop approximation and generalization error bounds for a transformer composed of a deep neural network and one linear attention layer, pretrained on nonparametric regression tasks sampled from general function spaces including the Besov space and piecewise γ -smooth class. We show that sufficiently trained transformers can achieve - and even improve upon - the minimax optimal estimation risk in context by encoding the most relevant basis representations during pretraining. Our analysis extends to high-dimensional or sequential data and distinguishes the pretraining and in-context generalization gaps. Furthermore, we establish information-theoretic lower bounds for meta-learners w.r.t. both the number of tasks and in-context examples. These findings shed light on the roles of task diversity and representation learning for ICL.

1 Introduction

Large language models (LLMs) have demonstrated remarkable capabilities in understanding and generating natural language data. In particular, the phenomenon of in-context learning (ICL) has recently garnered widespread attention. ICL refers to the ability of pretrained LLMs to perform a new task by being provided with a few examples within the context of a prompt, without any parameter updates or fine-tuning. It has been empirically observed that few-shot prompting is especially effective in large-scale models (Brown et al., 2020) and requires only a couple of examples to consistently achieve high performance (García et al., 2023). In contrast, Raventos et al. (2023) demonstrate that sufficient pretraining task diversity is required for the emergence of ICL. However, we still lack a comprehensive understanding of the statistical foundations of ICL and few-shot prompting.

A vigorous line of research has been directed towards understanding ICL of single-layer linear attention models pretrained on the query prediction loss of linear regression tasks (Garg et al., 2022; Akyürek et al., 2023; Zhang et al., 2023; Ahn et al., 2023; Mahankali et al., 2023; Wu et al., 2024). It has been shown that the global minimizer of the L 2 pretraining loss implements one step of GD on a least-squares linear regression objective (Mahankali et al., 2023) and is nearly Bayes optimal (Wu et al., 2024). Moreover, risk bounds with respect to the context length (Zhang et al., 2023) and number of tasks (Wu et al., 2024) have been obtained.

Other works have examined ICL of more complex multi-layer transformers. Bai et al. (2023); von Oswald et al. (2023) give specific transformer constructions which simulate GD in context, however it is unclear how such meta-algorithms may be learned. Another approach is to study learning with representations , where tasks consist of a fixed nonlinear feature map composed with a varying linear function. Guo et al. (2023) empirically found that trained transformers exhibit a separation where lower layers transform the input and upper layers perform linear ICL. Recently, Kim and Suzuki

(2024) analyzed a model consisting of a shallow neural network followed by a linear attention layer and proved that the MLP component learns to encode the true features during pretraining. However, they assumed the infinite task and sample size limit and did not study generalization capabilities.

Our contributions. In this paper, we analyze the optimality of ICL from the perspective of statistical learning theory. Our object of study is a transformer consisting of a deep neural network with N -dimensional output followed by one linear attention layer. The model is pretrained on n input-output samples from T nonparametric regression tasks, generated from a suitably decaying distribution on a general function space. Compared to previous works, we take a crucial step towards understanding practical multi-layer transformers by incorporating the representation learning capabilities of the DNN module. From a more abstract perspective, this work can also be situated as a nonlinear extension of meta-learning. Our contributions are highlighted below.

We develop a general framework for upper bounding the in-context estimation error of the empirical risk minimizer in terms of the approximation error of the neural network and separate in-context and pretraining generalization gaps depending on n, T , respectively.

In the Besov space setting, we show that ICL achieves nearly minimax optimal risk n -2 α 2 α + d when T is sufficiently large. Since LLMs are pretrained on vast amounts of data in practice, T can be taken to be nearly infinite, justifying the emergence of ICL at large scales. We extend the optimality guarantees to nearly dimension-free rates in the anisotropic Besov space and also to learning sequential data with deep transformers in the piecewise γ -smooth function class.

We show that ICL can improve upon the a priori optimal rate when the task class basis resides in a coarser Besov space by learning to encode informative basis representations, emphasizing the importance of pretraining on diverse tasks.

We also derive information-theoretic lower bounds for the minimax risk in both n, T , rigorously confirming that ICL is jointly optimal when T is large (Besov space setting), while any metalearning method is jointly suboptimal when T is small (coarser space setting). This separation aligns with empirical observations of a task diversity threshold (Raventos et al., 2023).

The paper is structured as follows. In Section 2, the regression tasks and transformer model are defined in an abstract setting. In Section 3, we present the general framework for estimating the ICL approximation and generalization error. In Section 4, we specialize to the Besov-type and piecewise γ -smooth class settings and show that transformers can achieve or exceed the minimax optimal rate in context. In Section 5, we derive minimax lower bounds. All proofs are deferred to the appendix; moreover, we provide numerical experiments validating our results in Appendix E.

1.1 Other Related Works

Meta-learning. The theoretical setting of ICL is closely related to meta-learning , where the goal is to infer a shared representation ψ ◦ with samples from a set of transformed tasks β ⊤ i ψ ◦ . When ψ ◦ is linear, fast rates have been established by Tripuraneni et al. (2020); Du et al. (2021), while the nonlinear case has been studied by Meunier et al. (2023) where ψ ◦ is a feature projection into a reproducing kernel Hilbert space. Our results can be viewed as extending this body of work to function spaces of generalized smoothness with a specific deep transformer architecture.

Optimal rates for DNNs. Our analysis extends established optimality results for classes of DNNs in ordinary supervised regression settings to ICL. Suzuki (2019) has shown that deep feedforward networks with the ReLU activation can efficiently approximate functions in the Besov space and thus achieve the minimax optimal rate. This has been extended to the anisotropic Besov space (Suzuki and Nitanda, 2021), convolutional neural networks for infinite-dimensional input (Okumoto and Suzuki, 2022), and transformers for sequence-to-sequence functions (Takakura and Suzuki, 2023). We remark that a work in progress (Imaizumi, 2024) also studies the sample complexity of ICL of transformers in a Sobolev space setting.

Pretraining dynamics for ICL. While how ICL arises from optimization is not fully understood, there are encouraging developments in this direction. Zhang et al. (2023) has shown for one layer of linear attention that running GD on the population risk always converges to the global optimum. This was extended to incorporate a linear output layer by Zhang et al. (2024), and to softmax attention

by Huang et al. (2023); Li et al. (2024); Chen et al. (2024). Kim and Suzuki (2024) considered a compound transformer equivalent to ours with a shallow MLP component and proved that the loss landscape becomes benign in the mean-field limit, deriving convergence guarantees for the corresponding gradient dynamics. These analyses indicate that the attention mechanism, while highly nonconvex, may possess structures favorable for gradient-based optimization.

2 Problem Setup

2.1 Nonparametric Regression

In this paper, we analyze the ability of a transformer to solve nonparametric regression problems in context when pretrained on examples from a family of regression tasks, which we describe below. Let X ⊆ R d be the input space ( d is allowed to be infinite), P X a probability distribution on X , and ( ψ ◦ j ) ∞ j =1 a fixed countable subset of L 2 ( P X ) . A regression function F ◦ β : X → R is randomly generated for each task by sampling the sequence of coefficients β ∈ R ∞ from a distribution P β on B ( R ∞ ) ; the class of tasks F ◦ is defined as

endowed with the induced distribution. Each task prompt contains n example input-response pairs { ( x k , y k ) } n k =1 . The covariates x k are i.i.d. drawn from P X and the responses are generated as

where the noise ξ k is assumed to be i.i.d. with mean zero and | ξ k | ≤ σ almost surely. 1 In addition, we independently generate a query token ˜ x and corresponding output ˜ y in the same manner.

We proceed to state our assumptions for the regression model. Informally, we suppose a relaxed version of sparsity and orthonormality of ψ ◦ j and suitable decay rates for the basis expansion. These will be subsequently verified for specific function spaces of interest with their natural decay rate.

Assumption 1 (relaxed sparsity and orthonormality of basis functions) . For N ∈ N , there exist integers ¯ N < ¯ N ≲ N with ¯ N -¯ N +1 = N such that ψ ◦ ¯ N , · · · , ψ ◦ ¯ N are independent and ψ ◦ 1 , · · · , ψ ◦ ¯ N -1 are all contained in the linear span of ψ ◦ N , · · · , ψ ◦ ¯ N . Moreover, there exist r, C 1 , C 2 , C ∞ > 0 such

¯ that Σ Ψ ,N := ( E x ∼P X [ ψ ◦ j ( x ) ψ ◦ k ( x )] ) ¯ N j,k N satisfies C 1 I N ⪯ Σ Ψ ,N ⪯ C 2 I N and

Denoting the ¯ N -basis approximation of F ◦ β as F ◦ β, ¯ N := ∑ ¯ N j =1 β j ψ ◦ j , by Assumption 1 there exist 'aggregated' coefficients ¯ β ¯ N , · · · , ¯ β ¯ N uniquely determined by β such that F ◦ β, ¯ N = ∑ ¯ N j = ¯ N ¯ β j ψ ◦ j . We define two types of coefficient covariance matrices

Assumption 2 (decay of β ) . For s, B > 0 it holds that ∥ F ◦ β ∥ L ∞ ( P X ) ≤ B for all F ◦ β ∈ F ◦ and

uniformly over β ∈ supp P β . Furthermore, Tr(Σ ¯ β,N ) is bounded for all N and

Remark 2.1. In the simple case where ( ψ ◦ j ) ∞ j =1 is a basis for L 2 ( P X ) , we may set ¯ N = 1 , ¯ N = N so that the dependency condition of Assumption 1 is trivially satisfied, moreover, Σ ¯ β,N = Σ β,N and boundedness of Tr(Σ ¯ β,N ) automatically follows from (4). However, the assumptions in the stated form also allow for hierarchical bases with dependencies such as wavelet systems. We also note that (3) and (4) entail basically the same rate but are not equivalent: the uniform bound | β j | 2 ≲ j -2 s -1 (log j ) -2 along with Assumption 1 implies (3). The (log j ) -2 term can be replaced with any g ( j ) such that ∑ ∞ j =1 j -1 g ( j ) is convergent.

2.2 In-Context Learning

We now describe our transformer model, which takes n context pairs X = ( x 1 , · · · , x n ) ∈ R d × n , y = ( y 1 , · · · , y n ) ⊤ ∈ R n and a query token ˜ x as input and returns a prediction for the corresponding output. The covariates are first passed through a nonlinear representation or feature mapping ϕ : X → R N , which we assume belongs to a sufficiently powerful class of estimators F N . Specifically:

Assumption 3 (expressivity of F N ) . ∥ ϕ ( x ) ∥ 2 ≤ B ′ N for some B ′ N > 0 for all x ∈ X , ϕ ∈ F N Moreover for some δ N > 0 , there exist ϕ ∗ N , · · · , ϕ ∗ ¯ N ∈ F N satisfying

¯

By choosing F N and δ N to satisfy the above assumption, we will be able to utilize established approximation and generalization guarantees for families of deep neural networks in Section 4.

The extracted representations ϕ ( X ) = ( ϕ ( x 1 ) , · · · , ϕ ( x n )) are then mapped to a scalar output via a linear attention layer parametrized by a matrix Γ ∈ S N for S N ⊂ R N × N ,

Finally, the output is constrained to lie on [ -¯ B, ¯ B ] by applying clip ¯ B ( u ) := max { min { u, ¯ B } , -¯ B } , yielding f Θ ( X , y , ˜ x ) := clip ¯ B ( ˇ f Θ ( X , y , ˜ x )) . We set S N = { Γ ∈ R N × N | 0 ⪯ Γ ⪯ C 3 I N } for some C 3 > 0 and fix ¯ B = B for simplicity.

The above setup is a restricted reparametrization of linear attention widely used in theoretical analyses (see e.g. Zhang et al., 2023; Wu et al., 2024, for more details), where the values only refer to y and the query and key matrices are consolidated into one matrix Γ . The form is equivalent to one step of GD with matrix step size and has been shown to be optimal for a single layer of linear attention for linear regression tasks (Ahn et al., 2023; Mahankali et al., 2023). The placement of the attention layer after the DNN module ϕ is justified by the observation that lower layers of trained transformers act as data representations on top of which upper layers perform ICL (Guo et al., 2023).

During pretraining, the model is presented with T prompts { ( X ( t ) , y ( t ) , ˜ x ( t ) ) } T t =1 where the tasks F ◦ β ( t ) ∈ F ◦ , β ( t ) ∼ P β and tokens X ( t ) = ( x ( t ) 1 , · · · , x ( t ) n ) , y ( t ) = ( y ( t ) 1 , · · · , y ( t ) n ) ⊤ , ˜ x ( t ) and ˜ y ( t ) are independently generated as described in Section 2.1, and is trained to minimize the empirical risk

Our goal is to verify the efficiency of ICL as a learning algorithm and show that learning the optimal ̂ Θ allows the transformer to solve new random regression problems y = F ◦ β ( x ) + ξ for F ◦ β ∈ F ◦ in context. To this end, we evaluate the convergence of the mean-squared risk or estimation error,

Note that we do not study whether the transformer always converges to ̂ Θ ; the training dynamics of a DNN is already a very difficult problem. For the attention layer, see the discussion in Section 1.1.

3 Risk Bounds for In-Context Learning

In this section, we outline our framework for analyzing the in-context estimation error ¯ R ( ̂ Θ) . Some additional definitions are in order. The ϵ -covering number N ( C , ρ, ϵ ) of a metric space C equipped with a metric ρ for ϵ > 0 is defined as the minimal number of balls in ρ with radius ϵ needed to cover C (van der Vaart and Wellner, 1996). The ϵ -covering entropy or metric entropy is given as V ( F , ρ, ϵ ) := log N ( F , ρ, ϵ ) . The ϵ -packing number M ( ϵ, C , ρ ) is given as the maximal cardinality of a ϵ -separated set { c 1 , . . . , c M } ⊆ C such that ρ ( c i , c j ) ≥ δ for all i = j . The transformer model class is defined as T N := { f Θ | Θ ∈ S N ×F N } .

̸

To bound the overall risk, we first decompose into the approximation and generalization gaps.

.

Theorem 3.1 (Schmidt-Hieber (2020), Lemma 4, adapted) . There exists a universal constant C such that for any ϵ > 0 such that V ( T N , ∥·∥ L ∞ , ϵ ) ≥ 1 ,

Proof. The convergence rate of the empirical risk minimizer is established for a fixed regression problem y = f ◦ ( z ) + ξ in Schmidt-Hieber (2020) when ξ is Gaussian; we modify the proof to incorporate bounded noise in Appendix B.1. The ICL setup can be reduced to the ordinary case as follows. We consider the entire batch ( β, X , ξ 1: n , ˜ x ) including the hidden coefficient β as a single datum z with output ˜ y . The true function is given as f ◦ ( z ) = F ◦ β (˜ x ) and the model class is taken to be T N implicitly concatenated with the generative process ( β, X , ξ 1: n , ˜ x ) ↦→ ( X , y , ˜ x ) . Then R (Θ) , V ( T N , ∥·∥ L ∞ , ϵ ) and T agree with the ordinary L 2 risk, model class entropy and sample size.

Here, the second term is the pretraining generalization error dependent on the number of tasks T ; the in-context generalization error dependent on the prompt length n manifests as part of the first term. This separation allows us to compare the relative difficulty of the two types of learning.

Bounding approximation error. In order to bound the first term, we analyze the risk of the choice

where ϕ ∗ is given as in Assumption 3 for a suitable δ N to be determined. The definition of Γ ∗ approximately generalizes the global optimum Γ = ( (1 + 1 n )Λ + 1 n tr(Λ) I d ) -1 for the Gaussian linear regression setup where x ∼ N (0 , Λ) (Zhang et al., 2023). Since Σ Ψ ,N ⪰ C 1 I N we have Γ ∗ ⪯ C -1 1 I N and hence we may assume Γ ∗ ∈ S N by replacing C 3 with C 3 ∨ C -1 1 if necessary.

Proposition 3.2. Under Assumptions 1-3, it holds that

The proof is presented throughout Appendix A. The overall scheme is to approximate F ◦ β (˜ x ) by its truncation F ◦ β, ¯ N (˜ x ) and the finite basis ψ ◦ ¯ N : ¯ N by ϕ ∗ ¯ N : ¯ N , which incurs errors N -2 s and the terms pertaining to δ N , respectively. The first three terms arise from the concentration of the n token representations ψ ◦ N : ¯ N ( x k ) . All hidden constants are at most polynomial in problem parameters.

¯

Bounding generalization error. To estimate the metric entropy of T N , we first reduce to the metric entropy of the representation class F N . Here, ∥·∥ L ∞ refers to the essential supremum over the support of P X and also over all N components for F N . The proof is given in Appendix B.2.

Lemma 3.3. Under Assumptions 1-3, there exists D > 0 such that for all ϵ sufficiently small,

4 Minimax Optimality of In-Context Learning

4.1 Besov Space and DNNs

We now apply our theory to study the sample complexity of ICL when F N consists of (clipped, see (6)) deep neural networks. These can be also seen as simplified transformers with attention layers and skip connections removed. To be precise, we define the set of DNNs with depth L , width W , sparsity S , norm bound M and ReLU activation η ( x ) = x ∨ 0 (applied element-wise) as

The Besov space is a very general class of functions including the Hölder and Sobolev spaces which captures spatial inhomogeneity in smoothness, and provides a natural setting in which to study the expressive power of deep neural networks (Suzuki, 2019). Here, we fix X = [0 , 1] d for simplicity.

Definition 4.1 (Besov space) . For 2 ≤ p ≤ ∞ , 0 < q ≤ ∞ , fractional smoothness α > 0 and r = ⌊ α ⌋ +1 , the r th modulus of f ∈ L p ( X ) is defined using the difference operator ∆ r h , h ∈ R d as

Also, the Besov (quasi-)norm is given as ∥·∥ B α p,q = ∥·∥ L p + | · | B α p,q where

and the Besov space is defined as B α p,q ( X ) = { f ∈ L p ( X ) | ∥ f ∥ B α p,q < ∞} . We write U ( B α p,q ( X )) for the unit ball in ( B α p,q ( X ) , ∥·∥ B α p,q ) .

We have that the Hölder space C α ( X ) = B α ∞ , ∞ ( X ) for order α > 0 , α / ∈ N and the Sobolev space W m 2 ( X ) = B m 2 , 2 ( X ) for m ∈ N as well as the embeddings B m p, 1 ( X ) ↪ → W p m ( X ) ↪ → B m p, ∞ ( X ) ; if α > d/p , B α p,q ( X ) compactly embeds into the space of continuous functions on X . See Triebel (1983); Giné and Nickl (2015) for more details. The difficulty of learning a regression function in the Besov is quantified by the minimax risk; the following rate is classical.

Proposition 4.2 (Donoho and Johnstone (1998)) . The minimax risk for an estimator ̂ f n with n i.i.d. samples D n = { ( x i , y i ) } n i =1 over U ( B α p,q ( X )) satisfies

Anatural basis system for B α p,q ( X ) is formed by the B-splines , which can be seen as a type of wavelet decomposition or multiresolution analysis (DeVore and Popov, 1988). As B-splines are piecewise polynomials, they can be efficiently approximated by DNNs with at most log depth (Suzuki, 2019). Definition 4.3 (B-spline wavelet basis) . The tensor product B-spline of order m ∈ N satisfying m>α +1 -1 /p , at resolution k ∈ Z d ≥ 0 and location ℓ ∈ I d k = ∏ d i =1 [ -m : 2 k i ] is

When k 1 = · · · = k d , we abuse notation and write ω d k,ℓ for k ∈ Z ≥ 0 in place of ω d ( k, ··· ,k ) ,ℓ .

4.2 Estimation Error Analysis

To apply our framework, we set the task class as the unit ball F ◦ = U ( B α p,q ( X )) and take as basis { ψ ◦ j | j ∈ N } = { 2 kd/ 2 ω d k,ℓ | k ∈ Z ≥ 0 , ℓ ∈ I d k } the set of all B-spline wavelets ordered primarily by increasing k and scaled to counteract the dilation in x . Abusing notation, we also write β k,ℓ to denote the coefficient in β corresponding to 2 kd/ 2 ω d k,ℓ . The set of B-splines at each resolution are independent, while those of lower resolution can always be decomposed into a linear sum of B-splines of higher resolution satisfying certain decay rates, which we prove in Proposition C.10.

From this setup, in Appendix C.1.1, we verify Assumptions 1 and 2 with r = 1 / 2 , s = α/d under: Assumption 4. F ◦ = U ( B α p,q ( X )) , α > d/p and P X has positive Lebesgue density ρ X bounded above and below on X . Also, all coefficients β k,ℓ are independent and

We can check that we have not given ourselves an easier learning problem with (5): the assumed variance decay rate is tight (up to the logarithmic factor k -2 ) in the sense that any f ∈ U ( B α p,q ( X )) can indeed be expanded into a sum of wavelets with the same coefficient decay when averaged over ℓ ∈ I d k . See Lemma C.1 and the following discussion. We also obtain the following in-context approximation and entropy bounds in Appendix C.1.2.

Lemma 4.4. For any δ N > 0 , Assumption 3 is satisfied by taking

where Π B ′ N is the projection in R N to the centered ball of radius B ′ N = O ( √ N ) and each ϕ j is a ReLU network such that L = O (log N +log δ -1 N ) and W,S,M = O (1) . Also, the metric entropy of F N is bounded as V ( F N , ∥·∥ L ∞ , ϵ ) ≲ N log N δ N ϵ .

Hence Assumptions 1-3 all follow from Assumption 4, and we conclude in Appendix C.1.3:

Theorem 4.5 (minimax optimality of ICL in Besov space) . Under Assumption 4, if n ≳ N log N ,

Hence if T ≳ n 2 α +2 d 2 α + d and N ≍ n d 2 α + d , in-context learning achieves the minimax optimal rate n -2 α 2 α + d up to a log factor.

The first term arises from the N -term truncation and oracle approximation error of the DNN module, and is equal to the N -term optimal error (D˜ ung, 2011a). The second and third term each correspond to the in-context and pretraining generalization gap. With regard to N , we see that n = ˜ Ω( N ) is enough to learn the basis expansion in context, while T = ˜ Ω( N 2 ) is necessary to learn the attention layer. However if T/N = o ( n ) , the third term dominates and the overall complexity scales suboptimally as T -α α + d , illustrating the importance of sufficient pretraining. This also aligns with the task diversity threshold observed by Raventos et al. (2023). Since the amount of training data for LLMs is practically infinite in practice, our result justifies the effectiveness of ICL at large scales with only a small number of in-context samples.

A limitation of ICL. In the regime 1 ≤ p < 2 , the approximation error is strictly worse without an adaptive representation scheme and the resulting rate is suboptimal (see Remark C.3). While DNNs can adapt to task smoothness in supervised settings (Suzuki, 2019), ICL and any other metalearning methods are fundamentally constrained to non-adaptive representations since they cannot update at inference time, and hence are bounded below by the best linear approximation rate or Kolmogorov width, which is strictly worse than the minimax optimal rate when p < 2 . Indeed, for any N -dimensional subspace L N ⊂ B α p,q ( X ) it holds that (Vybíral, 2008)

Remark 4.6. The N 2 log N term in the pretraining generalization gap is due to the covering bound of the attention matrix Γ , while the entropy of the DNN class is only N log N . Hence the task diversity requirement may be lessened to the latter by considering low-rank structure or approximation of attention heads (Bhojanapalli et al., 2020; Chen et al., 2021).

4.3 Avoiding the Curse of Dimensionality

The above rate inevitably suffers from the curse of dimensionality as d appears in the exponent of the optimal rate. We also consider the anisotropic Besov space (Nikol'skii, 1975), a generalization allowing for different degrees of smoothness ( α 1 , · · · , α d ) in each coordinate. Then the optimal rate is nearly dimension-free in the sense that the rate only depends on d through the quantity ˜ α := ( ∑ i α -1 i ) -1 , and becomes independent of dimension if only a few directions are important i.e. have small α i . Rigorous definitions, statements and proofs are provided in Appendix C.2.

Extending Theorem 4.5, we show that ICL again attains near-optimal estimation error in the anisotropic Besov space, circumventing the curse of dimensionality and theoretically establishing the efficiacy of in-context learning in high-dimensional settings.

Theorem 4.7 (informal version of Theorem C.7) . For the anisotropic Besov space of smoothness ( α 1 , · · · , α d ) , assume variance decay (4) with s = ˜ α > 1 /p . If T ≳ nN and N ≍ n 1 2 ˜ α +1 , in-context learning achieves the minimax optimal rate n -2 ˜ α 2 ˜ α +1 up to a log factor.

4.4 Learning a Coarser Basis

Thus far, we have demonstrated the importance of sufficient pretraining to achieve optimal risk; as another application of our framework, we illustrate how pretraining can actively mitigate the complexity of in-context learning. Consider the case where ( ψ ◦ j ) ∞ j =1 is no longer the B-spline basis of B α p,q ( X ) but instead is chosen from some wider function space, say the unit ball of B τ p,q ( X ) for a smaller smoothness τ < α . Without knowledge of the basis, the sample complexity of learning any regression function F ◦ β is a priori lower bounded by the minimax rate n -2 τ 2 τ + d by Proposition 4.2. For ICL, this difficulty manifests as an increase in the metric entropy of the class F N which must be powerful enough to approximate ψ ◦ 1: N (Corollary C.12), giving rise to the modified risk bound:

Corollary 4.8 (ICL for coarser basis) . Suppose α > τ > d/p , the basis ( ψ ◦ j ) ∞ j =1 ⊂ U ( B τ p,q ( X )) and Assumptions 1, 2 hold with r = 1 / 2 , s = α/d . Then if n ≳ N log N ,

Hence if T ≳ n 1+ d 2 α + d α + d τ and N ≍ n d 2 α + d , the risk converges as n -2 α 2 α + d log n .

The pretraining generalization gap is now dominated by the higher complexity N 1+ α τ + d τ log 3 N of the DNN class and strictly worse compared to N 2 log N for Theorem 4.5. The required number of tasks also suffers and the exponent is no longer 2 α +2 d 2 α + d ∈ (1 , 2) but scales as O ( d ) . Nevertheless, observe that the burden of complexity is entirely carried by T ; with sufficient pretraining, the third term can be made arbitrarily small and the ICL risk again attains n -2 α 2 α + d . Hence ICL improves upon the a priori lower bound n -2 τ 2 τ + d at inference time by encoding information on the coarser basis during pretraining. We remark that the result is also readily adapted to the anisotropic setting.

4.5 Sequential Input and Transformers

We now consider a more complex setting where the inputs x ∈ [0 , 1] d ×∞ are bidirectional sequences of tokens (e.g. entire documents) and ϕ is itself a transformer network. 2 In this infinite-dimensional setting, transformers can still circumvent the curse of dimensionality and in fact achieve near-optimal sample complexity due to their parameter sharing and feature extraction capabilities (Takakura and Suzuki, 2023). Our goal in this section is to extend this guarantee to ICL of trained transformers.

For sequential data, it is natural to suppose the smoothness w.r.t. each coordinate can vary depending on the input. For example, the position of important tokens in a sentence will change if irrelevant strings are inserted. To this end, we adopt the piecewise γ -smooth function class introduced by Takakura and Suzuki (2023), which allows for arbitrary bounded permutations among input tokens; see Appendix D.1 for definitions. Also borrowing from their setup, we consider multi-head sliding window self-attention layers with window size U , embedding dimension D , number of heads H with key, query, value matrices K ( h ) , Q ( h ) ∈ R D × d , V ( h ) ∈ R d × d and norm bound M defined as 3

We also consider a linear embedding layer Enc ( x ) = Ex + P , E ∈ R D × d with absolute positional encoding P ∈ R D of bounded norm. Then the class of depth J transformers is defined as

Our result, proved in Appendix D.2, reads:

Theorem 4.9 (informal version of Theorem D.1) . Suppose F ◦ consists of functions on [0 , 1] d ×∞ of bounded piecewise γ -smooth and L ∞ -norm with smoothness α ∈ R d ×∞ > 0 , and let γ be mixed or anisotropic smoothness with α † = max i,j α ij or ( ∑ i,j α -1 ij ) -1 , respectively. Under suitable regularity and decay assumptions, by taking F N to be a class of clipped transformers it holds that

This matches the optimal rate in finite dimensions independently of the (possibly infinite) length of the input or context window. The dynamical feature extraction ability of attention layers in the F TF class is essential in dealing with input-dependent smoothness, further justifying the efficiacy of ICL of sequential data.

5 Minimax Lower Bounds

In this section, we provide lower bounds for the minimax rate in both n, T by extending the theory of Yang and Barron (1999), which can be leveraged to yield results stronger than optimality in merely n . The bound is purely information-theoretic and hence applies to not just ICL but any meta-learning scheme for the regression problem of Section 2.1 from the data D n,T = { ( X ( t ) , y ( t ) ) } T +1 t =1 , where the index T +1 corresponds to the test task.

For this section we assume that the noise (1) is i.i.d. Gaussian, ξ k ∼ N (0 , σ 2 ) , instead of bounded; while the exact shape of the noise distribution is not important, having restricted support may convey additional information and affect the minimax rate. We also suppose for simplicity that the support of P β is included in B := { β ∈ R ∞ | | β j | 2 ≲ j -2 s -1 (log j ) -2 , j ∈ N } and that the aggregated coefficients ¯ β j for ¯ N ≤ j ≤ ¯ N satisfy E β [ ¯ β 2 j ] ≤ σ 2 β for some σ β dependent on N . The proof of the following statement is given in Appendix F.1.

Proposition 5.1. For ε n, 1 , ε n, 2 , δ n > 0 , let Q 1 and Q 2 be the ε n, 1 - and ε n, 2 -covering numbers of F N and B respectively, and M be the δ n -packing number of F ◦ . Suppose that the following conditions are satisfied:

Then the minimax rate is lower bounded as

Finally, Proposition 5.1 is applied to obtain concrete lower bounds for the settings studied in Section 4 throughout Appendices F.2-F.4.

Corollary 5.2 (minimax lower bound) . The minimax rates in the previous regression settings are lower bounded as follows.

(i) Besov space (Section 4.2): inf ̂ f sup f ◦ E D n,T [ ∥ ̂ f -f ◦ ∥ 2 ] ≳ n -2 α 2 α + d ,

(ii) Coarser basis (Section 4.4): inf ̂ f sup f ◦ E D n,T [ ∥ ̂ f -f ◦ ∥ 2 ] ≳ n -2 α 2 α + d +( nT ) -2 τ 2 τ + d ,

(iii) Sequential input (Section 4.5): inf ̂ f sup f ◦ E D n,T [ ∥ ̂ f -f ◦ ∥ 2 ] ≳ n -2 α † 2 α † +1 .

These results match the upper bounds for (i), (iii) and show that ICL is provably jointly optimal in n, T in the 'large T ' regime. Moreover, we can check for the coarser basis setting that insufficient pretraining T = O (1) indeed leads to the worse complexity n -2 τ 2 τ + d , while the faster rate n -2 α 2 α + d is retrieved when T ≳ n ( α -τ ) d (2 α + d ) τ . This aligns with the discussion in Section 4.4, showing that ICL is provably suboptimal in the 'small T ' regime.

Remark 5.3. The obtained upper and lower bounds in the coarser basis setting are not tight as T varies, hence it remains to be shown whether there exists a meta-learning algorithm that attains the lower bound (ii). The task diversity threshold for optimal learning suggested by the bounds are also different ( n 1+ d ( α + d ) (2 α + d ) v.s. n ( α -τ ) d (2 α + d ) τ ); it would be interesting for future work to resolve this gap.

6 Conclusion

In this paper, we performed a learning-theoretic analysis of ICL of a transformer consisting of a DNN and a linear attention layer pretrained on nonparametric regression tasks. We developed a general framework for bounding the in-context estimation error of the empirical risk minimizer in terms of both the number of tasks and samples, and proved that ICL can achieve nearly minimax optimal rates in the Besov space, anisotropic Besov space and γ -smooth class. We also demonstrated that ICL can improve upon the a priori optimal rate by learning informative representations during pretraining. We supplemented our analyses with corresponding minimax lower bounds jointly in n, T and also performed numerical experiments validating our findings. Our work opens up interesting approaches of adapting classical learning theory to study emergent phenomena of foundation models.

Limitations. Our transformer model is limited to a single layer of linear self-attention and does not consider more complex in-context learning behavior which may arise in transformers with multiple attention layers. Moreover, the obtained upper and lower bounds are not tight in certain regimes, suggesting future research directions for meta-learning.

Acknowledgments

JK was partially supported by JST CREST (JPMJCR2015). TS was partially supported by JSPS KAKENHI (24K02905, 20H00576) and JST CREST (JPMJCR2115). We would like to express our gratitude to Masaaki Imaizumi for valuable and insightful discussions on the topic in relation to his work in progress (Imaizumi, 2024).

References

Ahn, X. Cheng, H. Daneshmand, and S. Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. arXiv preprint arXiv:2306.00297 , 2023.

Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou. What learning algorithm is incontext learning? Investigations with linear models. In International Conference on Learning Representations , 2023.

Bai, F. Chen, H. Wang, C. Xiong, and S. Mei. Transformers as statisticians: provable incontext learning with in-context algorithm selection. In ICML Workshop on Efficient Systems for Foundation Models , 2023.

Bhojanapalli, C. Yun, A. S. Rawat, S. J. Reddi, and S. Kumar. Low-rank bottleneck in multi-head attention models. In International Conference on Machine Learning , 2020.

Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems , 2020.

Chen, T. Dao, E. Winsor, Z. Song, A. Rudra, and C. Ré. Scatterbrain: unifying sparse and low-rank attention approximation. In Advances in Neural Information Processing Systems , 2021.

Chen, H. Sheen, T. Wang, and Z. Yang. Training dynamics of multi-head softmax attention for in-context learning: emergence, convergence, and optimality. arXiv preprint arXiv:2402.19442 , 2024.

A. DeVore and V. A. Popov. Interpolation of Besov spaces. Transactions of the American Mathematical Society , 305(1):397-414, 1988.

L. Donoho and I. M. Johnstone. Minimax estimation via wavelet shrinkage. The Annals of Statistics , 26(3):879-921, 1998.

Du, W. Hu, S. Kakade, J. Lee, and Q. Lei. Few-shot learning via learning the representation, provably. In International Conference on Learning Representations , 2021.

D˜ ung. Optimal adaptive sampling recovery. Advances in Computational Mathematics , 34:1-41, 2011a.

D˜ ung. B-spline quasi-interpolant representations and sampling recovery of functions with mixed smoothness. Journal of Complexity , 27(6):541-567, 2011b.

García, Y. Bansal, C. Cherry, G. F. Foster, M. Krikun, F. Feng, M. Johnson, and O. Firat. The unreasonable effectiveness of few-shot learning for machine translation. In International Conference on Machine Learning , 2023.

Garg, D. Tsipras, P. Liang, and G. Valiant. What can Transformers learn in-context? A case study of simple function classes. In Advances in Neural Information Processing Systems , 2022.

Giné and R. Nickl. Mathematical foundations of infinite-dimensional statistical models . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2015.

Guo, W. Hu, S. Mei, H. Wang, C. Xiong, S. Savarese, and Y. Bai. How do Transformers learn in-context beyond simple functions? A case study on learning with representations. arXiv preprint arXiv:2310.10616 , 2023.

Hayakawa and T. Suzuki. On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces. Neural Networks , 123:343-361, 2020.

Huang, Y. Cheng, and Y. Liang. In-context convergence of Transformers. arXiv preprint arXiv:2310.05249 , 2023.

Imaizumi. Statistical analysis on in-context learning, 2024. Personal communication.

Kim and T. Suzuki. Transformers learn nonlinear features in context: nonconvex mean-field dynamics on the attention landscape. In International Conference on Machine Learning , 2024.

Li, M. Wang, S. Lu, X. Cui, and P.-Y. Chen. Training nonlinear Transformers for efficient in-context learning: a theoretical learning and generalization analysis. arXiv preprint arXiv:2402.15607 , 2024.

Mahankali, T. B. Hashimoto, and T. Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. arXiv preprint arXiv:2307.03576 , 2023.

Meunier, Z. Li, A. Gretton, and S. Kpotufe. Nonlinear meta-learning can guarantee faster rates. arXiv preprint arXiv:2307.10870 , 2023.

M. Nikol'skii. Approximation of functions of several variables and imbedding theorems , volume 205 of Grundlehren der mathematischen Wissenschaften . Springer Berlin, 1975.

Nishimura and T. Suzuki. Minimax optimality of convolutional neural networks for infinite dimensional input-output problems and separation from kernel methods. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id= EW8ZExRZkJ .

Okumoto and T. Suzuki. Learnability of convolutional neural networks for infinite dimensional input via mixed and anisotropic smoothness. In International Conference on Learning Representations , 2022.

Raventos, M. Paul, F. Chen, and S. Ganguli. Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression. In Advances in Neural Information Processing Systems , 2023.

Schmidt-Hieber. Nonparametric regression using deep neural networks with ReLU activation function. The Annals of Statistics , 48(4), 2020.

Suzuki. Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality. In International Conference on Learning Representations , 2019.

Suzuki and A. Nitanda. Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space. In Advances in Neural Information Processing Systems , 2021.

J. Szarek. Nets of Grassmann manifold and orthogonal group. Proceedings of Banach Space Workshop , pages 169-186, 1981.

Takakura and T. Suzuki. Approximation and estimation ability of Transformers for sequenceto-sequence functions with infinite dimensional input. In International Conference on Machine Learning , 2023.

Triebel. Theory of function spaces . Monographs in mathematics. Birkhäuser Verlag, 1983.

Triebel. Entropy numbers in function spaces with mixed integrability. Revista Matematica Complutense , 24:169-188, 2011.

Tripuraneni, C. Jin, and M. Jordan. Provable meta-learning of linear representations. In International Conference on Machine Learning , 2020.

A. Tropp. An introduction to matrix concentration inequalities. Foundations and Trends in Machine Learning , 8(1-2):1-230, May 2015. ISSN 1935-8237.

W. van der Vaart and J. A. Wellner. Weak convergence and empirical processes: with applications to statistics . Springer, New York, 1996.

von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning , 2023.

Vybíral. Function spaces with dominating mixed smoothness , volume 30 of Lectures in Mathematics . European Mathematical Society, 2006.

Vybíral. Widths of embeddings in function spaces. Journal of Complexity , 24(4):545-570, 2008.

Wu, D. Zou, Z. Chen, V. Braverman, Q. Gu, and P. L. Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? In International Conference on Learning Representations , 2024.

Yang and A. Barron. Information-theoretic determination of minimax rates of convergence. The Annals of Statistics , 27(5):1564-1599, 1999.

Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks , 94: 103-114, 2016.

Zhang, S. Frei, and P. L. Bartlett. Trained Transformers learn linear models in-context. arXiv preprint arXiv:2306.09927 , 2023.

Zhang, J. Wu, and P. L. Bartlett. In-context learning of a linear Transformer block: benefits of the MLP component and one-step GD initialization. arXiv preprint arXiv:2402.14951 , 2024.

Table of Contents

Appendix

A Proof of Proposition 3.2

A.1 Decomposing Approximation Error

Recall that

We introduce some additional notation in the following fashion. For brevity, we write ¯ N : ∞ instead of ( ¯ N +1) : ∞ as an exception.

Since clipping ˇ f Θ does not make its difference with F ◦ β ∈ [ -B,B ] larger, it holds that

due to Assumption 2 and ¯ N ≍ N . Expanding the attention output as

and the final term further as

we obtain that

We proceed to control each term separately, from which the statement of Proposition 3.2 will follow.

A.2 Bounding Term (8)

Denote the operator norm (with respect to L 2 norm of vectors) by ∥·∥ op . For (13), noting that Σ Ψ ,N is positive definite,

due to the independence of X , ˜ x and β . For the last inequality, we have used the fact that both the Σ 1 / 2 Ψ ,N Γ ∗ Σ Ψ ,N Γ ∗ Σ 1 / 2 Ψ ,N term and the matrix multiplied to it are positive semi-definite, and the former is bounded above as ≾ I N by Assumption 1.

Furthermore, we utilize the following result proved in Appendix A.5:

It follows that (13) is bounded as

since Tr(Σ ¯ β,N ) is bounded by Assumption 2.

Moreover for (14), we compute

A.3 Bounding Term (9)

Since the sequence of covariates x 1 , · · · , x n and noise ξ 1 , · · · , ξ n are each i.i.d.,

for independent samples x, x ′ , ˜ x ∼ P X . We now bound the three terms separately below. For (15), we have that

since Σ Ψ ,M ⪯ C 2 I N as M →∞ by Assumption 2. As sup x ∥ ψ ◦ ¯ N : ¯ N ( x ) ∥ 2 ≲ N -2 r by (2) and the diagonal of E β [ β ¯ N : M β ⊤ ¯ N : M ] decays faster than ¯ N -2 s M -1 (log M ) -2 when M > ¯ N due to (4), it follows that

Similarly, for (16), we have that

And for (17), we have that

Therefore, we obtain the following bound:

A.4 Bounding Terms (10) -(12)

For (10), we use the Cauchy-Schwarz inequality and Assumption 3 to bound

Similarly for (11) and (12), we have

and

and

This concludes the proof of Proposition 3.2.

A.5 Proof of Lemma A.1

We will make use of the following concentration bound and its corollary, proved at the end of this subsection.

Theorem A.2 (matrix Bernstein inequality) . Let S 1 , · · · , S n ∈ R N × N be independent random matrices such that E [ S i ] = 0 and ∥ S i ∥ op ≤ L almost surely for all i . Define Z = ∑ n i =1 S n and the matrix variance statistic v ( Z ) as

Then it holds for all t > 0 that

Proof. See Tropp (2015), Theorem 6.1.1.

Corollary A.3. For matrices S 1 , · · · , S n satisfying the conditions of Theorem A.2, it holds that

We will apply Corollary A.3 to the matrices

It is straightforward to verify that

almost surely by Assumption 1.

Next, we evaluate the matrix variance statistic. Since each S i is symmetric,

for a single sample x ∼ P X . The second term is further bounded as

again by Assumption 1.

Hence we may apply Corollary A.3 with v ( Z ) ≲ nN 2 r , L ≲ N 2 r to conclude:

Proof of Corollary A.3. From Theorem A.2 and with the change of variables λ = t 2 /n 2 , we have

Since the probability is also bounded above by 1, it follows that

and the expectation can be controlled as

For the first integral, we truncate at λ 1 := 4 v ( Z ) n 2 log 2 N so that

For the second integral, we truncate at λ 2 := ( 4 L 3 n log 2 N ) 2 so that

Adding up, we conclude that

as was to be shown.

B Proofs on Metric Entropy

B.1 Modified Proof of Theorem 3.1

For a full proof of the original statement, we refer the reader to Section B.1 of Hayakawa and Suzuki (2020), which corrects some technical flaws in the original proof. Here we only outline the necessary modification to incorporate the bounded noise setting.

Denote an ϵ -cover of the model space by f 1 , · · · , f M . The only step which relies on the normality of noise ξ 1: n is a concentration result for the random variables

where it is shown via the normality of ε j that

We will instead rely on Hoeffding's inequality. By writing ε j = w ⊤ j ξ 1: n = ∑ n i =1 w j,i ξ i where

since | w j,i ξ i | ≤ σ | w j,i | a.s. it follows that

for all u > 0 . Then the squared-exponential moment of each ε j is bounded as

by setting t = 1 4 σ 2 . Hence via Jensen's inequality we obtain

and thus

retrieving the original result up to a constant.

B.2 Proof of Lemma 3.3

Let us take two functions f Θ 1 , f Θ 2 ∈ T N for Θ i = (Γ i , ϕ i ) , i = 1 , 2 separated as

Then it holds that

Hence to construct an ϵ -cover G T of T N , it suffices to exhibit a δ 1 -cover G S of S N and a δ 2 -cover G F of F N for

and set G T = G S × G F . For the metric entropy, this implies

We further bound the metric entropy of S N with the following result, proved in Appendix B.3.

Substituting into the above, we conclude that

The choice of δ 2 is not important as long as the metric entropy of F N is at most polynomial in δ 2 .

B.3 Proof of Lemma B.1

Let Γ 1 , Γ 2 ∈ S N and consider their diagonalizations

where U i ∈ O N , the orthogonal group in dimension N , and 0 ≤ λ i,j ≤ C 3 for each 1 ≤ j ≤ N . Assuming

it follows that

Moreover, the covering number of O N in operator norm is given by the following result.

Theorem B.2 (Szarek (1981), Proposition 6) . There exist universal constants c 1 , c 2 > 0 such that for all N ∈ N and δ ∈ (0 , 2] ,

Hence we obtain that

Finally, we remark that if elements of the domain S N are not constrained to be symmetric, we can alternatively consider the singular value decomposition and separately bound entropy of the two rotation components, giving the same result up to constants.

C Details on Besov-type Spaces

C.1 Besov Space

C.1.1 Verification of Assumptions

We first give some background on wavelet decomposition. The decay rate s = α/d is intrinsic to the Besov space as shown by the following result, which allows us to translate between functions f ∈ B α p,q ( X ) and their B-spline coefficient sequences.

Lemma C.1 (DeVore and Popov (1988), Corollary 5.3) . If α > d/p and m > α + 1 -1 /p , a function f ∈ L p ( X ) is in B α p,q ( X ) if and only if f can be represented as

such that the coefficients satisfy

with appropriate modifications if p = ∞ or q = ∞ . Moreover, the two norms ∥ ˜ β ∥ b α p,q and ∥ f ∥ B α p,q are equivalent.

In particular, this implies that for any f ∈ U ( B α p,q ( X )) the p -norm average of ˜ β k,ℓ for ℓ ∈ I d k at resolution k is bounded as

and the coefficients β k,ℓ = 2 -kd/ 2 ˜ β k,ℓ w.r.t. the scaled basis (2 kd/ 2 ω d k,ℓ ) k,ℓ satisfy

Thus it is natural in a probabilistic sense to assume E [ β p k,ℓ ] 1 /p ≲ (2 kd ) -α/d -1 / 2 . This will be the case if for instance we sample ( β k,ℓ ) ℓ ∈ I d k uniformly from the p -norm ball (18). This matches Assumption 4 up to the logarithmic factor and hence the rate is nearly tight in variance, even though (18) only applies to the average over locations rather than each coefficient explicitly. See also the discussion in Lemma 2 of Suzuki (2019).

Assumption 1. We take m to be even for simplicity. The wavelet system ( ω d K,ℓ ) ℓ ∈ I d K at each resolution K is linearly independent; for any g ∈ L 2 ( X ) that can be expressed as

we have the quasi-norm equivalence (D˜ ung, 2011b, 2.15)

which implies that the covariance matrix E x ∼ Unif([0 , 1] d ) [ ψ ◦ ¯ N : ¯ N ( x ) ψ ◦ ¯ N : ¯ N ( x ) ⊤ ] is bounded above and below. Since we assume P X has Lebesgue density bounded above and below, it follows that Σ Ψ ,N is uniformly bounded above and below for all K ≥ 0 .

In contrast, any B-spline at a lower resolution k < K can be exactly expressed as a linear combination of elements of ( ω d K,ℓ ) ℓ ∈ I d K by repeatedly applying the following relation.

Lemma C.2 (refinement equation) . For even m and r = ( r 1 , · · · , r d ) ⊤ , 1 = (1 , · · · , 1) ⊤ ∈ R d it holds that

Proof. The relation for one-dimensional wavelets is given in equation (2.21) of D˜ ung (2011b),

from which it follows that

as was to be shown.

Therefore we select all B-splines at a fixed resolution K to approximate the target tasks,

It is straightforward to see that 0 ≤ ω d k,ℓ ( x ) ≤ 1 for all x ∈ X and moreover the B-splines (extended to all ℓ ∈ Z d ) form a partition of unity of R d at all resolutions:

Then for all x ∈ X we have the bound

and hence (2) holds with r = 1 / 2 .

and

Assumption 2. For any β ∈ supp P β we have that

Furthermore, the convergence rate of the truncated approximation F ◦ β, ¯ N is determined by the decay rate of β in Lemma C.1 as follows (it does not matter whether we bound F ◦ β, ¯ N or F ◦ β,N since ¯ N ≍ N ). We consider a truncation of all resolutions lower than K so that ¯ N,N ≍ 2 Kd . Then it holds that

where for the last two inequalities we have used the inequality ∥ z ∥ 2 ≤ D 1 / 2 -1 /p ∥ z ∥ p for z ∈ R D and p ≥ 2 in conjunction with (18). Thus our choice of s = α/d is justified. Under this choice, (5) directly implies (4) as

holds for the basis elements at each resolution K , that is for those numbered between N and ¯ N .

¯ Remark C.3. When 1 ≤ p < 2 , the truncation up to ¯ N does not suffice to achieve the N -2 α/d approximation rate, and basis elements must be judiciously selected from a wider resolution range. More concretely, a size N subset of all wavelets up to resolution K ′ = ⌈ K (1 + ν -1 ) ⌉ where ν = pα/ 2 d -1 / 2 > 0 must be used (Suzuki and Nitanda, 2021, Lemma 2). Hence the exponent is a factor of 1 + ν -1 worse w.r.t. N ′ ≍ 2 K ′ d , leading to the inevitable suboptimal rate.

To show boundedness of Tr(Σ ¯ β,N ) , we analyze the composition of the aggregated coefficients ¯ β using the following result.

Corollary C.4. For any 0 ≤ k < k ′ there exists constants γ k,k ′ ,ℓ,ℓ ′ ≥ 0 for ℓ ∈ I d k , ℓ ′ ∈ I d k ′ such that

holds for all ( β k,ℓ ) ℓ ∈ I d k . Moreover, it holds that

The statement follows directly from the more general Proposition C.10, stated and proved in Appendix C.3 below, by restricting to wavelets with uniform resolution across dimensions. Using Corollary C.4, we can refine each lower resolution component of F ◦ β to resolution K :

Thus each aggregated coefficient, indexed here by ℓ ∈ I d K , can be expressed as

Hence it follows that

from which we conclude that Tr(Σ ¯ β,N ) = ∑ ℓ ∈ I d K E β [ ¯ β 2 K,ℓ ] ≲ N · N -1 is uniformly bounded.

Finally for the verification of Assumption 3, see Appendix C.1.2.

C.1.2 Proof of Lemma 4.4

We use the following result to approximate each wavelet ω d K,ℓ at resolution K with the class (6). The proof, in turn, relies on the construction by Yarotsky (2016) of DNNs which efficiently approximates the multiplication operation.

Lemma C.5 (Suzuki (2019), Lemma 1) . For all δ > 0 , there exists a ReLU neural network ˜ ω ∈ F DNN ( L, W, S, M ) with

satisfying supp ˜ ω ⊆ [0 , m +1] d and ∥ ω d 0 , 0 -˜ ω ∥ L ∞ ( X ) ≤ δ

Here, δ is also dependent on N .

Now consider N identical copies of ˜ ω in parallel, where each module is preceded by the scaling ( x i ) d i =1 ↦→ (2 K x i -ℓ i ) d i =1 for ℓ ∈ I d K and whose output is scaled by 2 Kd/ 2 . In particular, these operations can be implemented by K ≲ log N consecutive additional layers with norm bounded by a constant. Hence each module ϕ ∗ ¯ N , · · · , ϕ ∗ ¯ N approximates the basis 2 Kd/ 2 ω d K,ℓ with 2 Kd/ 2 δ ≲ √ Nδ accuracy, and substituting δ N = √ Nδ gives that

with L ≲ log δ -1 +log N ≲ log δ -1 N +log N . Note that the sparsity S is only multiplied by a factor of N since different modules do not share any connections. Moreover the target basis has 2-norm bounded as

where we have again used the sparsity of ω d k,ℓ at each resolution. Hence we may clip the magnitude of the vector output ϕ by B ′ N and the approximation guarantee remains unchanged.

To bound the covering number of F N , we directly apply the following result.

Lemma C.6 (Suzuki (2019), Lemma 3) . The covering number of F DNN is bounded as

Since clipping the magnitude of the outputs does not increase the covering number, we conclude:

C.1.3 Proof of Theorem 4.5

By Lemma 3.3 and Lemma 4.4, the metric entropy of T N is bounded as

Combining with Theorem 3.1 and Proposition 3.2 with r = 1 / 2 and s = α/d gives

Substituting δ N ≍ N -1 -α/d and ϵ ≍ N -2 α/d yields the desired bound.

C.2 Anisotropic Besov Space

C.2.1 Definitions and Results

For 1 ≤ p, q ≤ ∞ , directional smoothness α = ( α 1 , · · · , α d ) ∈ R d > 0 and r = max i ≤ d ⌊ α i ⌋ +1 , we define ∥·∥ B α p,q = ∥·∥ L p + | · | B α p,q where

The anisotropic Besov space is defined as

The definition reduces to the usual Besov space if α 1 = · · · = α d ; see Vybíral (2006); Triebel (2011) for details. We also write α = max i α i , α = min i α i and the harmonic mean smoothness as

For the anisotropic Besov space, we need to redefine the wavelet basis so that the sensitivity to resolution k ∈ Z ≥ 0 differs for each component depending on α . Define the quantities

We then set for each k ≥ 0 and ℓ ∈ I d,α k

and take the scaled basis

with the natural hierarchy induced by k .

The minimax optimal rate for the anisotropic Besov space is equal to n -2 ˜ α 2 ˜ α +1 (Suzuki and Nitanda, 2021, Theorem 4). Our result for in-context learning is as follows.

Theorem C.7 (minimax optimality of ICL in anisotropic Besov space) . Let α ∈ R d > 0 with ˜ α > 1 /p and F ◦ = U ( B α p,q ( X )) . Suppose that P X has positive Lebesgue density ρ X bounded above and below on X . Also suppose that all coefficients are independent and

Then for n ≳ N log N we have

Hence if T ≳ nN and N ≍ n 1 2 ˜ α +1 , in-context learning achieves the rate n -2 ˜ α 2 ˜ α +1 log n which is minimax optimal up to a polylog factor.

C.2.2 Proof of Theorem C.7

The overall approach is similar to Appendix C.1. The decay rate of functions in the anisotropic Besov space is characterized by the following result which extends Lemma C.1.

Lemma C.8 (Suzuki and Nitanda (2021), Lemma 2) . If ˜ α > 1 /p and m>α +1 -1 /p , a function f ∈ L p ( X ) is in MB α p,q ( X ) if and only if f can be represented as

such that the coefficients satisfy

Moreover, the two norms ∥ ˜ β ∥ b α p,q and ∥ f ∥ B α p,q are equivalent.

We again select all B-splines ( ω d,α K,ℓ ) ℓ ∈ I d K at each resolution K to approximate the target functions. By repeatedly applying the refinement equation for one-dimensional wavelets as many times as needed to each dimension separately, we may express any B-spline at a lower resolution k < K as a linear combination of ( ω d,α K,ℓ ) ℓ ∈ I d K similarly to Lemma C.2. See Proposition C.10 for details. We thus have

Since ∥ k ∥ α/α = kα/ ˜ α + O k (1) always holds, it also follows that

and similarly ¯ N ≍ N . Therefore,

and the scaled coefficients decay in average as

For Assumption 2, we can check that

and for a resolution cutoff K > 0 , ¯ N ≍ 2 ∥ K ∥ α/α the truncation error satisfies

Thus we may set r = 1 / 2 , s = ˜ α and take the variance decay rate (20) as

For boundedness of Tr(Σ ¯ β,N ) , we use the following result which is also obtained from Proposition C.10 by considering resolution vectors ( ⌊ kα/α 1 ⌋ , · · · , ⌊ kα/α d ⌋ ) and ( ⌊ k ′ α/α 1 ⌋ , · · · , ⌊ k ′ α/α d ⌋ ) . Corollary C.9. For any 0 ≤ k < k ′ there exists constants γ k,k ′ ,ℓ,ℓ ′ ≥ 0 for ℓ ∈ I d,α k , ℓ ′ ∈ I d,α k ′ such that

holds for all ( β k,ℓ ) ℓ ∈ I d,α k . Moreover, it holds that

We apply Corollary C.9 to refine all components of F ◦ β, ¯ N to resolution K :

Hence it follows that

and we again conclude that Tr(Σ ¯ β,N ) is uniformly bounded.

The rest of the proof proceeds similarly to the ordinary Besov space.

C.3 Wavelet Refinement

In this subsection, we present and prove an auxiliary result concerning the refinement of B-spline wavelets and the recurrence relations satisfied by their coefficient sequences.

Proposition C.10. For any k, k ′ ∈ Z d ≥ 0 such that k ′ -k ∈ Z d ≥ 0 there exists constants γ k,k ′ ,ℓ,ℓ ′ ≥ 0 for ℓ ∈ I d k , ℓ ′ ∈ I d k ′ such that

holds for all ( β k,ℓ ) ℓ ∈ I d k . Moreover, it holds that

Proof. We proceed by induction on ∥ k ′ -k ∥ 1 . When k ′ = k + e j for some 1 ≤ j ≤ d , we can refine each ω d k,ℓ using equation (2.21) of D˜ ung (2011b) as

̸

̸

Since ℓ +( ℓ j + r -m 2 ) e j matches a given location vector ℓ ′ ∈ I d k + e j if and only if ℓ i = ℓ ′ i ( i = j ) and ℓ ′ j = 2 ℓ j + r -m 2 , comparing coefficients in (21) yields

̸

Here, 1 A denotes the indicator function for condition A . It follows that γ k,k + e j ,ℓ,ℓ ′ ≥ 0 and

by considering parities.

Now suppose the claim holds for a fixed difference ∥ k ′ -k ∥ 1 . Applying the above derivation to further refine resolution k ′ to k ′′ = k ′ + e j for arbitrary j gives

where

̸

̸

̸

k k Hence we obtain the recurrence relation γ k,k ′ + e j ,ℓ,ℓ ′′ = ∑ ℓ ′ ∈ I d k ′ 2 -m +1 / 2 1 { ℓ ′ i = ℓ ′′ i ( i = j ) } ( m ℓ ′′ j -2 ℓ ′ j + m 2 ) γ k,k ′ ,ℓ,ℓ ′ , from which we verify that γ k,k ′ + e j ,ℓ,ℓ ′′ ≥ 0 and ∑ ℓ ∈ I d k γ k,k ′ + e j ,ℓ,ℓ ′′ = ∑ ℓ ′ ∈ I d k ′ 2 -m +1 / 2 1 { ℓ ′ i = ℓ ′′ i ( i = j ) } ( m ℓ ′′ j -2 ℓ ′ j + m 2 ) ∑ ℓ ∈ I d k γ k,k ′ ,ℓ,ℓ ′ ≤ 2 -m +1 / 2 -∥ k ′ -k ∥ 1 / 2 ∑ ℓ ′ j ∈ Z ( m ℓ ′′ j -2 ℓ ′ j + m 2 ) = 2 -( ∥ k ′ -k ∥ 1 +1) / 2 , and furthermore ∑ ℓ ′′ ∈ I d k ′ + e j γ k,k ′ + e j ,ℓ,ℓ ′′ = ∑ ℓ ′ ∈ I d k ′ ∑ ℓ ′′ ∈ I d k ′ + e j 2 -m +1 / 2 1 { ℓ ′ i = ℓ ′′ i ( i = j ) } ( m ℓ ′′ j -2 ℓ ′ j + m 2 ) γ k,k ′ ,ℓ,ℓ ′ ≤ 2 -m +1 / 2 ∑ ℓ ′ ∈ I d k ′ ∑ ℓ ′′ j ∈ Z ( m ℓ ′′ j -2 ℓ ′ j + m 2 ) γ k,k ′ ,ℓ,ℓ ′ ≤ 2 1 / 2 ∑ ℓ ′ ∈ I d k ′ γ k,k ′ ,ℓ,ℓ ′ ≤ 2 ( ∥ k ′ -k ∥ 1 +1) / 2 .

This concludes the proof.

C.4 Proof of Corollary 4.8

In order to approximate arbitrary ψ ◦ j ∈ U ( B τ p,q ( X )) , we need the following construction instead of Lemma C.5. Note that N ′ corresponds to the number of B-splines used to approximate the target function and can be freely chosen to match the desired error, which however affects the covering number of F N .

Lemma C.11 (Suzuki (2019), Proposition 1) . Set m ∈ N , m>τ +2 -1 /p and ν = ( pτ -d ) / 2 d . For all N ′ ∈ N sufficiently large and ϵ = N ′-τ/d (log N ′ ) -1 , for any f ◦ ∈ U ( B τ p,q ( X )) there exists a ReLU network ˜ f ∈ F DNN ( L, W, S, M ) with

Also note that from Assumption 1 it follows that ∥ ψ j ∥ L ∞ ( P X ) ≤ C ∞ N 1 / 2 . Setting N ′ ≍ δ -d/τ N and applying the covering number bound in Lemma C.6, after some algebra we obtain the following counterpart to Lemma 4.4.

Corollary C.12. For any δ N > 0 , Assumption 3 is satisfied by taking

where B ′ N = C ∞ N 1 / 2 and

̸

̸

Also, the metric entropy of F N is bounded as

Then by combining with Lemma 3.3 and Proposition 3.2 via Theorem 3.1, it follows that

and

Substituting δ N ≍ N -1 -α/d and ϵ ≍ N -2 α/d concludes the desired bound.

D Details on Sequential Input

D.1 Definitions and Results

γ -smooth class. We first define the γ -smooth function class introduced by Okumoto and Suzuki (2022). Let r ∈ Z d ×∞ 0 and s ∈ ¯ N d ×∞ 0 , where ¯ N = N ∪{ 0 } and the subscript 0 indicates restriction to the subset of elements with a finite number of nonzero components. Consider the orthonormal basis ( ψ r ) r of L 2 ([0 , 1] d ×∞ ) given as

The frequency s component δ s ( f ) of f ∈ L 2 ([0 , 1] d ×∞ ) is defined as

For a monotonically nondecreasing function γ : ¯ N d ×∞ 0 → R and p ≥ 2 , q ≥ 1 , the γ -smooth norm and function class are defined as

and

The γ -smooth class over finite-dimensional input space [0 , 1] d × m is similarly defined.

In particular, we consider two specific cases of γ for the component-wise smoothness parameter α ∈ R d ×∞ > 0 , for which we also define the corresponding degrees of smoothness α † ∈ R > 0 . Denote by (˜ α j ) ∞ j =1 all components of α sorted by ascending magnitude.

Mixed smoothness: γ ( s ) = ⟨ α, s ⟩ , α † = ˜ α 1 = max i,j α ij .

Anisotropic smoothness: γ ( s ) = max i,j α ij s ij , α † = ( ∑ i,j α -1 ij ) -1 .

Furthermore, the weak l η -norm of α is defined as ∥ α ∥ wl η := sup j j η ˜ α -1 j for η > 0 .

Piecewise γ -smooth class. The piecewise γ -smooth class is an extension of the γ -smooth class allowing for arbitrary bounded permutations of the tokens of an input (Takakura and Suzuki, 2023). For a threshold V ∈ N and an index set Λ , let { Ω λ } λ ∈ Λ be a disjoint partition of supp P X and { π λ } λ ∈ Λ a set of bijections from [2 V +1] to [ -V : V ] . Further define the permutation operator Π : supp P X → R d × (2 V +1) as

Then the piecewise γ -smooth function class is defined as

where

Next, we state the set of assumptions inherited from Takakura and Suzuki (2023). In particular, the importance function makes precise a notion of relative importance between tokens which is preserved by permutations.

Assumption 5 (smoothness and importance function) . 1 < q ≤ 2 and:

The smoothness parameter α satisfies ∥ α ∥ wl η ≤ 1 and α ij = Ω( | i | η ) for some η > 1 . For mixed smoothness, we also require ˜ α 1 < ˜ α 2 .

There exists a shift-equivariant map µ : supp P X → R ∞ such that µ 0 ∈ U ( F γ ∞ ,q ) , ∥ µ 0 ∥ ≤ 1 and Ω λ = { x ∈ supp P X | µ ( x ) π λ (1) > · · · > µ ( x ) π λ (2 V +1) } for all λ ∈ Λ . µ is moreover well-separated, that is µ ( x ) π λ ( v ) -µ ( x ) π λ ( v +1) ≥ C µ v -ϱ for C µ , ϱ > 0 .

We focus on parameter ranges 1 < q ≤ 2 and η > 1 strictly for simplicity of presentation, but the cases q = 1 , q > 2 and η > 0 can be handled with some more analysis. Note that η > 1 ensures α † > 0 for anisotropic smoothness.

Additionally, the assumption pertaining to our ICL setup is stated as follows.

Assumption 6. For r ∈ Z d ×∞ 0 the coefficients β r corresponding to ψ r are independent and satisfy for s ∈ ¯ N d ×∞ 0 such that the frequency component δ s ( f ) contains the element ψ r ,

Also Σ Ψ ,N ≍ I N holds, for example P X is bounded above and below with respect to the product measure λ d ×∞ on B ([0 , 1] d ×∞ ) of the uniform measure λ on B ([0 , 1]) .

We then obtain the following result for ICL with transformers:

Theorem D.1 (minimax optimality of ICL for sequential input) . Let F ◦ = { f ∈ U ( P γ p,q ( P X )) | ∥ f ∥ L ∞ ( P X ) ≤ B } for some B > 0 where γ corresponds to mixed or anisotropic smoothness. Suppose Assumptions 5 and 6 hold. Then for n ≳ N log N we have

Hence if T ≳ nN 1 ∨ 1 /α † and N ≍ n 1 2 α † +1 , ICL achieves the rate n -2 α † 2 α † +1 polylog( n ) .

D.2 Proof of Theorem D.1

Since the system ( ψ r ) r is orthonormal, we may take ¯ N = 1 , ¯ N = N following Remark 2.1. We mainly utilize the following approximation and covering number bounds.

Theorem D.2 (Takakura and Suzuki (2023), Theorem 4.5) . For a function F ◦ ∈ U ( P γ p,q ( P X )) ,

∥ F ◦ ∥ L ∞ ( P X ) ≤ B and any K > 0 , there exists a transformer ̂ F ∈ F TF ( J, U, D, H, L, W, S, M ) such that

where

Theorem D.3 (Takakura and Suzuki (2023), Theorem 5.3) . For ϵ > 0 and B ≥ 1 it holds that

To analyze the decay rate in the γ -smooth class, we approximate a function f ∈ L 2 ([0 , 1] d ×∞ ) by the partial sum of its frequency components up to 'resolution' K , measured via the γ function:

The basis functions ψ r are thus ordered primarily ordered by increasing γ ( s ) .

Lemma D.4 (Okumoto and Suzuki (2022), Lemma 17) . For 1 ≤ q ≤ 2 it holds that

Note that if γ ( s ) < K then s ij < K/a ij ≲ K/ | i | η for all i, j for both types of smoothness and so ∥ s ∥ 0 ≲ dK 1 /η . In addition, the number of basis functions ψ r used in the sum for δ s ( f ) is exactly 2 ∥ s ∥ 1 = ∏ i,j 2 s ij . Theorem D.3 of Takakura and Suzuki (2023) shows that the number of basis elements used in the sum R K ( f ) satisfies

for both mixed and anisotropic smoothness. Hence the N -term approximation error decays as N -α † so that the choice s = α † leading to the assumed variance bound N -2 α † -1 ≍ 2 -(2+1 /α † ) K in (22) is justified. Moreover for large K ,

since ∥ r ∥ 0 = ∥ s ∥ 0 , so that (2) of Assumption 1 is satisfied with r = 1 / 2 . The second part of Assumption 1 holds since ( ψ r ) r ∈ Z d ×∞ 0 is orthonormal w.r.t. λ d ×∞ . Furthermore, the discussion thus far immediately extends to the piecewise γ -smooth class for any partition { Ω λ } λ ∈ Λ by composing with the permutation operator Π .

We proceed to use Theorem D.2 to approximate each basis function ψ r ◦ Π up to resolution K . Moreover, we can see from the proof of Lemma 17 of Okumoto and Suzuki (2022) that we do not need to account for the sup-norm scaling of ψ r and thus it suffices to find the parameter K ′ ∈ N such that the approximation error 2 -K ′ ≍ δ N . Hence combining Theorems D.2 and D.3, we conclude that

is sufficient to satisfy Assumption 3. Therefore, we can now apply our framework with B ′ N ≍ N to obtain the bound

Substituting δ N ≍ N -1 -α † and ϵ ≍ N -2 α † concludes the theorem.

QueryPrediction

QueryPrediction

QueryPrediction

Self-Attention(Linear)

Self-Attention(Softmax)

Feedforward(ReLU)

Output

ndino

Self-Attention(Softmax)

DNN Block

DNN Block

Feedforward(ReLU)

Feedforward (ReLU)

Feedforward(ReLU)

Feedforward (ReLU)

Feedforward (ReLU)

Self-Attention(Softmax)

Input

Input

Input

(a) linear model

(b) Softmax model

(c) full Transformer

train loss

test loss

16

soft

linear

16

linear

soft

14

full

14

full

12

12

10

10

8

8

6

6

4

4

2

2

0

0

0

10

20

30

40

50

0

10

20

30

40

50

epochs

epochs

E Numerical Experiments

In this section, we connect our theoretical contributions to practical transformers by conducting experiments verifying our results as well as justifying the simplified model setup and the empirical risk minimization assumption. We implement and compare the following toy models: (a) the simplified architecture studied in our paper; (b) the same model with linear attention replaced by softmax; and (c) a full transformer with 2 stacked encoder layers. The number of feedforward layers, widths of hidden layers, learning rate, etc. are set to equal for a fair comparison, see Figure 1 for details.

Figure 2 shows training (solid) and test loss curves (dashed) during pretraining. All 3 architectures exhibit similar behavior and converge to near zero training loss, justifying the use of our simplified model and also supporting the assumption that the empirical risk is minimized. Moreover, Figure 3 shows the converged losses over a wide range of N,n,T values. We verify that increasing N,n

(a) output width N

(b) prompt length n

(c) task number T

14

16

16

linear

14

soft

full

12

14

12

10

12

10

8

10

8

8

6

6

6

4

4

4

2

2

2

0

4

8

16

32

64

0

64

128

256

512

1024

0.1

0.25

0.5

1

2

4

N

n

T/n

leads to decreasing train and test error, corresponding to the approximation error of Theorem 3.1. We also observe that increasing T tends to improve the pretraining generalization gap up to a threshold, confirming our theoretical analysis of task diversity. Again, this behavior is consistent across the 3 architectures. We note that in the overparametrized regime when the number of total parameters ≳ nT , the trained model is likely not the empirical risk minimizer, which may also contribute to the large error for large N or small n, T .

F Proofs of Minimax Lower Bounds

F.1 Proof of Proposition 5.1

In this section, we develop our framework for obtaining minimax lower bounds in the ICL setup by adapting the information-theoretic approach of Yang and Barron (1999).

Let { ( ψ ( j ) , β ( j ) T +1 ) } j M =1 be a δ n -packing of the class F ◦ with respect to the L 2 ( X ) -norm such that

where M is the corresponding packing number. Then we have the following proposition as an application of Fano's inequality (Yang and Barron, 1999).

Proposition F.1. Let Θ be a random variable uniformly distributed over { ( ψ ( j ) , β ( j ) T +1 ) } j M =1 . Then, it holds that

where I X (1: T +1) (Θ , y (1: T +1) ) is the mutual information between Θ , y (1: T +1) for given X (1: T +1) .

The mutual information I X (1: T +1) (Θ , y (1: T +1) ) is formulated more concretely as

where p ( y | θ, X ) is the probability density of y conditioned on θ, X and p w is the marginal distribution of y (1: T +1) where w ( · ) ≡ 1 M is the probability mass function of Θ (i.e., p w ( ·| X (1: T +1) ) = ∑ θ ∈ supp Θ w ( θ ) p ( ·| θ, X (1: T +1) ) ). We let P y ( t ) | θ (and P y (1: t ) | θ ) be the distribution of y ( t ) conditioned on θ, X ( t ) (and X (1: t ) ) respectively, and let

be the marginal distribution of y (1: T +1) conditioned on X (1: T +1) .

Next, we define the set { ˜ ψ ( j ) } Q 1 j =1 to be a ε n, 1 -covering of F N w.r.t. the norm d ( ψ, ψ ′ ) := √ E x [ ∥ ψ ( x ) -ψ ′ ( x ) ∥ 2 ] with ε n, 1 -covering number Q 1 , and { ˜ β ( j ) } Q 2 j =1 to be a ε n, 2 -covering of B w.r.t. the L 2 norm with ε n, 2 -covering number Q 2 . By taking all combinations of ( ˜ ψ ( j ) , ˜ β ( j ′ ) ) for 1 ≤ j ≤ Q 1 and 1 ≤ j ′ ≤ Q 2 , we obtain the covering { ˜ θ ( j ) } Q j =1 with respect to the quantity ε 2 n = σ 2 β ε 2 n, 1 + C 2 ε 2 n, 2 where Q = Q 1 Q 2 and each ˜ θ ( j ) is given by ˜ θ ( j ) = ( ˜ ψ ( j 1 ) , ˜ β ( j 2 ) ) for some indices j 1 and j 2 .

Then as in the discussion of Yang and Barron (1999), the mutual information is bounded by

where D ( ·∥· ) is the Kullback-Leibler divergence and ˜ P y (1: T +1) = 1 Q ∑ Q j =1 P y (1: T +1) | ˜ θ ( j ) because ¯ P y (1: T +1) minimizes the right hand side. If we let

then each summand of the right-hand side is further bounded by

Then the KL-divergence can be bounded as

where the joint convexity of KL-divergence was used for the last inequality. Since the observation noise is assumed to be normally distributed, the integrand KL-divergence can be bounded as

Hence, its expectation with respect to β, X ( t ) becomes

In the same manner, we have that

The expectation of the right-hand side with respect to X ( T +1) , β ( j ) T +1 is bounded as

Therefore, the expected mutual information can be bounded as

Applying Proposition F.1 together with (7) concludes the proof.

F.2 Lower Bound in Besov Space

̸

Here, we derive the minimax lower bound when F ◦ = U ( B α p,q ( X )) . Recall that in this setting s = α/d . We fix a resolution K and then consider the set of B-splines ω d K,ℓ , ℓ ∈ I d K of cardinality N ′ ≍ 2 Kd . Considering the basis pairs ( ω d K, 1 , ω d K, 2 ) , . . . , ( ω d K,N ′ -1 , ω d K,N ′ ) , we can determine which one is employed to construct the basis ψ ( j ) . The Varshamov-Gilbert bound yields that for Ω = { 0 , 1 } N ′ / 2 , we can construct a subset Ω ′ = { w 1 , . . . , w 2 N ′ / 16 } ⊂ Ω such that | Ω | = 2 N ′ / 16 and w = w ′ ∈ Ω ′ has a Hamming distance not less than N ′ / 16 . Using this Ω ′ , we set N = N ′ / 2 and M = 2 N ′ / 16 and define ( ψ ( j ) ) j M =1 as ψ ( j ) i = ω d K, 2 i -1 if w j,i = 0 and ψ ( j ) i = ω d K, 2 i if w j,i = 1 . We use the same B-spline bases with resolution more than K for ψ ( j ) i ( i ≥ N ) across all j .

By the construction of ( ψ ( j ) ) , if we set β (1) = ( σ β , . . . , σ β , 0 , 0 , . . . )

Hence, for δ 2 n ≤ σ 2 β N/ 8 ≲ 1 , the δ n -packing number is not less than 2 N/ 8 . Moreover, the logarithmic δ n -packing number of { β ⊤ ψ ( j ) | β ∈ B} for a fixed j is Θ(min { δ -1 /s n , N log(1 /δ n ) } ) by the standard argument.

Hence taking δ n = N -s , we obtain log M ≳ N and the upper bound of the covering numbers log Q 1 +log Q 2 ≲ N for σ 2 β ε 2 n, 1 ≤ δ 2 n and ε 2 n, 2 = Cδ 2 n where C is a constant. Then, by choosing C appropriately and ε n, 1 ≲ N -1 -s (so that log Q 1 ≲ N ), as long as

is satisfied, the minimax rate is lower bounded by δ 2 . Taking N ≍ n 2 s +1 , we obtain the lower bound n 1

F.3 Lower Bound with Coarser Basis

We consider a generalized setting where X = R d × R d ×··· × R d ︸ ︷︷ ︸ ( N +1) times and take ψ ( j ) i ∈ U ( B τ p,q ( R d ))

and assume that β 1 ∈ [ -1 , 1] and β j ∈ [ -σ β , σ β ] where σ 2 β = ˜ Θ( N -2 s -1 ) . Since the logarithmic ˜ ε 1 -covering and packing numbers of U ( B τ p,q ( R d )) are Θ(˜ ε -d/τ 1 ) , those for the basis functions on j = 2 , . . . , N + 1 become Θ( N ˜ ε -d/τ 1 ) , and those for B are Θ( N log(1 + Nσ 2 β ε 2 n, 2 )) . Therefore, by taking ε 2 n, 1 = N ˜ ε 2 1 we see that

should be satisfied. Moreover, by taking ε (2 τ + d ) /τ n, 1 ≍ 1 /nT and ε 2 n, 2 ≍ N log(1 + N -2 s /ε 2 n, 2 ) /n we can balance both sides. In particular, we may set

Taking the balance with respect to N to maximize ε 2 n, 2 , we have N ≍ n d 2 α + d and ε 2 n, 1 ≍ ( nT ) -2 τ 2 τ + d . Therefore, the minimax rate is lower bounded as

F.4 Lower Bound in Piecewise γ -smooth Class

Suppose that we utilize the basis functions up to resolution K . Then, by the argument by Nishimura and Suzuki (2024), the number of basis functions ψ r in the K -th resolution is N ′ ≍ 2 K/a † . Moreover, the δ n -packing number of the γ -smooth class is also lower bounded by

̸

Here, by noticing the approximation error bound in Appendix D.2, we take N ′ ≍ δ -1 /a † n where the basis functions are chosen from the K th resolution. As in the case of the Besov space, we construct ( ψ ( j ) ) M ′ j =1 where M ′ = 2 N ′ / 16 and ψ ( j ) ( x ) ∈ R N for N = N ′ / 2 and ∥ ψ ( j ) -ψ ( j ′ ) ∥ 2 L 2 ( P X ) ≥ N/ 8 for j = j ′ . Following the same argument as in the Besov case, we need to take ε n, 1 and ε n, 2 as

up to logarithmic factors. This is satisfied by taking ε 2 n, 2 = Cδ 2 n ≍ n -2 a † 2 a † +1 with a constant C and balancing N so that σ 2 β ε 2 n, 1 = ( nT ) -2 a † 2 a † +1 σ 2 2 a † +1 β ≍ T -1 n -2 a † 2 a † +1 . Combining this evaluation and (25) yields that the minimax lower bound is given by

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See the Our Contributions paragraph for details.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See the Limitations paragraph.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Assumptions 1, 2, 3, 4, 5, 6. All proofs were provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Appendix E, Figure 1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: All experiments are toy simulations and data is i.i.d. random.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Appendix E, Figure 2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The median over 5 runs is reported in Figure 3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: All experiments are toy simulations.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research is theoretical and raises no ethical concerns.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The research is theoretical and raises no concerns on societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzed in-context learning of a transformer consisting of a DNN and a linear attention layer pretrained on nonparametric regression tasks.  The authors derived a general bound on the generalization error consisting of the approximation error, in-context generalization error and pertaining generalization error. They also showed that the icl prediction given by the pretrained TF is minimax optimal when the nonparametric regression functions are from the Besov space or  piecewise  $\\gamma$-smooth class.\n\nSTRENGTHS:\nThe paper is well-written and the theoretical results are solid. The authors made reasonable assumptions on the regression tasks and the TF function class that do not significantly simplify the problem. From a technical perspective, this work leverages the approximation ability of DNNs and the ICL ability of single-layer linear attention to derive the near-optimal generalization error bounds. A few generalizations are also considered, e.g., anisotropic Besov space and piecewise  $\\gamma$-smooth class.\n\nWEAKNESSES:\nThe paper considered a model different from the standard TF models in practice, in the sense that a trainable DNN-based feature map is applied to the tokens before the attention layer. The proof in this works heavily rely on this feature map, as it is used to approximate the basis functions of the Besov space.\n\nLack of simulation results. The work showed that the empirical risk minimizer is minimax-optimal but didn't analyze the training dynamics. So it would be good to have empirical evidence showing that pretraining indeed finds an empirical risk minimizer.\n\nQUESTIONS:\n1. The functions in the Besov space are well approximated by linear combinations of finite number of basis functions in the space. Suppose now the feature map $\\phi$ can well-approximate the basic functions (assumption 3) and is fixed to be $\\phi^*$, and only the linear weight in the attention layer is trainable. I wonder how much differences there are between this simplified setting and previous works on ICL for linear regression.\n\n2. I wonder if it is possible to theoretically analyze the training dynamics under the setting where the feature map is given and fixed and only the linear attention is trainable.\n\nLIMITATIONS:\nAs the author already mentioned, this work only analyzed the generalization error of the ERM but not the training dynamics. The transformer model in this work is limited to a single layer of self-attention and does not include the softmax activation and the MLP layer.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the ICL capabilities of transformers from a statistical learning theory perspective. It focuses on transformers with a deep neural network and a linear attention layer, pretrained on nonparametric regression tasks from function spaces like the Besov space and piecewise gamma-smooth class. The authors demonstrate that sufficiently trained transformers can achieve or even surpass the minimax optimal estimation risk by encoding relevant basis representations during pretraining. The paper also establish result that explains the pretraining and in-context generalization gaps, which is essential for understanding ICL.\n\nSTRENGTHS:\n1. The topic of this work is both interesting and important, addressing key aspects of ICL in transformers.\n\n2. The results developed, especially Theorem 4.5, are very encouraging and could help the community better understand LLMs and their ICL capacity.\n\n3. The analysis provided is rigorous and sound, offering new tools and methodologies for theoretical analysis in the ICL literature.\n\nWEAKNESSES:\n1. The writing and presentation of the paper need improvement. For instance, in Section 4, the flow feels disjointed, with several results appearing to be stitched together without a clear, organized connection.\n\n2. Although the ICL capabilities of transformers are powerful and of great interest, the simplified transformer model presented in this paper is not very realistic. While it is common to use simplified models for analyzing LLMs, the paper would greatly benefit from numerical experiments demonstrating the claimed theoretical findings.\n\nQUESTIONS:\nWhile reading this paper, I feel it provides new insights into understanding ICLs. However, I cannot find concrete evidence of how this theoretical framework can offer supportive insights for practitioners using ICL. The simplified model without numerical experiments cannot indicate how real LLMs behave. \n\nCould the authors provide more detailed explanations on how this work connects to the mechanisms of ICL and LLMs in practice?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work shows that ICL can perform non-parametric regression at an optimal rate.\n\nSection 3 gives an upper bound on ICL error in terms of a metric entropy of the representation class. Section 4 instantiates the bound for DNN representations and shows it to be optimal. Section 4.3 explores ways to reduce the dependence on dimension. Section 5 gives lower bounds for any ICL (not just transformers).\n\nThere is a sufficiently expressive representation class $\\mathcal{F}$ (here we take DNNs) from which we can draw a $\\phi^*$ such that linear attention on top of this solves regression well. In that sense, this paper generalizes the idea from linear attention for linear regression that the attention inverts the data-covariance.\n\nSTRENGTHS:\nThe paper contains a general analysis of ICL error for regression problems. The comparison to non-parametric rates is interesting and possibly the right way to extend the so far mostly linear analysis in the literature to problems beyond linear regression.\n\nWEAKNESSES:\nThis is general analysis of a very specific simplification of the transformer. The setting is that there are trainable layers before the attention layer, and then exactly one linear attention layer. Is there any hope to extend this analysis to more attention layers/ non-linear attention/ constant $N$? When $N$ is constant, the error is constant, which seems vacuous considering everything is bounded. Is that an artifact of analysis?\n\nIs it possible to have a setting where we removed the DNN and force the features themselves to satisfy Assumption 3? Would that reduce to the linear regression setting of prior work? If so, it seems very important to justify how we can find a $\\phi^*$ satisfying Assumption 3 for DNNs.\n\nQUESTIONS:\nA matter of notation: in assumption 1, why aren’t the basis functions the first N functions, rather than the functions from $\\underline{N}$ to $\\overline{N}$? Also, are the functions past $\\overline{N}$ not spanned by the basis? Remark 2.1 offers some explanation. I think it would be helpful to write the main paper in the setting where $\\psi$'s span the space and leave the generalization for the appendix.\n\nWhat role does the sparsity $S$ play, why is it set to be $O(1)$? Isn't is unrealistic that $N$ should scale in $n$ for a real transformer (one of the key selling points about transformers are how the parameters dont scale in the context length)? Do the authors think this is due to the analysis or is it fundamental for attention?\n\nIt would be helpful to instantiate a target function class in the main paper, specify what $\\alpha$ is, etc.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies in-context nonparametric learning using transformers. In the setting used in the main result of the paper, the transformer is trained on a dataset consisting of multiple sequences/tasks. For each task, the target function $F_\\beta$ is drawn from the span of a certain countable set of functions - in the setting used for the main result of the paper, the target function is drawn from a Besov space spanned by a B-spline wavelet basis. The task then consists of multiple in-context pairs $(x_k, y_k)$, where $y_k$ is $F_\\beta(x_k) + \\xi$, where $\\xi$ is random noise. Certain technical assumptions are made on the basis functions $\\psi_j$ and the coefficients $\\beta_j$: the $\\psi_j$ are assumed to have a property that is similar to linear independence/orthonormality, while the $\\beta_j$ are assumed to decay at a certain rate.\n\nThe model in this paper's setting consists of a feature map $\\phi$ applied to all of the $x_k$, followed by a linear attention layer applied to the $\\phi(x_k)$ and the $y_k$. The feature map $\\phi$ is assumed to be expressive enough to approximate the $\\psi_j$. In the main result of this paper, $\\phi$ is chosen from the class of deep neural networks (DNNs) with a logarithmic number of layers (logarithmic in the size of the feature map), and $O(1)$ width and $O(1)$ entries per layer. The class of feature maps $\\phi$ is denoted as $\\mathcal{F}_N$, and the overall model class is denoted as $\\mathcal{T}_N$.\n\nThe main result of the paper, Theorem 4.5, is that with a sufficient number of tasks during pretraining, in-context learning with transformers will achieve the optimal minimax risk. This is shown as follows. Theorem 3.1 first gives a bound on the expected test loss in terms of the covering number of the model class $\\mathcal{T}_N$, and the minimum test loss achievable by some member of $\\mathcal{T}_N$. Next, the paper gives a particular construction for a set of parameters which achieves low test loss. The weight matrix for the linear attention layer is chosen similarly to the optimal weight matrix given in Zhang, et al. (2023) and nonlinear feature map outputs a subset of the basis functions.\n\nAdditional results are also given. This paper shows how the curse of dimensionality can be avoided when the target function is drawn from an anisotropic Besov space. Also, under certain assumptions, even if the target function is drawn from a Besov space with smoothness $\\tau < \\alpha$, it is possible to achieve the same minimax rate as in the case where the target function is drawn from a Besov space with smoothness $\\alpha$. Finally, the paper gives similar results in the setting where the tokens are themselves sequences of unbounded length, and gives minimax lower bounds that match the upper bounds.\n\nSTRENGTHS:\n- This work seems to be the first to study in-context learning using transformers in Besov spaces (prior work studied settings such as regression with DNNs).\n- The paper is very well-written.\n\nWEAKNESSES:\nI think the main weakness is that the techniques used to show Theorem 4.5 are somewhat standard. \n- Theorem 3.1, which shows that the empirical risk minimizer achieves good test loss, seems to be using relatively standard arguments from previous work, based on covering numbers.\n- The construction used to show that low approximation error can be achieved also seems somewhat straightforward. The weight matrix for the attention layer is chosen similarly to Zhang, et al. (2023), and the feature map outputs a subset of the basis vectors.\n\nQUESTIONS:\n- In Lemma 4.4, how important is the assumption that the width/sparsity of the deep neural network is $O(1)$? Would the guarantee obtained in Theorem 4.5 change if the deep neural network is allowed to have more nonzero entries?\n- Could the setting in this paper be considered similar to the linear regression setting studied by previous works such as Zhang, et al. (2023), with the difference being that the linear regression weight vector is replaced by the vector $\\beta$ of coefficients for the basis functions?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-29"}