Title: Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling
Venue: NeurIPS 2024
Accepted: yes
Keywords: Transformer

expressive power

theoretical insights
Abstract: We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory. 
We investigate the mechanisms through which different components of Transformer, such as the dot-product self-attention, positional encoding and feed-forward layer, affect its expressive power, and we study their combined effects through establishing explicit approximation rates.
Our study reveals the roles of critical parameters in the Transformer, such as the number of layers and the number of attention heads.
These theoretical insights are validated experimentally and offer natural suggestions for alternative architectures.

Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling

Abstract

We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory. We investigate the mechanisms through which different components of Transformer, such as the dot-product self-attention, positional encoding and feed-forward layer, affect its expressive power, and we study their combined effects through establishing explicit approximation rates. Our study reveals the roles of critical parameters in the Transformer, such as the number of layers and the number of attention heads. These theoretical insights are validated experimentally and offer natural suggestions for alternative architectures.

1 Introduction

In recent years, Transformer networks (Vaswani et al., 2017) have emerged as foundational models, setting new benchmarks across various domains, including natural language processing (NLP), computer vision (CV), and protein folding. Despite their impressive practical achievements, the underlying mechanisms and theoretical foundations of Transformer networks remain largely elusive.

Transformer networks encompass various components, posing challenges to their comprehensive understanding. A typical Transformer comprises multiple layers, each consisting of a multi-head self-attention (Attn) sub-layer and a feed-forward network (FFN) sub-layer, integrated with residual blocks. FFN is a two-layer nonlinear network, while Attn includes dot-product (DP) and positional encoding (PE). To get a better understanding of how Transformer works in practice, we need to study several key issues. These include:

(i) How do the key hyper-parameters, for example, the number of layers, the number of Attn heads and the with of FFN layers, affect the performance of the Transformer network?

(ii) How do the Attn and FFN layers contribute differently to the overall performance?

(iii) How does DP attention work, and is the DP structure necessary?

(iv) How efficient is PE in modeling long-range correlations?

Extensive empirical research on Transformer components has led to the proposal of numerous alternatives to the current structure of Transformer. For example, several relative positional encodings (RPE) (Shaw et al., 2018; Raffel et al., 2020; Su et al., 2024; Press et al., 2022) have been proposed

to substitute the original absolute positional encoding (APE), yielding superior performance in challenging tasks like length generalization (Ontanón et al., 2022; Csordás et al., 2021; Anil et al., 2022). Additionally, the necessity of the computationally expensive DP in Attn layers has been widely questioned, and researchers proposed numerous alternatives of DP that show considerable efficacy in specific tasks (Kitaev et al., 2020; Wang et al., 2020; Choromanski et al., 2020; Tay et al., 2021; Allen-Zhu and Li, 2023). Nonetheless, these explorations have not yielded a satisfactory theoretical understanding of the mechanisms of these components.

In this work , we investigate the expressive power of Transformer and the underlying mechanisms of its components for sequence modeling. Our contributions are summarized as follows:

We categorize three types of sequence modeling tasks with varying complexity, which are relevant to a broad spectrum of application areas. Task I: Modeling fixed, long but sparse memories . This is relevant to sparse Boolean functions and the traditional n -gram model in NLP. Task II: Modeling adaptive, long but sparse memories . This is relevant to multi-step reasoning tasks as well as various NLP tasks such as dependency parsing, sentiment analysis, and continuation writing. Task III: Modeling essentially sparse memories . Examples include feature representation in CV and wavelet analysis in classical signal processing.

For these sequence modeling tasks, we theoretically investigate the expressive power of Transformer and its variants, establishing explicit approximation rates. Our meticulous analysis provides theoretical insights into the underlying mechanisms of Transformer components. Specifically,

The distinct roles of the number of layers, the number of Attn heads, and the width of FFN layers. Deeper Transformer are capable of handling memories with more intricate interrelationships, such as nested relationships (Thm 4.4). In contrast, for memories lacking such interrelationships, single-layer Transformer with sufficient number of Attn heads and FFN width should suffice (Thm 4.1). This is quite intuitive: If the content of the next token relies on a few previous tokens in an independent way, we can treat each such dependence by a separate attention head. There is no need for many layers. Additionally, increasing the depth can also alleviate the reliance on the number of heads and width (Prop 4.5).

The different roles of Attn layers and FFN layers. Our results consistently suggest that: FFN layers are tasked with approximating nonlinear memory functions and the readout function, while Attn layers are responsible for extracting the tokens from these memory locations.

The functionality and necessity of DP. For the relatively simple Task I, DP is not necessary and can be omitted (Thm 3.1). However, for the more complex Task II, the cooperation between DP and RPE provides the needed interaction between the temporal space and the token space, crucial for the extraction of adaptive memories (Thm 4.1 and 4.4). Additionally, for Task II, while the nonlinearity provided by DP is necessary (Prop 4.2), a computationally efficient alternative to DP exists, as we show in Prop 4.3.

The efficiency of RPE in modeling long-range correlations. Our results consistently suggest that the primary role of RPE is to approximate the memory kernels. Specifically, for Task III, we demonstrate that Transformer with suitable RPE can handle heavy-tailed memories, thus overcoming the Curse of Memory faced by recurrent neural networks (Thm 5.1). Moreover, our findings give theoretical support to the choice of RPE in practice.

Finally, we conduct experiments to validate our theoretical insights.

2 Preliminaries

Basic notations. We use bold-faced letters for vectors or matrices and lowercase letters for scalars, e.g. x = ( x 1 , · · · , x d ) ⊤ ∈ R d and W = ( W ij ) m × n ∈ R m × n . The standard Euclidean inner product between two vectors is denoted by ⟨· , ·⟩ , and the l p norm of a vector is represented by ∥·∥ p . We employ standard big-O notations O , Ω , Θ to hide absolute positive constants and use ˜ O , ˜ Ω , ˜ Θ to further hide logarithmic constants. For any positive integer n , let [ n ] = { 1 , · · · , n } . Denote by I { E } the indicator function for an event E . Denote by a ∨ b = max { a, b } for real number a, b .

2.1 Sequence modeling with long but sparse memories

Sequence modeling. For convenience, we consider input sequences of infinite length ( t ∈ Z ) . It is important to note, however, that our theoretical framework can be adapted to finite-length input sequences by masking distant tokens. Formally, the output sequence Y = ( y t ) t ∈ Z ∈ R c × Z is generated from the input sequence X = ( x t ) t ∈ Z ∈ X ⊂ R d × Z via an unknown mapping H · ( · ) dependent on the input sequence up to the prediction time, and this can be expressed as:

Our objective is to learn the mapping H · ( · ) . Additionally, we define the norm | | | H | | | := sup t ∈ Z sup X ∈X ∥ H t ( X ) ∥ . Without loss of generality, we assume ∥ x t ∥ 2 ≤ 1 for any X ∈ X and set the output dimension c = 1 for simplicity.

Long but sparse memories. To model such sequences, we define three types of memories: fixed, long but sparse memories; adaptive, long but sparse memories; and essentially sparse memories. These memory types are prevalent in sequence modeling tasks across diverse domains such as NLP, CV, signal processing, and sparse function representation. In Section 3, 4, and 5, we will formally define these different types and investigate Transformer's capacity to model them.

2.2 Transformer architecture

Transformer network. Transformer (Vaswani et al., 2017) is a network architecture designed for processing sequences and generating predictions. Given an input sequence X , Transformer executes the following steps. Initially, each d -dimensional (dim) input token is transformed into a D -dim vector through an embedding mapping such as x (0) t = W E x t + b E , where W E ∈ R D × d , b E ∈ R D . Subsequently, a typical L -layer Transformer with residual block operates according to the formulation:

At the l -th layer, FFN ( l ) ( · ) denotes a standard (point-wise) two-layer ReLU networks with m neurons: for a given input x ∈ R D , FFN ( l ) ( x ) = ∑ m k =1 a ( l ) k σ ( b ( l ) ⊤ k x + c ( l ) k ) , where σ ( · ) is the activation function such as ReLU. Additionally, in the final ( L -th) FFN layer, the residual block is omitted, commonly referred to as the readout function. Moreover, Attn ( l ) ( · ) refers to a multi-head self-attention, as elaborated below.

Multi-head self-attention. Our focus lies on standard dot-product Attn, denoted as Attn ( l ) ( · ) and consisting of H heads. When applied to an input sequence X , Attn operates as follows:

Here, the parameters W ( l,h ) Q , W ( l,h ) K , W ( l,h ) V , W ( l,h ) O correspond to the query, key, value, output matrices of the ( l, h ) -th head, respectively. softmax c represents taking softmax normalization across column. Furthermore, R ( l,h ) ∈ R Z × Z denotes the relative positional encoding matrix, which satisfies R ( l,h ) t,s = -∞ for t < s in the next-token prediction paradigm. Consequently, the t -th output of Attn is expressed as:

Logarithmic and Power relative positional encoding. As highlighted in Section A, among various types of RPEs, the RPEs used in T5 and KERPLE(log) demonstrate superior performance over Alibi, significantly outperforming other RPEs and APEs in the length generalization task (Kazemnejad et al., 2023; Chi et al., 2022). This finding motivates our focus on the T5-type, KERPLE(log), and Alibi-type RPEs throughout this paper. All of these RPE matrices are Toeplitz, with the form of

R t,s = r ( t -s ) . Notably, for T5 and KERPLE(log), r ( t -s ) undergoes an initial linear decrease followed by a logarithmic decrease as the relative distance t -s increases (Please refer to Section G.1 for more details). In contrast, for Alibi, r ( t -s ) decreases linearly. Inspired by these discussions, we examine the following RPEs with different decay rates:

We will study Transformer with ϕ type RPE ( type ∈ { log , lin } ). Specifically, the RPE in the ( l, h ) -th head (3) is as follows:

where p ( l,h ) ∈ R + is a trainable parameter.

Remark 2.1. For standard Transformer (2) incorporating Attn (3) with RPE (4), the parameters are: the embedding matrix W E ; a ( l ) k , b ( l ) k , c ( l ) k in the FFN layers; W ( l,h ) Q , W ( l,h ) K , W ( l,h ) V , p ( l,h ) , W ( l ) O in the Attn layers. Notably, the number of parameters is independent of the sequence length, thus enabling the model to handle input sequences of arbitrary length.

Remark 2.2. In the subsequent sections, we will analyze Transformer and its variants. For the sake of brevity, some shorthand notations are introduced here. For examples, Transformer (2) using ϕ log / ϕ lin RPE (4) is referred to as 'Transformer with log / lin -RPE'; Transformer with W ( l,h ) Q , W ( l,h ) K = 0 is called 'dot-product-free Transformer'.

2.3 Expressive power via approximation theory

This paper delves into the expressive power of Transformer through the lens of approximation theory, with a specific focus on establishing explicit approximation rates for Transformers in modeling long but sparse memories.

Approximation rates v.s. universal approximation. In approximation theory, results are generally categorized into two types: universal approximation (density-type) and approximation rates (Jacksontype) (Jackson, 1930). Universal approximation investigates whether the hypothesis class is dense in the target class. Although this property is fundamental, it does not offer detailed insights into approximation efficiency. In contrast, approximation rates go deeper, emphasizing the efficiency of the approximation. A typical example within this framework is the approximation theory of two-layer neural networks (2NNs).

Barron space of 2NNs . The well-known universal approximation result for 2NNs asserts that 2NNs can approximate any continuous function (Barron, 1992; 1993; 1994). Nonetheless, this result lacks a characterization of the approximation efficiency, i.e., how many neurons are needed to achieve a certain approximation accuracy? This gap was addressed by the Barron space theory (E et al., 2019; 2021; Ma et al., 2020). It is established that for any function within Barron space f ∈ B (Appendix G.2), 2NNs with m neurons (denoted by H m ) can approximate them efficiently, at a rate of inf f m ∈H m ∥ f -f m ∥ ≤ O ( ∥ f ∥ B / √ m ) , remarkably independent of the input dimension d , thus avoiding the Curse of Dimensionality (Bellman, 1966; Bach, 2017).

3 Fixed, long but M -sparse memories

3.1 Problem formulation

Fixed, long but M -sparse memories. In this section, we investigate a fundamental category of long but sparse memories. Our focus is on scenarios where the positions of the sparse memories remain fixed and are independent of the tokens. The target function is represented by:

where 1 ≤ T 1 < · · · < T M < + ∞ signify the fixed positions of the memories. Despite the memories being fixed (token-independent) and sparse (finite M ), the task can still be complex due to the potentially long-range memories ( T 1 , · · · , T M can be large enough).

Examples. (I) For Boolean inputs, (5) aligns with sparse Boolean functions , also studied in (Edelman et al., 2022; Bhattamishra et al., 2022). Notably, Bhattamishra et al. (2022) observed that Transformers outperform LSTMs in learning sparse parities. (II) Selecting the simplest case of T i = i in (5) corresponds to the traditional n -gram model , which consists of short and sparse memories.

Target class. We focus on target functions described in (5). The readout function f is considered within the standard Barron space B , i.e., which can be effectively approximated by 2NNs. Moreover, we assume that f is Lipschitz, denoted by f ∈ L . Thus, we can focus more on investigating the memory extraction power of Transformer. Formally, we define the target class for modeling fixed, long but M -sparse memories as:

Transformer hypothesis class. As mentioned in Section 1, one of our main aims is to study the necessity and roles of different components in Transformer, such as DP and RPE. This section focuses on the 'simplest' one-layer Transformer and investigates whether it can effectively model this task. Formally, our hypothesis class includes all one-layer DP-free Transformers, configured with H Attn heads and FFN width m :

3.2 Theoretical results and insights

Theorem 3.1 (Approximation rate) . For any target H ∈ H Fix (6) , rate n ∈ N + , and H,m ∈ N + , there exists a 1 -layer Transformer TF ∈ T F DPF , type (1 ,H,m ) (7) and a constant C ( n ) such that

Theorem 3.1 establishes the approximation rate of one-layer DP-free Transformer for modeling fixed, long but sparse memories. Here, the model complexity is governed by the number of Attn heads H and the width of FFN layers m , while the target complexity arises from the lengths of the memories T 1 , · · · , T M and the complexity of the readout function f . The approximation error comprises two components: the error in the FFN component E FFN and the error in the Attn component E Attn ( type ) . The error E FFN aligns with classical results, showcasing its effectiveness in approximating Barron functions. On the other hand, E Attn ( type ) hinges on the capacity of the Attn block for modeling long-range memories. Specifically, with increasing memory length, the necessary number of Attn heads grows at a small exponential rate for lin -RPE and at a polynomial rate for log -RPE.

The proof of Theorem 3.1 is deferred to Appendix B. We can draw some insights from Theorem 3.1 and its proof.

Different roles of the Attn layer and the FFN layer. The Attn and FFN layers fulfill distinct roles in this task. Specifically, the FFN layer efficiently approximates the nonlinear readout function f , while the Attn layer is responsible for extracting the token x t -T i by approximating the memory kernel I {· = T i } . These components together enable effective modeling of fixed, long, but sparse memories.

Non-necessity of DP. Theorem 3.1 suggests that the DP component in Attn is not necessary and can be omitted for modeling fixed, long but sparse memories. This is due to the relative simplicity of modeling fixed memory kernels. In a more complex scenario in Section 4, the role of the dot-product becomes important. In contrast to Edelman et al. (2022), which utilizes the property of DP to prove that Transformer can model sparse Boolean functions, our result reveals that one-layer Transformer can successfully tackle the same task even without the dot product in the attention layer .

Effect of RPE types on expressivity. Our result indicates that the type of the RPE used in the Attn layer subtly influences the Transformer's ability to model long-range memories. As the range of the memory increases, the required head number grows at a slightly exponential rate for lin -RPE

and at a polynomial rate for log -RPE. The subtle difference is attributed to the relative simplicity of approximating the memory kernel I {· = T i } . We will explore a more complex task in Section 5, where the impact of different types of RPE becomes even more pronounced.

4 K -Adaptive, long but M -sparse memories

4.1 Problem formulation

In this section, we delve into a more complex modeling scenario closely aligned with typical language processing tasks.

K -Adaptive, long but M -sparse memories. This section investigates the scenario where the positions of the sparse memories are 'adaptive', meaning they depend on the input tokens. The target function is formulated as:

where the positions of the memory tokens t 1 , · · · , t M follow a nested relationship:

Here, M denotes the number of memory tokens, and K measures the nesting complexity in the memory structure. We assume that memory functions g i generate positive integers for the input tokens, and there exist maximum values T i such that g i ≤ T i . In this adaptive framework, each position of the memory token depends on multiple input tokens and is nested within other memory structures, leading to potential influence of later memory tokens by the earlier ones.

To facilitate understanding, we first consider a warm-up case, i.e., K = 0 in (8). In this case, the positions of memories only depend on the current token, without interaction with each other. It can be represented as:

where t i = g ( x i ) , i ∈ [ M ] .

Target class. The target classes for modeling adaptive, long but sparse memories in both warm-up and general cases are as follows:

Examples. Adaptive memories are commonly encountered in practical scenarios. (I) Adaptive sparse Boolean functions , e.g., y t = x t · x t -g ( x t ) · x t -g ( x t -g ( x t ) ) , where X ∈ {± 1 } Z , g ( x ) = 1 for x = 1 and g ( x ) = 2 for x = -1 . This fits within our framework (8) with K = M = 2 . (II) Multi-step reasoning , e.g., modeling the K -adaptive, long, but K -sparse memories contains a complicated K -step reasoning task, which require the sequential search following the rule (( · · · (( x t ↦→ x t -t 1 ) ↦→ x t -t 2 · · · ) ↦→ x t -t K -1 ) ↦→ x t -t K . (III) In NLP tasks like dependency parsing, part-of-speech tagging, sentiment analysis, or continuation writing, the positions of relevant prefix tokens usually depend on the context itself, and can vary depending the content. Additionally, the nested structure is a fundamental characteristic of natural language (Hawkins, 2021).

Transformer hypothesis class. Some previous works Yun et al. (2019); Kim et al. (2022) treated the softmax with normalization as an approximation of hardmax, suggesting the potential importance of the normalization. In contrast, in this section, we remove the normalization in the denominator of softmax and investigate its ability for sequence modeling. Additionally, to address the discreteness of time and memory values, we consider Transformer with specific precision, as detailed in Appendix C. The precision technique is widely used in LLM training (Kalamkar et al., 2019), such as BFloat16 . Formally, the hypothesis class is defined as follows, encompassing all normalization-free L -layer Transformer, configured with H Attn heads and FFN width m and using type -RPE and specific precision.

4.2 Theoretical results and insights: The warm-up case

Theorem 4.1 (Approximation rate, warm-up case) . For any target H ∈ H (1 ,M ) (8) , rate n ∈ N + and H,m ∈ N + , there exists a two-layer Transformer TF ∈ T F type (2 ,H,m ) (12) and a constant C ( n ) such that: if the width satisfies m ≥ { ˜ Ω ( ∑ M i =1 ∥ g i ∥ 2 B ) , type = lin ˜ Ω ( ∑ M i =1 ∥ log g i ∥ 2 B T 2 i ) , type = log , then the following approximation rate holds:

In Theorem 4.1, we present the approximation rate of two-layer Transformer for the warm-up case: modeling 1 -adaptive, long but M -sparse memories. This theorem reveals that the approximation error comprises two distinct components: the error in the FFN component E FFN and the error in the Attn component E Attn ( type ) . A critical difference from 3.1 is the presence of the condition related to the width m of FFN layers. This term arises from using the FFN layer to approximate the memory function g i . Owing to the discreteness of memory g i and the implementation of rounding operations, the approximation within rounding accuracy all achieves zero error after rounding, while it can not get correct rounding beyond this accuracy. In contrast, the error E FFN is caused by using FFN to approximate the readout function f , the same as E FFN in Theorem 3.1.

The proof of Theorem 4.1 can be found in Appendix C.1. Theorem 4.1 and its proof offer several critical insights into the underlying mechanism of Transformer.

Distinct roles of Attn layers and FFN layers. Our proof elucidates that the FFN layers are tasked with approximating the readout function f and memory functions g i , while the Attn layers are responsible for the extraction of the adaptive memories. It is essential to clarify the difference between 'approximating memory functions' and 'memory extraction'. The former refers to utilizing some function to estimate the memory function g i , whereas the latter pertains to extracting the token x t -g i ( x t ) from the memory location.

Cooperation between DP and RPE. In the 2 -nd Attn layer, the extraction of the memory functions is achieved through an interplay between DP and RPE. Specifically, this is done through a nice interaction between the temporal space (provided by RPE) and the token space (provided by DP) . Please refer to Appendix C.1 for more details.

Rethinking DP in Attn. Our proof highlights that the core mechanism of Attn is to provide a nice interaction between the temporal space and the token space through the cooperation of DP and RPE. This leads us to the following question: Is DP in Attn necessary and replaceable? The following two propositions provide some hints.

Proposition 4.2 (DP vs. DP-free (informal)) . There exists a target H ∈ H Adap (1 , 1) (10) such that:

Proposition 4.2 reveal a significant distinction in the expressiveness of two network types for modeling adaptive, long, but sparse memories. Specifically, 1 -layer Attn with DP can effectively model this task, while 1 -layer DP-free Attn provably fails. This finding underscores the essential role of DP in providing the necessary nonlinearity for Attn to model adaptive memories. The formal version of Proposition 4.2 and its proof can be found in Appendix C.2.

Proposition 4.3 (Substitute for DP (informal)) . There exists a substitute structure for DP , requiring only O ( D ) parameters (compared to O ( D 2 ) in standard DP) that can effectively model H ∈ H adap (1 ,M ) (10) . Specifically, if we substitute DP with this structure, 1 -layer Transformer can achieve the same approximation rate as stated in Section 4.1.

Proposition 4.3 demonstrates the existence of a structurally simpler yet effective alternative to traditional DP for modeling (10). This alternative is proposed based on our insights into the role of Attn in facilitating the interaction between the temporal space and the token space. Specifically, we propose a more direct structure to achieve this interaction. More details are deferred to Appendix C.3.

4.3 Theoretical results and insights: The general case

Theorem 4.4 (Approximation rate, general case) . For any target H ∈ H Adap ( K,M ) , rate n ∈ N + , and H,m ∈ N + , there exists an L -layer ( L = K + 1 + I { M ≥ K + 1 } ) Transformer TF ∈ T F type ( L,H,m ) (12) and a constant C ( n ) such that: if the width satisfies if the width satis- fies m ≥ { ˜ Ω ( max i ∈ [ K ] ∨ ∑ M i = K +1 ∥ g i ∥ 2 B ) , type = lin , ˜ Ω ( max i ∈ [ K ] ∨ ∑ M i = K +1 ∥ log g i ∥ 2 B T 2 i ) , type = log , then the following approximation rate holds:

In Theorem 4.4, we establish the approximation rate of deep Transformer for modeling K -adaptive, long but M -sparse memories. Similar to that in Theorem 4.1, the approximation error divides into two distinct terms. A key difference from Theorem 4.1 is the impact of the nested relationships among the memory functions on the required number of layers, Attn heads, and the width of FFN layers. The nested structure within the initial K memories mandates sequential processing in the first K layers one by one. If M ≥ K +1 , then in the K +1 -th layer, the remaining M -K non-nested memory functions t K +1 , · · · , t M are concurrently processed. The proof of Theorem 4.4 is deferred to Appendix D.1.

Distinct roles of the number of layers L , the number of Attn heads H , and the width of FFN layers m . Theorem 4.4 and its proof highlight the distinct roles of three key hyper-parameters of Transformer: L , H , and m . Deeper Transformer are capable of handling the memories with more intricate nested relationships, requiring a K + 1 layer network for a nesting complexity of K . In contrast, the number of heads and width needed is dictated by the individual complexity of memory functions themselves ( ∥ g i ∥ B , ∥ log g i ∥ B , T i for memory g i ), necessitating that each layer's Attn heads and FFN width are sufficient to capture the memory functions extracted in that layer. This understanding is quite intuitive: If the content of the next token relies on a few previous tokens in an independent way, we can treat each such dependence with a separate attention head. There is no need for many layers.

Mitigating required head and width with depth. Recalling Theorem 4.1, the memories lacking nested relationships can be efficiently approximated by 2 -layer Transformer with a sufficient number of heads and width. The subsequent proposition further explores how increasing the depth of Transformer can influence its efficiency for modeling memories without nested relationships.

Proposition 4.5 (Deep network, warm-up case) . For any target H ∈ H Adap (1 ,M ) (8) , rate n ∈ N + , and H,m ∈ N + , there exists an M +1 -layer Transformer TF ∈ T F type ( M +1 ,H,m ) (12) and a constant C ( n ) such that: if the width satisfies m ≥ { ˜ Ω ( max i ∈ [ K ] ∥ g i ∥ 2 B ) , type = lin , ˜ Ω ( max i ∈ [ K ] ∥ log g i ∥ 2 B T 2 i ) , type = log , then the following approximation rate holds:

Upon comparing Proposition 4.5 with Theorem 4.1, a notable distinction becomes evident between 2 -layer and M +1 -layer Transformer in terms of the requirement of the number of Attn heads and the width of FFN layers. Specifically, for 2 -layer Transformer, the required width is proportionally linked

to the sum of all the memory functions' complexity ( ∥ g i ∥ B , ∥ log g i ∥ B , T i for memory function g i ). In contrast, for M +1 -layer Transformer, the required width correlates with the maximum complexity of the memory functions, much lower than that for 2 -layer Transformer. Similarly, the required number of heads for M +1 -layer Transformer is much fewer than that for 2 -layer Transformer. Please refer to Appendix D.2 for a detailed comparison. The observation suggests that increased depth can significantly reduce the demands on the number of heads and the width. The underlying reason is that deep networks can distribute the memories across different layers for processing, with each layer focusing on approximating only a single memory function.

5 Essentially M -sparse memories

5.1 Problem formulation

In language tasks, each token possesses clear semantic meaning. As a result, the structure of the memory is sparse in the original space. This aligns well with our modeling assumptions discussed in Section 3 and 4. However, in other machine learning tasks, we may encounter situations where the input tokens lack distinct semantic meaning. This might happen in image processing or classical signal processing. In these situations, the memory structure could potentially be dense in the original space. Nonetheless, the memory structure might exhibit sparsity in some transformed domain. We call such memory structure 'essentially sparse'. In this section, we study the situation in which the memory structure in long-ranged but essentially sparse. For simplicity, we consider the situation in which the positions of the memory kernels are fixed. The analysis can be easily extended to the situation with an adaptive memory structure.

Fixed, essentially M -sparse memory. Consider the following situation:

where ρ 1 ( · ) , · · · , ρ M ( · ) ∈ ℓ 1 ( N ) serve as memory kernels, and ( X ∗ ρ k )( t ) = ∑ + ∞ s =0 x t -s ρ k ( s ) denotes the convolution of the inputs with kernel ρ k .

Target class and Transformer hypothesis class. The target class for modeling essentially sparse memories is defined as:

For the hypothesis class, we consider one-layer dot-product-free Transformer with Attn head number H and FFN width m , as defined in (7).

Examples. Essentially sparse memories are prevalent in real-world scenarios:

(I) Image Tasks. In CV, a fundamental objective is identifying and representing meaningful 'features', such as ears, nose, etc. These features can often be modeled using convolution kernels, leading to a task in the form y = f ( X ∗ ρ eye , X ∗ ρ nose , X ∗ ρ ear ) . This is an extension of the task we discussed above, in which the kernel functions { ρ j } are data-dependent ('adaptive' in the terminology used in the previous section).

(II) Signal processing . In signal processing, it is commonly the case that the signals are highly sparse under Wavelet or Fourier transforms. For instance, let ψ ( · ) be a wavelet function and define ψ a,b ( t ) := ψ ( t -b a ) / √ | a | . Then we have y = f ( X ∗ ψ a 1 ,b 1 , · · · , X ∗ ψ a M ,b M ) where ( a 1 , b 1 ) , · · · , ( a M , b M ) might be data-dependent.

(III) Mathematical calculation. Consider algebraic operations where memory exhibits sparsity under specific linear transformations. For example, y t = 10 x t + x t -4 / ( ∑ 100 s =0 w s x t -10 -s ) -∑ + ∞ s =0 v s x t -100 -s can be represented in our framework as y = f ( X ∗ ρ 1 , · · · , X ∗ ρ 4 ) , where each ρ i represents a specific linear transformation.

5.2 Theoretical results and insights

Theorem 5.1 (Approximation rates) .

(A) Consider H Ess (14) with exponentially decayed memory kernels, i.e., there exists β > 0 such that

ρ 1 ( t ) , · · · , ρ M ( t ) = O ( e -βt ) . Then for any target H ∈ H Ess , rate n ∈ [ ⌊ 99 β ⌋ ] , and H,m ∈ N + , there exists a 1 -layer DP-free Transformer TF ∈ T F DPF , lin (1 ,H,m ) (7) and a constant C ( n ) such that

(B) Consider H Ess (14) with polynomially decayed memory kernels, i.e., there exists β > 1 such that ρ 1 ( t ) , · · · , ρ M ( t ) = O ( t -β ) . Then for any target H ∈ H Ess , rate n ∈ [ ⌊ 0 . 99 β ⌋ -1] , and H,m ∈ N + , there exists a 1 -layer DP-free Transformer TF ∈ T F DPF , log (1 ,H,m ) (7) and a constant C ( n ) such that

Theorem 5.1 illustrates that one-layer DP-free Transformer with lin -RPE is effective in modeling essentially sparse memories with exponentially decayed kernels, and one-layer DP-free Transformer with log -RPE can efficiently model the memories with polynomially decayed kernels. A key difference between Theorem 5.1 and Theorem 3.1 lies in the memory kernels they address. In Theorem 5.1, the Attn layer should approximate general memory kernels ρ i ( · ) , instead of approximating indicator kernels I {· = T i } in Theorem 3.1. The proof of Theorem 5.1 can be found in Appendix E.

Overcoming the Curse of Memory (CoM). For recurrent neural networks (RNN), it was discovered (Li et al., 2021; 2022) that both approximation and optimization become exceedingly difficult when the target has long-term memory. This phenomenon is referred as the ' curse of memory ', or 'CoM'. It was shown in (Li et al., 2021; 2022) that RNN requires an exponentially large number of neurons to approximate targets with heavy-tailed memory kernels, such as the ones that exhibit polynomial decay. In contrast, Theorem 5.1 reveals that Transformer with log -RPE efficiently handles polynomial decaying memory kernels, requiring only a polynomial number of neurons for effective approximation. This finding theoretically elucidates the superior performance of T5's RPE and KERPLE(log) in length generalization task in practice (Section G.1).

6 Experimental Validation

As summarized in Section 1, our theoretical analysis reveals novel insights into the expressive power and mechanisms of Transformer. To validate these insights, we conduct experiments ranging from simple toy models to more complex language model pre-training. Due to space constraints, detailed experimental validation and practical implications of our insights are presented in Appendix H.

7 Conclusion and Future Work

In this work, we investigate theoretically the expressive power and the mechanisms of Transformer for modeling long but sparse memories. Our analysis establishes explicit approximation rates and offers much-needed insights into the functionalities of the various components of Transformer. However, we still have a long way to go for a full theoretical understanding of Transformer. For instance, although we have investigated the mechanisms of Transformer in terms of expressive power, the evolution of the mechanisms during the training process remains elusive. Recent studies revealed that Transformer exhibits multi-phase learning dynamics (Boix-Adsera et al., 2023) and undergoes phase transitions (Olsson et al., 2022) during training, akin to the phenomenon of learning with increasing complexity in classical neural networks (Kalimeris et al., 2019; Xu et al., 2019; Rahaman et al., 2019; Abbe et al., 2023a; Wang and Ma, 2023). These and other issues will be studied in future work.

Acknowledgments

This work is supported in part by the National Key Basic Research Program of China (No. 2015CB856000). We thank Prof. Qianxiao Li, Prof. Lei Wu, Dr. Zhong Li, and Dr. Hongkang Yang for helpful discussions and anonymous reviewers for their valuable suggestions.

References

Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory , pages 2552-2623. PMLR, 2023a. 10

Emmanuel Abbe, Samy Bengio, Aryo Lotfi, and Kevin Rizk. Generalization on the unseen, logic reasoning and degree curriculum. International Conference on Machine Learning , 2023b. 17

Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661 , 2022. 17

Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 1, context-free grammar. arXiv preprint arXiv:2305.13673 , 2023. 2, 17

Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. Advances in Neural Information Processing Systems , 35:38546-38556, 2022. 2, 17

Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research , 18(1):629-681, 2017. 4, 64

Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. arXiv preprint arXiv:2306.04637 , 2023. 17

Andrew R Barron. Neural net approximation. In Proc. 7th Yale Workshop on Adaptive and Learning Systems , volume 1, pages 69-72, 1992. 4, 64

Andrew R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory , 39(3):930-945, 1993. 4, 64

Andrew R Barron. Approximation and estimation bounds for artificial neural networks. Machine Learning , 14(1):115-133, 1994. 4, 64

Richard Bellman. Dynamic programming. Science , 153(3731):34-37, 1966. 4, 64

Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020. 17

Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. On the ability and limitations of transformers to recognize formal languages. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020. 17

Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom. Simplicity bias in transformers and their ability to learn sparse boolean functions. arXiv preprint arXiv:2211.12316 , 2022. 5

Enric Boix-Adsera, Etai Littwin, Emmanuel Abbe, Samy Bengio, and Joshua Susskind. Transformers learn through gradual rank increase. arXiv preprint arXiv:2306.07042 , 2023. 10

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. 17

Emmanuel J Candès and Michael B Wakin. An introduction to compressive sampling. IEEE signal processing magazine , 25(2):21-30, 2008. 17

Ta-Chung Chi, Ting-Han Fan, Peter J Ramadge, and Alexander Rudnicky. Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems , 35:8386-8399, 2022. 3, 17

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 , 2020. 2, 17

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1-113, 2023. 17

Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber. The devil is in the detail: Simple tricks improve systematic generalization of transformers. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , 2021. 2, 17

Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. International Conference on Learning Representations , 2019. 16

David L Donoho. Compressed sensing. IEEE Transactions on information theory , 52(4):1289-1306, 2006. 17

Weinan E, Chao Ma, and Lei Wu. A priori estimates of the population risk for two-layer neural networks. Communications in Mathematical Sciences , 17(5):1407-1425, 2019. 4, 64

Weinan E, Chao Ma, and Lei Wu. The barron space and the flow-induced function spaces for neural network models. Constructive Approximation , pages 1-38, 2021. 4, 64

Benjamin L Edelman, Surbhi Goel, Sham Kakade, and Cyril Zhang. Inductive biases and variable creation in self-attention mechanisms. In International Conference on Machine Learning , pages 5793-5831. PMLR, 2022. 5, 16

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021. https://transformer-circuits.pub/2021/framework/index.html. 17

Guhao Feng, Yuntian Gu, Bohang Zhang, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: a theoretical perspective. arXiv preprint arXiv:2305.15408 , 2023. 17

WNelson Francis and Henry Kucera. Brown corpus manual. Letters to the Editor , 5(2):7, 1979. 17

Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems , 35:30583-30598, 2022. 17

Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. International Conference on Machine Learning , 2023. 16

Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io/ OpenWebTextCorpus , 2019. 66

Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics , 8:156-171, 2020. 17

Jeff Hawkins. A thousand brains: A new theory of intelligence . Basic Books, 2021. 6

Dunham Jackson. The theory of approximation , volume 11. American Mathematical Soc., 1930. 4, 65

Haotian Jiang and Qianxiao Li. Approximation theory of transformer networks for sequence modeling. arXiv preprint arXiv:2305.18475 , 2023. 17

Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. A study of bfloat16 for deep learning training. arXiv preprint arXiv:1905.12322 , 2019. 6, 23

Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Advances in neural information processing systems , 32, 2019. 10

Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems , 2023. 3, 17

Junghwan Kim, Michelle Kim, and Barzan Mozafari. Provable memorization capacity of transformers. In The Eleventh International Conference on Learning Representations , 2022. 6

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020. 2, 17

Zhong Li, Jiequn Han, Qianxiao Li, and Weinan E. On the curse of memory in recurrent neural networks: Approximation and optimization analysis. International Conference on Learning Representations , 2021. 10

Zhong Li, Jiequn Han, Weinan E, and Qianxiao Li. Approximation and optimization theory for linear continuous-time recurrent neural networks. Journal of Machine Learning Research , 23(42):1-85, 2022. 10

Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, et al. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. In Conference on Parsimony and Learning (Recent Spotlight Track) , 2023. 17

Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. arXiv preprint arXiv:2210.10749 , 2022. 17

Chao Ma and Lexing Ying. Why self-attention is natural for sequence-to-sequence problems? a perspective from symmetries. arXiv preprint arXiv:2210.06741 , 2022. 17

Chao Ma, Stephan Wojtowytsch, Lei Wu, and Weinan E. Towards a mathematical understanding of neural network-based machine learning: what we know and what we don't. arXiv preprint arXiv:2009.10713 , 2020. 4, 64, 65

Arvind Mahankali, Tatsunori B Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. arXiv preprint arXiv:2307.03576 , 2023. 17

William Merrill and Ashish Sabharwal. The expresssive power of transformers with chain of thought. arXiv preprint arXiv:2310.07923 , 2023a. 17

William Merrill and Ashish Sabharwal. The parallelism tradeoff: Limitations of log-precision transformers. Transactions of the Association for Computational Linguistics , 11:531-545, 2023b. 17

William Merrill, Ashish Sabharwal, and Noah A Smith. Saturated transformers are constant-depth threshold circuits. Transactions of the Association for Computational Linguistics , 10:843-856, 2022. 17

Yves Meyer. Wavelets and Operators: Volume 1 . Cambridge university press, 1992. 17

Tetsuya Nasukawa and Jeonghee Yi. Sentiment analysis: Capturing favorability using natural language processing. In Proceedings of the 2nd international conference on Knowledge capture , pages 70-77, 2003. 17

Joakim Nivre and Mario Scholz. Deterministic dependency parsing of english text. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics , pages 64-70, 2004. 17

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895 , 2022. 10, 17

Santiago Ontanón, Joshua Ainslie, Vaclav Cvicek, and Zachary Fisher. Making transformers solve compositional tasks. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , 2022. 2, 17

OpenAI. Gpt-4 technical report. https://cdn.openai.com/papers/gpt-4.pdf , 2023. 17

Jorge Pérez, Pablo Barceló, and Javier Marinkovic. Attention is turing complete. The Journal of Machine Learning Research , 22(1):3463-3497, 2021. 16

Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. International Conference on Learning Representations , 2022. 1, 17

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020. 1, 17, 64

Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning , pages 5301-5310. PMLR, 2019. 10

Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal , 27(3):379-423, 1948. 17

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics , 2018. 1, 17

Lingfeng Shen, Aayush Mishra, and Daniel Khashabi. Do pretrained transformers really learn in-context by gradient descent? arXiv preprint arXiv:2310.08540 , 2023. 17

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063, 2024. 1, 17

Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention for transformer models. In International Conference on Machine Learning , pages 10183-10192. PMLR, 2021. 2, 17

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023. 17

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 1, 3, 17, 66

Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning , pages 35151-35174. PMLR, 2023. 17

Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vladymyrov, Razvan Pascanu, et al. Uncovering mesa-optimization algorithms in transformers. arXiv preprint arXiv:2309.05858 , 2023. 17

Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023. 17

Mingze Wang and Chao Ma. Understanding multi-phase optimization dynamics and rich nonlinear behaviors of relu networks. Advances in Neural Information Processing Systems , 2023. 10

Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 , 2020. 2, 17

Colin Wei, Yining Chen, and Tengyu Ma. Statistically meaningful approximation: a case study on approximating turing machines with transformers. Advances in Neural Information Processing Systems , 35:12071-12083, 2022a. 16

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems , 35:24824-24837, 2022b. 17

Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. In International Conference on Machine Learning , pages 11080-11090. PMLR, 2021. 17

BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022. 17

Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. Communications in Computational Physics , 2019. 10

Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? arXiv preprint arXiv:1912.10077 , 2019. 6, 16

Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems , 33:17283-17297, 2020. 17

Zhongwang Zhang, Zhiwei Wang, Junjie Yao, Zhangchen Zhou, Xiaolong Li, Zhi-Qin John Xu, et al. Anchor function: a type of benchmark functions for studying language models. arXiv preprint arXiv:2401.08309 , 2024. 17

Appendix

A Detailed Related Works

Theoretical results of Transformer. We first review the expressive power results of Transformer. Yun et al. (2019) first proved the universal approximation property (UAP) of Transformer, highlighting the crucial role of PE in breaking permutation invariance. Edelman et al. (2022) demonstrated that Transformer can approximate fixed sparse functions. Dehghani et al. (2019); Pérez et al. (2021); Wei et al. (2022a) explored the Turing-completeness of infinite-precision and finite-precision Transformer. Giannou et al. (2023) showed that looped Transformer can implement practical computer programs.

Jiang and Li (2023) provided explicit approximation rates for Transformer in sequences modeling with inherent graph structures. Liu et al. (2022) found that Transformer can execute finite-state automata. Ma and Ying (2022) asserted the natural suitability of Attn for achieving permutation equivariance. Besides these affirmative results, several studies characterized the expressivity limitation of Transformers, particularly in modeling formal languages or simulating circuits (Hahn, 2020; Weiss et al., 2021; Bhattamishra et al., 2020; Merrill and Sabharwal, 2023b; Merrill et al., 2022). Additionally Feng et al. (2023); Merrill and Sabharwal (2023a) examined the expressivity of Transformer using Chain of Thought prompting (Wei et al., 2022b). Moreover, some studies showed that the in-context learning ability of Transformer is attainable by simulating gradient-based iterations across various layers (Garg et al., 2022; Akyürek et al., 2022; von Oswald et al., 2023; Von Oswald et al., 2023; Mahankali et al., 2023; Bai et al., 2023; Shen et al., 2023). Besides, experimental studies also provide insights into the mechanisms of Transformer through induction head (Elhage et al., 2021; Olsson et al., 2022), information flow (Wang et al., 2023), anchor functions (Zhang et al., 2024), etc.

Positional encoding. One core component of Transformer is the PE, which facilitates the representation of input sequence order. Theoretically, Transformer without PE lacks UAP and is restricted to representing permutation-invariant functions. PE was first introduced in Vaswani et al. (2017). It has limitations in encoding unseen positions. To overcome this difficulty, Shaw et al. (2018) introduced RPE. Subsequent studies proposed various different RPE types. Notable examples include T5's RPE (Raffel et al., 2020), Rotary RPE (Su et al., 2024) (utilized in PaLM (Chowdhery et al., 2023) and LlaMA (Touvron et al., 2023)), Alibi RPE (Press et al., 2022) (employed in BLOOM (Workshop et al., 2022)), and KERPLE (Chi et al., 2022). A prevailing belief is that RPEs can outperform APEs in the 'length generalization task' (Ontanón et al., 2022; Csordás et al., 2021)- the ability to generalize from smaller training contexts to larger ones, a critical challenge for Large Language Models (Anil et al., 2022; Abbe et al., 2023b). However, Press et al. (2022) revealed that the commonly used Rotary RPE may exhibit suboptimal performance in this task. The recent work (Kazemnejad et al., 2023; Chi et al., 2022) conducted systematic experiments comparing the length generalization capabilities of Transformers with various RPEs and APEs, suggesting that the RPEs used in T5 and KERPLE(log) demonstrate superior performance over other types .

Rethinking dot-product. Another critical component of Transformer is the DP structure. Due to its quadratic cost as a function of the sequence length, the necessity of DP has always been questioned. Numerous variants of DP have been proposed, demonstrating competitive performance across diverse tasks. Representative examples include Longformer (Beltagy et al., 2020), Big Bird (Zaheer et al., 2020), Reformer (Kitaev et al., 2020), Linformer (Wang et al., 2020), Performer (Choromanski et al., 2020), Synthesizer (Tay et al., 2021), etc. In particular, a recent study (Allen-Zhu and Li, 2023) compared standard and DP-free Transformers in modeling "context-free grammar". Their findings suggested that the presence of DP has a marginal impact on performance. These evidences motivate us to rethink the necessity of DP in Transformer .

Sparsity (Donoho, 2006; Candès and Wakin, 2008) has gained considerable attention in sequence modeling. In classical signal processing, there is a prevailing notion that valuable signals are extremely sparse. For example, when representing an image, one often finds that only a few wavelet coefficients hold significant values in wavelet space (Meyer, 1992). In NLP, the starting point off the traditional n -gram model (Shannon, 1948) is that the next token only relies on a few previous tokens. Such models, however, overlook long-range information, often resulting in suboptimal performance. For NLP tasks such as dependency parsing (Nivre and Scholz, 2004), sentiment analysis (Nasukawa and Yi, 2003), part-of-speech tagging (Francis and Kucera, 1979), and continuation writing (Brown et al., 2020; OpenAI, 2023), it is indeed often the case that only a limited subset of preceding information is crucial for accurate prediction. However, these relevant information can be quite distant. For instance, the resolution of a mystery novel may hinge on elements introduced at the outset. Moreover, for Transformer networks, extensive research into the visualization and interpretability has revealed that (i) the learned activation maps of FFN layers are extremely sparse (Li et al., 2023); (ii) the learned self-attention matrices exhibit notable sparsity, yet it does not closely resemble a diagonal configuration (Elhage et al., 2021). These observations suggest that the prediction of the next token is influenced by a small number of previous tokens which might be far away. Therefore, being able to represent sparse but long-range dependence is important for sequence modeling .

B Proof of Section 3

B.1 Proof of Theorem 3.1

In this subsection, we give the detailed proofs of fixed, long but sparse memory:

where 1 ≤ T 1 < · · · < T M < + ∞ signify the fixed positions of the memories.

Theorem B.1 (Restatement of Theorem 3.1) . For any target H ∈ H Fix (6) , rate n ∈ N + , and H,m ∈ N + , there exists a 1 -layer Transformer TF ∈ T F DPF , type (1 ,H,m ) (7) and a constant C ( n ) such that

where E FFN = ˜ O ( ∥ f ∥ B √ m ) and

Proof of Theorem B.1.

First, we choose the embedding dimension D = ( M + 1) d , and select the simple embedding W E = ( I d × d , 0 ) ⊤ ∈ R D × d , b E = 0 ∈ R D .

Then for any input sequence X = ( x t ) t ∈ Z , the token after embedding satisfies:

For one-layer Dot-product-free Transformer TF ∈ T F DPF , type (1 ,H,m ) with ϕ type , the output token TF t ( X ) of t -th input token x t satisfies:

where

This proof can be summarized as the following process:

Now we give the formal proof.

Step I. Extract the memory locations by (Dot-product-free) Attn layer.

We consider to use H k attention heads (from ∑ k -1 i =1 H i +1 -th head to ∑ k i =1 H i -th head) to extract it, and it satisfies to ∑ M k =1 H k = H .

For simplicity, we denote the following projection matrices:

Now we consider the extraction of k -th memory x t -T k ( 1 ≤ k ≤ M ).

Case type = lin .

By Lemma F.1, for any rate n ∈ N + , there exists an constant C ( n ) and a function

such that β h > 0 and

Therefore, for these attention heads ( ∑ k -1 i =1 H i +1 ≤ h ≤ ∑ k i =1 H i ), we can choose

where δ ( k +1 , 1) ∈ R D × D means that: it equals to I d × d for the ( k +1 , 1) -th d × d blocks, and 0 d × d for the other d × d blocks.

Then it holds that:

This implies:

moreover, the following estimate holds:

· Case type = log .

By Lemma F.4, for any rate n ∈ N + , there exists an constant C ( n ) and a function

such that β h > 1 and

Therefore, for these attention heads ( ∑ k -1 i =1 H i +1 ≤ h ≤ ∑ k i =1 H i ), we can choose

where δ ( k +1 , 1) ∈ R D × D means that: it equals to I d × d for the ( k +1 , 1) -th d × d blocks, and 0 d × d for the other d × d blocks.

Then it holds that:

moreover, the following estimate holds:

Then we combine the results for all k ∈ [ M ] for these two cases. By choose W O = I D , we have:

Consequently, one detail is to assign the head number { H k } M k =1 such that the error's sum E Attn ( type ) is as small as possible. Our way is solving the minimization problem:

which suggests that we should choose the head number:

Thus, we obtain the bound in Step I:

.

Furthermore, by choosing E Attn ( type ) ≤ 1 , it holds that

Step II. Approximate the readout function by FFN layer.

In this step, we aim to approximate the function f using two-layer network. By Lemma G.6, there exists a two layer neural network with m neurons defined on R D

The final bound.

For any t and X ∈ X , it holds that

where

Due to the arbitrariness of t and X , the proof is completed.

such that

C Proof of Section 4.2

In this section, we give the detailed proofs of the approximation theory of Transformer for modeling the warm-up case of adaptive, long but sparse memory:

where the adaptive memory satisfies to:

Moreover, g k ( · ) generate positive integers for the input tokens, and there exist maximum values T k such that 1 ≤ g k ( x t ) ≤ T k holds for any x t and k ∈ [ M ] .

To tackle the discrete values of the time and the memory values g k ( x t ) , a modified version of standard FFN, termed 'FFN with precision', us cibsudered. This approach ensures that the output of FFN undergoes a simple rounding operation. Notably, the precision technique is widely used in LLM training (Kalamkar et al., 2019), such as BFloat16 . Specifically, for Transformer using RPE with type , we use the following FFN with precision:

where [ · ] signifies rounding to the nearest integer, i.e., [ x ] = arg min n ∈ Z | n -x | ( x ∈ R ).

It is important to note that the rounding obtained by using the operator log[exp( z )] , used in (15), is quite fine , which is much finer than the vanilla rounding obtained by [ z ] . To elaborate, the following proposition is presented:

Proposition C.1. For any z ≥ 1 , the following holds:

C.1 Proof of Theorem 4.1

Theorem C.2 (Restatement of Theorem 4.1) . For any target H ∈ H Adap (1 ,M ) (8) , rate n ∈ N + , and H,m ∈ N + , there exists a two-layer Transformer TF ∈ T F NF , type (2 ,H,m ) (12) and a constant C ( n ) such that: if the width satisfies

then the following approximation rate holds:

where E FFN = ˜ O ( ∥ f ∥ B √ m ) and

Proof of Theorem C.2.

First, we choose the embedding dimension D = ( M +1)( d +1) , and select a simple embedding W E = ( I d × d , 0 ) ⊤ ∈ R D × d , b E = 0 ∈ R D .

Then for any input sequence X = ( x t ) t ∈ Z , the token after embedding satisfies:

To tackle the discrete values of g m ( x t ) , we utilize ˜ FFN , FFN with precision (15). It ensures that the output of FFN undergoes a simple rounding operation.

Thus, for two-layer normalization-free Transformer TF ∈ T F NF , type (2 ,H,m ) with ϕ type , the output token x (2) t of t -th input token satisfies:

where

This proof can be summarized as the following process:

Now we give the formal proof.

Step I. Identity map.

For the first Attn layer, we only need to do the identity map by taking W (1) 0 = 0 . Then x (1 / 2) t = x (0) t . Step II. Approximate the adaptive memory function by the first FFN layer.

Case type = lin . Our main idea is that using the first FFN layer to express ( x ⊤ t , 0 ⊤ , g 1 ( x t ) , · · · , g M ( x t ) , 1) ⊤ exactly.

First, we consider to approximate the r -th memory function g ( x ) by standard FFN.

For any r ∈ [ M ] , by Lemma G.6, there exists a two-layer neural network with m neurons r r

defined on R d such that

Therefore, if we choose the following holds:

Noticing g r ( x t ) ∈ N + , we have [ f 2NN (1 ,r ) ( x t ) ] = g r ( x t ) , which implies:

Consequently, in order to construct the form ( 0 ⊤ , g 1 ( x t ) , · · · , g M ( x t ) , 1) ⊤ ∈ R D , we need to arrange the parameters a (1 ,r ) k , b (1 ,r ) k , and c (1 ,r ) k ( k ∈ [ m r ] , r ∈ [ M ]) appropriately.

Denote ¯ b (1 ,r ) k = ( b (1 ,r ) k ⊤ , 0 ⊤ ) ⊤ ∈ R D for k ∈ [ m r ] , r ∈ [ M ] . Consider the following two-layer neural network with 1 + ∑ M r =1 m r neurons defined on R D :

It is easy to verify that for any x (1 / 2) t , it holds that

Moreover, it satisfies that

Thus, we have achieved our goal in this step:

Case type = log . Our main idea is that using the first FFN layer to express ( x ⊤ t , 0 ⊤ , log g 1 ( x t ) , · · · , log g M ( x t ) , log 2) ⊤ exactly.

First, we consider to approximate the r -th memory function log g ( x ) by standard FFN.

r For any r ∈ [ M ] , by Lemma G.6, there exists a two-layer neural network with m r neurons

defined on R d such that

Therefore, if we choose the following holds:

which ensures

Noticing g r ( x t ) ∈ N + , we have [ exp ( f 2NN (1 ,r ) ( x t ) )] = g r ( x t ) , which implies:

Consequently, in order to construct the form ( 0 ⊤ , log g 1 ( x t ) , · · · , log g M ( x t ) , log 2) ⊤ , we need to arrange the parameters a (1 ,r ) k , b (1 ,r ) k , and c (1 ,r ) k ( k ∈ [ m r ] , r ∈ [ M ]) appropriately.

Denote ¯ b (1 ,r ) k = ( b (1 ,r ) k ⊤ , 0 ⊤ ) ⊤ ∈ R D for k ∈ [ m r ] , r ∈ [ M ] . Consider the following two-layer neural network with 1 + ∑ M r =1 m r neurons defined on R D :

It is easy to verify that for any x (1 / 2) t , it holds that

Moreover, it satisfies that

Thus, we have achieved our goal in this step:

As established in the proof above, the width m must satisfy:

Step III. Extract the adaptive memories by the second Attn layer.

We consider to use H k attention heads (from ∑ k -1 i =1 H i +1 -th head to ∑ k i =1 H i -th head) to extract it, and it satisfies to ∑ M k =1 H k = H .

For simplicity, we denote the following projection matrices in R D × D :

Now we consider the extraction of k -th adaptive memory x t -g k ( x t ) ( 1 ≤ k ≤ M ).

Case type = lin .

By Lemma F.2, for any rate n ∈ N + , there exists a constant C ( n ) and a function

such that β h > 0 and

Moreover, Noticing that 1 ≤ g k ( x t ) ≤ T k holds for any X = ( x t ) t ∈ Z ∈ X , the following holds:

Therefore, for these attention heads ( ∑ k -1 i =1 H i +1 ≤ h ≤ ∑ k i =1 H i ) , we can choose:

where δ ( r × r ) ( p 1 ,p 2 ) means that: it equals to I r × r for the ( p 1 , p 2 ) -th r × r blocks, and 0 r × r for the other r × r blocks.

Then it is easy to verify:

which implies:

Then it holds that:

moreover, the following estimate holds:

· Case type = log .

By Lemma F.5, for any rate n ∈ N + , there exists a constant C ( n ) and a function

such that β h > 1 and

Moreover, Noticing that 1 ≤ g k ( x t ) ≤ T k holds for any X = ( x t ) t ∈ Z ∈ X , the following holds:

Therefore, for these attention heads ( ∑ k -1 i =1 H i +1 ≤ h ≤ ∑ k i =1 H i ) , we can choose:

where δ ( r × r ) ( p 1 ,p 2 ) means that: it equals to I r × r for the ( p 1 , p 2 ) -th r × r blocks, and 0 r × r for the other r × r blocks.

Then it is easy to verify:

which implies:

Then it holds that:

moreover, the following estimate holds:

Then we combine the estimate for all k ∈ [ M ] for these two cases. It holds that

Similar to the proof of Theorem B.1, we choose the head number:

Thus, we obtain the final bound in Step III:

.

Furthermore, by choosing E Attn ( type ) ≤ 1 , it holds that

Step IV. Approximate the nonlinear function by 2 -nd FFN layer.

In this step, we aim to approximate the function f using two-layer network. By Lemma G.6, there exists a two-layer neural network with m neurons

defined on R ( M +1) d such that

Denote ¯ b (2) k = ( b (2) k ⊤ , 0 ⊤ ) ⊤ ∈ R D for k ∈ [ m ] . And we consider the following two-layer neural network with m neurons defined on R D :

It is easy to verify:

Moreover,

The final bound.

For any t and ∥ X ∥ ∈ X , it holds that

where

Moreover, recalling our proof in Step II, we further need the hard condition:

Due to the arbitrariness of t and X , the proof is completed.

Remark C.3. The core step in this proof is Step III , where the extraction of the memory functions is achieved through a a nice interaction between the temporal space (provided by RPE) and the token space (provided by DP) . Specifically, the memory functions g i ( x t ) (in token space) are mapped into the temporal space s , resulting in the form of x s -g i ( x t ) for DP Attn with lin -RPE or log( s/g i ( x t )) for DP Attn with log -RPE.

C.2 Proof of Proposition 4.2

For Proposition 16, we denote the following one-layer Attn hypothesis class:

Proposition C.4 (The formal version of Proposition 4.2) . Consider 1 -layer Attn. Then, there exists a target H ∈ H adap (1 , 1) (10) such that:

(A) (Attn with DP) For any ϵ > 0 , there exists a 1 -layer Attn Attn DP ∈ AT T N type (1 ,H ) such that

(B) (Attn without DP) For any 1 -layer DP-free Attn Attn DPF ∈ AT T N DPF , type (1 ,H ) , a uniform lower bound holds:

Proof of Proposition C.4.

Consider the following target function H ∈ H Adap 1 , 1 . Let the input sequence X ∈ X = {-1 , 0 , 1 } Z , and we consider the target

Part (A). The Efficiency of Attn with DP.

First, we choose the embedding dimension D = 2 , and select simple embedding W E = b E = (1 , 0) ⊤ ∈ R 2 × 1 . Then for any input sequence X = ( x t ) t ∈ Z , the token after embedding satisfies:

We consider one-layer normalization-free Self-attention with ϕ exp , which has the form:

By Lemma F.2 (for n = 1 ), there exists a constant C > 0 and a function

such that β h > 0 and

where the adaptive memory is

Moreover, Noticing that 0 ≤ g ( x t ) ≤ 2 holds for any X = ( x t ) t ∈ Z ∈ X , the following holds:

Therefore, for attention heads ( 1 ≤ h ≤ H ), we can choose:

where δ ( r × r ) ( p 1 ,p 2 ) means that: it equals to I r × r for the ( p 1 , p 2 ) -th r × r blocks, and 0 r × r for the other r × r blocks.

Then it is easy to verify:

Thus, the following estimate holds:

Due to the arbitrariness of t and X , the proof is completed: for any ϵ > 0 , we only need to use H = Ω(1 /ϵ ) heads to approximate it.

Part (B). The Hardness Result of Attn without DP.

We consider one-layer Dot-product-free Self-attention with ϕ exp or ϕ log . For any input X , the corresponding output can be written as:

For simplicity, we denote:

then we have the following estimate:

C.3 Proof of Proposition 4.3

In this subsection, we propose a structurally simpler yet effective alternative to traditional Dot-product in Self-attention. This alternative is proposed based on our insights into the role of Attn in facilitating interaction between the temporal-space and the token-space. Specifically, we propose a more direct structure to achieve the interaction ϕ type ( s ) -ϕ type ( w ⊤ x t ) .

Definition C.5 (TMX Transformer) . We define the TMX (' t minus x ') Transformer as follows. In standard Transformer (2), we modify the term

in Multi-head Dot-product Self-attention to the new formulation:

where the parameters w ( l,h ) ∈ R D × 1 .

Notice that in TMX Transformer, the revised term requires only O ( D ) parameters, much less than O ( D 2 ) in standard Dot-product Transformer.

Consequently, we define the following TMX Transformer hypothesis class.

Proposition C.6 (The formal version of Proposition 4.3) . Under the same conditions in Theorem 4.1, there exists a two-layer TMX Transformer TF ∈ T F TMX , type (2 ,H,m ) (17) such that it can achieve the same approximation rate as standard Transformer presented in Theorem 4.1.

Proof of Proposition C.6.

It is worth noting that TMX Transformer only replaces 〈 W ( l,h ) Q x t , W ( l,h ) K x t -s 〉 + p ( l,h ) ϕ type ( s ) in standard Transformer with p ( l,h ) ( ϕ type ( s ) -ϕ type ( w ( l,h ) ⊤ x t ) ) . Therefore, the proof is highly similar to that of Theorem C.2. We only need to prove that TMX Attn can also achieve Step I and Step III in the proof of Theorem C.2.

Step I. Step I is trivial due to the same use of the residual block.

Step III. Extract the adaptive memories by the second Attn layer.

We still consider to use H k attention heads (from ∑ k -1 i =1 H i + 1 -th head to ∑ k i =1 H i -th head) to extract it, and it satisfies to ∑ M k =1 H k = H .

Now we consider the extraction of k -th adaptive memory x t -g k ( x t ) ( 1 ≤ k ≤ M ).

· Case type = lin .

For the proof of standard Transformer (the proof of Theorem C.2), for the attention heads ( ∑ k -1 i =1 H i +1 ≤ h ≤ ∑ k i =1 H i ) , we can construct specific p (2 ,h ) , W (2 ,h ) Q , W (2 ,h ) K , W (2 ,h ) V such that

In this proof, we only need to prove that we can also construct specific w ( l,h ) , W (2 ,h ) V such that

Recalling the proof of Theorem C.2,

Therefore, we can choose

where δ ( r × r ) ( p 1 ,p 2 ) means that: it equals to I r × r for the ( p 1 , p 2 ) -th r × r blocks, and 0 r × r for the other r × r blocks.

The the following holds:

Case type = log .

For the proof of standard Transformer (the proof of Theorem C.2), for the attention heads ( ∑ k -1 i =1 H i +1 ≤ h ≤ ∑ k i =1 H i ) , we can construct specific p (2 ,h ) , W (2 ,h ) Q , W (2 ,h ) K , W (2 ,h ) V such that

In this proof, we only need to prove that we can also construct specific w ( l,h ) , W (2 ,h ) V such that

Recalling the proof of Theorem C.2,

Therefore, we can choose

where δ ( r × r ) ( p 1 ,p 2 ) means that: it equals to I r × r for the ( p 1 , p 2 ) -th r × r blocks, and 0 r × r for the other r × r blocks.

The the following holds:

The rest of the proof is exactly the same as the proof of Theorem C.2, and we do not repeat it.

D Proof of Section 4.3

D.1 Proof of Theorem 4.4

In this subsection, we give the detailed proofs for the general case of K -adaptive, long but M -sparse memory:

where the adaptive memories satisfy:

where 1 ≤ t k ≤ T k holds for any k ∈ [ M ] .

Theorem D.1 (Restatement of Theorem 4.4) . For any target H ∈ H Adap ( K,M ) , rate n ∈ N + , and H,m ∈ N + , there exists an L -layer ( L = K +1+ I { M ≥ K +1 } ) Transformer TF ∈ T F NF , type ( L,H,m ) (12) and a constant C ( n ) such that: if the width satisfies

then the following approximation rate holds:

where E FFN = ˜ O ( ∥ f ∥ B √ m ) and

Proof of Theorem D.1.

First, we choose the embedding dimension D = ( M +1)( d +1) , and select the same embedding matrix W E = ( I d , 0 ) ⊤ ∈ R D × d , b E = 0 = ∈ R D as the proof of Theorem C.2. Moreover, we still use the network with precision ˜ FFN defined in Appendix C.1 to tackle the discrete values of memories.

Then for any input sequence X = ( x t ) t ∈ Z , the token after embedding satisfies:

Thus, for L -layer ( L = K +1+ I { M ≥ K +1 } ) normalization-free Transformer TF ∈ T F NF , type ( L,H,m ) with ϕ type , the output token x ( K +1) t of t -th input token satisfies:

.

where

Since the proof of this theorem is similar to the proof of Theorem C.2, we mainly discuss the differences.

The proof can be summarized as the following process:

Case type = lin .

Regime M ≥ K +1 .

•

For simplicity, we denote the following projection matrices:

Step 1 is trivial due to the use of the residual block.

Step 2. In the same way as Step II in the proof of Theorem C.2, we obtain the conclusion in this step: If the width of FFN satisfies

then the following holds:

Thus, (E1) holds for l = 2 .

Case type = lin .

-FFN layers.

We use l -th ( 2 ≤ l ≤ K ) FFN layer to express l -th memory t l exactly. By Lemma G.6, there exists a two-layer neural network with m neurons defined on R ld

such that

For l -th FFN layer, we only need to arrange the parameters a ( l ) k , b ( l ) k , and c ( l ) k ( k ∈ [ m ] , 2 ≤ l ≤ M ) .

Denote ¯ b ( l ) k = ( b ( l ) k ⊤ , 0 ⊤ ) ⊤ ∈ R D for k ∈ [ m ] , 2 ≤ l ≤ K -1 . Consider the following two-layer neural network with m neurons defined on R D :

It is easy to verify

Notice that if the width m satisfies

and the input x ( l -1 / 2) t satisfies

the following holds:

Noticing t l = g l ( x t , · · · , x t -t l -1 ) ∈ N + , we have

Thus, it holds that:

-Attn layers.

By Lemma F.2, for any rate n ∈ N + , there exists a constant C ( n ) and K functions

such that β l,h > 0 and

Moreover, Noticing that 1 ≤ g l ( · ) ≤ T l holds for any X = ( x t ) t ∈ Z ∈ X and 2 ≤ l ≤ K , the following holds:

Therefore, for the attention heads h ( h ∈ [ H ] ) in each layer l ( 1 ≤ l ≤ K ) , we can choose:

where δ ( r × r ) ( p 1 ,p 2 ) means that: it equals to I r × r for the ( p 1 , p 2 ) -th r × r blocks, and 0 r × r for the other r × r blocks.

· Case type = log .

-FFN layers.

We use l -th ( 2 ≤ l ≤ K ) FFN layer to express l -th memory t l exactly. By Lemma G.6, there exists a two-layer neural network with m neurons defined on R ld

such that

For l -th FFN layer, we only need to arrange the parameters a ( l ) k , b ( l ) k , and c ( l ) k ( k ∈ [ m ] , 2 ≤ l ≤ M ) .

Denote ¯ b ( l ) k = ( b ( l ) k ⊤ , 0 ⊤ ) ⊤ ∈ R D for k ∈ [ m ] , 2 ≤ l ≤ K -1 . We consider the following l -th layer 2NN with m neurons defined on R D :

It is easy to verify

Notice that if the width m satisfies

and the input x ( l -1 / 2) t satisfies

the following holds:

which ensures

Noticing t l = g l ( x t , · · · , x t -t l -1 ) ∈ N + , we have

Thus, it holds that:

-Attn layers.

By Lemma F.5, for any rate n ∈ N + , there exists a constant C ( n ) and K functions

such that β l,h > 0 and

Moreover, Noticing that 1 ≤ g l ( · ) ≤ T l holds for any X = ( x t ) t ∈ Z ∈ X and 1 ≤ l ≤ K , the following holds:

Therefore, for the attention heads h ( h ∈ [ H ] ) in each layer l ( 1 ≤ l ≤ K ) , we can choose:

where δ ( r × r ) ( p 1 ,p 2 ) means that: it equals to I r × r for the ( p 1 , p 2 ) -th r × r blocks, and 0 r × r for the other r × r blocks.

Similar to the estimate in Step II and Step III in the proof of Theorem C.2, it is easy to prove the following estimates by induction.

If the width satisfies

and the head number satisfies

then the following estimates hold:

(E1) for any 2 ≤ l ≤ K ,

(E2) for any 1 ≤ l ≤ K ,

(E3) for any 1 ≤ l ≤ K ,

The Remained Steps.

Regime M ≥ K +1 .

In the similar way as Step 3 ∼ Step 2 K -1 in this proof and Step II, Step III in the proof of Theorem C.2, it is easy to verify the following estimate.

If the width satisifes

and the head number satisfies

then the following estimates hold:

Step 2 K +4 and the final bound.

In the same way as Step IV in the proof of Theorem C.2, there exists FFN , such that the following estimate holds for any t and X :

where

Recalling our analysis, we need the head number satisfies

Due to

when we is large enough, this condition holds naturally and do not affect the approximation rate.

Moreover, we need the following condition on the width:

.

· Regime M = K .

Step 2 K +2 and the final bound.

In the same way as Step IV in the proof of Theorem C.2, there exists FFN , such that the following estimate holds for any t and X :

where

Recalling our analysis, we need the head number satisfies

Due to

when H is large enough, this condition holds naturally and do not affect the approximation rate.

Moreover, we need the following condition on the width:

Combining these two regimes, we complete our proof.

D.2 Proof of Proposition 4.5

Proof of Proposition 4.5.

This proposition is a direct corollary of Theorem 4.4. It can be seen as a special case of M = K in Theorem 4.4.

Therefore, under the same conditions, there exists a K + 1 -layer Transformer TF ∈ T F NF , type ( K +1 ,H,m ) (12) and a constant C ( n ) such that: if the width satisfies

then the following approximation rate holds:

where E FFN = ˜ O ( ∥ f ∥ B √ m ) and

.

Comparison between Proposition 4.5 and Theorem 4.1.

We compare 2 -layer Transformer and M +1 -layer Transformer regarding the requirement of the number of heads and width.

The required width of FFN layers.

-For 2 -layer Transformer, the required width of FFN layers m (2) need is proportionally linked to the sum of all the memory functions' complexity:

-For M +1 -layer Transformer, the required width of FFN layers m ( M +1) need correlates with the maximum complexity of the memory functions:

It is easy to see that:

The required number of Attn heads. To achieve the same E Attn ( type ) = ϵ ,

-for 2 -layer Transformer, the required number of Attn heads H (2) need satisfies:

.

-for M +1 -layer Transformer, the required number of Attn heads H ( M +1) need satisfies:

It is easy to see that:

This finding suggests that increased depth can significantly reduce the demands on the number of heads and the width. The underlying reason is that deep networks can distribute memories across different layers for processing, with each layer focusing on approximating only a single memory function.

E Proof of Section 5

E.1 Proof of Theorem 5.1

In this subsection, we give the detailed proofs of the warm-up case of (fixed) essentially sparse memories as follows:

where ρ 1 ( · ) , · · · , ρ M ( · ) ∈ ℓ 1 ( N ) serve as memory kernels, and ( X ∗ ρ k )( t ) = ∑ + ∞ s =0 x t -s ρ k ( s ) denotes the convolution of the inputs with kernel ρ k .

Theorem E.1 (Restatement of Theorem 5.1) .

(A) Consider H Ess (14) with exponentially decayed memory kernels, i.e., there exists β > 0 such that ρ 1 ( t ) , · · · , ρ M ( t ) = O ( e -βt ) . Then for any target H ∈ H Ess , rate n ∈ [ ⌊ 99 β ⌋ ] , and H,m ∈ N + , there exists a 1 -layer DP-free Transformer TF ∈ T F DPF , exp (1 ,H,m ) (7) and a constant C ( n ) such that

(B) Consider H Ess (14) with polynomially decayed memory kernels, i.e., there exists β > 1 such that ρ 1 ( t ) , · · · , ρ M ( t ) = O ( t -β ) . Then for any target H ∈ H Ess , rate n ∈ [ ⌊ 0 . 99 β ⌋ -1] , and H,m ∈ N + , there exists a 1 -layer DP-free Transformer TF ∈ T F DPF , poly (1 ,H,m ) (7) and a constant C ( n ) such that

Proof of Theorem E.1.

The proof of this theorem is highly similar to the proof of Theorem B.1. The only difference is that the Attn layer needs to be used to approximate general memory kernel ρ k ( · ) instead of simple I {· = T k } . But for the completeness of the proof in this section, we still provide the detailed proof.

First, we choose the embedding dimension D = Md , and select the simple embedding W E = ( I d × d , 0 ) ⊤ ∈ R D × d , b E = 0 ∈ R D .

For any input sequence X = ( x t ) t ∈ Z , the token after embedding satisfies:

Then for one-layer Dot-product-free Transformer TF ∈ T F DPF , type (1 ,H,m ) without residual blocks, the output token TF t ( X ) of t -th input token x t satisfies:

where

This proof can be summarized as the following process:

Now we give the formal proof.

Step I. Extract the memory locations by (Dot-product-free) Attn layer.

We consider to use H k attention heads (from ∑ k -1 i =1 H i +1 -th head to ∑ k i =1 H i -th head) to extract it, and it satisfies to ∑ M k =1 H k = H .

Now we consider the extraction of k -th memory ( X ∗ ρ k )( t ) ( 1 ≤ k ≤ M ).

Case (A). Approximating exponentially decayed memories by type = lin . Because there exists β > 0 such that ρ k ( t ) = O ( e -βt ) , by Lemma F.3, for any n ∈ [ ⌊ 99 β ⌋ and m ∈ N + , there exists an absolute constant C ( n ) only depending on n and a function

such that β h > 0 and

Therefore, for these attention heads ( ∑ k -1 i =1 H i +1 ≤ h ≤ ∑ k i =1 H i ), we can choose

where δ ( k, 1) ∈ R D × D means that: it equals to I d × d for the ( k, 1) -th d × d blocks, and 0 d × d for the other d × d blocks.

Then it holds that:

This implies:

]

moreover, the following estimate holds:

· Case (B). Approximating polynomially decayed memories by type = log .

Because there exists β > 0 such that ρ k ( t ) = O ( t -β ) , by Lemma F.6, for any n ∈ [ ⌊ 0 . 99 β ⌋ -1] and m ∈ N + , there exists an absolute constant C ( n ) only depending on n and a function

such that β h > 1 and

Therefore, for these attention heads ( ∑ k -1 i =1 H i +1 ≤ h ≤ ∑ k i =1 H i ), we can choose

where δ ( k, 1) ∈ R D × D means that: it equals to I d × d for the ( k, 1) -th d × d blocks, and 0 d × d for the other d × d blocks.

Then it holds that:

This implies:

Then we combine the estimate for all k ∈ [ M ] for these two cases. By choose W O = I D , we have:

Consequently, one detail is to assign the head number { H k } M k =1 such that the error's sum E Attn ( type ) is as small as possible. Here, we simply choose the same H k :

Thus, we obtain the bound in Step I:

Furthermore, by choosing E Attn ≤ 1 , it holds that

Step II. Approximate the readout function by FFN layer.

In this step, we aim to approximate the function f using two-layer network. By Lemma G.6, there exists a two-layer neural network with m neurons defined on R D

The final bound.

For any t and X ∈ X , it holds that

where

Due to the arbitrariness of t and X , the proof is completed.

such that

F Key Lemmas about Approximation

F.1 Approximation by the sum of exponential decay

Lemma F.1 (Exp decay, fixed Delta function) . For any T ∈ N + , n, m ∈ N + , there exists and absolute constant C ( n ) only depending on n and a ϕ exp m ( t ) = m ∑ k =1 α k e -β k t such that

where β k > 0 holds for any k ∈ [ m ] .

Proof of Lemma F.1.

Let α, γ > 0 be constants, and they will take specific values at the end of the proof.

First, recall the standard bump function on [ -1 , 1] :

and we can define the following constants for T ≥ 1 :

Then we consider the following bump function Ψ T ∈ C ∞ ([0 , 1]) :

where V T is a scaling constant such that Ψ T ( e -αT ) = e γT .

First, we consider the approximation of Ψ T on [0 , 1] .

Notice that Ψ T ∈ C ∞ ([0 , 1]) , and Ψ ( k ) T (0) = 0 for any k ∈ N . For the standard bump function Ψ , for any n ∈ N + , there exists an absolute constant M ( n ) > 0 only depending on n , such that max 0 ≤ k ≤ 10 sup x ∈ [ -1 , 1] ∣ ∣ Ψ ( k ) ( x ) ∣ ∣ ≤ M ( n ) .

Notice that for any k ∈ N and x ∈ [0 , 1] ,

Therefore, the following upper bound holds:

By Lemma G.5, for any m ∈ N + , there exists a polynomial Q m ( x ) = m -1 ∑ k =0 α k x k such that

Now we use the transform x = e -αt on the function Ψ and consider

It is easy to verify that Φ T satisfies that

Moreover, we consider the function

Then by choosing α = γ = 0 . 01 , the following error estimate holds:

Finally, notice that P m ( t ) = e -γt Q m ( e -αt ) = m -1 ∑ k =0 α k e -(0 . 01+0 . 01 k ) , so we can select ϕ exp m ( t ) := P m ( t ) .

Lemma F.2 (Exp decay, adaptive Delta function) . For any T ∈ N , n, m ∈ N + , there exists an absolute constant C ( n ) only depending on n and a ϕ exp m ( t ; B ) = m ∑ k =1 α k e -β k ( t -B ) such that

where β k > 0 holds for any k ∈ [ m ] .

Proof of Lemma F.2.

The key point of the proof is to note that the adaptability of B can be eliminated by the translation operator t -B .

First, recall our proof of Lemma F.1. For the same Ψ T ( · ) , for any n, m ∈ N + , there exists an absolute constant C ( n ) only depending on n and a polynomial Q m ( x ) = m -1 ∑ k =0 α k x k such that

Moreover, using the transform x = e -0 . 01( t -B + T ) ( t ≥ 0) on the function Ψ and consider

It is easy to verify that Φ T ( · ; · ) satisfies that

And we consider the function

Then, for any 1 ≤ B ≤ T , the following error estimate holds:

Due to the arbitrariness of B , the proof is completed.

Lemma F.3 (Exp decay, fixed Delta function) . Consider a exponentially decayed memory ρ ( · ) : there exists β > 0 such that ρ ( t ) = O ( e -βt ) . Then for any n ∈ [ ⌊ 99 β ⌋ ] and m ∈ N + , there exists an absolute constant C ( n ) only depending on n and a ϕ exp m ( t ) = m ∑ k =1 α k e -β k t such that

where β k > 0 holds for any k ∈ [ m ] .

Proof of Lemma F.3.

There exists C > 0 such that | ρ ( t ) | ≤ Ce -βt .

Let α, γ > 0 be constants, and they will take specific values at the end of the proof.

First, recall the standard bump function on [ -1 , 1] :

and we can define the following constants for T ≥ 1 :

and we consider the following bump function Ψ T ∈ C ∞ ([0 , 1]) :

where V T is a scaling constant such that Ψ T ( e -αT ) = e γT ρ ( T ) .

Consequently, we consider the sum of bump functions on [0 , 1] :

It is easy to verify that ( µ T 1 -σ T 1 , µ T 1 + σ T 1 ) ∩ ( µ T 2 -σ T 2 , µ T 2 + σ T 2 ) = ∅ for any T 1 = T 2 and

.

̸

First, we study the property of φ ( · ) .

We denote the absolute constants M k = sup x | φ ( k ) ( x ) | . Notice that for any k ∈ N ,

Therefore, it holds that

Therefore, if β ≥ γ + kα , then the following uniform bounds holds:

Consequently, we consider the smoothness of Φ at x = 0 . Recalling the previous results, for any x ∈ (0 , 1] , we have

Thus, by induction, it is easy to verify that for any i < β -γ α ( i ∈ N ),

Therefore, for any n < β -γ α ( n ∈ N ), φ ( k ) (0) = 0 holds for any 0 ≤ k ≤ n . Moreover, there exists absolute constant C ( n, α ) such that:

By Lemma G.5, for any m ∈ N + , there exists a polynomial Q m ( x ) = m -1 ∑ k =0 α k x k such that

Now we use the transform x = e -αt ( t ≥ 0) on the function φ and consider

It is easy to verify that Φ satisfies that

Moreover, we consider the function

Then for any n < β -γ α ( n ∈ N ), the following error estimate holds:

By choosing α = 5 · 10 -3 and γ = 10 -2 β , it holds that 99 β < β -γ 2 α = β -γ α . Thus, we obtain our result: for any n ∈ [ ⌊ 99 β ⌋ ] ( β ≥ 1 / 99

), the following error estimate holds:

F.2 Approximation by the sum of polynomial decay

Lemma F.4 (Poly decay, fixed Delta function) . For any T, n, m ∈ N + , there exists an absolute constant C ( n ) only depending on n and a ϕ poly m ( t ) = m ∑ k =1 α k t -β k such that

where β k > 1 holds for any k ∈ [ m ] .

Let α, γ > 0 be constants, and they will take specific values at the end of the proof First, recall the standard bump function on [ -1 , 1] :

and we can define the following constants for T ≥ 1 :

Then we consider the following bump function Ψ T ∈ C ∞ ([0 , 1]) :

where V T is a scaling constant such that Ψ T ( 1 T α ) = T 1+ γ . First, we consider the approximation of Ψ T on [0 , 1] .

Notice that Ψ T ∈ C ∞ ([0 , 1]) , and Ψ ( k ) T (0) = 0 for any k ∈ N . For the standard bump function Ψ , for any n ∈ N + , there exists an absolute constant M ( n ) > 0 only depending on n , such that max 0 ≤ k ≤ n sup x ∈ [ -1 , 1] ∣ ∣ Ψ ( k ) ( x ) ∣ ∣ ≤ M ( n ) .

Notice that for any k ∈ N and x ∈ [0 , 1] ,

Therefore, the following upper bound holds:

By Lemma G.5, for any m ∈ N + , there exists a polynomial Q m ( x ) = m -1 ∑ k =0 α k x k such that

Now we use the transform x = 1 t α ( t ≥ 1) on the function Ψ and consider

It is easy to verify that Φ T satisfies that

Moreover, we consider the function

Then by choosing α = γ = 0 . 01 , the following error estimate holds:

Finally, notice that P m ( · ) satisfies to P m ( t ) = 1 t 1+ γ Q m ( 1 t α ) = m -1 ∑ k =0 α k t -(1 . 01+0 . 01 k ) , so we can select ϕ poly m ( t ) := P m ( t ) .

Lemma F.5 (Poly decay, adaptive Delta function) . For any T, n, m ∈ N + , there exists an absolute constant C ( n ) only depending on n and a ϕ poly m ( t ; B ) = m ∑ k =1 α k ( t/B ) -β k such that

where β k > 1 holds for any k ∈ [ m ] .

Proof of Lemma F.5.

The key point of the proof is to note that the adaptability of B can be eliminated by the rescaling operator t/B .

First, recall our proof of Lemma F.4. For the same Ψ T ( · ) , for any n, m ∈ N + , there exists an absolute constant C ( n ) only depending on n and a polynomial Q m ( x ) = m -1 ∑ k =0 α k x k such that

We use the transform x = 1 t 0 . 01 ( t ≥ 1) on the function Ψ and consider

It is easy to verify that Φ T ( · ; · ) satisfies that

And we consider the function

Then, for any 1 ≤ B ≤ T , the following error estimate holds:

Due to the arbitrariness of B , the proof is completed.

Lemma F.6 (Poly decay, fixed Delta function) . Consider a polynomially decayed memory ρ ( · ) : there exists β > 1 such that ρ ( t ) = O ( t -β ) . Then for any n ∈ [ ⌊ 0 . 99 β ⌋ -1] and m ∈ N + , there exists an absolute constant C ( n ) only depending on n and a ϕ poly m ( t ) = m ∑ k =1 α k t -β k such that

where β k > 1 holds for any k ∈ [ m ] .

Proof of Lemma F.6.

There exists C > 0 such that | ρ ( t ) | ≤ C/t β .

Let α, γ > 0 be constants, and they will take specific values at the end of the proof First, recall the standard bump function on [ -1 , 1] :

and we can define the following constants for T ≥ 1 :

and we consider the following bump function Ψ T ∈ C ∞ ([0 , 1]) :

.

where V T is a scaling constant such that Ψ T ( 1 T α ) = T 1+ γ ρ ( T ) Consequently, we consider the sum of bump functions on [0 , 1] :

It is easy to verify that ( µ T 1 -σ T 1 , µ T 1 + σ T 1 ) ∩ ( µ T 2 -σ T 2 , µ T 2 + σ T 2 ) = ∅ for any T 1 = T 2 and

First, we study the property of φ ( · ) .

We denote the absolute constants M k = sup x | φ ( k ) ( x ) | . Notice that for any k ∈ N ,

.

Therefore, it holds that

Therefore, if k ≤ β -(1+ γ ) 1+ α , the following uniform bounds hold:

Consequently, we consider the smoothness of Φ at x = 0 .

Recalling the previous results, for any x ∈ (0 , 1] , we have

Thus, by induction, it is easy to verify that for any i < β -(1+ γ ) 1+ α ( i ∈ N ),

̸

Therefore, for any n < β -(1+ γ ) 1+ α ( n ∈ N + ), φ ( k ) (0) = 0 holds for any 0 ≤ k ≤ n . Moreover, the following uniform bound holds:

By Lemma G.5, for any m ∈ N + , there exists a polynomial Q m ( x ) = m -1 ∑ k =0 α k x k such that

Now we use the transform x = 1 t α ( t ≥ 1) on the function φ and consider

It is easy to verify that Φ satisfies that

Moreover, we consider the function

Then for any n < β -(1+ γ ) 1+ α ( n ∈ N ), the following error estimate holds:

By choosing α = 10 -2 and γ = 10 -4 β , we have 0 . 99 β -1 = β -γ 1+ α -1 = β -(1+ γ + α ) 1+ α < β -(1+ γ ) 1+ α . Thus, we obtain our result: for any n ∈ [ ⌊ 0 . 99 β ⌋ -1] ( β ≥ 2 / 0 . 99 ), the following error estimate holds:

G Some Background and Proof Preparation

G.1 T5's relative positional encoding

The T5's Relative Positional Encoding is primary focus of this study. Its standard form in practical applications (Raffel et al., 2020) adheres to R t,s = r ( t -s ) , where

Here, D is a large integer, signifying the longest distance of concern, while B is a small integer. One can see that for n < B , r ( · ) exhibits polynomial decay, whereas for B < n < D , r ( · ) demonstrates logarithmic decay. Consequently, the overall decay rate of r ( · ) is logarithmic .

The following Table further provides an example of standard T5's Relative Positional Encoding.

G.2 Barron space theory

The well-known universal approximation result for 2NNs asserts that 2NNs can approximate any continuous function (Barron, 1992; 1993; 1994). Nonetheless, this result lacks a characterization of the approximation efficiency, i.e., how many neurons are needed to achieve a certain approximation accuracy? This gap was addressed by the Barron space theory (E et al., 2019; 2021). It is established that for any function within Barron space f ∈ B , 2NNs with m neurons (denoted by H m ) can approximate them efficiently, at a rate of inf f m ∈H m ∥ f -f m ∥ ≤ O ( ∥ f ∥ B / √ m ) , remarkably independent of the input dimension d , thus avoiding the Curse of Dimensionality (Bellman, 1966; Bach, 2017). Specifically, the Barron space is defined by:

Definition G.1 (Barron space (E et al., 2019; 2021; Ma et al., 2020)) . Consider functions f : X → R that admit the following representation:

For any p ∈ [1 , + ∞ ] , we define the Barron norm:

Then the Barron space are defined as:

Proposition G.2. For any p ∈ [1 , + ∞ ] , B p = B ∞ and ∥ f ∥ B p = ∥ f ∥ B ∞ .

Remark G.3. From the Proposition above, the Barron spaces B p are equivalent for any p ∈ [1 , + ∞ ] . Consequently, in this paper, we use B and ∥·∥ B to denote the Barron space and Barron norm.

Remark G.4. For Barron space B , both Direct and Inverse Approximation Theorems hold (E et al., 2021). In this paper, we mainly utilize the Direct Approximation Theorem, stated in Lemma G.6.

G.3 Useful approximation lemmas

Lemma G.5 (Jackson (1930)) . Let f ∈ C n ([0 , 1]) with f (0) = f ′ (0) = · · · = f ( n ) (0) = 0 . Then for any m ∈ N + , there exists a polynomial Q m ( x ) = m -1 ∑ k =0 α k x k such that

where M ( n ) = max k ≤ n ∥ ∥ f ( k ) ∥ ∥ L ∞ ([0 , 1]) .

Lemma G.6 (Ma et al. (2020)) . For any f ∈ B and m ∈ N , there exists a two-layer ReLU neural network f m ( x ) = m ∑ k =1 a k σ ( b ⊤ k x + c k ) with m neurons such that

H Experiments

H.1 Restatement of our theoretical insights

As detailed in Section 1, our theoretical analysis reveals the following novel insights into the expressive power and mechanisms of Transformer:

Insight (1). The distinct roles of the number of layers, the number of Attn heads, and the width of FFN layers. (1a) Deeper Transformers can handle tasks with memories with more intricate interrelationships, such as nested relationships (Type II). (1b) In contrast, for tasks with memories lacking such interrelationships (Type I), a single-layer Transformer with sufficient Attn heads and FFN width should suffice.

Insight (2). The different roles of Attn layers and FFN layers. (2a) FFN layers are tasked with approximating nonlinear memory functions and the readout function, (2b) while Attn layers are responsible for extracting the tokens from the memory locations.

Insight (3). The functionality and necessity of Dot-product (DP). (3a) For the relatively simple Task I, DP is not necessary and can be omitted. (3b) However, for the more complex Task II, DP provides necessary nonlinearity: the cooperation between DP and RPE provides the needed interaction between the temporal space and the token space.

Insight (4). The efficiency of Relative Positional Encoding (RPE) in modeling long-range correlations. The primary role of RPE is to approximate the memory kernels. (4a) Transformer with log-type RPE can handle heavy-tailed memories. (4b) Transformer with lin-type RPE can handle light-tailed memories.

H.2 Experimental Validation

To validation of our theoretical insights (1a) ∼ (4b) , we conduct 8 experiments, from simple toy models to more complex LLM pre-training. The experiments are conducted on 1 A100.

H.2.1 Validation of Insight (1a)

Objective. As indicated in Section 4, numerous NLP tasks exhibit complex interrelationships among tokens and belong to our Task II. This experiment aims to verify our Insight (1a): for such tasks, increasing the number of layers L is more efficient than increasing the number of Attn heads H .

Setup. Specifically, we pretrain decoder-only Transformers (Vaswani et al., 2017) with different L and H on the OpenWebText dataset (Gokaslan and Cohen, 2019) for 10,000 iterations (approximately 1B tokens) on 1 A100, using cross-entropy loss and AdamW with the same hyperparameters. To ensure comparability, we meticulously balance the total number of parameters across both experimental setups.

Results and conclusion. The final validation losses are shown in Table 2. By comparing these two subtables, the benefits brought by increasing L are much greater than the benefits brought by increasing H ( 0 . 802 > 0 . 136 ), thereby corroborating our Insight (1a).

H.2.2 Validation of Insight (1b)

Objective. As mentioned in Section 3, sparse Boolean functions have no interactions among the memories and belong to our Task I. This experiment aims to verify our Insight (1b): for such tasks, a single-layer Transformer equipped with a sufficient number of Attn heads H and FFN width m suffices, and there is no need to increase the number of layers L .

Setup. Specifically, we train single-layer DP-free Transformers with different H and m to learn a sparse Boolean target function f ∗ : f ∗ ( x ) := g ∗ ( x 48 , x 56 , x 99 ) := ∑ 64 k =1 ReLU( ⟨ w ∗ k , ( x 48 , x 56 , x 99 ) ⟩ ) for input sequence x = ( x 1 , · · · , x 1000 ) ∈ {± 1 } 1000 , where w ∗ k are generated by w ∗ k ∼ N (0 , I 3 ) . Training proceeds for 10,000 iterations (1M samples) using squared loss and AdamW with the same hyperparameters.

Results and conclusion. The final validation losses are shown in Table 3. As shown in this table, a single-layer Transformer equipped with a sufficient H (32) and m (256) is adequate for representing this sparse Boolean function. This empirical evidence supports our Insight (1b).

H.2.3 Validation of Insight (2a)

Objective. This experiment aims to verify our Insight (2a): to learn a sparse Boolean function with a 'complex' readout function and 'simple' memories, increasing the FFN width m can significantly improve the performance, whereas increasing the number of Attn heads H brings almost no benefit.

Setup. Specifically, we train single-layer DP-free Transformers with different H and m to learn a sparse Boolean function with a "complex" readout function ( g ∗ ) and a "simple" single memory ( x 99 ): f ∗ ( x ) := g ∗ ( x 99 ) := ∑ 64 k =1 ReLU( w ∗ k · x 99 ) for any input sequence x = ( x 1 , · · · , x 1000 ) ∈ {± 1 } 1000 , where w ∗ k are generated by w ∗ k ∼ N (0 , 1) . Training proceeds for 10,000 iterations (1M samples) using squared loss and AdamW with the same hyperparameters.

Results and conclusion. The final validation losses are shown in Table 4. The tables indicate that, for learning a sparse Boolean function with a 'complex' readout function and 'simple' memories, increasing m can significantly improve the performance ( 0 . 49 → 0 . 002 ), almost completing this task perfectly. Conversely, increasing H fails to yield substantial improvement. This empirical evidence supports our Insight (2a).

H.2.4 Validation of Insight (2b)

Objective. Contrasting with Experiment (2a), this experiment aims to verify our Insight (2b): for learning a sparse Boolean function with a 'simple' readout function and 'complex' memories, increasing the number of Attn headers H can substantially improve the performance while increasing FFN width m will offer almost no benefit.

Setup. Specifically, we train single-layer DP-free Transformers with different H and m to learn a sparse Boolean function with a 'simple' linear readout function ( g ∗ ) and relatively 'complex' memories ( x 48 , x 56 , x 99 ): f ∗ ( x ) := g ∗ ( x 48 , x 56 , x 99 ) := x 48 + x 56 + x 99 for any input sequence x = ( x 1 , · · · , x 1000 ) ∈ {± 1 } 1000 . Training processes for 10,000 iterations (1M samples), using squared loss and AdamW with the same hyperparameters.

Results and conclusion. The final validation losses are presented in Table 5. The tables indicate that, for learning a sparse Boolean function with a 'simple' readout function and 'complex' memories, increasing m can significantly improve the performance ( 1 . 16 → 10 -6 ), closely achieving task perfection. In contrast, increasing m brings almost no benefits. This empirical evidence supports our Insight 2(b).

H.2.5 Validation of Insight (3a)

Objective. As mentioned in Section 3, learning sparse Boolean functions has no interactions among the memories and belongs to our Task I. This experiment aims to verify our insight (3a): for such

tasks, a DP-free Transformer equipped with a sufficient number of Attn heads H and FFN width m is sufficiently capable. Moreover, there is no need to use DP structure in Attn.

Setup. Specifically, we train single-layer DP-free Transformers with different H and m and with DP or without DP to learn a sparse Boolean target function f ∗ : f ∗ ( x ) := g ∗ ( x 48 , x 56 , x 99 ) := ∑ 64 k =1 ReLU( ⟨ w ∗ k , ( x 48 , x 56 , x 99 ) ⟩ ) for input sequence x = ( x 1 , · · · , x 1000 ) ∈ {± 1 } 1000 , where w ∗ k are generated by w ∗ k ∼ N (0 , I 3 ) . Training proceeds for 10,000 iterations (1M samples) using squared loss and AdamW with the same hyperparameters.

Results and conclusion. The final validation losses are shown in Table 6. The findings illustrate that a DP-free Transformer equipped with a sufficient H (32) and m (256) is adept at accurately representing the given sparse Boolean function. Additionally, the Incorporation of the DP structure into the layers contributes marginally to performance enhancement. This substantiates our Insight (3a).

H.2.6 Validation of Insight (3b)

Objective. As indicated in Section 4, numerous NLP tasks exhibit complex interrelationships among tokens and belong to our Task II. This experiment aims to verify our Insight (3b): for such tasks, the utilization of DP structure in Attn layers is necessary.

Setup. Specifically, we pre-train Transformers with DP or without DP on the OpenWebText dataset for 10,000 iterations (approximately 1B tokens) on 1 A100, using cross-entropy loss and AdamW with the same hyperparameters.

Results and conclusion. The final validation losses are presented in Table 7. As shown in the table, for NLP pre-training tasks, Transformer incorporating DP structure is more efficient than Transformer without DP ( 5 . 796 < 5 . 830 , 5 . 374 < 5 . 486 , 4 . 994 < 5 . 274 ), thereby supporting our Insight 3(b).

H.2.7 Validation of Insight (4a)

Objective. This experiment aims to verify our Insight (4a): for learning Task III with heavy-tailed memories, Transformers with log-type RPE are efficient, whereas those with lin-type RPE fail.

Setup. Specifically, we train single-layer, FFN-free, DP-free Transformers with log-type RPE or lintype RPE and varying numbers of Attn heads H . The target function involves a heavy-tailed memory kernel ρ ( t ) = t -0 . 5 : f ∗ ( x ) := ∑ 1000 s =1 x s ρ (1000 -s ) for any input sequence x = ( x 1 , · · · , x 1000 ) ∈ {± 1 } 1000 . Training processes for 10,000 iterations (1M samples) using squared loss and AdamW with the same hyperparameters.

Results and conclusion. The final validation losses are shown in Table 8. As shown in the table, to learn heavy-tailed memories, even single-head Transformer with log-type RRE can complete it

perfectly ( < 10 -5 ). Conversely, Transformers employing lin-type RRE exhibit limited improvement even with up to 64 heads ( 0 . 19 ). This empirical evidence supports our Insight (4a).

H.2.8 Validation of Insight (4b)

Objective. In contrast to Experiment (4a), this experiment aims to verify that for learning our Task III with light-tailed memories, Transformers with lin-type RPE are efficient, whereas those with log-type RPE fail.

Setup. Specifically, we train single-layer, FFN-free, DP-free Transformers with log-type RPE or lintype RPE and varying numbers of Attn heads H . The target function involves a heavy-tailed memory kernel ρ ( t ) = e -5 t : f ∗ ( x ) := ∑ 1000 s =1 x s ρ (1000 -s ) for any input sequence x = ( x 1 , · · · , x 1000 ) ∈ {± 1 } 1000 . Training processes for 10,000 iterations (1M samples) using squared loss and AdamW with the same hyperparameters.

Results and conclusion. The final validation losses are shown in Table 9. As shown in the table, to learn light-tailed memories, even single-head Transformer with lin-type RRE can complete it perfectly ( < 10 -7 ). Conversely, Transformers employing log-type RRE exhibit limited improvement even with up to 64 heads ( 5 . 3 × 10 -4 ). This empirical evidence supports our Insight (4b).

H.3 Practical Implications

Our theoretical insights and empirical evidence can directly lead to the following 8 practical implications, such as the strategic selection of Transformer hyperparameters for specific tasks.

Implication (1a). (supported by Insight (1a) and Experiment (1a))

For sequence modeling tasks with complex interrelations between memories, such as many NLP applications, enhancing the number of layers L is more beneficial than increasing the number of Attn heads H and FFN width m .

Implication (1b).

(supported by Insight (1b) and Experiment (1b))

For simple sequence modeling tasks with almost no memory intercorrelation, such as the learning of sparse Boolean function, improving performance necessitates only sufficient H and m in a single-layer Transformer, without a need to increase L .

Implication (2a). (supported by Insight (2a) and Experiment (2a))

For sequence modeling tasks with complex readout or memory functions, increasing m can significantly improve performance.

Implication (2b). (supported by Insight (2b) and Experiment (2b))

For sequence modeling tasks with multiple memories, increasing H can markedly improve performance.

Implication (3a). (supported by Insight (3a) and Experiment (3a))

For simple sequence modeling tasks with almost no memory correlations, such as learning sparse Boolean functions, omitting the DP structure in Attn layers can still perform well.

Implication (3b). (supported by Insight (3b) and Experiment (3b))

For sequence modeling tasks with complex correlations between memories, such as many NLP tasks, preserving the DP structure in attention layers is crucial for achieving high performance due to its indispensable nonlinearity.

Implication (4a). (supported by Insight (4a) and Experiment (4a))

For sequence modeling tasks with heavy-tailed memories, the employment of log-type RPE (such as T5's RPE and KERPLE (log)) is recommended over lin-type RPE (such as Alibi).

Implication (4b). (supported by Insight (4b) and Experiment (4b))

For sequence modeling tasks with light-tailed memories, the employment of lin-type RPE (such as Alibi) is recommended over log-type RPE (such as T5's RPE and KERPLE (log)).

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We believe that the abstract and introduction reflect the contributions and scope of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In Section 2, 3, 4, and 5; Appendix B, C, D, E, and F.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We believe that all of the experimental results are reproducible in our work.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The code or data of the experiments are simple and easy to reproduce following the description in the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: the approximation error is deterministic and there is no need to consider the error bars here.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Section H.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have confirmed that the research is conducted with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper conducts a systematic study on the expressive power and mechanisms of Transformers for sequence modeling. It explores various properties of Transformers, including the number of layers, attention heads, width, and the use of dot production. The theoretical results are supported by experimental evidence.\n\nSTRENGTHS:\nThis paper appears to be technically solid. It provides a detailed formulation of the problem and explores the expressive capabilities of Transformers across various tasks. Notably, it includes experiments in Appendix H to validate its theoretical insights. However, I have not verified the correctness of the proofs.\n\nThe paper contains some intriguing findings. I particularly want to highlight the exploration between dropout (DP) and relative position encoding (RPE), which I find especially interesting.\n\nWEAKNESSES:\nThe key weakness of this paper lies in its unclear presentation of motivation, innovation, and contributions. The detailed reasons are as follows:\n\n1) The motivation of this paper appears weak. The authors state their motivation as \"to gain a better understanding of how Transformers work in practice.\" However, there are already numerous studies exploring the expressive power of Transformers, covering most of the topics in this paper. It is unclear why the authors believe the current understanding of Transformers is insufficient and why the community needs this paper.\n\n2) The technical contributions of this work are not clearly presented. It is difficult to discern which conclusions are derived from known techniques and which are based on new techniques proposed by the authors. Additionally, the nature of these new techniques is not clearly explained.\n\n3) The paper's analysis is based on dividing tasks into three categories: modeling fixed, long but sparse memories; modeling adaptive, long but sparse memories; and modeling essentially sparse memories. However, the necessity of this categorization and the relationships between these tasks are unclear. The authors claim these categories are prevalent, but the supporting evidence is weak.\n\nQUESTIONS:\n1) This paper includes numerous proofs, making it challenging to verify their correctness. Could the authors provide a proof sketch for Theorem 4.4?\n\n2) The tightness of the bound is also uncertain. Could the authors provide a numerical result demonstrating the bound in a toy case?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the approximation capabilities of transformers with relative positional encodings and derives approximation rates for three types of sequence modeling tasks. Each task corresponds to a particular choice of target function class. Namely, a first class of mappings are fixed, long but sparse memories (equation (5)), and the authors focus on dot-product free transformers. A second class are adaptive, long but sparse memories (equation (8)), and the authors consider deep transformers with specific precision and no SoftMax. The last class of mappings are essentially sparse memories and consist of convolving the input sequence with different kernels where, again, the considered Transformers are dot-product free.\nEach approximation rate is commented on the role of the feedforward and self-attention components, the role of dot-product attention (that is, the fact that the attention maps depend on the input sequence through pairwise interactions), and the effect of relative position encoding (whether it is of linear or logarithmic type).\n\nSTRENGTHS:\n- The paper is well written and easy to follow. \n\n- The mathematical results are all supported with detailed and rigorous proofs.\n\n- Each Theorem is structured in the same way which eases the understanding of the whole paper. \n\n- The paper adresses the important question of approximation rates of Transformers in sequence modeling.\n\n- The paper establishes rigorous approximation results for Transformers on sequences modeling tasks. These results are significant and give insights on the role of the different Transformer components on their approximation capabilities. \n\n- The three problem formulations are clearly defined and commented.\n\nWEAKNESSES:\n- Unless I missed something, the \"true\" attention mechanism (line 104) is never used in the paper to derive approximation rates. In sections 3 and 5 the attention is DPF, while in section 4 there is no softmax and quantization is used. I don't think this is a major problem, but it should be emphasised (perhaps by extending remark 2.2) that you never consider the transformer with all the bells and whistles. \n\n- I understand that the dimension of the embeddings grows with $Md$. This is not mentioned in the main text, nor commented on in the supplement. How does this compare with previous work? \n\n- Also, you mention that the number of heads needed grows at least polynomially with the $T_i$'s. I understand that this is due to the particular choice of RPE. Would other types of PE give a better dependency in the $T_i$'s ?  How does the current dependency compare with previous works? \n\n- Paragraph l. 274 to l. 277 is unclear. When do you use this alternative in the rest of the paper? \n\n- Typo in l. 281\n\n- l. 282, I don't understand the notation for the lower bound on $m$.\n\nQUESTIONS:\n- Please see some questions in the Weaknesses part. \n\n- What do you mean by rate $n$ in all the theorems ? Is there an optimal choice for $n$ ? \n\n- What is the main reason why you did not consider the SoftMax in section 4 ? \n\n- How does $C(n)$ behaves with $n$ ? \n\n- Could you approximate the model with specific precision with another Transformer which does not rely on specific precision ?  \n\n- Overall, the spirit of the approximation in the paper is always to form augmented tokens $(x_{t}, x_{t-t_1}, \\dots x_{t-t_M})$ using attention and to apply the MLP on top of it. Do you think this is the \"optimal\" way ? And if so, what are the reasons behind it ?\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a thorough analysis of the expressive power of Transformers. It does so by investigating the relative importance of different architectural components, such as the mixing layer (dot product attention), the feed-forwad block and positional embeddings. More specifically, the paper studies three different tasks, characterised by different levels of complexity, sharing some key properties with relevant realistic applications such as n-gram models, sparse Boolean functions (Task I), Adaptive sparse Boolean functions, parsing, part-of-speech tagging (Task II), feature extraction in image and signal processing (Task III). On each of such tasks, the paper studies the approximation rates of transformer models defined in terms of their number of layers $L$ and heads $H$, the width of the feed-forward module $m$ and the type of relative positional embedding utilised (i.e. either Alibi or T5 style). Through its theoretical analysis the paper shows that the necessary model's complexity needs to adapt to the complexity of the task. Moreover by investigating the role of different components, the paper shows that dot-product attention is necessary for more complex tasks (Task II) while its dot-product free variant provably fails. Nevertheless, the authors show that the task can still be solved with a more parameter efficient alternative. In addition, the role of the feed-forward blocks is mainly to approximate readout functions and memory functions, aligning with classical results on approximating Barron functions. The theoretical results are corroborated with empirical results.\n\nSTRENGTHS:\n* The paper meaningfully contributes to the very relevant research line on theoretical analysis of transformers models. \n* The paper originally addresses important questions, such as the relative role of different components in the transformer architecture and the necessity of the dot-product attention, through a principled approach and in a controlled framework.\n* Generally the paper is very well written (modulo some sparse and minor typos) and pleasant to read.\n\nWEAKNESSES:\n* Some concepts could be better introduced: for example, it was not clear to the reviewer what the authors meant by \"memories\" in the introduction of the paper. While this gets clearer later, it would be helpful to clarify it earlier.\n* Layer normalization, an important component of modern transformers, does not seem to be included in the analysis. What effect do you expect it to have on your results?\n* The role of the embedding dimension $D$ is not considered in the reported rates. What is the effect of varying $D$ on the approximation rates of the model?\n\nQUESTIONS:\nSee weaknesses section.\n\nLIMITATIONS:\nThe limitations of the paper are discussed in section 7", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}