Title: Representational Strengths and Limitations of Transformers
Venue: NeurIPS 2023
Accepted: yes
Keywords: self-attention

approximation theory

communication complexity
Abstract: Attention layers, as commonly used in transformers, form the backbone of modern deep learning, yet there is no mathematical description of their benefits and deficiencies as compared with other architectures. In this work we establish both positive and negative results on the representation power of attention layers, with a focus on intrinsic complexity parameters such as width, depth, and embedding dimension. On the positive side, we present a sparse averaging task, where recurrent networks and feedforward networks all have complexity scaling polynomially in the input size, whereas transformers scale merely logarithmically in the input size; furthermore, we use the same construction to show the necessity and role of a large embedding dimension in a transformer. On the negative side, we present a triple detection task, where attention layers in turn have complexity scaling linearly in the input size; as this scenario seems rare in practice, we also present natural variants that can be efficiently solved by attention layers. The proof techniques emphasize the value of communication complexity in the analysis of transformers and related models, and the role of sparse averaging as a prototypical attention task, which even finds use in the analysis of triple detection.

Representational Strengths and Limitations of Transformers

Abstract

Attention layers, as commonly used in transformers, form the backbone of modern deep learning, yet there is no mathematical description of their benefits and deficiencies as compared with other architectures. In this work we establish both positive and negative results on the representation power of attention layers, with a focus on intrinsic complexity parameters such as width, depth, and embedding dimension. On the positive side, we present a sparse averaging task , where recurrent networks and feedforward networks all have complexity scaling polynomially in the input size, whereas transformers scale merely logarithmically in the input size; furthermore, we use the same construction to show the necessity and role of a large embedding dimension in a transformer. On the negative side, we present a triple detection task, where attention layers in turn have complexity scaling linearly in the input size; as this scenario seems rare in practice, we also present natural variants that can be efficiently solved by attention layers. The proof techniques emphasize the value of communication complexity in the analysis of transformers and related models, and the role of sparse averaging as a prototypical attention task, which even finds use in the analysis of triple detection.

1 Introduction

In recent years, transformer networks [Vaswani et al., 2017] have been established as a fundamental neural architecture powering state-of-the-art results in many applications, including language modeling [OpenAI, 2023], computer vision [Dosovitskiy et al., 2021], and protein folding [Jumper et al., 2021]. The key building block of transformer models is the self-attention unit , a primitive that represents interactions among input elements as inner-products between low-dimensional embeddings of these elements.

The success of transformer models is linked to their ability to scale their training and generalization performance to larger datasets and sequence lengths. Their representational capacity, however, underlies this scaling power, and is tied to the inductive biases of their learning algorithms. Empirically, transformer models trained with gradient-based learning algorithms exhibit biases towards certain algorithmic primitives [Edelman et al., 2022, Liu et al., 2022] and learn representations that may encode domain-specific information in the self-attention units [Clark et al., 2019, Hewitt and Manning, 2019, Rogers et al., 2020, Chen et al., 2022]. These examples indicate that transformer architectures not only provide computational benefits, but also have representational capabilities that are particularly well-matched to practical tasks.

In this paper, we investigate these inductive biases by identifying 'natural' computational tasks for which transformers are well-suited, especially compared to other neural network architectures, as well as tasks that highlight the limitations of transformers. The tasks-sparse averaging, pair-matching,

Matus Telgarsky

and triples-matching-represent primitive operations that aggregate structural information encoded in embeddings. We use these tasks to elucidate the relationship between the embedding dimension m of a self-attention unit and its expressivity, and to showcase the fundamental representational limitations of self-attention layers.

In our model, the primary computational bottleneck faced by a transformer in computing a 'sequenceto-sequence' 1 function f : X N → Y N is the constrained processing of pairs of input elements { x i , x j } ∈ ( X 2 ) ; we allow transformers unbounded computational power when processing the individual elements x i ∈ X . This is motivated by modern scaling regimes where the context length N has rapidly increased, the self-attention embedding dimension m remains much smaller than N , and the parameterization of multi-layer perceptrons (MLPs) that operate on individual elements is much larger than m . Indeed, the largest GPT-3 model [Brown et al., 2020] features a context length N = 2048 , an embedding dimension m = 128 , and MLPs with a 12288-dimensional parameterization; the context length of GPT-4 is as large as N = 32000 . As such, we are interested in the capabilities of transformers with N o (1) total 'size', as opposed to N Ω(1) . The nature of the bottleneck in our model makes the tools of communication complexity indispensable for formalizing computational limits.

1.1 Our contributions

Sparse averaging separations among atomic self-attention units. The q -sparse averaging task q SA aims to capture the essential approximation-theoretic properties of self-attention units. In q SA , the i th input x i is a pair ( y i , z i ) , where z i ∈ R d ′ is the data part of x i , simply a vector in R d ′ , whereas and y i ∈ ( [ N ] q ) is the indexing part, which specifies q locations in the input sequence; the i th output element in q SA is obtained by averaging the q data parts z j given by j ∈ y i , meaning

(See also Definition 4.) As summarized in the following informal theorem, our analysis of q SA in Section 3 and Appendix A illustrates the ability of the self-attention primitive to associate arbitrary subsets of input elements (as opposed to just 'local' subsets, as specified by some sequential/topological structure), measures the expressive power accrued by increasing the embedding dimension m of a self-attention unit, and indicates the representational limitations of 'traditional' neural architectures on basic computational tasks.

Informal Theorem 1. The task q SA for q ∈ Z + satisfies the following properties (see Definition 4 for a formal definition and approximation metric).

There exists a unit of self-attention f with an m -dimensional embedding that approximates q SA if and only if m ≳ q (Theorems 2 and 4).

Any fully-connected neural network whose output approximates q SA requires its first hidden layer to have width at least Ω( Nd ) (Theorem 10).

Any recurrent neural network whose iterates approximate q SA requires a hidden state of at least Ω( N ) bits (Theorem 11).

We consider the q SA implementation in Item 1 efficient since the dimension of the model parameters grows with poly( q, d, log N ) , whereas the latter two are inefficient since their parameter (or state) dimension grows as poly( N ) . The proofs of the positive results employ embeddings for each index j and each subset y i that have large inner products if and only if j ∈ y i . The negative results involve communication complexity reductions and geometric arguments. These arguments naturally introduce a dependence on bits of precision, which we suppress above within the notation ' ≳ '; we note that these bounded-precision results are arguably more relevant to modern networks, which uses as few as 4 or even 2 bits of numerical precision.

Contrast between pairwise and triple-wise matching with self-attention layers. We frame standard transformer architectures as being able to efficiently represent functions that are decomposable

into sparse pairwise interactions between inputs. To do so, we introduce two sequential tasks and prove a collection of constructions and hardness results that characterize the abilities of transformers to solve these tasks.

Given an input sequence X = ( x 1 , . . . , x N ) ∈ [ M ] N (for some M = poly( N ) ), we formalize the problems of similar pair detection ( Match2 ) and similar triple detection ( Match3 ) as

✶ For both tasks, note that the output is an N -dimensional vector whose i th element is 1 if and only if the sequence X includes a pair or triple containing x i . In this sense, the problems differ from 2SUM and 3SUM, which are not sequence-to-sequence tasks.

We believe these two tasks are intrinsically 'pairwise' and 'triple-wise', respectively; moreover, since we also believe self-attention performs a fundamentally 'pairwise' operation, we will use Match2 and Match3 to show a sharp gap in the representation power of self-attention.

Informal Theorem 2.

A single unit of standard self-attention with input and output MLPs and an O ( d ) -dimensional embedding can compute Match2 (Theorem 6).

A single layer of standard multi-headed self-attention cannot compute Match3 unless its number of heads H or embedding dimension m grows polynomially in N (Theorem 7).

A standard transformer model can efficiently compute a modified version of Match3 that makes assumptions about embedding structure or locality (Theorems 8 and 9).

Under a generalized notion of 'third-order tensor self-attention' introduced in Appendix C.3, Match3 is efficiently computable with a single unit of third-order attention (Theorem 18).

While the above result demonstrates the limitations of multi-headed self-attention and illustrates the importance of learning embeddings with contextual clues, we believe that a stronger result exists. Specifically, we conjecture that even multi-layer transformers are unable to efficiently compute Match3 without hints or augmentation.

Informal Conjecture 1. Every multi-layer transformer that computes Match3 must have width, depth, embedding dimension, or bit complexity at least N Ω(1) .

In Appendices C.5 and C.6, we give a heuristic information-theoretic argument to support this conjecture, prove a matching upper-bound, and finally prove analogous results for graph-augmented transformers with respect to the problem of cycle detection in directed and undirected graphs.

1.2 Related work

Several computational and learning-theoretic aspects of transformers, distinct from but related to the specific aims of the present paper, have been mathematically studied in previous works.

Universality and Turing-completeness. To demonstrate the power of transformers, universal approximation results for transformers [Yun et al., 2020, Wei et al., 2022]-analogous to results for feedforward networks [Hornik et al., 1989]-establish the capability for sufficiently large networks to accurately approximate general classes of functions. Note, however, that the precise minimal dependence of the required size (e.g., number of attention units, depth of the network) as a function of the input size N does not directly follow from such results, and it is complicated by the interleaving of other neural network elements between attention layers. (Approximate) Turing-completeness of transformers demonstrates their power in a different manner, and such results have been established, first assuming infinite precision weights [Pérez et al., 2019] and later also with finite-precision [Wei et al., 2022]. Such results are more closely aligned with our aims, because Turing machines represent a uniform model of computation on inputs of arbitrary size. Wei et al. [2022] showed that Turing machines that run for T steps can be approximated by 'encoder-decoder' transformers of depth log( T ) and size polynomial in log( T ) and the number of states of the Turing machine (but the decoder runs for T steps).

Formal language recognition. The ubiquity of transformers in natural language understanding has motivated the theoretical study of their ability to recognize formal languages. On the positive side, Bhattamishra et al. [2020] constructed transformers that recognize counter languages, and Yao et al. [2021] showed that transformers of bounded size and depth can recognize Dyck languages that have bounded stack depth. Liu et al. [2022] showed that the computations of finite-state automata on sequences of length N can be performed by transformers of depth log( N ) and size polynomial in the number of states. On the negative side, Hahn [2020] showed limitations of modeling distributions over formal languages (including Dyck) with fixed-size transformers (though this result does not imply quantitative lower bounds on the size of the transformer). Hahn [2020], as well as Hao et al. [2022], also establish the inability of 'hard attention' Transformers to recognize various formal languages and circuit classes by leveraging depth reduction techniques from circuit complexity [Furst et al., 1984].

Learnability. The sample complexity of learning with low-weight transformers can be obtained using techniques from statistical learning theory and, in turn, establish learnability of certain boolean concept classes (e.g., sparse parity) [Edelman et al., 2022, Bhattamishra et al., 2022] using transformerbased hypothesis classes. Our q SA function is inspired by these classes, and we establish concrete size lower bounds for approximation (and hence also learnability) by transformers. We note that our constructions use bounded-size weights, and hence, in principle, the aforementioned sample complexity results can be combined with our results to analyze empirical risk minimization for learning transformers. Prior work of Likhosherstov et al. [2021] also shows how sparse attention patterns can be achieved by self-attention units (via random projection arguments); however, when specialized to q SA , their construction is suboptimal in terms of the sparsity level q .

Related models. Graph neural networks (GNNs), like transformers, process very large inputs (graphs) using neural networks that act only on small collections of the input parts (vertex neighborhoods). Many classes of GNNs are universal approximators for classes of invariant and equivariant functions [Maron et al., 2019, Keriven and Peyré, 2019]. At the same time, they are restricted by the distinguishing power of certain graph isomorphism tests [Xu et al., 2018, Morris et al., 2019, Chen et al., 2019], and lower bounds have been established on the network size to approximate such tests [Aamand et al., 2022]. Loukas [2019] established a connection between GNNs and the LOCAL [Angluin, 1980] and CONGEST [Peleg, 2000] models for distributed computation, and hence directly translates lower bounds for CONGEST-notably cycle detection problems-into size lower bounds for GNNs. Our lower bounds for cycle detection using transformers also leverage a connection to the CONGEST model. However, transformers do not have the same limitations as GNNs, since the computational substrate of a transformer does not depend on the input graph in the way it is with GNNs. Thus, we cannot directly import lower bounds for CONGEST to obtain lower bounds for transformers.

Transformers are also related to other families of invariant and equivariant networks. Our focus on Match2 and Match3 (and related problems) was inspired by the separation results of Zweig and Bruna [2022] between models for processing sets: Deep Sets [Qi et al., 2017, Zaheer et al., 2017], which are 'singleton symmetric', and the more expressive Relational Pooling networks [Santoro et al., 2017], which are only 'pairwise symmetric'.

1.3 Conclusion and future work

Our primary contributions are to present a multi-faceted story about transformer approximation: firstly, q SA separates transformer models approximation-theoretically from RNNs and MLPs, and moreover the attention embedding dimension both necessary and sufficient for q SA scale directly with q , meaning q SA also functions to characterize representation power amongst different transformers. Secondly, while single units of self-attention can solve the Match2 task, even wide layers of selfattention with high-dimensional embeddings cannot solve Match3 , and we believe that deeper models cannot as well. This question of deeper models is stated as a formal conjecture and addressed heuristically in Appendix C.6, using both information- and communication-theoretic proof techniques, both of which we feel are significant steps towards a complete proof.

While our investigation is purely approximation-theoretic, we also include in Appendix D a preliminary empirical study, showing that attention can learn q SA with vastly fewer samples than recurrent

networks and MLPs; we feel this further emphasizes the fundamental value of q SA , and constitutes an exciting direction for future work.

Beyond the explicit open question in Informal Conjecture 1, we anticipate that future research could connect the separation results proved in this work to formal linguistic theory and empirical work on attention matrix interpretation. This work examines Match2 and Match3 because we believe that the former could represent a key primitive for language processing tasks such as co-referencing, while the latter represents a natural extension of the former that likely is not necessary for language modeling. Rather, it may be possible that language modeling performs triple-wise modeling for tasks such as the identification of subject, verb, and object components by relying on pairwise matching constructions and 'clues' learned within an embedding, such as those encoded in the toy problems Match3Bigram and Match3Local . That is, transformers serve as a useful foundational model for language modeling because of their abilities to integrate contextual clues and pairwise communication, and while they are not extensible to 'purely triple-wise problems,' most practical sequential problems have some efficient decomposition to pairwise structures that can be easily exploited by these architectures. Future work by linguists, theoretical computer scientists, and empirical NLP practitioners could assess how foundational our primitives are and study whether there are any practical triple-wise problems that transformer models fail to solve.

2 Preliminaries

Let B d = { x ∈ R d : ∥ x ∥ 2 ≤ 1 } denote the unit ball in R d , and let [ n ] = { 1 , 2 , . . . , n } denote the first n positive integers. The expression ✶ { P } equals 1 if predicate P is true and 0 otherwise. The row-wise softmax operator applied to matrix A ∈ R N × M returns

2.1 Attention units and transformer architectures

We first introduce the concept of self-attention, which is used as the building block of all transformer architectures included in this paper.

Definition 1. For input dimension d , output dimension d ′ , embedding dimension m , precision p , and matrices Q,K ∈ R d × m and V ∈ R d × d ′ (encoded using p -bit fixed-point numbers), a self-attention unit is a function f Q,K,V : R N × d → R N × d ′ with

Let A d,m,d ′ ,p = { f Q,K,V : Q,K,V } denote all such self-attention units.

Self-attention units can be computed in parallel to create multi-headed attention.

Definition 2. For head-count H and self-attention units f 1 , . . . , f H ∈ A d,m,d ′ ,p , a multi-headed attention layer is a function L f 1 ,...,f H : R N × d → R N × m with L f 1 ,...,f H ( X ) = ∑ H h =1 f h ( X ) . Let A H d,m,d ′ ,p contain all such L f 1 ,...,f H .

Transformer models are composed of two components: multi-headed attention layers (as above) and element-wise multi-layer perceptrons. Due to universal approximation results, we model multi-layer perceptrons as arbitrary functions mapping fixed-precision vectors to themselves.

Definition 3. A multi-layer perceptron (MLP) layer is represented by some ϕ : R d → R d ′ , whose real-valued inputs and outputs can be represented using p -bit fixed-precision numbers. We apply ϕ to each element (i.e., row) of an input X ∈ R N × d , abusing notation to let ϕ ( X ) = ( ϕ ( x 1 ) , . . . , ϕ ( x N )) ∈ R N × d ′ . Let Φ d,d ′ ,p denote all such MLPs.

We concatenate the notation of each class of functions to denote function composition. For example, for output dimension d ′ , we use A ′ d,m,d ′ ,p := A m,m,d ′ ,p Φ d,m,p and A H ′ d,m,d ′ ,p := A H m,m,d ′ ,p Φ d,m,p to represent single-headed and multi-headed attention units with an input MLP respectively. (The capabilities and limitations of these models are studied in Section 3.) For depth D , we let

represent a full transformer model comprising D layers of H -headed self-attention with interspersed MLPs.

While two key features of transformer architectures-the residual connection and the positional embedding-are conspicuously missing from this formalism, the two can be implemented easily under the framework. We can include a positional embedding by encoding the index as a coordinate of the input, i.e. x i, 1 = i . Then, the subsequent MLP transformation ϕ ( X ) can incorporate i suitably into the embedding. A residual connection can be included additively as input to a multi-layer perceptron layer (as is standard) by implementing an 'approximate identity' attention head f with Q,K and V = I m set to ensure that f ( X ) ≈ X . 2

We periodically consider transformers implemented with real-valued arithmetic with infinite bit complexity; in those cases, we omit the bit complexity p from the notation.

Finally, we assume for the proof of Theorem 3 that the model is permitted to append a single <END> token at the end of a sequence. That is, we say that a model f ∈ T D,H d,m,d ′ ,p represents a target h : R N × d → R N × d ′ if f ( X ′ ) 1: N = g ( X ) when X ′ = ( x 1 , . . . , x N , x ′ ) for constant-valued x ′ ∈ R d .

3 Sparse averaging with attention units

We present the sparse averaging task to highlight the ability of transformer architectures to simulate a wide range of meaningful interactions between input elements. This task demonstrates how the embedding dimension of a self-attention unit modulates the expressive capabilities of the architecture, while showcasing the inabilities of fully-connected and recurrent neural networks to capture similar interactions (see Appendix A).

Definition 4. For sparsity q , problem dimension d ′ , and input dimension d = d ′ + q +1 , consider an input X = ( x 1 , . . . , x N ) ∈ R N × d with x i = ( z i ; y i ; i ) for z i ∈ B d ′ and y i ∈ ( [ N ] q ) . 3 Let the q -sparse average be

.

For accuracy ϵ > 0 , a function f : R N × d → R N × d ′ ϵ -approximates q SA if for all X ,

Figure 1a visualizes the sparse averaging task as a bipartite graph between subsets y i and elements z i with corresponding averages. Theorems 2 and 4 jointly show that the minimum embedding dimension m of single self-attention units A ′ d,m,d ′ ,p that O ( 1 q ) -approximate q SA scales linearly with q . We believe that the sparse averaging problem is thus a canonical problem establishing the representational capabilities and inductive biases of self-attention units.

3.1 Self-attention can approximate q SA when m ≳ q

Our principle positive result shows that the sparse averaging task q SA can be approximately solved using fixed-precision arithmetic self-attention units with embedding dimension m growing with q log N .

Theorem 2 (Fixed-precision) . For any N , any m ≥ Ω( d ′ + q log N ) , any ϵ ∈ (0 , 1) , and p = Ω(log( q ϵ log N )) , there exists some f ∈ A ′ d,m,d ′ ,p that ϵ -approximates q SA .

While the full proof appears in Appendix B.1, we briefly sketch the argument here. Because the output of a self-attention unit is a convex combination of rows of the value matrix ϕ ( X ) V ∈ R N × d ′ , a natural way to approximate q SA with a unit of self-attention is to let each value be the corresponding

(c) Key and query embeddings that produce the self-attention matrix in (b).

(a)

T

= 0

.

(b)

T

= 1000

.

0.0

0.2

0.4

0.6

0.8

1.0

(c)

T

= 40000

.

1.0

0.8

0.6

0.4

0.2

0.0

6

vector in the average (i.e. V T ϕ ( x i ) = z i ) and choose the key and query functions in order to ensure that the attention matrix satisfies

To do so, let each key K T ϕ ( x i ) represent a fixed vertex on a convex polytope, which depends only on index i and is constructed from random binary vectors. We select each query Q T ϕ ( x i ) to ensure that ϕ ( x i ) T QK T ϕ ( x j ) is a fixed large value if j ∈ y i and a slightly smaller value otherwise. We obtain the precise query, key, and value embeddings by employing tools from dual certificate analysis from the theory of compressed sensing.

Wevisualize this construction in Figure 1b and 1c for q = 3 and d ′ = 4 , which presents the associated attention and value matrices necessary for the construction, and plots a polytope of keys (red dots) with each face corresponding to each subset y i (green dots). The construction is empirically relevant; Figure 2 shows that a unit of self-attention trained on data generated by the q SA task recovers a similar attention matrix to the one stipulated in our construction and visualized in Figure 1b.

The logarithmic dependence of the embedding dimension m on the sequence length N can be eliminated by considering self-attention units with real-valued arithmetic with infinite bit complexity.

Theorem 3 (Infinite-precision) . For fixed N , m ≥ Ω( d ′ + q ) and ϵ > 0 , there exists some f ∈ A ′ d,m,d ′ that ϵ -approximates q SA .

bits

The proof of Theorem 3 employs a similar polytope-based construction in Appendix B.2, relying on a cyclic polytope rather than one drawn from discrete boolean vectors. Theorem 16 proves the near-optimality of that bound by employing a geometric argument to show that a variant of q SA can only be approximated by a restricted family of self-attention units with a sufficiently high-dimensional embedding.

3.2 Self-attention cannot approximate q SA when m ≲ q

We show that the construction used to prove Theorem 2 is nearly optimal.

Theorem 4. For any sufficiently large q , any N ≥ 2 q +1 , and any d ′ ≥ 1 , there exists a universal constant c such that if mp ≤ cq , then no f ∈ T 1 , 1 d,m,d ′ ,p exists that 1 2 q -approximates q SA .

(By choosing p = O (log( q log N )) , Theorem 2 is shown to be optimal up to logarithmic factors of q and doubly-logarithmic factors of N .)

The proof of Theorem 4 employs a standard communication complexity argument based on a reduction from the following set disjointness problem in the two-party communication model, in which each party possesses a subset of an n element domain (encoded as n -bit strings), and they wish to jointly determine whether their subsets are disjoint. We note that communication complexity is commonly-used technique for proving lower bounds on the representational power of circuits and feedforward neural networks [see, e.g., Karchmer and Wigderson, 1988, Ben-David et al., 2002, Martens et al., 2013, Vardi et al., 2021].

Fact 5 (Set disjointness communication lower bound [Yao, 1979]) . Suppose Alice and Bob are given inputs a, b ∈ { 0 , 1 } n , respectively, with the goal of jointly computing DISJ( a, b ) = max i a i b i by alternately sending a single bit message to the other party over a sequence of communication rounds. Any deterministic protocol for computing DISJ( a, b ) requires at least n rounds of communication.

Our proof designs a communication protocol that Alice and Bob use to jointly compute DISJ( a, b ) when n = q in O ( mp ) rounds of communication, under the assumption that such an f exists that closely approximates q SA .

Alice encodes her input a in a single subset by letting y 2 q +1 = { 2 i + a i -1 : i ∈ [ q ] } .

Bob uses his input b to assign z 2 i -1 to 2 b i -1 and z 2 i = -1 for all i ∈ [ q ] .

All other input components are set to constant values known by both parties.

Alice sends her mp -bit query embedding Q T ϕ ( x 2 q +1 ) bit-by-bit to Bob, who approximately computes q SA by determining the outcome of f . The crux of the reduction shows that q SA( X ) 2 q +1 = -1 if and only if a i b i = 0 for all i ∈ [ q ] , which allows Bob to determine DISJ( a, b ) .

We visualize the protocol in Figure 3 and give the proof in Appendix B.3. The proofs of Theorems 7, 11, 21, and 23 employ similar communication complexity reductions to DISJ .

4 Standard transformer models can only efficiently represent intrinsically pairwise functions

In this section, we argue that the standard transformer architecture is unable to efficiently represent functions that do not decompose into a small number of pairwise-symmetric functions. We do this by contrasting the (in)approximability of intrinsically pairwise and triple-wise functions, respectively Match2 and Match3 (defined in (1) and (2)), and their variants.

4.1 Efficient computation of Match2 with standard self-attention

We first show that Match2 can be efficiently approximated by a single standard (pairwise) selfattention unit.

Theorem 6. For any input size N , input range M = N O (1) , and fixed-precision bit complexity p = O (log M ) , there exists a transformer architecture f ∈ T 1 , 1 1 ,m, 1 ,p with a single self-attention unit with embedding dimension m = 3 such that for all X ∈ [ M ] N , f ( X ) = Match2( X ) .

The proof, given in Appendix C.1 uses both a 'blank token' and a trigonometric positional embedding, which ensures that

for some sufficiently large constant c . This embedding ensures that a cell of the attention matrix softmax( ϕ ( X ) QK T ϕ ( X ) T ) i,j is extremely close to zero, unless x i = -x j (mod M ) .

4.2 Hardness of computing Match3 with a multi-headed self-attention layer

Although Match2 can be efficiently represented using a single unit of standard self-attention, representing Match3 using an entire layer of multi-headed attention units is impossible unless either the number of heads H , the embedding dimension m , or the precision p grows as N Ω(1) .

Theorem 7. There is universal constant c > 0 such that for sufficiently large N , and any M ≥ N +1 , if mpH ≤ cN/ log log N , then there is no f ∈ T 1 ,H 1 ,m, 1 ,p satisfying f ( X ) = Match3( X ) for all X ∈ [ M ] N .

We give the proof in Appendix C.2. Like that of Theorem 4, the proof relies on a reduction from set disjointness in two-party communication. The proof of the lower bound applies a domain-restricted variant of Match3 , which actually makes the problem substantially simpler to solve. In Remark 1, we show how this variant of Match3 introduces a depth separation between the representational powers of single-layer and two-layer transformer models.

As mentioned in the introduction, we also conjecture that multiple layers of multi-headed attention are subject to the same impossibility (Conjecture 19). The impossibility is specific to standard (pairwise) attention; in Appendix C.4, we show that Match3 can be efficiently computed with a single unit of third-order self-attention.

4.3 More efficient constructions for simplified Match3 computations

While the previous sections suggests that no efficient construction exists to compute Match3 with standard transformer models, practical examples of triple detection abound. For example, a transformerbased language model will likely succeed in linking a subject/verb/object triple because all three tokens likely inhabit the same local region and because the model could agglomerate the triple by first identifying a pair and then adding the third. Here, we introduce two variants on the Match3 problem that have additional structure to serve as hints. The first variant specifies triple sums comprising the input element and a neighboring pair elsewhere in the sequence: for each i ∈ [ N ] ,

Match3Bigram( X ) i = ✶ {∃ j s . t . x i + x j + x j +1 = 0 (mod M ) } . The second focuses on localized sums, where are all components of a triple must be within a fixed range of constant width K ≪ N : for each i ∈ [ N ] ,

Match3Local( X ) i = ✶ {∃ j 1 , j 2 s . t . x i + x j 1 + x j 2 = 0 (mod M ) , | i -j 1 | , | i -j 2 | ≤ K } . We show that the two can be efficiently represented using compact standard transformer models.

Theorem 8. For any N , M = N O (1) , and p = O (log M ) , there exists a transformer architecture f ∈ T D, 1 1 ,m, 1 ,p with embedding dimension m = 3 and depth D = 2 such that for all X ∈ [ M ] N × d , f ( X ) = Match3Bigram( X ) .

Informally, the first layer of the construction uses a sinusoidal positional encoding to compute each bigram sum x j + x j +1 in the j th element of the sequence. The second layer applies the Match2 construction provided by Theorem 6 to determine whether there exists a j for each i such that x i + x j + x j +1 = 0 (mod M ) .

Theorem 9. For any d , N , M = N O (1) , p = O (log M ) , and K ≤ N , there exists a transformer architecture f ∈ T 1 , 1 1 ,m, 1 ,p with embedding dimension m = O ( K log N ) and bit-complexity p = O (log( K log N )) such that for all X ∈ [ M ] N × d , f ( X ) = Match3Local( X ) .

Proof. We implement the localized construction by using Theorem 2 to construct a specific sparse simultaneous average of the inputs with q := 2 K +1 and d ′ := 2 K +1 . To do so, we use the input MLP to convert x i to the embedding ( z i ; y i ; i ) , for zero-padded input

for ¯ i = i (mod 2 K +1) and subset

This construction ensures that the i th element of self-attention output computes (a rotation of) ( x i -K , x i -K +1 , . . . , x i + K ) . An output MLP can then verify whether any matching triples involving x i exist among those vectors.

Acknowledgments and Disclosure of Funding

We are grateful for many discussions with and feedback from Navid Ardeshir, Peter Bartlett, Alberto Bietti, Yuval Efron, Christos Papadimitriou, Shivam Nadimpalli, Rocco Servedio, Yusu Wang, and Cyril Zhang. This work was supported in part by NSF grants CCF-1740833 and IIS-1563785, a JP Morgan Faculty Award, and an NSF Graduate Research Fellowship.

References

Anders Aamand, Justin Chen, Piotr Indyk, Shyam Narayanan, Ronitt Rubinfeld, Nicholas Schiefer, Sandeep Silwal, and Tal Wagner. Exponentially improving the complexity of simulating the Weisfeiler-Lehman test with graph neural networks. In Advances in Neural Information Processing Systems 35 , 2022.

Dana Angluin. Local and global properties in networks of processors. In Proceedings of the Twelfth Annual ACM Symposium on Theory of Computing , 1980.

Shai Ben-David, Nadav Eiron, and Hans Ulrich Simon. Limitations of learning via embeddings in euclidean half spaces. Journal of Machine Learning Research , 3(Nov):441-461, 2002.

Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. On the ability and limitations of transformers to recognize formal languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , 2020.

Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom. Simplicity bias in transformers and their ability to learn sparse boolean functions. arXiv preprint arXiv:2211.12316 , 2022.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020.

Emmanuel J Candes and Terence Tao. Decoding by linear programming. IEEE transactions on information theory , 51(12):4203-4215, 2005.

Nuo Chen, Qiushi Sun, Renyu Zhu, Xiang Li, Xuesong Lu, and Ming Gao. Cat-probing: A metric-based approach to interpret how pre-trained models for programming language attend code structure. arXiv preprint arXiv:2210.04633 , 2022.

Zhengdao Chen, Soledad Villar, Lei Chen, and Joan Bruna. On the equivalence between graph isomorphism testing and function approximation with GNNs. In Advances in Neural Information Processing Systems 32 , 2019.

Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04341 , 2019.

Amit Daniely. Depth separation for neural networks. In Satyen Kale and Ohad Shamir, editors, Proceedings of the 2017 Conference on Learning Theory , volume 65 of Proceedings of Machine Learning Research , pages 690-696. PMLR, 07-10 Jul 2017. URL https://proceedings.mlr. press/v65/daniely17a.html .

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2021.

Benjamin L. Edelman, Surbhi Goel, Sham M. Kakade, and Cyril Zhang. Inductive biases and variable creation in self-attention mechanisms. In International Conference on Machine Learning , 2022.

Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In Vitaly Feldman, Alexander Rakhlin, and Ohad Shamir, editors, 29th Annual Conference on Learning Theory , volume 49 of Proceedings of Machine Learning Research , pages 907-940, Columbia University, New York, New York, USA, 23-26 Jun 2016. PMLR. URL https://proceedings. mlr.press/v49/eldan16.html .

Merrick Furst, James B Saxe, and Michael Sipser. Parity, circuits, and the polynomial-time hierarchy. Mathematical systems theory , 17(1):13-27, 1984.

David Gale. Neighborly and cyclic polytopes. In Proc. Sympos. Pure Math , volume 7, pages 225-232, 1963.

Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Trans. Assoc. Comput. Linguistics , 8:156-171, 2020. doi: 10.1162/tacl\_{a}{\_{0}{0}{3}}{0}6. URL https: //doi.org/10.1162/tacl_a_00306 .

Yiding Hao, Dana Angluin, and Robert Frank. Formal language recognition by hard attention transformers: Perspectives from circuit complexity. Trans. Assoc. Comput. Linguistics , 10:800810, 2022. URL https://transacl.org/ojs/index.php/tacl/article/view/3765 .

John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2019.

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Netw. , 2(5):359-366, July 1989.

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021.

Mauricio Karchmer and Avi Wigderson. Monotone circuits for connectivity require super-logarithmic depth. In Proceedings of the Twentieth Annual ACM Symposium on Theory of Computing , 1988.

Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neural networks. In Advances in Neural Information Processing Systems 32 , 2019.

Valerii Likhosherstov, Krzysztof Choromanski, and Adrian Weller. On the expressive power of self-attention matrices. arXiv preprint arXiv:2106.03764 , 2021.

Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. CoRR , abs/2210.10749, 2022. doi: 10.48550/arXiv.2210.10749.

Andreas Loukas. What graph neural networks cannot learn: depth vs width. arXiv preprint arXiv:1907.03199 , 2019.

Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. In International Conference on Machine Learning , 2019.

James Martens, Arkadev Chattopadhya, Toni Pitassi, and Richard Zemel. On the representational efficiency of restricted boltzmann machines. In Advances in Neural Information Processing Systems 26 , 2013.

Shahar Mendelson, Alain Pajor, and Nicole Tomczak-Jaegermann. Reconstruction and subgaussian operators in asymptotic geometric analysis. Geometric and Functional Analysis , 17(4):1248-1282, 2007.

Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAI Conference on Artificial Intelligence , 2019.

OpenAI. Gpt-4 technical report, 2023.

David Peleg. Distributed computing: a locality-sensitive approach . SIAM, 2000.

Jorge Pérez, Javier Marinkovi´ c, and Pablo Barceló. On the turing completeness of modern neural network architectures. arXiv preprint arXiv:1901.03429 , 2019.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017.

Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics , 8:842-866, Dec 2020. ISSN 2307-387X. doi: 10.1162/tacl_a_00349. URL http://dx.doi.org/10.1162/ tacl_a_00349 .

Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems 30 , 2017.

Norbert Sauer. On the density of families of sets. Journal of Combinatorial Theory, Series A , 13(1): 145-147, 1972.

Saharon Shelah. A combinatorial problem; stability and order for models and theories in infinitary languages. Pacific Journal of Mathematics , 41(1):247-261, 1972.

Matus Telgarsky. Benefits of depth in neural networks. In Vitaly Feldman, Alexander Rakhlin, and Ohad Shamir, editors, 29th Annual Conference on Learning Theory , volume 49 of Proceedings of Machine Learning Research , pages 1517-1539, Columbia University, New York, New York, USA, 23-26 Jun 2016. PMLR. URL https://proceedings.mlr.press/v49/telgarsky16.html .

Vladimir Naumovich Vapnik and Aleksei Yakovlevich Chervonenkis. The uniform convergence of frequencies of the appearance of events to their probabilities. Doklady Akademii Nauk , 181(4): 781-783, 1968.

Gal Vardi, Daniel Reichman, Toniann Pitassi, and Ohad Shamir. Size and depth separation in approximating benign functions with neural networks. In Conference on Learning Theory , 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30 , 2017.

Colin Wei, Yining Chen, and Tengyu Ma. Statistically meaningful approximation: a case study on approximating turing machines with transformers. Advances in Neural Information Processing Systems , 35:12071-12083, 2022.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

Andrew Chi-Chih Yao. Some complexity questions related to distributive computing (preliminary report). In Proceedings of the Eleventh Annual ACM Symposium on Theory of Computing , 1979.

Shunyu Yao, Binghui Peng, Christos H. Papadimitriou, and Karthik Narasimhan. Self-attention networks can process bounded hierarchical languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing , 2021.

Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? In International Conference on Learning Representations , 2020.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems 30 , 2017.

Günter M Ziegler. Lectures on polytopes. Graduate Texts in Mathematics , 152, 2006.

Aaron Zweig and Joan Bruna. Exponential separations in symmetric neural networks. CoRR , abs/2206.01266, 2022. doi: 10.48550/arXiv.2206.01266.

A Fully-connected neural networks and recurrent neural networks cannot efficiently approximate q SA

A.1 Only wide fully-connected neural networks can approximate q SA

In this section, we show that any fully-connected neural network that approximates q SA : R Nd → R Nd ′ must have width m = Ω( N ) . 4 We consider networks of the form f ( x ) = g ( Wx ) for some weight matrix W ∈ R m × Nd (the first layer weights) and arbitrary function g : R m → R Nd ′ (computed by subsequent layers of a neural network).

Theorem 10. Suppose q ≤ N 2 . Any fully-connected neural network f defined as above that 1 2 q -approximates q SA satisfies m ≥ rank( W ) ≥ Nd ′ 2 .

Proof. For simplicity, we arrange the input as

and W = [ ˜ W ; V 1 ; . . . ; V N ] with z 1 , . . . , z N ∈ B d ′ , ˜ W ∈ R m × N ( d -d ′ ) , and V 1 , . . . , V N ∈ R m × d ′ . If rank( W ) ≤ Nd ′ 2 -1 , then so too is rank([ V q ; . . . ; V N ]) ≤ Nd ′ 2 -1 , and [ V q ; . . . ; V N ] has a nontrivial null space containing a nonzero vector u = ( u q ; . . . ; u N ) ∈ R ( N -q ) d ′ . Let

V j z j = V j z ′ j = 0 for all j ∈ [ N ] ; and

Therefore, for any y 1 , . . . , y N ∈ ( [ N ] q ) , respective x = (1; . . . ; N ; y 1 ; . . . ; y N ; z 1 ; . . . ; z N ) and x ′ = (1; . . . ; N ; y 1 ; . . . ; y N ; z ′ 1 ; . . . ; z ′ N ) satisfy f ( x ) = f ( x ′ ) . Consider y with y j = (1 , . . . , q -1 , j ) for each j ∈ { q, . . . , N } . Then,

Hence, ∥ q SA( x ) j ∗ -q SA( x ′ ) j ∗ ∥ 2 ≥ 2 q . Because f ( x ) = f ( x ′ ) ,

so f can approximate q SA to accuracy no better than 1 q .

A.2 Only high-memory recurrent neural networks can approximate q SA

In this section, we show that any memory-bounded algorithm that approximates q SA : R N × d → R N × d ′ must use a large 'hidden state' (memory) as it processes the input elements. This lower bound applies to various recurrent neural network (RNN) architectures.

A memory-bounded algorithm with an m -bit memory processes input X ∈ R N × d sequentially as follows. There is an initial memory state h 0 ∈ { 0 , 1 } m . For i = 1 , 2 , . . . , N , the algorithm computes the i -th output f ( X ) i ∈ R d ′ and the updated memory state h i as a function of the input x i ∈ R d and previous memory state h i -1 :

where g i : R d ×{ 0 , 1 } m → R d ′ ×{ 0 , 1 } m is permitted to be an arbitrary function, and f : R N × d → R N × d ′ is the function computed by the algorithm.

Our lower bound applies to algorithms that only need to solve the subclass of 'causal' instances of q SA in which the input X = (( z i , y i , i )) i ∈ [ N ] ∈ R N × d is promised to satisfy y i = ∅ for all i ≤ N/ 2 + 1 , and y i ⊆ { 1 , . . . , N/ 2 + 1 } for all i > N/ 2 + 1 .

Theorem 11. For any ε ∈ (0 , 1) , any memory-bounded algorithm that ε -approximates q SA (for q = 1 and d ′ = 1 ) on the subclass of 'causal' instances must have memory m ≥ ( N -1) / 2 .

Proof. Consider an m -bit memory-bounded algorithm computing a function f : R N × d → R N that ε -approximates q SA (for q = 1 and d ′ = 1 ). We construct, from this algorithm, a communication protocol for DISJ (with N = 2 n +1 ) that uses m bits of communication.

Let a, b ∈ { 0 , 1 } n be the input for DISJ provided to Alice and Bob, respectively. The protocol is as follows.

Alice constructs inputs x i = ( z i , ∅ , i ) for i = 1 , . . . , n +1 , where for each i = 1 , . . . , n ,

Bob constructs inputs x n +1+ i = (0 , y n +1+ i , n +1+ i ) for i = 1 , . . . , n , where

Observe that, for this input X = ( x 1 , . . . , x 2 n +1 ) , we have

Alice simulates the memory-bounded algorithm on the first n + 1 inputs x 1 , . . . , x n +1 , and sends Bob the m -bit memory state h n +1 . This requires m bits of communication.

Starting with h n +1 , Bob continues the simulation of the memory-bounded algorithm on these n additional inputs x n +2 , . . . , x 2 n +1 .

If any output f ( X ) n +1+ i for i = 1 , . . . , n satisfies

then Bob outputs 1 (not disjoint); otherwise Bob outputs 0 (disjoint).

The approximation guarantee of f implies that sign( f ( X ) n +1+ i ) = q SA( X ) n +1+ i for all i = 1 , . . . , n , so Bob outputs 1 if and only if a and b are not disjoint. Because this protocol for DISJ uses m bits of communication, by Fact 5, it must be that m ≥ n = ( N -1) / 2 .

We note that the proof of Theorem 11 can be simplified by reducing from the INDEX problem, which has a 1-way communication lower bound of n bits. This suffices for "single pass" algorothms, such as standard RNNs. However, the advantage of the above argument (and reducing from DISJ ) is that it easily extends to algorithms that make multiple passes over the input. Such algorithms are able to capture bidirectional recurrent neural net and related models. A straightforward modification of the protocol in the proof of Theorem 11 shows that Ω( N ) memory is required for any algorithm that makes O (1) passes over the input (and computes the outputs in a final pass).

and

B Supplementary results for Section 3

B.1 Proof of Theorem 2

Theorem 2 (Fixed-precision) . For any N , any m ≥ Ω( d ′ + q log N ) , any ϵ ∈ (0 , 1) , and p = Ω(log( q ϵ log N )) , there exists some f ∈ A ′ d,m,d ′ ,p that ϵ -approximates q SA .

Proof. Before explaining how they are produced by the input MLP, we introduce the corresponding key, value, and query inputs. The values will simply be ϕ ( X ) V = ( z 1 , . . . , z N ) . For some m ′ = m -d 2 , let ϕ ( X ) K = ( u 1 , . . . , u N ) ∈ R N × m ′ be embedded key vectors, where u 1 , . . . , u N ∈ {± 1 / √ m ′ } m ′ are the columns of a m ′ × N matrix satisfying the ( q, 1 / 4) -restricted isometry and orthogonality property (Definition 5), as guaranteed to exist by Lemma 12 and the assumption on m ′ . Let α := ⌈ 2 log(4 N/ϵ ) ⌉ . By Lemma 13, for each y ∈ ( [ N ] q ) , there exists w y ∈ R m ′ with ∥ w y ∥ 2 ≤ 2 √ q satisfying

Given the bounded precision of the model, we are not free to represent the vectors w y exactly. Under p -bit precision for p sufficiently large, we there exists a vector of p -bit floating point numbers ˜ w y ∈ R m ′ for every w y with ∥ w y ∥ 2 ≤ 2 √ q satisfying ∥ ˜ w y -w y ∥ 2 ≤ ϵ 4 α . As an immediate consequence, |⟨ u i ′ , ˜ w y ⟩ - ⟨ u i ′ , w y ⟩| ≤ ϵ 4 α for all i ′ and y (by Cauchy-Schwarz). The remainder of the proof demonstrates that the necessary properties of the argument hold even with this approximation.

We now describe how to structure the neural network. We define an MLP ϕ : R d → R m as ϕ ( x i ) = ϕ ( z i ; y i ; i ) = ( z i ; α ˜ w y i ; u i ) , which works simply by using a look-up table on the values of u i and ˜ w y i from keys i and y i respectively. Then, we define Q,K,V as sparse boolean-valued matrices that simply copy their respective elements from ϕ ( X ) .

We analyze the output of the softmax. If i ′ ∈ y i , then

An analogous argument shows that

Likewise, if i ′ ̸∈ y i , then

We thus conclude that that we meet the desired degree of approximation for such m :

B.1.1 Restricted isometry and orthogonality property

̸

The proof relies on the restricted isometry and orthogonality property from the compressed sensing literature. For v ∈ R N , let supp( v ) = { i ∈ [ N ] : v i = 0 } .

Definition 5. We say a matrix U ∈ R m × N satisfies the ( q, δ ) -restricted isometry and orthogonality property if

for all vectors v, v ′ ∈ R N with | supp( v ) | ≤ q , | supp( v ′ ) | ≤ 2 q , and supp( v ) ∩ supp( v ′ ) = ∅ .

The first result shows the existence of a sign-valued matrix U that satisfies the desired distancepreserving property.

Lemma 12 (Consequence of Theorem 2.3 of Mendelson et al. [2007] and Lemma 1.2 of Candes and Tao [2005]) . There is an absolute constant C > 0 such that the following holds. Fix δ ∈ (0 , 1 / 2) and q ∈ N . Let U denote a random m × N matrix of independent Rademacher random variables scaled by 1 / √ m . If m ≥ C ( q log N ) /δ 2 , then with positive probability, U satisfies the ( q, δ ) -restricted isometry and orthogonality property.

Sparse subsets of the columns of such a U can then be linearly separated from all other columns.

Lemma 13 (Consequence of Lemma 2.2 in Candes and Tao [2005]) . Fix δ ∈ (0 , 1 / 2) and q ∈ N . Let matrix U = [ u 1 , . . . , u N ] ∈ R m × N satisfy the ( q, δ ) -restricted isometry and orthogonality property. For every vector v ∈ { 0 , 1 } N with supp( v ) ≤ q , there exists w ∈ R m satisfying

B.2 Proof of Theorem 3

Theorem 3 (Infinite-precision) . For fixed N , m ≥ Ω( d ′ + q ) and ϵ > 0 , there exists some f ∈ A ′ d,m,d ′ that ϵ -approximates q SA .

The proof relies on the properties of neighborly polytopes , which we define.

Definition 6 (Ziegler [2006]) . A polytope P is q -neighborly if every subset of q ′ ≤ q vertices forms a ( q ′ -1) -face.

We give a q -neighborly polytope below that we use for the construction. For vectors v 1 , . . . , v N ∈ R m ′ , let Conv( v 1 , . . . , v N ) = { ∑ N i =1 α i v i : α ∈ [0 , 1] N , ∑ i α i = 1 } denote their convex hull.

Fact 14 (Theorem 1 of Gale [1963]) . For t ∈ R , let θ ( t ) = ( t, . . . , t m ′ ) ∈ R m ′ . Then, for all distinct t 1 , . . . , t N ∈ R , the cyclic polytope Conv( θ ( t 1 ) , . . . , θ ( t N )) is m ′ 2 -neighborly.

The proof of Theorem 3 is immediate from the aforementioned fact and the following lemma.

Lemma 15. Suppose there exists u 1 , . . . , u N ∈ R m ′ such that Conv( u 1 , . . . , u N ) is q -neighborly. Then, for any ϵ > 0 , there exists some f ∈ A ′ d,m,d ′ with fixed key vectors ϕ ( X ) K = ( u 1 , . . . , u N ) that ϵ -approximates q SA .

Proof. The construction employs a similar look-up table MLP ϕ to the one used in the proof of Theorem 2. We let the key and value embeddings be

To set the query vectors, observe that for any face F of a polytope P , there exists a hyperplane H F such that F ⊂ H F and P \ F lies entirely on one side of H F . Thus, for every y ∈ ( [ N ] q ) , there exists w ′ y ∈ R m ′ and b y ∈ R such that

We construct the MLP to satisfy ϕ ( x i ) = ( z k ; w y i ; u i ; 1) ∈ R m for m = 2 m ′ +2 and set parameter weights accordingly. Following the softmax analysis of Theorem 3, a sufficiently large choice of α ensures that max i ∈ [ N ] ∥ f ( X ) i -q SA( X ) i ∥ 2 ≤ ϵ .

B.3 Proof of Theorem 4

Theorem 4. For any sufficiently large q , any N ≥ 2 q +1 , and any d ′ ≥ 1 , there exists a universal constant c such that if mp ≤ cq , then no f ∈ T 1 , 1 d,m,d ′ ,p exists that 1 2 q -approximates q SA .

Proof. We first embed every instance of DISJ with n = q into an instance of q SA and prove that they correspond. We assume the existence of the a transformer f ∈ T 1 , 1 d,m,d ′ ,p that 1 2 q -approximates q SA and implies the existence of an O ( mp ) -bit communication protocol that computes DISJ . An application of Fact 5 concludes the proof.

Consider an instance of DISJ with a ∈ { 0 , 1 } q and b ∈ { 0 , 1 } q known by Alice and Bob respectively. We design an instance X = ( z i ; y i ; i ) i ∈ [ N ] of q SA . For each j ∈ [2 q ] , let y 2 q +1 = { 2 i + a i -1 : i ∈ [ q ] } . Additionally, let

All other inputs are set arbitrarily. Then,

Hence, q SA( X ) 2 q +1 = -e 1 if and only if DISJ( a, b ) = 0 .

It remains to show that this implies the existence of an efficient communication protocol that computes DISJ( a, b ) . By the existence of f , there exist Q,K,V : R d → R m and ψ : R m → R d ′ such that

The protocol is as follows:

From a , Alice determines y 2 q +1 and then computes Q ( x 2 q +1 ) ∈ R m , which she sends to Bob. This transmission uses O ( mp ) bits.

Bob determines z 1 , . . . , z 2 q from b . Using those and the information from Alice, he computes f ( X ) 2 q +1 . He returns 1 if and only if f ( X ) T 2 q +1 e 1 ≥ -1 + 1 q .

The protocol computes DISJ( a, b ) because f is a 1 2 q -approximation of q SA . Because any such protocol requires sharing Ω( q ) bits of information, we conclude that mp ≤ cq for some c .

B.4 Optimality of Theorem 3 under restricted architectures

While the near-optimality of the bounded-precision self-attention construction in Theorem 2 is assured by the communication complexity argument of Theorem 4, it is not immediately apparent whether Theorem 3 is similarly optimal among infinite-precision self-attention models. Theorem 16 proves that this is indeed the case for a restricted family of architectures that resembles cross-attention rather than self-attention.

Theorem 16. For input x 1 , . . . , x N satisfying x i = ( z i ; y i ; i ) , suppose ϕ ( x i ) T Q = w ( y i , i ) , ϕ ( x i ) T K = u ( i ) , and ϕ ( x i ) T V = z i . Then, for any q < N and m ≤ q (1 -C log N q ) for some universal C , there do not exist w : R d × [ N ] → R m and u : [ N ] → R m such that the resulting self-attention unit 1 2 q -approximates q SA .

The architectural assumptions of this statement are strong. For each element x i = ( z i ; y i ; i ) , its value embedding must reproduce its target z i ; its key embedding depends exclusively on the index i ; and its query embedding only on the indices y i and i . Indeed this attention unit more closely resembles cross-attention rather than self-attention, in which the problem is formulated as two sequences (( z 1 , 1) , . . . , ( z N , N )) and ( y 1 ; 1) , . . . , ( y N ; N ) that are passed to the key and value inputs and the query inputs respectively. We leave open the problem of generalizing this result to include all infinite-precision cross-attention or self-attention architectures, but we note that the constructions in Theorems 2 and 3 can be implemented under such architectural assumptions.

The proof relies on a geometric argument about how the convex hull of fixed key embeddings U = ( u (1) , . . . , u ( N )) lacks neighborliness and hence cannot separate every sizeq subsets of values embeddings z 1 , . . . , z N from the other values.

Proof. It suffices to show that for any fixed key embedding U , there exists some y i and setting of z 1 , . . . , z N such that

where w ( X ) = ( w ( y 1 , 1) , . . . , w ( y N , N )) ∈ R N × m and U = ( u (1) , . . . , u ( N )) ∈ R N × m .

By Fact 17, for some y 1 ∈ ( [ N ] q ) , there are no w and τ ∈ R satisfying w ( y 1 , 1) T u i ′ ≥ τ if and only if i ′ ∈ y 1 . Hence, for any fixed w , there exists i 1 ∈ y 1 and i 2 ∈ [ N ] \ y 1 such that w ( y 1 , 1) T u i 2 > w ( y 1 , 1) T u i 1 . Given the value embeddings z i 1 = e 1 , z i 2 = e 2 and z i = e 3 for all i ̸∈ { i 1 , i 2 } , we have

Fact 17. If m ′ < q (1 -log N Cq ) , then the columns of any U = ( u 1 , . . . , u N ) ∈ R N × m ′ can be partitioned into sets U 1 and U 2 with | U 1 | = q that are not linearly separable. Hence, Conv( u 1 , . . . , u N ) is not q -neighborly.

Proof. By the Sauer-Shelah Lemma [Sauer, 1972, Shelah, 1972, Vapnik and Chervonenkis, 1968] and the fact that the VC dimension of m ′ -dimensional linear thresholds is m ′ +1 , the maximum number of partitions of the columns of U that can be linearly separated is at most

for a sufficiently large choice of C given universal constant C ′ . If the fact were to be false, then at least ( N q ) ≥ ( N q ) q such partitions must exist, which contradicts the above bound.

C Supplementary results for Section 4

C.1 Proof of Theorem 6

Theorem 6. For any input size N , input range M = N O (1) , and fixed-precision bit complexity p = O (log M ) , there exists a transformer architecture f ∈ T 1 , 1 1 ,m, 1 ,p with a single self-attention unit with embedding dimension m = 3 such that for all X ∈ [ M ] N , f ( X ) = Match2( X ) .

Proof. As discussed in Section 2.1, we allow a single blank token to be appended to the end of the sequence X and assume the existence of a positional encoding. That is, we consider input X ′ = ( x 1 , . . . , x N , x ′ ) with x i, 0 = i and x ′ = ⃗ 0 to be the input to the target attention model. We define input MLP ϕ : R → R 3 and parameterizations Q,K,V ∈ R 3 × 3 such that

V T ϕ ( x i ) = ⃗ 1 , Q T ϕ ( x ′ ) = ⃗ 0 , K T ϕ ( x ′ ) = e 3 , and V T ϕ ( x ′ ) = ⃗ 0 . By elementary trigonometric identities, the following is true about the corresponding inner products:

As a result, ( Q T ϕ ( x i )) T K T ϕ ( x j ) = cd if and only if x i + x j = 0 (mod M ) . Otherwise, ( Q T ϕ ( x i )) T K T ϕ ( x j ) ≤ c (1 -1 M 2 ) . (Here, the O (log M ) -bit fixed-precision arithmetic is sufficient to numerically distinguish the two cases.) For each i ∈ [ N ] let

represent the total number of matches the input belongs to. If we take c = M 2 log(6 N ) , then

̸

We conclude that for any i ∈ [ N ] ,

where ≤ is a partial ordering with v ≤ v ′ if v i ≤ v ′ i for all i . Since the latter case holds only when β i ≥ 1 , the final step of the proof is design an output MLP ψ such that ψ ( z ) = 1 if z ≥ 1 3 and ψ ( z ) = 0 if z ≤ 1 6 , which can be crafted using two ReLU gates.

C.2 Proof of Theorem 7

Theorem 7. There is universal constant c > 0 such that for sufficiently large N , and any M ≥ N +1 , if mpH ≤ cN/ log log N , then there is no f ∈ T 1 ,H 1 ,m, 1 ,p satisfying f ( X ) = Match3( X ) for all X ∈ [ M ] N .

Proof. The proof relies on a reduction to Fact 5 that embeds inputs to the set-disjointness problem of cardinality n = N -1 2 into a subset of instances passed to Match3 . For the sake of simplicity, we assume in the construction that N is odd; if it were not, we could replace it with N -1 and set the final element such that it never belongs to a triple.

We consider the following family of inputs to Match3 :

Note that Match3( X ) 1 = 1 if and only if there exists i ∈ { 2 , . . . , N +1 2 } such that x i = i and x i + N -1 2 = ( M -i ) . Given input ( a, b ) ∈ { 0 , 1 } n ×{ 0 , 1 } n to DISJ , let x i +1 = 1 if and only if a i = 0 , and let x i + N +1 2 = 1 if and only if b i = 0 . Then, Match3( X ) 1 = 1 iff DISJ( a, b ) = 1 .

Suppose f ( X ) = Match3( X ) for all X ∈ [ M ] N for some f ∈ T 1 ,H 1 ,m, 1 ,p . Weshow that f simulates an O ( mpH ) -bit communication protocol for testing DISJ . By definition of the standard self-attention unit with multi-layer perceptrons, note that f ( X ) 1 = ψ ( ∑ H h =1 f h ( ϕ ( X ))) for ϕ : R → R m , ψ : R m →{ 0 , 1 } , and

for Q h , K h , V h : R m × m .

If we assume that this construction exists and is known explicitly by both Alice and Bob, we design a communication protocol for Alice and Bob to solve DISJ by sharing O ( mpH ) bits with one another. Let Alice possess a ∈ { 0 , 1 } n and Bob b ∈ { 0 , 1 } n , with n = N -1 2 .

Alice and Bob compute ( x 2 , . . . , x N +1 2 ) and ( x N +3 2 , . . . , x N ) from a and b respectively.

Alice computes an O ( p log log N ) -bit approximation of the logarithm of the first half of the softmax normalization term for each attention head and sends the result to Bob. That is, she sends Bob

for each h ∈ [ H ] . This requires transmitting O ( pH log log N ) bits.

Bob finishes the computation of normalization terms

for each h and sends the result back to Alice (up to O ( p log log N ) -bits of precision). This again requires transmitting O ( pH log log N ) bits.

Alice computes the partial convex combination of the first N +1 2 value vectors stipulated by the attention matrix

for each h and sends the partial combinations to Bob. This requires transmitting O ( mpH log log N ) bits (using the same precision as above).

Bob finishes the computation of the convex combinations

Bob concludes the protocol by computing and outputting f ( X ) 1 , using his knowledge of each f h ( X ) and of ψ .

By the equivalences previously established, Bob returns 1 if and only if DISJ( a, b ) = 1 . Because the protocol requires O ( mpH log log N ) bits of communication, we can only avoid contradicting Fact 5 if mpH ≥ Ω( n/ log log N ) = Ω( N/ log log N ) .

Remark 1. The domain restrictions to Match3 stipulated in Equation (3) make the Match3 problem substantially easier to solve than the full-domain case. Indeed, under the domain restrictions,

which is computable by a two-layer single-headed transformer network with constant embedding dimension. The first layer computes each Match2( X ) i with the construction in the proof of Theorem 6, and the second computes the maximum of the previous outputs by using those outputs as key vectors.

While Informal Conjecture 1 suggests that two layers are insufficient to compute the full-domain version of Match3 , this restricted variant introduces a concise depth separation (see Eldan and Shamir [2016], Telgarsky [2016], Daniely [2017]) between one- and two-layer transformer models.

C.3 Higher-order tensor attention

We introduce a novel category of higher-order tensor-based transformer models in order to show that problems like Match3 that are hard to compute with standard transformer models can be made solvable. An s -order transformer is designed to efficiently compute dense s -wise interactions among input elements in an analogous manner to how standard transformers compute pairwise interactions. (We think of a standard transformer as second-order.) Before defining the new type of attention, we introduce notation to express the needed tensor products.

For vectors v 1 ∈ R N 1 and v 2 ∈ R N 2 , let v 1 ⊗ v 2 ∈ R N 1 N 2 denote their Kronecker product by ( v 1 ⊗ v 2 ) ( i 1 -1) N 2 + i 2 = v 1 i 1 v 2 i 2 . The column-wise Kronecker product of matrices A 1 ∈ R N 1 × m and A 2 ∈ R N 2 × m is

The following generalizes the definition of self-attention.

Definition 7. For order s ≥ 2 , input dimension d , output dimension d ′ , embedding dimension m , bit complexity p , and matrices Q,K 1 , . . . , K s -1 ∈ R d × m and V 1 , . . . , V s -1 ∈ R d × d ′ (encoded with p -bit fixed-point numbers), an s -order self-attention unit is a function f Q,K,V : R N × d → R N × d ′ with

The input to the row-wise softmax is an N × N s -1 matrix. Let A ⊗ s d,m,d ′ ,p denote the set containing all such attention units.

Note that A ⊗ 2 d,m,d ′ ,p = A d,m,d ′ ,p . Because s -order self-attention units have the same domain and codomain as standard self-attention, multiple units can be analogous combined to construct multiheaded attention units and full transformer models. We define A M, ⊗ s d,m,d ′ ,p and T D,H, ⊗ s d,m,d ′ ,p accordingly.

The purpose of the s -order transformer model as a theoretical construct is to posit how strictly generalizing the architecture in order to permit higher order outer products transfers the expressive powers of standard transformer architectures to more sophisticated interactions among elements of the input sequence X . The model is not defined to be immediately practical, due to its steep computational cost of evaluation.

However, the trade-offs involved in using such architectures resemble those already made by using transformer models instead of fully-connected networks. Transformers are already computationally wasteful relative to the number of the parameters, and these models likely succeed only because extremely efficient factorized parameterization exist. Likewise, third-order transformers could indeed be practical if even more factorization proves useful, since the computational costs may prove mild if the embedding dimension m , number of heads H , and depth D necessary to succeed on a task exceed the sequence length N for standard second-order transformers.

C.4 Efficient representation of Match3 with third-order self-attention

Theorem 18 ( Match3 construction with third-order self-attention) . For any sequence length N , input range M = N O (1) , and fixed-precision bit complexity p = O (log M ) , there exists a third-order transformer architecture f ∈ T 1 , 1 , ⊗ 3 1 ,m, 1 ,p with a single self-attention unit with embedding dimension m = 5 such that for all X ∈ [ M ] N , f ( X ) = Match3( X ) .

Proof of Theorem 18. The proof is almost identical to that of Theorem 6, except that we instead use a different key and query transforms to express a different trigonometric function:

Together, these ensure that the resulting tensor products reduce to a trigonometric expression that is maximized when x i + x j 1 + x j 2 = 0 (mod M ) . That is,

We similarly let V 1 ϕ ( x i ) = V 2 ϕ ( x i ) = ⃗ 1 and V 1 ϕ ( x ′ ) = V 2 ϕ ( x ′ ) = ⃗ 0 . The remaining choice of c and the output MLP, and the analysis of the softmax proceeds identically to the previous proof.

C.5 Heuristic argument for Informal Conjecture 1

Conjecture 19 (Formal version of Informal Conjecture 1) . For sufficiently large N and any d ≥ 1 , for all M ≥ N +1 and mpHD ≤ N Ω(1) , there is no f ∈ T D,H 1 ,m, 1 ,p satisfying f ( X ) = Match3( X ) for all X ∈ [ M ] N .

We believe that the conjecture holds due to a heuristic information-theoretic argument. Define the distribution D over inputs X ∈ R N that will be used to show that the model cannot compute Match3 for M = N 4 with high probability. We draw X from D as follows:

( E 1 ) With probability 1 2 , draw each x i iid from Unif([ M ]) .

( E 2 ) With probability 1 2 , draw j 1 , j 2 , j 3 iid from Unif( ( [ N ] 3 ) ) . For all i = j 3 , draw each x i iid from Unif([ M ]) . Let x j 3 = -x j 1 -x j 2 (mod M ) .

̸

Note that under event E 1 , a three matching elements exist with probability at most 1 N , and

Under event E 2 , a triple of matching elements is always planted, so Match3( X ) = ⃗ 0 . It would suffice to prove that-unless a transformer is sufficiently large-it is impossible to determine whether Match3( X ) = ⃗ 0 with probability at least 0 . 9 .

̸

Under D , any subset of { x 1 , . . . , x N } consists of iid integers drawn uniformly from [ M ] , unless all of x j 1 , x j 2 , x j 3 appear in the subset. Consider a transformer architecture with p -bit precision, m -dimensional embeddings, H heads per layer, and D layers. We argue informally that a single-element output of a self-attention unit can take into account information about mp more inputs x 1 , . . . , x N than that it had in the previous layer. By induction, after D layers of H -headed self-attention with interleaved MLPs, each element is a function of at most mpHD inputs. Until an element exists that is a function of at least two of the three of x j 1 , x j 2 , x j 3 , we assume that the elements 'known' by each output are chosen independently of the indices j 1 , j 2 , j 3 . (Given two elements of the triple, the third element can be identified with a single self-attention unit.) Hence, we argue that it suffices to show that the probability any two elements of the triple j 1 , j 2 , j 3 occurring within any of the N sets of mpHD inputs is vanishingly small for sufficiently large transformer parameters. The probability of single collection having any of two of the three inputs is at most

Thus, the probability that any collection has all three inputs is no more than 3( empHD ) 2 /N . If mpHD = O ( √ N ) , then the randomly chosen triple will not jointly appear as the outcome of a single

element of a self-attention unit with probability at least 0 . 9 , and the transformer will be unexpected to successfully distinguish between the two cases.

Should the conjecture hold, it would represented a tight lower bound on the size of the smallest standard transformer architecture necessary to compute Match3 .

Theorem 20 (Tightness of Conjecture 19) . For any sequence length N , if the input range satisfies M = N O (1) and the transformer size parameters satisfy p ≥ log( M ) , H = 1 , m ≥ 4 , and mD ≥ CN 2 for some universal constant C , then there exists a transformer architecture f ∈ T D,H 1 ,m, 1 ,p such that f ( X ) = Match3( X ) .

Proof. We construct an architecture that collects a group of candidate pairs in each layer of singleheaded self-attention and verifies whether there exists a triple incorporating each pair that satisfies the summation property. Then, all candidate triples are disposed of, and the subsequent layer collects a new family of candidates.

To do so, we first let ℓ := ⌊ m 2 ⌋ -1 ≥ 1 represent the total number of pairs shared in each layer of attention. We let P = ( [ N ] 2 ) represent a collection of all pairs of indices and partition it into D subsets P 1 , . . . , P D , each containing ℓ distinct pairs. (Since | P | = N ( N +1) 2 , any D satisfying the theorem's preconditions is sufficiently large for this to be a proper partition.) Our construction ensures that there exist x i + x j 1 + x j 2 = 0 (mod M ) for ( j 1 , j 2 ) ∈ P k , then the k th layer of self attention will verify its existence and mark x i as belonging to the match. Throughout the network, we maintain that the first two dimensions of any embedding of the i th element correspond to x i ∈ [ M ] and a bit indicating whether a match has been found yet containing x i .

Consider the first layer of self-attention, and let P 1 = { ( i 1 , j 1 ) , . . . , ( i ℓ , j ℓ ) } . We set the input MLP ϕ 1 : R d → R m and respective matrices Q 1 , K 1 ∈ R m × m such that

for sufficiently large c . We additionally let

By making use of a residual connection, we ensure that the i th outcome of the self-attention is ( x i , 0 , x i 1 , x j 1 , . . . , x i ℓ , x j ℓ ) . We encode an MLP to compute

We repeat this construction D times, with the only modifications being the replacement of P 1 and the fact that the second dimension of the embedding remains 1 after being set to that value. After D layers, the final MLP outputs the value of the second dimension, which will be 1 if and only if the respective x i belongs to a three-way match.

C.6 Sharper separations for embedded subgraph detection problems

In pursuit of proving separations analogous to the one between Theorem 18 and Conjecture 19, we draw techniques for proving lower bounds for graph problems in the CONGEST model of distributed computation with restricted bandwidth [Peleg, 2000]. 5

The problems we consider take, as input, the adjacency matrix X ∈ { 0 , 1 } N × N of an N -vertex graph G = ( V , E ) with V = [ N ] , so x i,j = ✶ { ( i, j ) ∈ E} . We may regard each row of X as a highdimensional ( d = N ) embedding of the i -th vertex containing information about which (outgoing) edges are incident to the i -th vertex. We consider the following problems:

The former treats X as a directed graph (where X need not be symmetric) and asks whether each input belongs to a directed 3-cycle. The latter insists that X be an undirected graph by enforcing symmetry and determines membership in (undirected) 5-cycles.

However, solving these problems with any transformer model of constant order trivially requires having the product of the precision p , embedding dimension m , heads per layer H , and depth D grow polynomially with N , since each attention unit is limited to considering at most pm bits of information from each input. Such a lower bound is not interesting for dense graphs, where every vertex may have Ω( N ) incident edges; the bottleneck is not due to any feature of standard attention units (and would persist with higher-order attention).

To circumvent this issue, we consider an augmented self-attention unit, which permits each element of the self-attention tensor to depend on both its respective inner product and on the presence of edges among corresponding inputs.

Definition 8. For order s ≥ 2 , input dimension d , output dimension d ′ , embedding dimension m , bit complexity p , matrices Q,K 1 , . . . , K s -1 ∈ R d × m and V 1 , . . . , V s -1 ∈ R d × d ′ (encoded with p -bit fixed-point numbers), and cell-wise attention tensor function κ : { 0 , 1 } s ( s -1) × R → R , an s -order graph self-attention unit is a function f Q,K,V : R N × d → R N × d ′ with

For attention tensor A ∈ R N ⊗ s , we abuse notation by writing κ ( X,A ) as short-hand for the particular cell-wise application of a fixed function, incorporating information about all relevant edges:

Let AG ⊗ s d,m,d ′ ,p and T G D,H, ⊗ s d,m,d ′ ,p denote all such attention units and all such transformers respectively.

Now, we provide four results that exhibit separations between orders of graph self-attention.

Theorem 21 (Hardness of representing Cycle5 with standard graph transformer) . For sufficiently large N , any f ∈ T G D,H N,m, 1 ,p satisfying f ( X ) = Cycle5( X ) for all X ∈ { 0 , 1 } N × N with X = X T requires mpHD = Ω( N/ log 2 N ) .

Theorem 22 (Efficient construction of Cycle5 with fifth-order graph transformer) . For sequence length N and bit-complexity p = O (log N ) , there exists a fourth-order graph transformer architecture f ∈ T G 1 , 1 , ⊗ 5 N, 1 , 1 ,p with a single graph self-attention unit such that for all X ∈ { 0 , 1 } N × N with X = X T , f ( X ) = Cycle5( X ) .

Theorem 23 (Hardness of representing DirectedCycle3 with standard graph transformer) . For sufficiently large N , any f ∈ T G D,H N,m, 1 ,p satisfying f ( X ) = DirectedCycle3( X ) for all X ∈ { 0 , 1 } N × N requires mpHD = Ω( N/ log 2 N ) .

Theorem 24 (Efficient construction of DirectedCycle3 with fourth-order graph transformer) . For sequence length N and bit-complexity p = O (log N ) , there exists a third-order graph transformer architecture f ∈ T G 1 , 1 , ⊗ 3 N, 1 , 1 ,p with a single graph self-attention unit such that for all X ∈ { 0 , 1 } N × N , f ( X ) = DirectedCycle3( X ) .

The proofs of Theorems 22 and 24 are immediate from the construction. Because each cell of the self-attention tensor has explicit access the the existence of all relevant edges, κ can be configured to ensure that cell's value is large if and only if the requisite edges for the desired structure all exist. Taking a softmax with a blank element (like in Theorem 6) ensures that the outcome of the

self-attention unit for a given element distinguishes between whether or not it belongs to a 5-cycle or a directed 3-cycle. The output MLP ensure that the proper output is returned.

Weprove Theorems 21 and 23 by introducing a particular CONGEST communication graph that can be used to simulate any model in T G D,H d,m,d ′ ,p (and hence, also any model in T D,H d,m,d ′ ,p ) in O ( mHD log N ) rounds of communication. Then, we show for each problem that we can encode each instance of the set disjointness communication problem as an instance of Cycle5 (or DirectedCycle3 ) and derive a contradiction from the communication graph.

C.6.1 A CONGEST communication graph that generalizes standard graph transformer computation

The key principle of our analysis is that the predominant limitation of a transformer model is in its communication bandwidth and not its computational abilities. We model transformers as having element-wise multi-layer perceptron units with unbounded computational ability (but bounded precision inputs and outputs) and self-attention units, which compute linear combinations of inputs in a carefully regimented way that limits the ability of individual elements to share information with one another. Here, we introduce a specific CONGEST graph for each sequence length N and show that every transformer has a communication protocol that simulates its computation in this graph.

For fixed N , we design an undirected CONGEST graph G N = ( V N , E N ) with O ( N 2 ) nodes, each having degree at most 3. (Note that this graph is not the same as the graph provided as input X to a transformer; this graph is consistent across all transformers taking input of sequence size N .) Let u 1 , . . . , u N be nodes in V N corresponding to each input. For every pair i, j ∈ [ N ] , let v i,j be a node as well. For each i ∈ [ N ] , let B i = ( V i , E i ) be a balanced binary trees having root u i and leaves v i, 1 , . . . , v i,N , v 1 ,i , . . . , v N,i . Hence, each B i has O ( N ) vertices of degree 3 and is of depth O (log N ) . Let V N = V 1 ∪ · · · ∪ V N and E N = E 1 ∪ · · · ∪ E N . Noting that E 1 , . . . , E N are disjoint and that V 1 , . . . , V N are disjoint, except for leaves v i,j , we ascertain that G N contains O ( N 2 ) vertices of degree at most 3 and has diameter O (log N ) . We visualize the graph G N with a highlighted tree B 1 in Figure 4.

Lemma 25. For any transformer f ∈ T G D,H d,m,d ′ ,p and any X ∈ R N × d with p -bit fixed-precision numbers, there exists a CONGEST communication protocol on the graph G N that shares p bits of information between adjacent vertices per round satisfying the following characteristics:

Before any communication begins, each node u i is provided with x i and each node v i,j is provided with x i,j and x j,i .

After T = O ( HD ( m +log N )) rounds of communication, each node u i outputs f ( X ) i .

Proof. It suffices to give a protocol that computes the outcome of a single-headed unit of graph self-attention with parameters Q,K,V ∈ R m × m and κ : {-1 , 1 } 2 × R → R and transmits its i th output back to u i in O ( m log N ) rounds of p -bit communication. The remainder of the argument involves computing the outcomes of all element-wise MLPs within respective vertices u 1 , . . . , u N (since we assume each node to have unbounded computational power in the CONGEST model) and to repeat variants of the protocol HD times for every individual self-attention unit. Because the protocol is designed for a particular transformer architecture f , we can assume that every node in the CONGEST graph has knows every parameter of f .

We give the protocol in stages. We assume inductively that every input to f , y 1 , . . . , y N ∈ R m , is known by its respective vertex u 1 , . . . , u N .

Every vertex u i computes Q T y i ∈ R m and propagates it to every vertex v i, 1 , . . . , v i,N . This can be done in O ( m +log N ) rounds by transferring one p -bit fixed-precision number per round from an element of the binary tree B i to each of its children per round. Because the respective edges E 1 , . . . , E N are disjoint, this operation can be carried out in parallel.

Each u i computes K T y i , V T y i ∈ R m and propages them to v 1 ,i , . . . , v N,i in O ( m + log N ) rounds.

Each v i,j , using their knowledge of x i,j and x j,i , computes α i,j := exp( κ ( x i,j , x j,i , y T i QK T y j )) . This takes zero rounds.

Each u i computes ∑ N j =1 α i,j by propagating each α i,j in v i,j up B i to u i , iteratively summing terms passed up. This takes O (log N ) rounds.

Similarly, u i computes ∑ N j =1 α i,j V T y j in O ( m log N ) rounds. Then, it computes

which is the target output of the self-attention unit.

Because all steps are achievable in parallel with O ( m +log N ) rounds, the claim follows.

C.6.2 Reduction from set disjointness

Before proving Theorems 21 and 23 by embedding an instance of a transformer model into an instance of each subgraph identification problem, we first introduce a partition of the vertices V N of the CONGEST graph into those possessed by Alice and Bob for use in a two-party communication protocol. We call those two sets V N a and V N b .

Note that the previous section made no assumptions about the organization of edges in the binary tree. We thus add an additional condition: that each binary tree B i can be oriented to respect the left-to-right ordering v i, 1 , v 1 ,i , . . . , v i,N , v N,i . Let u i ∈ V N a if and only if i ≤ N 2 , and v i,j ∈ V N a if and only if min( i, j ) ≤ N 2 . We label are remaining nodes in B i by labeling a parent node w p as a function of its child nodes w ℓ and w r using the following rules:

(a) If w ℓ , w r ∈ V N a , then let w p ∈ V N a .

(b) If w ℓ , w r ∈ V N b , then let w p ∈ V N b .

(c) Otherwise, let w p ∈ V N a if and only if root u i ∈ V N a .

This partition, which we visualize in Figure 5, bounds the number of bits Alice and Bob can exchange by simulating a protocol on CONGEST graph G N .

Lemma 26. Suppose Alice and Bob simulate an R -round p -bit protocol on CONGEST communication graph G N where Alice has access to all vertices V N a and Bob V N b . No other communication is permitted besides sharing bits as permitted by the CONGEST protocol between neighboring vertices. Then, Alice and Bob exchange at most O ( pRN log N ) bits.

Proof. It suffices to show that the partition V N a , V N b induces a cut of size at most O ( N log N ) ; this ensures that each can send no more than O ( pN log N ) bits per round.

Per the rules defined above, an edge in ( w p , w ℓ ) and ( w p , w r ) is cut if and only if they are described by case (c). Within each tree B i under the orientation described above, an inductive argument shows that in every layer, all elements in V N a are to the left of all elements in V N b . Thus, there exists at most one parent of that layer that belongs to case (c), and thus, no more than one cut edge per layer. Because each tree has O (log N ) layers and because there are N trees, the partition cuts at most O ( N log N ) edges.

It remains to embed an instance of DISJ in V N a , V N b for each problem such that its output corresponds identically with that of DISJ .

Proof of Theorem 21. Assume for the sake of simplicity that N is divisible by 5. Let a, b ∈ { 0 , 1 } n for n = N 2 25 be an input to DISJ , and let Alice and Bob possess a and b respectively. We index those vectors as a = ( a 1 , 1 , a 1 , 2 , . . . , a N/ 5 ,N/ 5 -1 , a N/ 5 ,N/ 5 ) and b = ( b 1 , 1 , . . . , b N/ 5 ,N/ 5 ) for ease of analysis. We design input matrix X ∈ { 0 , 1 } N × N as follows:

If i ∈ (0 , N 5 ] and j ∈ ( N 5 , 2 N 5 ] , then x i,j = x j,i = a i,j -N/ 5 .

If i ∈ ( N 5 , 3 N 5 ] and j ∈ ( 2 N 5 , 4 N 5 ] , then x i,j = x j,i = δ i,j -N/ 5 .

If i ∈ ( 3 N 5 , 4 N 5 ] and j ∈ ( 4 N 5 , N ] , then x i,j = x j,i = b j -4 N/ 5 ,i -3 N/ 5 .

If i ∈ ( 4 N 5 , N ] and j ∈ (0 , N 5 ] , then x i,j = x j,i = δ i,j +4 N/ 5 .

Otherwise, x i,j = 0 .

This ensures that X has a 5-cycle if and only there exist i, j ∈ (0 , N 5 ] such that a i,j b i,j = 1 6 . In addition, note that under the protocol in Lemma 25, Alice's and Bob's inputs a and b are known exclusively by nodes belonging to V N a and V N b respectively.

Consider any transformer architecture f ∈ T G D,H N,m, 1 ,p that computes Cycle5 . By Lemma 25, there exists a protocol on the CONGEST graph G N that computes Cycle5 after O ( HD ( m +log N )) rounds of communication of p -bits each. If Alice and Bob simulate this protocol, and output 1 if and only if at least one of their outputs indicates the existence of a Cycle5 , then they successfully decide DISJ . By Lemma 26, this communication algorithm solves DISJ after exchanging O ( mpHDN log 2 N ) bits of communication. However, Fact 5 implies that no communication algorithm can do so without exchanging Ω( n ) = Ω( N 2 ) bits, which concludes the proof.

Proof of Theorem 23. The proof is identical to its predecessor, but uses a different embedding of an instance a, b ∈ { 0 , 1 } n to DISJ . Let n = N 2 16 . Then:

If i ∈ (0 , N 4 ] and j ∈ ( N 2 , 3 N 4 ] , then x i,j = a i,j -N/ 2 .

If i ∈ ( N 2 , 3 N 4 ] and j ∈ ( 3 N 4 , N ] , then x i,j = b j -3 N/ 4 ,i -N/ 2 .

If i ∈ ( 3 N 4 , N ] and j ∈ (0 , N 4 ] , then x i,j = δ i,j +3 N/ 4 .

Otherwise, x i,j = 0 .

This construction ensures that a directed 3-cycle exists if and only if a corresponding pair of elements in a and b are both 1.

D Experiment details

This section describes the experimental setup behind Figure 2, and provides further experiments suggesting an implicit bias of transformers for q SA , in particular when compared with MLPs and RNNs.

Experimental setup. Experiments used synthetic data, generated for q SA with n = 1000 training and testing examples, a sequence length N = 20 , q = 3 , with the individual inputs described in more detail as follows.

The positional encoding of element i is a random vector sampled uniformly from the sphere in R d 0 with d 0 := ⌈ 1 + 2 ln( N ) ⌉ , a quantity which agrees with the theory but was not tuned.

A sequence element then consists of the data portion z ∈ R d 1 where d 1 = 4 , also sampled from the unit sphere, then the positional encoding of this sequence element, and then q further positional encodings identifying elements to average to produce the output; this differs from (and is more tractable than) the presentation in Section 3, where the positional encoding is provided as an integer and the MLP layer input to our attention layers is expected to choose a sufficient positional encoding.

As such, the total dimension of a sequence element is d 1 +( q +1) d 0 = 32 . The architectures are detailed as follows.

The attention is identical to the description in the paper body, with the additional detail of the width and embedding dimension m being fixed to 100 .

Figure 6 also contains an MLP, which first flattens the input, then has a single hidden ReLU layer of width 256, before a final linear layer and an output reshaping to match the desired output sequence shapes.

Figure 6 also contains an LSTM, which is a standard pytorch LSTM with 2 layers and a hidden state size 800 , which is 200 times larger than the target output dimension 4 .

0

5

10

15

20

25

30

35

40

0

1

2

3

4

5

6

Attention (training)

Attention (testing)

LSTM (training)

LSTM (testing)

MLP (training)

MLP (testing)

Experiments fit the regression loss using Adam and a minibatch size of 32, with default precision, and take a few minutes to run on an NVIDIA TITAN XP, and would be much faster on standard modern hardware.

Further discussion of Figure 2 and Figure 7. In Figure 2 and Figure 7, we plot (post-softmax) alignment matrices after T ∈ { 0 , 1000 , 40000 } iterations of Adam. The alignment matrices in Figure 2 are taken from the training example whose loss is the median loss across all examples. Figure 7 is similar, but additionally shows the examples of minimal and maximal loss.

Further discussion of Figure 6. Figure 6 plots training and testing error curves for the same attention architecture as in Figure 2, but with further MLP and LSTM architectures as described above. but also an MLP trained on flattened (vectorized) error bars reflect 5 separate training runs from random initialization. A few variations of these architectures were attempted, however curves did not qualitatively change, and in particular, only the attention layer achieves good generalization across all attempts.

(a) Min loss,

T

= 0

.

(b) Min loss,

T

= 1000

.

0.0

0.2

0.4

0.6

0.8

1.0

(c) Min loss,

T

= 40000

.

(d) Median loss,

T

= 0

.

(e) Median loss,

T

= 1000

.

0.0

0.2

0.4

0.6

0.8

1.0

(f) Median loss,

T

= 40000

.

(g) Max loss,

T

= 0

.

(h) Max loss,

T

= 1000

.

0.0

0.2

0.4

0.6

0.8

1.0

(i) Max loss,

T

= 40000

.

■

日

口

口

■

口

日

口

口

口

日

■

日

18

6

+

口

日

甲日

日

日

日

日

口

口

口

口

品

1.

口

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the representation power of attention layers in transformer networks and compares them with other neural network architectures. The authors establish both positive and negative results on the benefits and limitations of attention layers, focusing on intrinsic complexity parameters such as width, depth, and embedding dimension. The positive results include the demonstration of a sparse averaging task where transformers scale logarithmically in the input size, compared to polynomial scaling in recurrent networks and feedforward networks. They also show the necessity and role of a large embedding dimension in transformers. On the negative side, they present a triple detection task where attention layers have linear complexity in the input size. However, they also provide variants of the task that can be efficiently solved by attention layers. The paper's contributions include the formalization of computational limits using communication complexity, the establishment of the representational capabilities and limitations of self-attention units, and the demonstration of the impossibility of computing Match3 using standard multi-headed attention layers in an efficient manner.\n\nSTRENGTHS:\n- This paper brings some original contributions to the understanding of attention layers in transformer networks. It provides a mathematical analysis of the representation power of attention layers and compares them to other neural network architectures. The paper introduces some interesting tasks, such as sparse averaging, pair matching, and triple matching, to evaluate the capabilities and limitations of attention layers. This approach introduces fresh perspectives on the benefits and deficiencies of attention layers and offers valuable insights into their role in deep learning.\n\n- The paper demonstrates a high level of quality in terms of its analysis and proofs. It presents rigorous mathematical formulations and provides formal definitions for the tasks and architectures under study. The proposed theorems and conjectures are well-supported and backed by detailed proofs, showcasing the authors' expertise in the subject matter. The use of communication complexity techniques adds depth and reliability to the analysis. The paper also includes supplementary proofs and explanations in the appendices, further enhancing the quality and comprehensiveness of the work.\n\n- The paper addresses an important gap in the literature by providing a mathematical analysis of the benefits and limitations of attention layers in transformers. The findings have implications for the design and optimization of deep learning models, especially in natural language processing and other sequential tasks. The identification of tasks that highlight the strengths and weaknesses of attention layers can guide future research in developing more efficient and effective architectures. The paper also raises intriguing conjectures, such as the impossibility of efficiently computing Match3, which can inspire further investigations and spark valuable discussions in the research community.\n\nWEAKNESSES:\n- While the paper provides rigorous mathematical analysis and proofs, it lacks empirical evaluation of the proposed tasks and architectures. Including experiments using real-world datasets could provide practical validation of the theoretical findings and further strengthen the paper's conclusions. Empirical evaluation could also provide insights into the computational efficiency and generalization performance of attention layers compared to other architectures.\n\n- Providing concrete examples of how the theoretical findings can be applied to practical deep learning problems, such as natural language processing or computer vision tasks, would enhance the paper's relevance and impact.\n\n- While the paper is generally well-written, the mathematical formulations and proofs can be complex and challenging to follow for readers without a strong background in the subject area. Simplifying and clarifying the presentation of the mathematical concepts and providing more intuitive explanations or examples could improve the accessibility of the paper for a wider audience.\n\nQUESTIONS:\nPlease see above comments.\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents the representational strength and some limitations of the transformer architecture.\n1. Strength: separation between a unit of self-attention and a one-hidden layer neural or a recurrent neural network.\nThe authors present a task where the complexity of the latter networks scale with N (number of tokens) and the self-attention does not.\n2. Limitation: the task Match2 can be computed with a self-attention unit that scales with the input dimension d, yet, a modification of the task, Match3, cannot be computed with a single transformer layer. Match3 can be computed with a standard and modified transformer model. The first makes assumptions over the input the second model modifies the self-attention module.\n\nSTRENGTHS:\nThe paper is very well written. I could easily understand and follow all the definitions and the Theorems presented in the paper's main text.\n\nWEAKNESSES:\nThe weakness of this paper lies in its relevance to the understanding of transformer networks.\n\nThe paper presents problems qSA, Match2, and Match3, whose relevance to understanding neural networks in practice is unclear. This is even stated in the paper: \"Future work by linguists, theoretical computer scientists, and empirical NLP practitioners could assess how foundational our primitives are and study whether there are any practical triple-wise problems that transformer models fail to solve.\"\n\nI think that when studying theoretical aspects of neural networks, it is the responsibility of the authors to motivate their theoretical results and why their assumptions or framework are relevant. Otherwise, such theoretical content belongs in a mathematical venue.\n\nMinor comment:\nWhen reading Section 1.1 for the first time, I could not follow the authors' intentions/message. Only after carefully reading the rest of the text and definitions could I understand it. As written now, I think it does not convey information properly for someone just interested in having a quick general idea of the results.\n\nQUESTIONS:\nI would be happy to change my mind about the paper since the paper is written technically very well. Better than most theoretical papers I encounter! \n\nCan the authors at least give some intuitive explanations why they think the tasks presented in the paper are relevant to NLP or any other ML task? Maybe a small experimental study on a small transformer network and show that the functions qSA, Match2, and Match3 somewhat resemble the functions the transformer networks actually compute?\n\nMore so, through the years, I haven't encountered a single theory paper about the representational power of neural networks that actually sheds any light on understanding neural networks better in the practical sense of explaining their performance (other than the original universal approximation paper). For example, one can study the VC dimension of different architectures, but this does not bring us any step closer to understanding neural networks in practice. \n\nIn its current form, I think the paper is suited to a pure theoretical venue such as a math journal or a theoretical CS conference. To make it more suitable for an ML venue, a reasonable amount of effort is required to motivate theoretical setup and questions. In that case, it's likely that the scope of the paper would not fit a conference paper but in a journal.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper focuses on the representational capabilities of attention layers in transformer models and showcases both the strengths and limitations. On one hand, the paper proves that attention layers excel at the presented sparse averaging task compared to RNNs and FNNs. On the negative side, they have complexity scaling linearly in the input size for the triple detection task. The paper theoretically showcases the strengths and limitations of the expressivity of attention layers, especially the role of embedding dimension of the attention layer.

STRENGTHS:
To my knowledge, the presented theoretical results are novel in the study of transformers and their representational capabilities. However, this work is out of my expertise and I cannot go into depth about the significance of the work. I do appreciate the authors in providing some empirical evidence about the theoretical findings of the work.

WEAKNESSES:
As said previously, this is not within my area of expertise. I am however interested in how relevant/connected are the proposed tasks (e.g. sparse averaging and triple detection) to empirical studies such as language or visual data modeling. I think the work would be improved if more insights could be provided as to how the theoretical results can be connected to real-world practices.

QUESTIONS:
My questions and suggestions are listed above.

LIMITATIONS:
This is a theory paper and the authors provided open directions for future work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper mainly investigates the inductive biases of attention-based models. They propose three computational tasks that show the limitations of Transformers, namely, sparse averaging, pair matching, and triples-matching. Specifically, they analyze the representational power of embedding dimensions and show that the sparse averaging task scales more favorably in the input size as opposed to the other two tasks. Various proofs are provided to support the investigations and conclusions.

STRENGTHS:
* Interesting and relevant topic on interpreting the Transformer computations and learning progress. 
* Good connections with relevant prior works in setting the backgrounds for inspiring the tasks and their practical implications proposed

WEAKNESSES:
* Initially the notation is a bit confusing, especially in section 1.1 that details the contributions. Variables $y$ and $z$ should be more clearly defined and explained when mentioning results of the theoretical analysis.
* The intro can maybe be reworked to slowly work in the details and notation of the method instead of initially presenting the mechanisms that the different tasks enable

QUESTIONS:
* It seems somewhat obvious that the representational capacity of performant Transformeres is related to the embedding space and operations for comparison performed on thus. Have the authors experimented with different contexts aside from the embedding space?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the representational power of transformers. They quantify how the transformers are superior to other neural network architectures, as well as the limitation of the transformer architecture. They focus on the $q$-sparse averaging (qSA) task which amounts to  averaging $d$-dimensional input vectors over subsets of size $q$ over a set of size $N$. They show by an explicit construction a unit of self-attention with $m \\geq q$ dimensional embedding can approximate qSA. Using tools from communication complexity between two parties, they show fully connected networks require $\\Omega(Nd)$ hidden layers, and recurrent neural networks require $\\Omega(N)$ bits of information. They further show that standard transformers can approximate functions that intrinsically depends on input pairs, while failing to approximate functions that depends on triplets instead. Finally, they propose a variant of transformers that can approximate the triplet-functions.\n\nSTRENGTHS:\n- The authors leverage the communication complexity to show that the self attention layers add more representational capacity as compared to fully connected and recurrent neural networks. \n\n- They use Match2 and Match3 problems to show rigorously that 2nd-order functions represent the threshold of efficient approximability for transformers.\n\n- The variants of Match3 problems hint at transformers leveraging some local structures present in the data to go beyond pair matching. This reconciles the Match3 impossibility result with the much presumably higher order successes of transformer models. \n\n- They additionally show higher order transformers can solve the higher order function approximations.\n\nWEAKNESSES:\n- The fixed precision (Theorem 2) result still uses precision that needs to grow with the data size and approximation parameters. Although it is a good first step, it leaves open the approximability with constant precision (which is used in most practical cases). \n\n- The ``third-order tensor self-attention\" is an important claim in the paper, but it is fully omitted in the main paper. It will be good to at least introduce the basic ideas in the main paper.\n\nQUESTIONS:\n- Given a fixed constant precision $p$ what is the bottleneck in creating the transformers that can approximate q-SA? What if $z_i$s are themselves restricted to some $O(p)$ precision?\n\n- Is the use of communication complexity in the approximability of Transformers (and broader Neural networks) novel in this work?\n\nLIMITATIONS:\nIt is hard to foresee any potential negative societal impact of this theoretical work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-28"}