A Formal Framework for Understanding Length Generalization in Transformers

## Abstract

Abstract A major challenge for transformers is generalizing to sequences longer than those observed during training. While previous works have empirically shown that transformers can either succeed or fail at length generalization depending on the task, theoretical understanding of this phenomenon remains limited. In this work, we introduce a rigorous theoretical framework to analyze length generalization in causal transformers with learnable absolute positional encodings. In particular, we characterize those functions that are identifiable in the limit from sufficiently long inputs with absolute positional encodings under an idealized inference scheme using a norm-based regularizer. This enables us to prove the possibility of length generalization for a rich family of problems. We experimentally validate the theory as a predictor of success and failure of length generalization across a range of algorithmic and formal language tasks. Our theory not only explains a broad set of empirical observations but also opens the way to provably predicting length generalization capabilities in transformers.

## 1 Introduction

A key problem in neural sequence modeling is generalization from shorter to longer sequences – *length generalization*.
A wide range of empirical research has found that transformers’ ability at length generalization is mixed, with success found on some problems and failure on others (e.g. Bhattamishra et al.,, ; Anil et al.,, ; ; Kazemnejad et al.,, ; ; Awasthi and Gupta,, ; Jelassi et al.,, ; ; Chang and Bisk,, ).
For instance, while transformer decoders can easily copy long strings (Bhattamishra et al.,, ), length generalization is substantially more brittle and depends on the absence of repetitions in the string (; Jelassi et al.,, ).
Similarly, while transformers can in theory simulate many finite-state automata in principle (Liu et al.,, ), their success of length generalization in practice varies widely across different automata (Liu et al.,, ; Bhattamishra et al.,, ).
Theoretical understanding of these phenomena is largely lacking, making it difficult to anticipate on which problems transformers will succeed or fail to generalizing beyond the length of their training inputs.

An important step towards theoretical understanding was made in the RASP-L Conjecture ().
This conjecture states that transformers show good length generalization exactly on those problems that have simple programs in RASP-L, a fragment of the RASP language (Weiss et al.,, ) with substantial restrictions on the ways in which positional information can be used.
While provided empirical evidence in support of this idea, two important gaps remain:
First, the RASP-L language has not been fully formalized and its expressiveness itself is not well understood; thus, it is largely open how to prove that a certain problem is indeed not representable in it.
Second, while compelling empirical evidence supports a link between definability in RASP fragments and length generalization, no formal proof exists.

We present a general theoretical framework analyzing length generalization as ultimate identifiability in the limit: When the input-output behavior of a function is observed at longer and longer input lengths, we ask under what conditions a learner can at some point converge on inferring the ground-truth function.
We answer this in the positive for a specific idealized learning strategy and a well-defined class of functions:
whenever a function belongs to this class, transformers are guaranteed to length-generalize in an idealized setting.

Our results apply to multilayer transformers, focusing on causal transformers with absolute positional encodings (APE) or without positional encodings (NoPE).
A key technical challenge in analyzing length generalization for absolute positional encodings is the scaling of the transformer’s parameter count with the input length.
To address this, we define a transformer-like limiting object, the *Limit Transformer*, which encapsulates the computations of a sequence of transformers operating on longer and longer inputs into a single object.
We then define an idealized inference procedure in which transformers are fitted to reproduce a target function on successively longer inputs while minimizing a specific norm-based regularizer.
Our main theoretical result states that the inference procedure will ultimately lead to length generalization for sufficiently long training inputs, provided the ground-truth function is expressible by a single such limiting object across all input lengths:

###### Theorem 1 (Informal Version of Theorem 7 ) .

Let $f$ be the target function expressible by a single Limit Transformer at all input lengths, subject to restrictions on the use of positional information.
Choose transformers $T_{n}$ ($n=1,2,3,\dots$) with context size $n$, where $T_{n}$ reproduces the behavior of $f$ up to length $\frac{n}{2}$, while minimizing a norm-based regularizer.
Then, for large $n$, $T_{n}$ will match the output of the target function $f$ up to length $\leq n$.

We then show that the expressivity of Limit Transformers can be understood for many functions.
We extend a recently introduced RASP variant (C-RASP, Yang and Chiang,, ) to provide lower bounds, showing that transformers will succeed at length generalization on various concrete problems under the inference procedure.
Conversely, we employ communication complexity to obtain upper bounds on the class of functions for which length generalization is predicted.
Experiments confirm the success of the theory at predicting empirical length generalization behavior across various algorithmic tasks and formal languages.
Overall, our results formalize the RASP-L Conjecture and take a step toward a theoretical understanding of length generalization.

###### Theorem 1 (Informal Version of Theorem 7 ) .

## 2 Model of Transformers

##### Positional Encoding Scheme

We study two positional encoding schemes. One uses no positional encoding at all; we refer to this as NoPE (No Positional Encoding).
The other one uses Absolute Positional Encodings (APE), with learned per-position embedding vectors ${\bm{p}}_{1},\dots,{\bm{p}}_{N}$.
We follow in requiring transformers to be able to perform a task at different offsets within a longer context.
Whereas concatenated different examples of a task, we simply encode an input $x$ of length $|x|=k\leq N$ using positional encodings ${\bm{p}}_{1+o},\dots,{\bm{p}}_{k+o}$ where $o$ is an offset such that $k+o\leq N$, and require that the transformer correctly performs the task independently of the offset $o\geq 0$. This mimics the computations in language models, where the same reasoning task can typically appear at different places in a long context.
For simplicity, we treat positions outside of the input, including those preceding the offset, as empty.

##### Parameterization

We focus on transformers with causal masking; for simplicity, we will use the term “transformer” for these throughout.
A transformer $T$ is parameterized by a finite alphabet $\Sigma$, a width $d\in\mathbb{N}$, a token embedding matrix ${\bm{E}}\in\mathbb{R}^{|\Sigma|\times d}$,
a context width $N(T)\in\mathbb{N}\cup\{+\infty\}$, positional encodings $\{{\bm{p}}_{i}\in\mathbb{R}^{d}:1\leq i<N(T)+1\}$, a depth $L\in\mathbb{N}$ and head count $H\in\mathbb{N}$, key, query, and value matrices $\{{\bm{K}}_{l,h},{\bm{Q}}_{l,h},{\bm{V}}_{l,h}\in\mathbb{R}^{d\times d}:1\leq l\leq L,1\leq h\leq H\}$,
MLP matrices and biases $\{{\bm{A}}_{l}\in\mathbb{R}^{d\times d},{\bm{B}}_{l}\in\mathbb{R}^{d\times d};{\bm{b}}_{l}\in\mathbb{R}^{d}:1\leq l\leq L\}$, and an unembedding matrix ${\bm{U}}\in\mathbb{R}^{|\Sigma|\times d}$.
For matrices, we use $\|\cdot\|$ and $\|\cdot\|_{F}$ to denote the spectral and Frobenius norm respectively.

##### Computation of Activations and Outputs

We assume a standard causal transformer, with a few technical points: We explicitly scale attention logits with the logarithm of the input length, omit layer norm, allow Heaviside activations in addition to ReLU activations, and assume that, while the transformer may overall compute at infinite precision, attention logits are operated over at fixed fractional precision.
We next define all computations formally.
Reserving a special SOS symbol not in $\Sigma$, written as “$”, we take the set of input strings to be $\mathfrak{S}$, the set of strings $x\in\Sigma^{*}$ where $x_{1}=\$$ and $ does not occur in $x_{2\dots|x|}$.
We now define the computation of the transformer $T$ on an input $x\in\mathfrak{S}$ where $1\leq|x|<N(T)+1$ (that is – $|x|\leq N(T)$ if $N(T)<\infty$, and $|x|$ is any finite length otherwise).
If $L$ is the number of layers, then we write the output of layer $l=1,\dots,L$ at position $i=1,\dots,N(T)$ as ${\bm{y}}_{i}^{(l)}\in\mathbb{R}^{d}$.
Let $o\geq 0$ be any offset such that $|x|+o<N(T)+1$ – that is, $x$ still fits into the transformer’s context width if encoded at offset $o$.
Given this offset, we set

$$ ${\bm{y}}_{i}^{(0)}={\bm{E}}_{x_{i}}+{\bm{p}}_{i+o}\ \ \ \ \ i=1,\dots,|x|$ (1) $$

where $x_{i}\in\Sigma$ is the input symbol at position $i$.
Attention logits, at query position $i$ and key position $j$ are computed as

$$ $a_{i,j}^{(l,h)}=({\bm{y}}_{j}^{(l-1)})^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{y}}_{i}^{(l-1)}\ \ \text{for}\ 1\leq j\leq i\leq|x|;\ l=1,\dots,L;\ h=1,\dots,H$ (2) $$

We assume standard softmax attention, but incorporate scaling with $\log|x|$ following prior work finding it necessary to theoretically represent sparse functions and circumvent theoretical limitations of soft attention (Chiang and Cholak,, ; Edelman et al.,, ):

$$ ${\bm{Y}}_{i}^{(l)}:={\bm{y}}_{i}^{(l-1)}+\sum_{h=1}^{H}\frac{\sum_{j=1}^{i}\exp\left(\log|x|\cdot a^{(l,h)}_{i,j}\right){\bm{V}}_{l,h}{\bm{y}}_{j}^{(l-1)}}{\sum_{j=1}^{i}\exp\left(\log|x|\cdot a^{(l,h)}_{i,j}\right)}$ (3) $$

After each attention block, the activations are passed through a one-layer MLP:

$$ ${\bm{y}}_{i}^{(l)}:={\bm{Y}}_{i}^{(l)}+{\bm{B}}_{l}\cdot\psi_{l}({\bm{A}}_{l}{\bm{Y}}_{i}^{(l)}+{\bm{b}}_{l})$ (4) $$

where we allow the activation function $\psi_{l}$ to be, in each coordinate, either ReLU or Heaviside (see Appendix [D.1](#A4.SS1) for discussion of this).
We omit layer norm, as it plays no important role in our results, but it can be accounted for. See Appendix [D.3](#A4.SS3) for the role of layer norm.

We assume an infinite-precision setup for the activations, with the restriction that attention logits ([2](#S2.E2)) and the output of the $\exp(\cdot)$ function are both rounded to $p$ fractional bits of precision before further processing. This is a mild restriction preventing tiny changes in attention patterns from potentially snowballing into large changes in the output due to infinite precision.
See Appendix [D.2](#A4.SS2).

We conceive of a transformer $T$ as a map from strings $x\in\mathfrak{S}$ ($|x|\leq N(T)$) to vectors of next-token prediction logits, $T(x,o)\in\mathbb{R}^{|x|\times|\Sigma|}$, where for $i=1,\dots,|x|$, $T(x)_{i}={\bm{U}}{\bm{y}}_{i}^{(L)}$ for the unembedding matrix ${\bm{U}}\in\mathbb{R}^{|\Sigma|\times d}$, and $o$ is the offset.
In line with the assumed setup, we focus on transformers whose input-output behavior is *invariant across offsets*: $T(x,o)=T(x,o^{\prime})$ for any $0\leq o,o^{\prime}\leq N(T)-|x|$.
Let $\mathcal{F}(\Sigma)$ be the class of all maps $f$ mapping $x\in\mathfrak{S}$ to $f(x)\in\mathbb{R}^{|x|\times|\Sigma|}$.

## 3 Theoretical Framework

### 3.1 Limit Transformers

Our theory addresses the setting of transformers with absolute positional encodings, where the width may grow with the input length.
Importantly, we cannot view the ground-truth function as realized by a single transformer: Even if one assigned such a transformer an infinite number of positional encodings, it would still effectively only be able to distinguish between a bounded number of positions, because the width of positional encodings within a single transformer is bounded.
Instead, we will derive a parameterization of transformers that allows us to convert sequences of transformers operating on longer and longer sequences to a single limiting transformer-like object.
Our key technical idea is to reparameterize the transformer in terms of product functions, inner products of parameter vectors as mediated by parameter matrices, such as

$$ $\displaystyle{\bm{E}}_{\sigma}^{T}{\bm{K}}_{1,h}^{T}{\bm{Q}}_{1,h}{\bm{E}}_{\tau}$ $\displaystyle{\bm{p}}_{i}^{T}{\bm{K}}_{1,h}^{T}{\bm{Q}}_{1,h}{\bm{p}}_{j}$ (5) $\displaystyle{\bm{p}}_{i}^{T}{\bm{K}}_{2,h}^{T}{\bm{Q}}_{2,h}{\bm{V}}_{1}{\bm{p}}_{j}$ $\displaystyle{\bm{U}}_{\sigma}{\bm{V}}_{2}{\bm{V}}_{1}{\bm{E}}_{\sigma}$ $$

and various others; see Appendix [F.1](#A6.SS1) for the full formal definition.
We first note that the transformer’s computations are uniquely specified by such products.
The number of products as in ([5](#S3.E5)) depends, among others, on $|\Sigma|$, $L$, $H$, $N(T)$, but crucially not on $d$.
We will use this parameterization
to translate sequences $T_{1},T_{2},T_{3},\dots$ of transformers running on inputs of length $1,2,3,\dots$ to limiting transformer-like objects that are applicable at all input lengths, while keeping width $d$ bounded even if the widths of $T_{n}$ diverge to infinity.
This limiting object, a Limit Transformer, differs from an ordinary transformer, as defined in Section [2](#S2), just in a few respects. Formally:

###### Definition 2 .

A *Limit Transformer* is a transformer $T$ where:

- 1.
$N(T)=+\infty$
- 2.
All parameters (including positional encodings ${\bm{p}}_{i}$, and the output of $\phi_{l,h}$) are expressed in $p$-bit precision, for some $p\in\mathbb{N}$
- 3.
In deviation from ordinary transformers, attention logits on input length $N$ are computed as
$a_{i,j}^{(l,h)}=({\bm{y}}^{(l-1)}_{j})^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{y}}^{(l-1)}_{i}+\phi_{l,h}(j,i)$
(6)
where $\phi_{l,h}:\mathbb{N}\times\mathbb{N}\rightarrow\mathbb{R}$.

Intuitively, a Limit Transformer can use positional information in two ways: through bounded-width and bounded-precision positional encodings ${\bm{p}}_{i}$, and through potentially more complicated functions $\phi_{l,h}$.
Given a transformer, we obtain a Limit Transformer by encoding products of the form ${\bm{p}}_{i}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{p}}_{j}$ into the functions $\phi_{l,h}$.
All other product functions involving positional encodings are expressed in terms of the positional encodings of the Limit Transformer.
Our main result will link length generalization to expressibility by Limit Transformers satisfying specific properties:

###### Definition 3 .

A function $f:\mathbb{N}\times\mathbb{N}\rightarrow\mathbb{R}$ is “translation-invariant” if $f(i,j)=f(i+\tau,j+\tau),\forall i\leq j,\forall\tau\geq 0$, and “local” if there is $\tau$ such that $f(i,j)=0$ when $j>i+\tau$.
A Limit Transformer satisfies
(1) Periodic if ${\bm{p}}_{i}={\bm{p}}_{i+\Delta}$ for all $i$ for some $\Delta>0$, and
(2) Local if each $\phi_{l,h}$ is translation-invariant and local.

Given a set $\Theta_{n}$ of transformers, the parameterization in terms of inner products permits a translation from a transformer $T\in\Theta_{n}$ to a Limit Transformer, whose width is bounded in terms of $\mathcal{R}(T)$, and which further satisfies Periodic and Local. This is formalized in Lemma [52](#theorem52) in the Appendix.

###### Definition 2 .

###### Definition 3 .

### 3.2 Definition of Inference Procedure

To define the inference procedure, we specify the following hypothesis class at each input length $n$:

###### Definition 4 (Hypothesis Class) .

For each $n=1,2,3,\dots$, define the hypothesis class $\Theta_{n}$ as the set of transformers $T$ (as defined in Section [2](#S2)) where
(1) $N(T)=n$,
(2) each parameter vector and matrix of $T$ is represented at $p$ bits of precision, for some $p\in\mathbb{N}$,
(3) each product function involving positional encodings is translation-invariant.
That is, every product function involving exactly one positional encoding is constant across positions, and for every $1\leq i,j,i+\Delta,j+\Delta\leq n$,

$$ ${\bm{p}}_{i}^{T}{\bm{M}}_{1}\dots\bm{M}_{k}{\bm{p}}_{j}={\bm{p}}_{i+\Delta}^{T}{\bm{M}}_{1}\dots\bm{M}_{k}{\bm{p}}_{j+\Delta}$ (7) $$

whenever ${\bm{M}}_{1}\dots\bm{M}_{k}$ is a product of parameter matrices linking the input layer.(^1^11Such as ${\bm{K}}_{1,h}^{T}{\bm{Q}}_{1,h}$, ${\bm{V}}_{2,h}^{T}{\bm{K}}_{3,h^{\prime}}^{T}{\bm{Q}}_{3,h^{\prime}}{\bm{V}}_{1,h^{\prime\prime}}$, and similar. See Appendix [F.2](#A6.SS2) for a formal definition.)

Note that the width $d$ of the transformers $T\in\Theta_{n}$ is unconstrained.
The most interesting requirement here is the third one: We ask that, while the positional encodings ${\bm{p}}_{i}$ will typically vary with position, their contributions to the transformer’s computations are offset-independent. This is a stronger requirement than for the input-output behavior to be offset-independent: we ask for the transformer’s “algorithm” itself to be the same across offsets.
This is a substantive condition, but we believe it to be a natural requirement in the context of length generalization.
Our inference procedure will use a regularizer $\mathcal{R}$ favoring simpler hypotheses. The following will be sufficient:

###### Definition 5 (Regularizer) .

Let $T\in\Theta_{n}$, thus $N(T)=n$.
Define $\mathcal{R}(T)$ as the sum of
(1) $L+H$;
(2) the precision $p$ used in Definition [4](#theorem4); the precision $p$ used for rounding logits and the output of $\exp(\cdot)$ (Section [2](#S2));
(3) $\max_{l,h}rank({\bm{V}}_{l,h})$;
(4) $\max_{l,h}\|{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}\|$; $\max_{l,h}\|{\bm{V}}_{l,h}\|$; $\max_{l}\|{\bm{A}}_{l}\|_{F}$, $\|{\bm{B}}_{l}\|_{F}$; $\|{\bm{U}}\|$;
(5) $\max_{i}\|{\bm{p}}_{i}\|_{2}$, $\max_{\sigma}\|{\bm{E}}_{\sigma}\|_{2}$, $\max_{l}\|{\bm{b}}_{l}\|_{2}$;
(6) the term

$$ $\sum_{l=1}^{L}\sum_{h=1}^{H}\sum_{j=1}^{N(T)}\left|{\bm{p}}_{1}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{p}}_{j}\right|^{2}$ (8) $$

The idea of ([8](#S3.E8)) is to discourage accidental attention between far-away positions that do not appear together during training, which could hamper length generalization.
Due to translation invariance, this term entails a bound on products for all pairs ${\bm{p}}_{i},{\bm{p}}_{j}$ ($i\leq j$) entering causal attention.
While such a regularizer is not part of standard training, standard initialization tends to lead to bounded values for ([8](#S3.E8)) when $d$ is large (Appendix [G.1](#A7.SS1)); it thus captures an implicit bias of standard initialization and training.
Importantly, the width $d$ does not explicitly enter ${\mathcal{R}}$; as a consequence, for any sufficiently large $C$, the number of transformers $T_{n}\in\Theta_{n}$ with ${\mathcal{R}}(T_{n})\leq C$ is infinite, simply because $d$ is not constrained.
Nonetheless, this regularizer will be sufficient for identification under our idealized inference procedure, which observes the input-output behavior of the target function $f$ on inputs of length $\leq\frac{n}{2}$ and selects a transformer $T$ with maximal context window $n$, $T\in\Theta_{n}$ that exactly fits that input-output behavior while minimizing the regularizer $\mathcal{R}(T)$:

###### Definition 6 (Inference Procedure) .

Given a function $f\in\mathcal{F}(\Sigma)$, the *Inference Procedure* obtains a sequence of transformers $T_{1},T_{2},\dots\in\Theta_{n}$ as follows. Define $U_{n}$ as the set of $T\in\Theta_{n}$ matching the behavior of $f$ on all inputs of length $\leq\frac{n}{2}$.
Then choose $T_{n}\in U_{n}$ such that

$$ $\mathcal{R}(T_{n})\leq\frac{1}{n}+\inf_{T\in U_{n}}\mathcal{R}(T)$ (9) $$

In ([9](#S3.E9)), we do not simply ask for minimizing the regularizer, as the set of elements of $U_{n}$ with $\mathcal{R}(T)$ smaller than a given value need not be finite and thus a minimum need not be attained by any $T_{n}$.
Importantly, we only ask $T_{n}$ to match the behavior of $f$ up to length $\frac{n}{2}$, formalizing the idea of training on shorter inputs and testing on longer ones; our identifiability guarantee will provide conditions under which $T_{n}$ will end up matching $f$ correctly up to length $n$ – representing length generalization.
While we take the testing length to be twice the training length, there is nothing special about this; our analysis works whenever the training length diverges to infinity.

###### Definition 4 (Hypothesis Class) .

###### Definition 5 (Regularizer) .

###### Definition 6 (Inference Procedure) .

### 3.3 Main Result: Convergence of Inference Procedure

Our main result asymptotically characterizes length generalization under the inference procedure from Definition [6](#theorem6).
For functions representable by Limit Transformers satisfying Local and Periodic, we guarantee that *any* run of the Inference Procedure will ultimately achieve length generalization, so that transformers with context length $n$ chosen to fit the target function on inputs with length $\leq\frac{n}{2}$ will, when $n$ is sufficiently large, also perform correctly at all lengths $\leq n$.
Formally,

###### Theorem 7 (Guaranteed Length Generalization in the Limit) .

Let $f\in\mathcal{F}(\Sigma)$.
Then the following are equivalent:

- 1.
$f$ is expressible by a Limit Transformer satisfying Periodic and Local.
- 2.
(Guaranteed Length Generalization)
Applying the Inference Procedure from Definition [6](#theorem6) to $f$ generates a sequence $T_{1},T_{2},\dots$ with $\sup_{n=1,2,3,\dots}{\mathcal{R}}(T_{n})<\infty$, for which there is some $N_{0}$ such that, for all $m>N_{0}$, $T_{m}$ matches $f$ on all inputs of any length $k\leq m$.

The formal proof is in Appendix [B.1](#A2.SS1).
Intuitively, if $f$ is expressible by a Limit Transformer satisfying Periodic and Local, then, even though the Inference Procedure produces infinitely many distinct *transformers* $T_{1},T_{2},\dots$, these can only traverse a finite set of underlying *algorithms*, each described by some Limit Transformer.
Periodic and Local ensure that the Limit Transformer’s parameter count effectively remains finite, as its position-related parameters can be fully specified in terms of ${\bm{p}}_{1},\dots,{\bm{p}}_{\Delta-1}$ and $\phi(1,1),\dots,\phi(1,\tau)$.
The regularizer bounds width, depth, and precision of the Limit Transformers; this keeps the set of algorithms traversed finite.
Each of these finitely many algorithms will either be ruled out at some input length $n$ or else match the behavior of $f$ at all input lengths.
At some finite $N_{0}$, only the latter type of algorithm remains; hence, transformers produced after this point will match the target function.
The proof also entails a result on NoPE length generalization: Applying the inference procedure to $f$ while constraining ${\bm{p}}_{i}\equiv 0$ will lead to length generalization when $f$ is expressible by a Limit Transformer where all ${\bm{p}}_{i}$ and all $\phi_{l,h}$ are zero (Corollary [18](#theorem18)).
While Theorem [7](#theorem7) guarantees length generalization from length $\frac{n}{2}$ to length $n$ for expressible problems, it does not rule out length generalization for inexpressible problems.
Such a statement becomes possible if we allow arbitrary scaling of training vs. testing lengths (Appendix [B.4](#A2.SS4)).
Besides length generalization guarantees, Limit Transformers are also useful in providing *expressiveness* results for transformers with absolute positional encodings (Appendix [C](#A3)).

###### Theorem 7 (Guaranteed Length Generalization in the Limit) .

## 4 Which Functions are Identifiable? Expressiveness of Limit Transformers and C-RASP

We have found that, if a target function $f$ is expressible by a Limit Transformer satisfying Periodic and Local, then Theorem [7](#theorem7) indicates length generalization under our Inference Procedure.
In order to understand the ramifications of this result, we now study what functions Limit Transformers can express – for these functions, Theorem [7](#theorem7) will then guarantee length generalization.

### 4.1 Simple Example: Induction Head

We consider the task of predicting the next token in proportion to the frequency at which different tokens had previously followed tokens matching the current one:

$$ $f(x_{1}\dots x_{N})_{i,\sigma}=\frac{\#\{k<i:x_{k}=x_{i},x_{k+1}=\sigma\}}{\#\{k<i:x_{k}=x_{i}\}}$ (10) $$

We can construct a Limit Transformer with two layers and one head, wherein the first layer $\phi(i,j)=1$ if $i+1=j$ and 0 0 else; each head copies the preceding symbol’s embedding.
In the second layer, attention focuses on positions with the same symbol. The transformer outputs next-token predictions in proportion to bigram frequencies in the context, up to approximation error $O(\frac{1}{n})$ (due to logit scaling).
Hence, Theorem [7](#theorem7) guarantees that the Inference Procedure will length-generalize on ([10](#S4.E10)), providing a length generalization guarantee for an induction head circuit (Olsson et al.,, ).
A special case of ([10](#S4.E10)) occurs when each symbol occurs exactly once; here, such an induction head circuit suffices to copy a string (), we thus obtain a length generalization guarantee for copying such strings (see Section [5](#S5)).

### 4.2 Length Generalization for C-RASP

We next present a large class of functions for which Theorem [7](#theorem7) guarantees length generalization.
We extend the C-RASP formalism (Yang and Chiang,, ) with positional information, and then show that any function defined by a C-RASP program is expressible by a Limit Transformer; hence, transformers will, by Theorem [7](#theorem7), length-generalize on those functions.
We first define C-RASP:

###### Definition 8 ( C-RASP ) .

Let $\Sigma$ be an alphabet, let $\Phi$ be a set of *unary relations* $\phi:\mathbb{N}\rightarrow\{0,1\}$, and let $\Psi$ be a set of *binary relations* $\psi:\mathbb{N}\times\mathbb{N}\rightarrow\{0,1\}$.
A $\textbf{C-RASP}[\Phi,\Psi]$ program $P$ is defined as a sequence $P_{1},\ldots,P_{k}$ of C-RASP operations. There are two sorts of operations:
Boolean-Valued Operations
Initial
$P(i):=Q_{\sigma}(i)$
for $\sigma\in\Sigma$
Boolean
$P(i):=\lnot P_{1}(i)$
$P(i):=P_{1}(i)\land P_{2}(i)$
Constant
$P(i):=\top$
Positional
$P(i):=\phi(i)$
for $\phi\in\Phi$
Comparison
$P(i):=C_{1}(i)\leq C_{2}(i)$
Count-Valued Operations
Counting
$C(i):=\textsc{{\#}}\left[j\leq i,\psi(i,j)\right]\;P(j)$
for $\psi\in\Psi\cup\{\top\}$
Conditional
$C(i):=P(i)\;\mathbf{?}\;C_{1}(i)\;\textbf{:}\;C_{2}(i)$
Addition
$C(i):=C_{1}(i)+C_{2}(i)$
Subtraction
$C(i):=C_{1}(i)-C_{2}(i)$
Constant
$C(i):=1$

A Counting operation returns the number of positions $j\leq i$ where $P(j)$ and $\psi(i,j)$ hold. A conditional operation returns $C_{1}(i)$ if $P(i)$, and $C_{2}(i)$ otherwise. We use the value of the *last* Boolean-valued operation, at the last position of the string, to determine acceptance using a C-RASP program. That is, if the program is run on input $w$ with final operation $L$, then we accept $w$ if and only if $L(|w|)$ is true.
$\textbf{C-RASP}[\text{periodic},\text{local}]$ is the class of C-RASP programs where each $\phi(i)$ is periodic in $i$, and each $\psi(i,j)$ is translation-invariant and local (Definition [3](#theorem3)). We also write $\textbf{C-RASP}[\text{periodic},\text{local}]$ for the class of all languages accepted by some $\textbf{C-RASP}[\text{periodic},\text{local}]$ program. As an example, we present a program recognizing $L=\Sigma^{*}ab\Sigma^{*}$ over $\Sigma=\{a,b\}$:

Any $\textbf{C-RASP}[\text{periodic, local}]$ program can be translated to a Limit Transformer with corresponding positional functions.
We say a Limit Transformer $T$ *accepts* an input if the value in the last dimension in the last position of the output is greater than 0 0, and rejects otherwise.

###### Theorem 9 .

For every $\textbf{C-RASP}[\Phi,\Psi]$ program $P$ with local functions $\Psi$ and periodic functions $\Phi$ there exists a Limit Transformer $T_{\infty}$ that satisfies Periodic and Local such that for all $w\in\Sigma^{*}$, $P$ accepts $w$ iff $T_{\infty}$ accepts $\$w$.
If $P$ uses no local or periodic relations, then $T$ requires no functions $\phi_{l,h}$ or positional encodings ${\bm{p}}_{i}$.

The proof is in Appendix [B.6](#A2.SS6).
As a consequence, the Inference Procedure will ultimately length-generalize on inputs from a function $f$ expressible by a $\textbf{C-RASP}[\text{periodic},\text{local}]$ program.
If the C-RASP program requires no positional functions (i.e., it is in $\textbf{C-RASP}[\emptyset]$), then length generalization will succeed even with NoPE transformers.
We establish that various functions are in C-RASP:

###### Theorem 10 .

Membership in the following languages is definable in $\textbf{C-RASP}[\emptyset]$: (1) MAJORITY, (2) DYCK-1, (3) $a^{n}b^{n}c^{n}$.

The proof is in Appendix [C.1](#A3.SS1).
By Theorem [9](#theorem9), these positive results translate into statements about length generalization under the Inference Procedure.
For these tasks, length generalization even with NoPE is empirically already well-documented (Bhattamishra et al.,, ).
Further, ${\textbf{C-RASP}[\text{local}]}$ can implement
versions of the Induction Head task from Section [4.1](#S4.SS1), see Appendix [C.2.1](#A3.SS2.SSS1)–[C.2.2](#A3.SS2.SSS2).
C-RASP also helps understand why transformers show varying abilities even on simple finite-state languages (Bhattamishra et al.,, ; Liu et al.,, ; ), a fact poorly understood theoretically.
For instance, we have the following:

###### Lemma 11 .

Consider the alphabet $\Sigma=\{a,b,e\}$.

- 1.
$PARITY:=b^{*}(ab^{*}ab^{*})^{*}\not\in{\textbf{C-RASP}[\text{periodic},\text{local}]}$
- 2.
$(aa)^{*}\in\textbf{C-RASP}[\text{periodic},\text{local}]$ and $(aa)^{*}\not\in\textbf{C-RASP}[\emptyset]$
- 3.
$\Sigma^{*}be^{*}\not\in\textbf{C-RASP}[\text{periodic},\text{local}]$
- 4.
$L\in\textbf{C-RASP}[\emptyset]$ for piecewise testable $L$

The proof builds on logics with majority quantifiers, whose expressiveness covers C-RASP (Appendix [C.3](#A3.SS3)).
Notably, all of these languages are recognizable by simple finite-state automata which are expressible by transformers (Liu et al.,, ), but empirical length generalization behavior differs in line with C-RASP expressiveness (Section [5](#S5)).
PARITY (1) has long been found difficult for transformers (e.g. Hahn,, ; Bhattamishra et al.,, ; Anil et al.,, ; Chiang and Cholak,, ; Delétang et al.,, ; Hahn and Rofin,, ).
Result (2) exemplifies the effect of different positional relations.
The language $\Sigma^{*}be^{*}$ (3) is a simple model of FlipFlop (Liu et al.,, ), a language on which transformers empirically struggle to generalize perfectly despite its simplicity for recurrent models (Liu et al.,, ; Sarrof et al.,, ). The class (4) is useful for determining the expressibility of languages in $\textbf{C-RASP}[\emptyset]$, as in Section [E.1.2](#A5.SS1.SSS2).

###### Definition 8 ( C-RASP ) .

###### Theorem 9 .

###### Theorem 10 .

###### Lemma 11 .

### 4.3 Limitations: Logarithmic Communication Complexity

Having shown that various functions are definable by Limit Transformers, we now provide a simple technique for showing that various functions are *not* definable by Limit Transformers.
Informally, any function satisfying the conditions in Theorem [7](#theorem7) has logarithmic communication complexity.
Formally:

###### Theorem 12 .

Let $T$ be a Limit Transformer satisfying Periodic and Local.
On an input $x\in\Sigma^{2N}$, assume Alice has access to $x_{1\dots N}$ and Bob has access to $x_{N+1\dots 2N}$.
Then Alice can communicate $C\log N$ bits to Bob, where $C$ depends on $T$ but not $N$, so that Bob can compute each activation in the second half, ${\bm{y}}_{i}^{(l)}$ ($N+1\leq i\leq 2N$).

The proof is in Appendix [B.3](#A2.SS3).
In principle, computing activations in the second half of the input requires full knowledge of the first half of the input, because positions in the second half can freely attend to positions in the first half.
In this situation, one would expect Bob to need full knowledge of $\frac{N}{2}$ input symbols from Alice’s part, exponentially more than the $C\log N$ claimed in the theorem.
This is indeed needed if $T$ performs the task of, say, checking if $x_{1\dots,N}$ and $x_{N+1\dots 2N}$ are identical.
However, if $T$ satisfies Periodic and Local, attention must largely be determined by the presence of tokens and token sequences; when an attention head’s behavior is determined by positional information, it can only focus its attention on a local neighborhood or equally distribute it over a periodic pattern. Intuitively, in such cases, the set of possible queries and keys can be grouped into a finite partitioning, of size bounded independently of $N$. It then suffices for Alice to communicate, for each possible group of keys, an aggregate of the value vectors at the positions where a matching key is computed.
The proof (Appendix [B.3](#A2.SS3)) formalizes this.
As a corollary:

###### Corollary 13 .

The following problems are not expressible by Limit Transformers satisfying Periodic and Local: (1) copying arbitrary strings, (2) addition of $n$-digit numbers.

This is proven in Appendix [B.3](#A2.SS3).
As a consequence, any run of the Inference Procedure on these functions will output solutions $T_{1},T_{2},T_{3},\dots$ for which the depth, number of heads, parameter norms or ranks, MLP dimensions, or precision $p$, must increase with the input length $n$; indeed, length generalization is empirically challenging for these functions (Section [5](#S5)).

###### Theorem 12 .

###### Corollary 13 .

## 5 Experiments

We evaluate the expressiveness of Limit Transformers and C-RASP as a predictor of empirical length generalization of NoPE and APE transformers.
Based on Theorems [7](#theorem7) and [9](#theorem9), we expect that APE transformers should length-generalize on problems with a C-RASP[periodic,local] program and that NoPE transformers will be successful in those cases where we found a $\textbf{C-RASP}[\emptyset]$ program.
We test this prediction on a suite of algorithmic problems and formal languages, largely taken from prior empirical work on length generalization (Bhattamishra et al.,, ; ), but evaluated within a uniform framework.

Figure: Figure 1: Experimental results (y-axis: accuracy), at lengths $\leq 50$ (Bin 1, training), $[51,100]$ (Bin 2), and $[101,150]$ (Bin 3, generalization), for APE (solid) and NoPE (dotted). Green lines indicate that we found a C-RASP program ($\textbf{C-RASP}[\text{periodic},\text{local}]$ for APE, $\textbf{C-RASP}[\emptyset]$ for NoPE), red lines indicate that we proved nonexistence, or found no program. Random baselines are indicated in gray in (left), and very close to zero in (right). On the algorithmic problems (left), we replicate prior empirical findings; C-RASP expressiveness predicts observed length generalization. On the regular languages (right, with same $x$ and $y$-axes as left, Table [2](#A5.T2)), length generalization tracks C-RASP expressiveness established in Lemma [11](#theorem11) ((1) = $(aa)^{*}$, (17) = $\Sigma^{*}be^{*}$) and other results (see Appendix [E.1](#A5.SS1)). C-RASP expressiveness performs much better than circuit complexity and standard notions of regular language complexity in predicting length generalization (Appendix, Figures [3](#A5.F3)–[4](#A5.F4)).
Refer to caption: /html/2410.02140/assets/x1.png

##### Setup

For each task, the model is trained on inputs whose Len is in the range $[l_{min},50]$, where $l_{min}$ is the minimum length for this task.
Len is the length of the input in the algorithmic tasks (Appendix [E.2](#A5.SS2)), and the overall sequence length in the formal language tasks.
The model is tested on 3 test sets, where Len is in the range $[l_{min},50]$, $[51,100]$, $[101,150]$; these lengths are based on the source of the regular languages benchmark (Bhattamishra et al.,, ).
We trained using a standard AdamW setup; see details in Appendix [E.3](#A5.SS3).
Hyperparameters are selected by searching in order of increasing complexity until we find a setting that performs well up to length $\leq 100$. We interpret results on lengths $[101,150]$ as a measure of length generalization.
Each model has as many positional encodings as needed to encode the longest inputs (at least 150); each input is presented with a random offset in agreement with the theoretical setup.
On algorithmic sequence-to-sequence tasks, we train with cross-entropy loss on the output.
On formal languages, where next-symbol predictions are generally not deterministic, we instead train the model to predict the *set of legal next symbols*, with each such set coded as an atomic symbol (as in Bhattamishra et al.,, ; Sarrof et al.,, ).
At test time, predictions are considered correct on a sequence if and only if the output at every step is correct; the random baseline is thus very low on the formal language benchmark.
We report accuracy, the fraction of test sequences where predictions are correct at every step.

##### Algorithmic Problems

We evaluate on 8 algorithmic problems, which largely overlap with , but are tailored to those where C-RASP expressiveness can be clearly settled.
Tasks are defined formally in Appendix [E.2.1](#A5.SS2.SSS1).
A new problem here is BINARY MAJORITY INTERLEAVE, which interleaves multiple MAJORITY functions and can be solved by C-RASP using periodic functions.
Length generalization behavior matches C-RASP expressiveness; $\textbf{C-RASP}[\emptyset]$ expressiveness predicts the success of NoPE (see Figure [1](#S5.F1)).
In agreement with prior empirical results (; Jelassi et al.,, ), COPY is difficult in the presence of repetition and easy when it is avoided; these findings match C-RASP expressiveness (Corollary [13](#theorem13) and Section [4.1](#S4.SS1)).

##### Formal Languages

We applied the experimental framework to 17 regular languages assembled by Bhattamishra et al., (), who evaluated length generalization in transformers and LSTMs.
Whereas LSTMs perform strongly across the board, the behavior of transformers on these regular languages has so far eluded theoretical understanding.
While it is known that transformers struggle with PARITY, it has remained unclear why they would struggle to length-generalize on some seemingly very simple languages.
We found C-RASP[periodic,local] programs for 13 of the languages and proved nonexistence for the others (Appendix [E.1.2](#A5.SS1.SSS2)).
Length generalization succeeded in those cases where we had found a C-RASP[periodic,local] program (see Figure [1](#S5.F1) right).
In those cases where a $\textbf{C-RASP}[\emptyset]$ program exists, generalization also succeeded with NoPE.
Generalization failed for languages where no C-RASP program exists, such as $\Sigma^{*}be^{*}$ (#17 in Figure [1](#S5.F1); Lemma [11](#theorem11)).

## 6 Discussion

Prior work has empirically found that transformers’ length generalization capabilities differ between tasks, but theoretical understanding has been lacking.
We have introduced a formal framework analyzing length generalization in an idealized inference procedure.
The framework explains what is common across the diverse tasks where prior research has empirically observed successful length generalization, in terms of expressiveness in two simple mathematical formalisms, Limit Transformers and C-RASP.
We also proved that various problems, on which length generalization is less successful empirically, are not expressible in one or both of these formalisms.
Beyond length generalization, the framework further sheds light on the expressiveness of APE transformers.
Our results on length generalization study an idealized regularizer and assume perfect fitting of the training distribution. Making the guarantee from Theorem [7](#theorem7) more realistic by incorporating SGD training dynamics and subsampling of training data is an interesting problem for future research.

Our results can be viewed as formalizing the RASP-L Conjecture ().
Both Limit Transformers and C-RASP[periodic,local] formalize intuitions underlying RASP-L in restricting how positional information can be used.
An important advance over is that we settle the expressiveness of these formalisms for many problems, and are able to explicitly prove a variety of problems with poor empirical length generalization, such as copying with repeated strings, to be inexpressible by Limit Transformers.
Our results provide a step towards rigorously confirming the idea that expressiveness in such restricted formalisms predicts length generalization.

##### Expressiveness of Transformers

A substantial line of research has studied the in-principle expressiveness of transformers (Strobl et al.,, ).
Transformers express a subset of the class $\textbf{TC}^{0}$ (; Strobl,, ), but it is unknown if this inclusion is proper.
*All* problems considered in Section [5](#S5) are in $\textbf{TC}^{0}$, but empirical length generalization behavior largely tracks C-RASP[periodic,local] expressiveness, which defines a proper subclass of $\textbf{TC}^{0}$ (Appendix [C.3.1](#A3.SS3.SSS1)).
While it remains open if the expressive power of transformers exhausts $\textbf{TC}^{0}$, our results suggest a separation between $\textbf{TC}^{0}$ and those problems for which length generalization is possible with absolute positional encodings.
In particular, our results suggest that the existence of APE transformers that perform a task across larger ranges of input lengths is linked to the expressiveness of Limit Transformers (Section [C](#A3)).
It is an open question how far new, yet-to-be-discovered positional encoding schemes may increase the range of length generalization;
empirical evidence indicates that NoPE and APE may be hard to beat by other general-purpose encodings (Kazemnejad et al.,, ).

The proof of Theorem [12](#theorem12) is closely linked to previous communication-complexity bounds for transformer layers (Sanford et al.,, ; ; Peng et al.,, ; Bhattamishra et al.,, ), which importantly were shown only for individual layers, not multilayer transformers.
Indeed, Bhattamishra et al., () showed that such a logarithmic bound is not in general possible for arbitrary multilayer transformers.
In contrast, our result applies even at multilayer models, which is enabled by the restrictions on the ways in which positional information can be used in a Limit Transformer.

##### Length Generalization of Transformers

Various studies have empirically evaluated length generalization in transformers.
Our work is most closely related to , discussed above.
Bhattamishra et al., () study length generalization on formal languages; we find that C-RASP[periodic,local] expressiveness explains behavior on their benchmark well (Section [5](#S5)).
Anil et al., () show that language models, finetuned on various reasoning problems, do not length-generalize well.
evaluate length generalization of NoPE transformers on real-world tasks.
Kazemnejad et al., () explore length generalization across different positional encoding schemes, finding NoPE to perform surprisingly well.
show that length generalization for addition improves with specific encoding schemes and input formats.
Jelassi et al., () show that transformers can succeed in length generalization on copying when inputs avoid $n$-gram repetition.
Chang and Bisk, () empirically find limitations in generalization in counting.
In contrast to the rich landscape of empirical studies, theoretical understanding of length generalization has been limited.
Most relevant, Ahuja and Mansouri, () study length generalization in simple neural architectures, including a one-layer transformer setup with linear (not softmax) attention. Our results, in contrast, apply to multi-layer softmax transformers and make statements about many concrete problems that have been studied empirically.
Some other works (e.g. Hou et al.,, ; Xiao and Liu,, ) provide length-generalizing constructions for certain problems but leave open whether learning would lead to such constructions.
show that GD training leads to length generalization on a specific token selection task.

##### Limitations

The main limitation of our results is that we study idealized asymptotic identification of a global minimum with perfect knowledge of behavior on the training distribution (cf. Sec. [6](#S6) and Q.4 in App. [A](#A1) for more discussion).
Extending Theorem [7](#theorem7) to account for subsampling of the training data and learning dynamics is an important problem for future research.
In particular, providing a practical upper bound on the threshold $N_{0}$ at which length generalization is expected is an interesting problem.
Our study focuses on absolute positional encodings; extending it to other positional encodings (e.g. Su et al.,, ; Press et al.,, ; Ruoss et al.,, ) is another important problem for future research.

## 7 Conclusion

We have introduced a theoretical framework that unifies a broad array of empirical findings about successes and failures of length generalization in transformers with absolute positional encodings.
Our framework is based on the analysis of an idealized inference procedure, for which length generalization provably happens whenever the ground-truth function is expressible with only limited access to positional information.
By providing upper and lower bounds on the expressiveness of these objects, we accurately predict the success and failure of length generalization across various algorithmic tasks and formal languages.

## Contributions

MH coordinated the project.
MH and XH developed the conceptual framework of Theorem [7](#theorem7), with input from the other authors.
XH and YS contributed Section [5](#S5).
AY contributed Section [4.2](#S4.SS2) with input from MH and AK.
MH, XH, AY, YS jointly developed the translation to Limit Transformers; MH worked out the formalization.
SB contributed Proposition [54](#theorem54), Lemma [56](#theorem56), and provided conceptual input throughout the project.
AK contributed to settling the C-RASP expressiveness of formal languages.
PN and HZ provided conceptual and writing-level input over the course of the project.
MH drafted the remaining portions of the paper and the proof of Theorem [7](#theorem7), including definitions and lemmas.

## Acknowledgments

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 232722074 – SFB 1102.
MH thanks Lena Strobl, Dana Angluin, David Chiang, Mark Rofin, Anthony Lin, and Georg Zetzsche for conversations on related topics.

## Appendix A FAQ

(1) What is the point of introducing Limit Transformers?

Limit Transformers are a mathematical formalism helping us prove a length generalization guarantee (Theorem [7](#theorem7)) for a broad class of functions, not just one specific function. They thus serve as an object that can help us prove things about standard transformers.

(2) What is the relation between Limit Transformers and C-RASP? Why use two different formalisms?

Limit Transformers are closely connected to standard transformers and provide a convenient formalism for formalizing a length generalization guarantee in our inference setup (Theorem [7](#theorem7)); they also provide bounds on APE transformer expressiveness as a side result (Appendix [C](#A3)).
C-RASP is a formalism based on the RASP language (Weiss et al.,, ), intended to provide a formal abstraction of the kinds of computations that transformers can perform in a human-readable format.
Limit Transformers with Periodic and Local express all the functions definable in C-RASP[periodic,local], though it is open if this inclusion is strict.
We provide rigorous tools for understanding the expressiveness of both formalisms.
For Limit Transformers, we prove a logarithmic communication complexity bound (Theorem [12](#theorem12)).
C-RASP brings additional use in understanding expressiveness from two angles.
First, one can conveniently prove functions expressible by writing down programs, as we did in Section [4.2](#S4.SS2).
Second, to prove negative results, we can bring to bear a set of deep results about logics using majority quantifiers (Krebs,, ), which allow us to settle the expressiveness of many problems provably.
Positive results translate into positive results about Limit Transformer expressiveness and hence, length generalization under our idealized learning setup.
While it is open if problems not expressible in C-RASP cannot in principle show length generalization, experimental results suggest that such an implication might hold in many cases.

(3) Why are Limit Transformers needed – can’t one just consider transformers whose parameters have infinite precision and hence can accommodate infinitely many different positional encodings ${\bm{p}}_{i}$?

The key advantage of Limit Transformers is that they effectively have finite parameter counts whenever they satisfy Local and Periodic, which is useful in establishing Theorem [7](#theorem7).
In an ordinary transformer, due to fixed width, effectively distinguishing unboundedly many positions requires infinitely many parameters ${\bm{p}}_{i}$.
Even then, a function as simple as $\phi(i,j)=\delta_{ij}$ cannot be exactly represented for infinitely many $i,j$ by a product ${\bm{p}}_{i}{\bm{Q}}^{T}{\bm{K}}{\bm{p}}_{j}$ at bounded width $d$.

(4) Why is the idealized setup considered for the analysis, as opposed to more practical frameworks of learning?

Proving guarantees in a more practical setting (SGD training dynamics, subsampling of training data) would, of course, be ideal.
However, such guarantees have been notoriously difficult to establish for deep learning models (Neyshabur et al.,, ). Standard frameworks for learning, such as PAC-learning, assume that the training and test distributions are the same, which precludes out-of-distribution guarantees such as length generalization. Even within the PAC-learning framework, obtaining nontrivial guarantees for deep neural networks remains challenging without making strong assumptions. Instead of analyzing the learning and generalization of Transformers trained with gradient-based methods, our work aims to understand the length generalization properties of Transformers from an architectural perspective. A substantial body of work (cf. Section [6](#S6)) has empirically investigated the length generalization properties of Transformers and found a complex array of empirical behavior, while theoretical understanding has been very limited. Hence, consolidating the theoretical relation between these empirical observations and the computational model of Transformers seems like an important direction. Our work provides a formal framework, based on an idealized model of learning, that separates the tasks on which Transformers succeed and those on which they fail to length-generalize.
The learning model considered in our work is closely related to the “identification in the limit” setting, which has been widely studied for decades in the context of learning automata and grammars (De la Higuera,, ).
Our framework is successful in explaining a wide range of empirical observations (Figure [1](#S5.F1)). This is a substantial advance, as no prior theoretical framework has been able to explain the empirical patterns in Figure [1](#S5.F1) to the extent that our framework can.
We hope that further work can build on these insights to establish guarantees that reproduce this success while narrowing the gap between theoretical analysis and practical learning.

(5) Why does the length generalization condition in Theorem [7](#theorem7) ask for $\sup_{n}\mathcal{R}(T_{n})<\infty$? Isn’t asking for length generalization sufficient?

If $\sup_{n}\mathcal{R}(T_{n})=\infty$, a transformer $T$ minimizing $\mathcal{R}(T)$ while fitting behavior at some length will be unlikely to work at substantially longer lengths because performing the task correctly at longer and longer lengths requires unbounded increase in $\mathcal{R}(T)$.
It might still happen that generalization from length $\frac{n}{2}$ to length $n$ is possible in certain problems not expressible by Limit Transformers.
However, this will depend on the problem and the specific scaling of test lengths relative to training lengths; for problems not satisfying the conditions in Theorem [7](#theorem7), length generalization will fail when the test length is made sufficiently longer than the training length, even as the training length diverges to infinity. We make this formal in Section [B.4](#A2.SS4).

(6) Given a task, how can one settle Limit Transformer and C-RASP expressiveness?

Showing that a task is definable by Limit Transformers or C-RASP simply requires providing an explicit construction, as we exemplify for various tasks (Section [C.1](#A3.SS1)).
For showing that a task is not definable in these formalisms, we provide a battery of methods that allow us to provide an answer for many tasks: communication complexity (Theorem [12](#theorem12)) applies to both formalisms; for showing non-definability in C-RASP, reduction to specific languages already proven not to be expressible (such as Parity and $L_{bb}$, see Appendix [E.1.2](#A5.SS1.SSS2)) is frequently useful.

(7) Why does the guarantee specifically apply to Periodic and Local Limit Transformers? What is special about such positional relations?

Local positional relations are important because, if a product function of the form ${\bm{p}}_{i}^{T}{\bm{Q}}^{T}{\bm{K}}{\bm{p}}_{j}$, where the rank of ${\bm{Q}},{\bm{K}}$ is not constrained, takes nonzero values at unboundedly long distances $j-i$, there is no general reason why the function should length-generalize.
Independent initialization of the ${\bm{p}}_{i}$’s tends to lead to values close to zero for most of these products (Appendix [G.1](#A7.SS1)); our Inference Procedure incorporates this via the term ([8](#S3.E8)).
Given this, one expects a learned model to still exhibit small products at distances not present in the training distribution, and hence a failure of length generalization in the presence of nonlocal product functions.

The situation is different for products involving ${\bm{V}}_{l,h}$ matrices, whose rank is penalized by $\mathcal{R}(\cdot)$; these are able to represent not local, but periodic functions. In the finite-precision setup, a translation-invariant product function of the form ${\bm{p}}_{i}^{T}{\bm{M}}_{1}\dots\bm{M}_{k}{\bm{p}}_{j}$ must be periodic in $j-i$ whenever one of the matrices ${\bm{M}}_{1}\dots\bm{M}_{k}$ has bounded rank as the number of positions considered diverges to infinity, with period bounded in terms of the rank (Lemma [48](#theorem48)). Hence, in a transformer $T\in\Theta_{n}$, any product function involving one or more ${\bm{V}}_{l,h}$ matrices needs to be periodic with period bounded in terms of $\mathcal{R}(T)$.

## Appendix B Proofs about Limit Transformers

### B.1 Proof of Theorem 7

We re-state and then prove Theorem [7](#theorem7):

###### Theorem 14 (Guaranteed Length Generalization in the Limit, restated from Theorem 7 ) .

Let $f\in\mathcal{F}(\Sigma)$.
Then the following are equivalent:

- 1.
$f$ is expressible by a Limit Transformer satisfying Periodic and Local.
- 2.
(Guaranteed Length Generalization)
Consider the inference procedure from Definition [6](#theorem6) applied to $f$ with ${\mathcal{R}}$, generating a sequence $T_{1},T_{2},\dots$.
For *any* such sequence, there is some $N_{0}$ such that, for all $m>N_{0}$, $T_{m}$ matches $f$ on all inputs of any length $k\leq m$, and $\sup_{n=1,2,3,\dots}{\mathcal{R}}(T_{n})<\infty$.

###### Remark 15 .

We note that a limit transformer $T_{\infty}$ representing $f$ need not itself be offset-invariant. It is sufficient to have

$$ $T_{\infty}(x,0)=f(x)$ (6) $$

Lemma [47](#theorem47) shows that such a function has a sequence of transformers $T_{n}\in\Theta_{n}$ which are offset-invariant, even without assuming $T_{\infty}$ to be offset-invariant.

###### Theorem 14 (Guaranteed Length Generalization in the Limit, restated from Theorem 7 ) .

###### Remark 15 .

##### High-Level Proof Sketch

The key to the proof is a compactness property: Any sequence $T_{1},T_{2},\dots$ ($T_{i}\in\Theta_{i}$) where $\sup_{i}\mathcal{R}(T_{i})<\infty$ has a subsequence of transformers whose behavior across inputs can be summarized into a single Limit Transformer.
For 1$\Rightarrow$2, given a sequence generated by the Inference Procedure, we show that ${\mathcal{R}}$ stays bounded and use the compactness property to show that a subsequence exhibits behavior equivalent to $f$.
To show that, in fact, *all* possible sequences $T_{n}$ generated by the Inference Procedure ultimately exhibit behavior equivalent to $f$, when $n$ is large, we show that subsequences failing to length-generalize would exhibit increasing attention dot products between far-away positions as input length increases. However, due to the penalty on attention dot products in ${\mathcal{R}}$, any such sequence would, for large $n$, need to have a higher value of ${\mathcal{R}}$ than sequences avoiding such an increase.
For 2$\Rightarrow$1, we obtain the Limit Transformer from the compactness property applied to the sequence generated by the Inference Procedure. The penalty on attention dot products enforces that it satisfies Local; the bounds on the MLP and value matrices enforce that the positional encodings in the Limit Transformer can be taken to be periodic.

##### Preliminaries and Formal Proof

We now proceed to the formal proof. We make crucial use of the two technical Lemmas [47](#theorem47) and [52](#theorem52), which provide translations between ordinary transformers and Limit Transformers.

The following definition will be used:

###### Definition 16 .

If $T\in\Theta_{i}$, then define $\mathcal{R}_{-}(T)$ to be $\mathcal{R}(T)$ minus the term in Eq. ([8](#S3.E8)). That is,

$$ $\mathcal{R}(T)=\mathcal{R}_{-}(T)+\sum_{l,h}\sum_{1\leq j\leq N(T)}|{\bm{p}}_{1}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{p}}_{j}|^{2}$ (7) $$

The following lemma will be used for both directions of the main theorem:

###### Lemma 17 .

Let $T_{1},T_{2},\dots$, where $T_{n}\in\Theta_{n}$, be a sequence generated by the Inference Procedure based on the functional behavior of a function $f\in\mathcal{F}$, and such that

$$ $\sup_{n=1,2,3,\dots}{\mathcal{R}}(T_{n})<\infty$ (8) $$

Then $f$ is expressible by a Limit Transformer satisfying Periodic and Local, and there is some $N_{0}$ such that, for all $m>N_{0}$, $T_{m}$ matches $f$ on all inputs of length $k\leq m$.

###### Proof.

From the sequence $T_{1},T_{2},\dots$ generated by the Inference Procedure, we obtain, using Lemma [52](#theorem52), Limit Transformers $\tilde{T}_{1},\tilde{T}_{2},\dots$ such that $\sup_{i}\mathcal{R}_{\infty}(\tilde{T}_{i})<\infty$
where

$$ $\tilde{T}_{i}(x,o)=T_{i}(x,o),\ \ \ \ \forall i,o,x;|x|+o\leq i$ (9) $$

and, in each $T_{n}$, $\tilde{T}_{n}$,

$$ $\phi_{l,h}(i,j)={\bm{p}}_{i}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{p}}_{j}$ (10) $$

Due to $\sup_{i}\mathcal{R}_{\infty}(\tilde{T}_{i})<\infty$, we know that, except for the functions $\phi_{l,h}$, only a finite number of Limit Transformer parameter settings will be traversed by $\tilde{T}_{i}$.
Each function $\phi_{l,k}$ is local; however, a priori, they might not be local for any single finite $\tau$ across the different $\tilde{T}_{n}$.
We will show that this is not possible, i.e., we will show that all $\phi_{l,k}$ are local for a single finite $\tau$ across the different $\tilde{T}_{n}$.
This will occupy us for the remainder of the proof.

First, we note that $\mathcal{R}(T_{n})$ converges because $\inf_{T\in U_{n}}\mathcal{R}(T)$ is bounded and monotonically increasing in $n$.
For each $\tau$ and each $n$, we consider

$$ $\displaystyle D_{n}(\tau)$ $\displaystyle=\sum_{l,h}\sum_{i=1}^{\tau}|\phi_{l,h}(1,i)|^{2}\leq{\mathcal{R}}(T_{n})$ $$

where the $\phi_{l,h}$ function is taken from $\tilde{T}_{n}$ when defining $D_{n}$.

Consider $\mathcal{R}_{-}(T_{n})$ (Equation [7](#A2.E7)).
Let

$$ $R_{0}:=\liminf_{n\rightarrow\infty}\mathcal{R}_{-}(T_{n})$ (11) $$

and let $\nu_{1},\nu_{2},\nu_{3},\dots$ be such that

$$ $\lim_{i\rightarrow\infty}\mathcal{R}_{-}(T_{\nu_{i}})=R_{0}$ (12) $$

Then, for some $D_{0}$,

$$ $\lim_{i\rightarrow\infty}D_{\nu_{i}}(\nu_{i})=D_{0}$ (13) $$

and

$$ $\lim_{n\rightarrow\infty}\mathcal{R}(T_{n})=\lim_{i\rightarrow\infty}\mathcal{R}(T_{\nu_{i}})=R_{0}+D_{0}$ (14) $$

Indeed,

$$ $D_{0}=\limsup_{n\rightarrow\infty}D_{n}(n)$ (15) $$

because(^2^22In general, if $a_{n}+b_{n}$ converges and $a_{n},b_{n}$ are bounded, then the limit $\lim(a_{n}+b_{n})$ equals $\limsup a_{n}+\liminf b_{n}$. For, assume $\limsup a_{n}+\liminf b_{n}>\lim(a_{n}+b_{n})$ (similar if $>$ is replaced by $<$). Then let $i(n)$ be a subsequence such that $a_{i(n)}\rightarrow\limsup a_{n}$. Then $\lim(a_{n}+b_{n})=\lim(a_{i(n)}+b_{i(n)})=\limsup a_{n}+\lim b_{i(n)}\geq\limsup a_{n}+\liminf b_{n}>\lim(a_{n}+b_{n})$, contradiction.)

$$ $D_{0}+R_{0}=\lim_{n\rightarrow\infty}(\mathcal{R}_{-}(T_{n})+D_{n}(n))=\liminf_{n\rightarrow\infty}\mathcal{R}_{-}(T_{n})+\limsup_{n\rightarrow\infty}D_{n}(n)$ (16) $$

Define, for each $\tau\in\mathbb{N}$,

$$ $D_{\infty}(\tau)=\liminf_{i\rightarrow\infty}D_{\nu_{i}}(\tau)$ (17) $$

As this function is monotonically increasing, and as $\phi_{l,h}$ has bounded precision, there must be $\tau_{\infty}$ such that $D_{\infty}(\tau_{\infty})=\lim_{\tau\rightarrow\infty}D_{\infty}(\tau)$.

Now define a sequence $T_{n}^{\prime}$ by selecting, for each $n$, an $i(n)$ such that $\nu_{i(n)}\geq n$ and

$$ $\displaystyle D_{\nu_{i(n)}}(n)=\liminf_{j\rightarrow\infty}D_{\nu_{j}}(n)=D_{\infty}(n)\leq D_{\infty}(\tau_{\infty})$ $$

(with equality when $\nu_{j}\geq\tau_{\omega}$); then define $T_{n}^{\prime}$ as the restriction of $T_{\nu_{i(n)}}$ to positions up to $n$.
As $T_{\nu_{i(n)}}$ agrees with the behavior of $f$ up to length $\frac{\nu_{i(n)}}{2}\geq\frac{n}{2}$, we also find that $T_{n}^{\prime}$ agrees with the behavior of $f$ up to length $\frac{n}{2}$.
Then

$$ $\displaystyle\limsup_{n\rightarrow\infty}\mathcal{R}(T_{n}^{\prime})=$ $\displaystyle\limsup_{n\rightarrow\infty}\mathcal{R}_{-}(T_{n}^{\prime})+D_{\nu_{i(n)}}(n)$ $\displaystyle=$ $\displaystyle\limsup_{n\rightarrow\infty}\mathcal{R}_{-}(T_{\nu_{i(n)}})+D_{\infty}(\tau_{\infty})$ $\displaystyle=$ $\displaystyle R_{0}+D_{\infty}(\tau_{\infty})$ $$

Since $T_{n}$ was created by the Inference Procedure, we have

$$ $\limsup_{n\rightarrow\infty}\mathcal{R}(T_{n}^{\prime})\geq\lim_{n\rightarrow\infty}\mathcal{R}(T_{n})$ (18) $$

On the other hand, since $\mathcal{R}(T_{n}^{\prime})\leq\mathcal{R}(T_{\nu_{i(n)}})$, we also have

$$ $\limsup_{n\rightarrow\infty}\mathcal{R}(T_{n}^{\prime})\leq\lim_{n\rightarrow\infty}\mathcal{R}(T_{n})$ (19) $$

giving

$$ $\limsup_{n\rightarrow\infty}\mathcal{R}(T_{n}^{\prime})=\lim_{n\rightarrow\infty}\mathcal{R}(T_{n})=D_{0}+R_{0}$ (20) $$

Hence,

$$ $\displaystyle R_{0}+D_{\infty}(\tau_{\infty})=$ $\displaystyle\limsup_{n\rightarrow\infty}\mathcal{R}(T_{n}^{\prime})$ $\displaystyle=$ $\displaystyle\lim_{n\rightarrow\infty}\mathcal{R}(T_{n})$ $\displaystyle=$ $\displaystyle R_{0}+D_{0}$ $$

and $D_{\infty}(\tau_{\infty})=D_{0}$.
Now assume there are infinitely many $n$ such that $\phi_{l,h}$ is not $\tau_{\infty}$-local in $T_{n}$, hence, infinitely many $n$ such that $D_{n}(n)\geq D_{n}(\tau_{\infty})+2^{-2p}$. Then:

$$ $D_{0}=\limsup_{i\rightarrow\infty}D_{n}(n)\geq\limsup_{n\rightarrow\infty}D_{n}(\tau_{\infty})+2^{-2p}\geq\liminf_{i\rightarrow\infty}D_{\nu_{i}}(\tau_{\infty})+2^{-2p}=D_{0}+2^{-2p}$ (21) $$

This is a contradiction.

We thus have shown that the functions $\phi_{l,k}(\tilde{T}_{n})$ must be local for a uniform $\tau_{\infty}$.
We thus know that the sequence $\tilde{T}_{i}$ only traverses a finite set of possible Limit Transformers.
The set of traversed functions becomes stationary at some $i=N_{0}$; all of these must be functionally equivalent to $f$.
Hence, $T_{i}$ is functionally equivalent to $f$ at all lengths $\leq i$ as soon as $i$ exceeds some threshold $N_{0}$.
∎

We now prove the theorem.

###### Proof of the Theorem.

Both directions are corollaries of Lemma [17](#theorem17).

###### Definition 16 .

###### Lemma 17 .

###### Proof.

###### Proof of the Theorem.

##### 2 ⇒ ⇒ \Rightarrow 1:

This directly follows from Lemma [17](#theorem17).

##### 1 ⇒ ⇒ \Rightarrow 2:

By Lemma [47](#theorem47), for each $i=1,2,3,\dots$, there are $\widehat{T}_{i}\in\Theta_{i}$ such that $R:=\sup_{i}\mathcal{R}(\widehat{T}_{i})<\infty$ such that

$$ $\widehat{T}_{i}(x,o)=f(x,o),\ \ \ \ \forall i,o,x;|x|+o\leq i$ (22) $$

such that

$$ ${\bm{p}}_{i}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{p}}_{j}=\phi_{l,h}(i,j)$ (23) $$

By Local,

$$ ${\mathcal{R}}(\widehat{T}_{i})<\infty$ (24) $$

and we conclude

$$ $\limsup_{i\rightarrow\infty}{\mathcal{R}}(T_{i})\leq\limsup_{i\rightarrow\infty}{\mathcal{R}}(\widehat{T}_{i})<\infty$ (25) $$

Lemma [17](#theorem17) now provides $N_{0}>0$ and a function $g$ such that for all $m>N_{0}$,

$$ $T_{m}(x,o)=g(x),\forall x:|x|+o\leq m$ (26) $$

On the other hand, for any string $x\in\mathfrak{S}$, we have

$$ $f(x)=T_{n}(x,0),\forall n\geq 2|x|$ (27) $$

Hence, $f\equiv g$ and for all $m>N_{0}$,

$$ $T_{m}(x,o)=f(x),\forall x:|x|+o\leq m$ (28) $$

∎

### B.2 Result for NoPE Transformers

###### Corollary 18 .

For ease of the reader, we mark the differences to Theorem [7](#theorem7) in blue font.

Let $f\in\mathcal{F}(\Sigma)$.
Then the following are equivalent:

- 1.
$f$ is expressible by a Limit Transformer satisfying where all ${\bm{p}}_{i}\equiv 0$, $\phi_{l,h}\equiv 0$.
- 2.
(Guaranteed Length Generalization)
Consider the inference procedure from Definition [6](#theorem6) applied to $f$ with ${\mathcal{R}}$ while constraining all ${\bm{p}}_{i}\equiv 0$, generating a sequence $T_{1},T_{2},\dots$.
For *any* such sequence, there is some $N_{0}$ such that, for all $m>N_{0}$, $T_{m}$ matches $f$ on all inputs of any length $k\leq m$, and $\sup_{n=1,2,3,\dots}{\mathcal{R}}(T_{n})<\infty$.

###### Proof.

Retracing the proof of Lemma [47](#theorem47) shows that, when translating a Limit Transformer to an ordinary transformer, the positional encodings can be taken to be zero when ${\bm{p}}_{i}\equiv 0$, $\phi_{l,h}\equiv 0$ in the Limit Transformer.
Retracing the proof of Lemma [52](#theorem52) shows that, when ${\bm{p}}_{i}\equiv 0$ in a transformer, the resulting Limit Transformer will have zero positional encodings and zero outputs for all $\phi_{l,h}$.
The proof of Theorem [7](#theorem7) then applies equally to show Corollary [18](#theorem18).
∎

###### Corollary 18 .

###### Proof.

### B.3 Logarithmic Communication Complexity for Limit Transformer

###### Theorem 19 (Restated from Theorem 12 ) .

Let $T$ be a Limit Transformer satisfying Periodic and Local.
Assume that $T$ operates in precision $O(\log N)$, i.e., attention weights are rounded to $O(\log N)$ precision.
On an input $x\in\Sigma^{2N}$, assume Alice has access to $x_{1\dots N}$ and Bob has access to $x_{N+1\dots 2N}$.
There is a communication protocol in which Alice and Bob exchange at most $C\log N$ bits, where $C$ depends on $T$ but not $N$ or $x$, and Bob can compute each activation in the second half, ${\bm{y}}_{i}^{(l)}$ ($N+1\leq i\leq 2N$).
Further, $C$ is bounded linearly by $\mathcal{R}_{\infty}(T)$.

###### Proof.

First, note that all activations ${\bm{y}}_{i}^{(l)}$ are computed at $\log N$ precision because parameters are at fixed precision and the output of $\exp(\cdot)$ in the softmax attention computation is computed at fixed fractional precision.
We first consider the attention logits, in the case where $j<N\leq i$:

$$ $\displaystyle a_{i,j}^{(l,h)}=\operatorname{Round}_{p}[({\bm{y}}^{(l-1)}_{j})^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{y}}^{(l-1)}_{i}+\phi_{l,h}(i,j)]$ $$

where $\operatorname{Round}_{p}[\dots]$ rounds each entry to the closest number with $p$ fractional bits. It is certainly sufficient to have access to

$$ $\displaystyle a_{i,j}^{(l,h)}=\left(\operatorname{Round}_{p^{\prime}}[{\bm{y}}^{(l-1)}_{j}]\right)^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{y}}^{(l-1)}_{i}+\operatorname{Round}_{p^{\prime}}[\phi_{l,h}(i,j)]$ $$

where $p^{\prime}$ depends on $p$ and the largest singular value of ${\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}$, which is a finite constant.
We can thus partition the positions $j=1,\dots,N-1$ into a bounded number of sets, indexed by

- 1.
$\operatorname{Round}_{p^{\prime}}[{\bm{y}}^{(l-1)}_{j}]$
- 2.
$\max(N-j,N-L)$ where $L=\max\{k:\phi_{l,h}(1,k)\neq 0\}$.

Due to the finite precision rounding of logits and the locality of positional relations, we can maintain a finite set of keys and queries (though not values). This is fundamental to getting a logarithmic communication bound.

We show the claim by induction over the layers.

We can write

$$ $\displaystyle{\bm{Y}}_{i}^{(l)}=$ $\displaystyle{\bm{y}}_{i}^{(l-1)}+\sum_{h=1}^{H}\frac{\sum_{j=1}^{i}\exp(\log|x|\cdot a^{(l,h)}_{i,j}){\bm{V}}_{l,h}{\bm{y}}_{j}^{(l-1)}}{\sum_{j=1}^{i}\exp(\log|x|\cdot a_{i,j})}$ $$

The residual stream is known to Bob by inductive hypothesis.
We need to understand the term inside the sum. The green terms are fully known to Alice, and the blue ones are fully known to Bob by inductive hypothesis:

$$ $\displaystyle\frac{\sum_{j=1}^{i}\exp(\log|x|\cdot a^{(l,h)}_{i,j}){\bm{V}}_{l,h}{\bm{y}}_{j}^{(l-1)}}{\sum_{j=1}^{i}\exp(\log|x|\cdot a_{i,j})}$ $\displaystyle=$ $\displaystyle\frac{\color[rgb]{0,0.69921875,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.69921875,0}\sum_{j=1}^{N-1}\exp(\log|x|\cdot a^{(l,h)}_{i,j}){\bm{V}}_{l,h}{\bm{y}}_{j}^{(l-1)}}{{\color[rgb]{0,0.69921875,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.69921875,0}\sum_{j=1}^{N-1}\exp(\log|x|\cdot a_{i,j})}+{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\sum_{j=N}^{i}\exp(\log|x|\cdot a_{i,j})}}$ $\displaystyle+\frac{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\sum_{j=N}^{i}\exp(\log|x|\cdot a^{(l,h)}_{i,j}){\bm{V}}_{l,h}{\bm{y}}_{j}^{(l-1)}}{{\color[rgb]{0,0.69921875,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.69921875,0}\sum_{j=1}^{N-1}\exp(\log|x|\cdot a_{i,j})}+{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\sum_{j=N}^{i}\exp(\log|x|\cdot a_{i,j})}}$ $$

Alice can communicate the green terms for every set in the partitioning of the indices $j<N$ defined above.
In fact, it is sufficient to communicate the number of relevant positions and the sum of the vectors ${\bm{V}}_{l,h}{\bm{y}}_{j}^{(l-1)}$.
∎

###### Corollary 20 (Restated from Corollary 13 ) .

The following problems are not expressible by Limit Transformers satisfying Periodic and Local: (1) copying strings with repeated n-grams, (2) addition of $n$-digit numbers.

###### Proof.

Formally, we define *copying* as the task of, given a prefix $\$x\#$, autoregressively predicting $x$.
Copying *with repeated n-grams* means that there is no restriction on the repetition of consecutive subspans of $x$ of any length; this is in contrast to copying tasks with restrictions on the repetition of n-grams (for some $n$) in $x$ (Jelassi et al.,, ; ), which we study separately (Appendix [E.2](#A5.SS2)).

Formally, we define *addition* as the task of, given a prefix $\$x+y=$, where $x,y$ are binary strings, to output the sum of the numbers denoted by $x,y$ in binary.

The communication complexity lower bound for copying follows from a standard communication complexity lower bound for determining string equality.
The bound follows for addition since the special case of adding 0 to a number amounts to copying.
∎

###### Remark 21 .

Analogous bounds follow for various other algorithmic and formal language problems. For instance, the special case of multiplying with 1 amounts to copying; hence, such a bound holds for multiplication.
For the unbounded-depth Dyck over two bracket types, we can consider a word of the form $(_{i_{1}}\dots(_{i_{N}})_{j_{N}}\dots)_{j_{1}}$, which is in the Dyck language if and only if $i_{k}=j_{k}$ for all $k$, again allowing a reduction to the communication complexity lower bound for determining string equality.

###### Theorem 19 (Restated from Theorem 12 ) .

###### Proof.

###### Corollary 20 (Restated from Corollary 13 ) .

###### Proof.

###### Remark 21 .

### B.4 Statement of Main Theorem for Arbitrary Training Lengths

Our main theorem considers generalization from length $\frac{n}{2}$ to length $n$.
Here, we discuss an alternative version applying to arbitrary scaling of training vs testing lengths.
In particular, in such a setup, we explicitly obtain *failure of length generalization for inexpressible functions*, though potentially requiring testing on lengths more than twice the lengths used in training.
We use the following definition:

###### Definition 22 .

A *training length* is a function $t:\mathbb{N}\rightarrow\mathbb{N}$ satisfying $\lim_{t\rightarrow\infty}t(n)=+\infty$ and $t(n)\leq n$ for all $n$.

If $t(n)$ is a training length, then the $t(n)$-Inference Procedure determines $T_{n}\in\Theta(n)$ to match $f$ at all inputs of lengths $\leq t(n)$ while minimizing $\mathcal{R}(T_{n})$ up to $\frac{1}{n}$.

The special case of $t(n)=\frac{n}{2}$ is the Inference Procedure from Definition [6](#theorem6).

We then state:

###### Theorem 23 .

Let $f\in\mathcal{F}(\Sigma)$.
The following are equivalent:

- 1.
$f$ is expressible by a Limit Transformer satisfying Periodic and Local.
- 2.
Let $t(n)$ be any training length. Then the $t(n)$-Inference Procedure will output solutions $T_{1},T_{2},\dots$ such that, for some $N_{0}$, for all $m>N_{0}$, $T_{m}$ matches $f$ at all lengths $\leq m$.
*Intuitively, this says that, when selected to fit the behavior of $f$ on sufficiently long inputs of length $t(n)$, the output of the Inference Procedure will generalize to unboundedly longer inputs of length $n$, where $n$ can be arbitrarily larger than $t(n)$.*

###### Corollary 24 .

Assume $f\in\mathcal{F}(\Sigma)$ is not expressible by a Limit Transformer satisfying Periodic and Local.
Then, for some training length $t(n)$, the $t(n)$-Inference Procedure outputs a sequence $T_{n}$ where infinitely many $T_{n}$ fail to match $f$ at length $n$.

###### Remark 25 .

There are two important differences compared to Theorem [7](#theorem7).
First, the second condition refers to length generalization for all arbitrary training lengths $t(n)$, not specifically training length $\frac{n}{2}$.
Second, the second condition does not ask for $\sup_{i}\mathcal{R}(T_{i})<\infty$, but simply asks for $T_{n}$ to ultimately length generalize.

###### Proof of Theorem 23 .

1$\Rightarrow$2
The proof of Theorem [7](#theorem7) remains valid in this direction without any changes, as it does not specifically rely on the training lengths being half the overall context size.

2$\Rightarrow$1
We show the contrapositive.
Assume $f$ is not expressible by a Limit Transformer satisfying Periodic and Local.
Then, using the same arguments as in the proof of Lemma [17](#theorem17)(^3^33Assume there is a sequence $T_{n}\in\Theta_{n}$ that matches $f$ and has $\liminf_{n\rightarrow\infty}\mathcal{R}(T_{n})<\infty$. Translating each element to a Limit Transformer leads to a sequence where, except perhaps for the functions $\phi_{l,h}$, only a finite number of settings will be traversed. Now, as in the proof of Lemma [17](#theorem17), one can use $D_{\infty}(\tau)$ to construct a sequence of Limit Transformers that are local for a single $\tau$. The important difference to Lemma [17](#theorem17) is that here we are not assuming the sequence $(T_{n})_{n}$ to be constructed by the inference procedure, but we nonetheless obtain such a sequence.), any sequence $T_{n}\in\Theta_{n}$ that matches $f$ will have $\liminf_{n\rightarrow\infty}\mathcal{R}(T_{n})=\infty$ ($\dagger$).
Now consider $k\in\mathbb{N}$; we will assign every $k$ a number $n_{k}>k$, starting with $n_{0}=0$.
For each $n>k$, there is $\hat{T}_{k,n}\in\Theta_{n}$ that matches $f$ up to length $k$ while $U_{k}:=\sup_{n}\mathcal{R}(\hat{T}_{k,n})<\infty$ for every fixed $k$.
Now select $n_{k}>n_{k-1}$ such that no $T\in\Theta_{n_{k}}$ with $\mathcal{R}(T)\leq U_{k}+1$ matches $f$ at length $n_{k}$; this is possible because of ($\dagger$).
We thus obtain a sequence $(k,n_{k})\in\mathbb{N}\times\mathbb{N}$.
By construction, there are infinitely many distinct different values $n_{k}$.
Then define

$$ $t(n):=\max\left(\{k:n_{k}\leq n\}\right)$ (29) $$

Then $t(n)$ is a training length.
By definition, the $t(n)$-Inference Procedure will, whenever $n$ is one of the $n_{k}$’s, find a transformer $T_{n_{k}}$ with $\mathcal{R}(T_{n_{k}})\leq U_{k}+\frac{1}{n_{k}}$ that fails to match $f$ at length $n=n_{k}$.
∎

###### Definition 22 .

###### Theorem 23 .

###### Corollary 24 .

###### Remark 25 .

###### Proof of Theorem 23 .

### B.5 Corollary about Expressivity

We have introduced Limit Transformers as a formalism for distilling computations of transformers performing on longer and longer sequences into a single limiting object, helping understand length generalization. Here, we show that they also provide a simple lower bound for the expressiveness of causal transformers across input lengths:

###### Corollary 26 .

Let $f\in\mathcal{F}(\Sigma)$.
Assume $f$ is expressible by a Limit Transformer satisfying Periodic and Local.
Then at each input length $N$, there exists a transformer $T_{N}$ performing $f$ on all inputs of length up to $N$ such that:

- 1.
The parameters of $T_{N}$ are expressed at $p$ bit precision, with $p$ independent of $N$
- 2.
The number of heads and layers of $T_{N}$ is bounded independently of $N$.
- 3.
The width $d$ of $T_{N}$ is bounded as $O(N)$.

We note that an important aspect is that $T_{N}$ performs correctly not just at length $N$, but at *all lengths up to* $N$. This distinguishes the result from constructions guaranteeing the existence of a transformer at a fixed length. For instance, Bhattamishra et al., () provide a transformer for testing equality between length $N$-strings (which could also be used for copying), but this construction uses specific positional encodings that depend on the input length.
In contrast, the result here provides conditions under which a transformer can perform a task at all lengths up to a given bound; in this stronger setup, no APE transformer for copying with uniform complexity bounds as provided by Corollary [26](#theorem26) is known, and the problem is indeed not expressible by Limit Transformers satisfying Periodic and Local (Corollary [13](#theorem13)).
In contrast, Corollary [26](#theorem26) provides APE constructions performing correctly *up to* any given length for a wide class of problems including C-RASP[periodic,local].

Another important feature is that the construction provides a fixed precision for the parameters, as is the case in real-world implementations.
We note that, if parameters are at fixed precision, it is generally not possible to find a single transformer across all input lengths in the APE setting; hence, it is unavoidable that the width of the transformers will need to increase as the input length increases.
Importantly, many other aspects of the transformer’s complexity, such as the number of heads and layers, remain bounded.

###### Proof.

The statement is an immediate corollary of Lemma [47](#theorem47), which provides transformers $T_{1},T_{2},\dots$ with bounded $\mathcal{R}(T_{N})$, which by Definition [5](#theorem5) entails a uniform bound on precision, heads, and layers. The construction provided in the proof of Lemma [47](#theorem47) provides a width bounded as $O(N)$.
∎

###### Corollary 26 .

###### Proof.

### B.6 From C-RASP to Limit Transformers

The proofs are adaptations of the proofs from Yang and Chiang, ().

###### Theorem 27 (Restated from Theorem 9 ) .

For every $\textbf{C-RASP}[\Phi,\Psi]$ program $P$ with local functions $\Psi$ and any periodic functions $\Phi$ there exists a Limit Transformer $T_{\infty}$ that satisfies Periodic and Local such that for all $w\in\Sigma^{*}$, $P$ accepts $w$ iff $T_{\infty}$ accepts $\$w$.
If $P$ uses no local or periodic relations, then $T$ requires no functions $\phi_{l,h}$ or positional encodings ${\bm{p}}_{i}$.

###### Remark 28 .

We note that the Limit Transformer $T_{\infty}$ provided by the proof of Theorem [27](#theorem27) emulates the C-RASP program $P$ at zero offset: That is, $P$ accepts $w$ iff a predetermined entry in the last output dimension of $T_{\infty}(\$w,0)$ is above some threshold.
In principle, its computations may not be offset-invariant, i.e., for the constructed $T_{\infty}$, the output $T_{\infty}(\$w,o)$ may depend on $o$.
Importantly, the proof of Theorem [7](#theorem7) does not require a Limit Transformer computing $f$ to be offset-invariant, but just requires it to compute $f$ when the offset is zero.
This is because Lemma [47](#theorem47) ensures that, for any Limit Transformer $T_{\infty}$ satisfying Local and Periodic, even if it is not offset-invariant, there are transformers $T_{n}\in\Theta_{n}$ whose behavior matches $T_{\infty}(\cdot,0)$.

###### Proof of Theorem 27 .

C-RASP has two sorts of operations, a Boolean sort and a Count sort. We will simulate each operation in the transformer by storing the Boolean values as $\{0,1\}$, and storing the counts as $\frac{c}{i+1}$.
That is, we say that a Limit Transformer $T_{\infty}$ simulates a C-RASP program $P$ if for every operation $P_{k}$ of $P$ there is a dimension $d_{k}$ in $T$ such that when $P_{k}(i)$ when run on $w$ is true iff $T_{\infty}(\$w)_{i+1,d_{k}}=1$ (and 0 0 otherwise) for Boolean operation and $P_{k}(i)=c$ iff $T_{\infty}(\$w)_{i+1,k}=\frac{c}{i+1}$ for count operations.

The theorem will be shown by induction on the length of $P$. As a clarifying note, we use 0 0-indexing everywhere in this proof. If $P$ is of length 0 0, we only have initial $Q_{\sigma}(i)$ vectors, which can be simulated by appropriately setting the word embedding. Otherwise, assume all programs of length $\leq k$ are simulated by some transformer, and we have cases for each type of operation $P_{k+1}(i)$ can be. All cases are identical to Yang and Chiang, () except for comparison, conditional, and counting.

First, we must address the SOS token $\$$. There exists a transformer layer that sets the entire vector to $\mathbf{0}$ in the initial position while leaving all other layers untouched. For instance, we may use a conditional operation, as described later in the proof.

If $P_{k+1}(i):=\phi(i)$, a periodic positional function in $\Phi$, then it is simulated in $T_{\infty}$ by appropriately setting ${\bm{p}}_{i}$ in the positional encoding.

If $P_{k+1}(i):=P(i)\;\mathbf{?}\;C_{1}(i)\;\textbf{:}\;C_{2}(i)$, we can implement the following function: for $P\in\{-1,1\}$ and $V\in[0,1]$

$$ $f(P,V)=\begin{cases}V&P=-1\\ 0&P=1\end{cases}$ $$

This is achieved by $f(P,V)=\operatorname{ReLU}(V-P)-\operatorname{ReLU}(-P)$. Thus, the desired Conditional Output can be defined in a single FFN as $f(P,V_{1})+f(-P,V_{2})$, where the first layer and $\operatorname{ReLU}$ compute each $f$ term and the second layer adds them together.

If $P_{k+1}(i):=C_{1}(i)\leq C_{2}(i)$. By the inductive hypothesis $C_{1}(i)$ and $C_{2}(i)$ are stored in dimensions $d_{1}$ and $d_{2}$ as the value $\frac{C_{1}(i)}{i+1}$ and $\frac{C_{2}(i)}{i+1}$. It suffices to check that $\frac{C_{2}(i)}{i+1}=\frac{C_{1}(i)}{i+1}\geq 0$.

To compute this, we use the Heaviside activation function, which we used in our model of MLPs as discussed in [D.1](#A4.SS1).

Thus, there exists an MLP which, letting $x_{1}$ and $x_{2}$ be the values in dimensions $d_{1}$ and $d_{2}$, computes $(\operatorname{hs}(x_{2}-x_{1})+1)/2$ in the dimension reserved for $P_{k+1}$, which will be the Boolean value in $\{0,1\}$ corresponding to $C_{1}(i)\leq C_{2}(i)$.

If $C(i):=\textsc{{\#}}\left[j\leq i\right]\;P(j)$ (using $\psi(i,j)=\top$), then the desired sum is computed using uniform attention since the boolean representation of $P(j)$ is just 0 0 or $1$. We enforced that $P(0)$ is false, so it does not contribute to the sum. This is described in more detail in Yang and Chiang, (), though the case here is simpler.

If $C(i):=\textsc{{\#}}\left[j\leq i,\psi(i,j)\right]\;P(j)$, we can think of it as implementing $\textsc{{\#}}\left[j\leq i\right]\;\psi(i,j)\land P(j)$. Suppose $\psi$ is a local function of the following form

$$ $\psi(i,j)=\begin{cases}1&j=i-\ell\\ 0&\text{else}\end{cases}$ $$

Then $C(i)$ will either be $1$ or 0 0 depending if $P(i-\ell)$ is true or false. If we set the query and key matrices to 0 0 we get

$$ $s_{ij}=\log N\cdot\psi(i,j)$ $$

We assume the $\log$ is base 2, but the argument is similar for others. Then we can have attention compute

$$ $c_{i,k}=\frac{\displaystyle\sum_{j\leq i}\exp\left(\log N\cdot\psi(i,j)\right)\cdot P(j)}{\displaystyle\sum_{j\leq i}\exp\left(\log N\cdot\psi(i,j)\right)}=\frac{\displaystyle\sum_{j\leq i}N^{\left(\frac{\psi(i,j)}{\ln 2}\right)}\cdot P(j)}{\displaystyle\sum_{j\leq i}N^{\left(\frac{\psi(i,j)}{\ln 2}\right)}}$ $$

If $P(i-\ell)$ and $\lnot P(j)$ for $j\neq i-\ell$, then we have a lower bound:

$$ $\frac{N^{\left(\frac{1}{\ln 2}\right)}}{N^{\left(\frac{1}{\ln 2}\right)}+i-1}\leq c_{i,k}$ $$

If $\lnot P(i-\ell)$ and $P(j)$ for $j\neq i-\ell$ then we have an upper bound:

$$ $c_{i,k}\leq\frac{i-1}{N^{\left(\frac{1}{\ln 2}\right)}+i-1}$ $$

Since $N^{\frac{1}{\ln 2}}\geq i$, and we know that $P(i-\ell)\iff c_{i,k}\geq\frac{1}{2}$, we can construct an MLP that computes the correct value. It will output either $\frac{0}{i+1}$ or $\frac{1}{i+1}$, in the dimension reserved for $P_{k+1}(i)$, for instance by using a conditional operation that checks that the output of the attention layer $c_{i+1,k}\geq\frac{1}{2}$, which was shown in an earlier case.
∎

###### Theorem 27 (Restated from Theorem 9 ) .

###### Remark 28 .

###### Proof of Theorem 27 .

## Appendix C Expressivity Proofs for C-RASP

### C.1 C-RASP Constructions

#### C.1.1 Majority

MAJORITY is the language of strings over $\Sigma=\{0,1\}$ with at least as many $1$’s as 0 0’s.

#### C.1.2 Dyck-1

Dyck-1 is the language of strings over $\Sigma=\{0,1\}$ with at least as many $1$’s as 0 0’s.

#### C.1.3 a n ​ b n ​ c n superscript 𝑎 𝑛 superscript 𝑏 𝑛 superscript 𝑐 𝑛 a^{n}b^{n}c^{n}

Let $\Sigma=\{a,b,c\}$. This is another example of a counter language which C-RASP can express and which transformers have been observed to length generalize on (Bhattamishra et al.,, ).

#### C.1.4 Existential Quantification

This is generally a useful primitive, so to save a little space we can add a macro for existential quantification towards the left in C-RASP. This is easily defined using counting:

And we abbreviate this using $P(i):=\overleftarrow{\exists}A(i)$. We demonstrate its use below.

#### C.1.5 Piecewise Testable Languages

Piecewise testable languages are Boolean combinations of languages of the form $\Sigma^{*}a_{1}\Sigma^{*}a_{2}\Sigma^{*}\ldots\Sigma^{*}a_{n}\Sigma^{*}$. This allows us to check for the presence of noncontiguous substrings, which contrasts with the proof in [C.3.2](#A3.SS3.SSS2) that implies the presence of contiguous substrings cannot be expressed in $\textbf{C-RASP}[\emptyset]$.

It suffices to show programs for languages of the form $L=\Sigma^{*}a_{1}\Sigma^{*}a_{2}\Sigma^{*}\ldots\Sigma^{*}a_{n}\Sigma^{*}$, since Boolean combinations are recognizable using Boolean operations of C-RASP. For $L$ we have the following C-RASP program which has the final accepting operation $L_{n}$:

### C.2 C-RASP ​ [ periodic , local ] C-RASP periodic local \textbf{C-RASP}[\text{periodic},\text{local}] Constructions

#### C.2.1 Induction Head (Argmax Version)

As an example consider $\Sigma=\{a,b,c\}$. Predicate $NEXT_{a}(i)$ is true iff the next token should be an $a$. First we can define predecessor

$$ $\displaystyle CP_{a}(i)$ $\displaystyle:=\textsc{{\#}}\left[j\leq i,j=i-1\right]\;Q_{a}(j)$ $\displaystyle PRED_{a}(i)$ $\displaystyle:=CP_{a}(i)\geq 1$ $$

Then we can count bigram occurence by counting

$$ $\displaystyle C_{ab}$ $\displaystyle:=\textsc{{\#}}\left[j\leq i\right]\;Q_{b}(j)\land PRED_{a}(j)$ $$

Then each $NEXT_{a}(i)$ predicate can be defined by checking the current symbol and finding the most frequently occuring bigram.

This corresponds to testing, for the $f$ in Equation [10](#S4.E10), for which $\sigma$ the entry $f(x_{1}\dots x_{N})_{N,\sigma}$ is maximal.

#### C.2.2 Induction Head (All possible next symbols)

Consider $\Sigma$. For $a\in\Sigma$, predicate $NEXT_{a}(i)$ is true iff the next token can possibly be an $a$. As in Section [C.2.1](#A3.SS2.SSS1), first, we can define predecessor

$$ $\displaystyle CP_{a}(i)$ $\displaystyle:=\textsc{{\#}}\left[j\leq i,j=i-1\right]\;Q_{a}(j)$ $\displaystyle PRED_{a}(i)$ $\displaystyle:=CP_{a}(i)\geq 1$ $$

Then we can check for bigram occurrence by counting

$$ $\displaystyle CBIGRAM_{ab}$ $\displaystyle:=\textsc{{\#}}\left[j\leq i\right]\;Q_{b}(j)\land PRED_{a}(j)$ $\displaystyle EXISTS_{ab}$ $\displaystyle:=CBIGRAM_{ab}(i)\geq 1$ $$

If a bigram $\sigma a$ ever occurred previously in the string, nonzero probability is assigned to predicting $a$ when at symbol $\sigma$. Then each $NEXT_{a}(i)$ predicate can be defined as follows

where $\bigvee_{\sigma\in\Sigma}$ can be expressed using the Boolean operations $\wedge$ and $\neg$ as defined in Section [4.2](#S4.SS2).
This corresponds to testing, for the $f$ in Equation [10](#S4.E10), for which $\sigma$ we have $f(x_{1}\dots x_{N})_{N,\sigma}>0$.

##### Generating based on this program

Consider an input prefix of the form $\#x\#$, where $\#$ denotes a separator symbol. If we iteratively generate the next symbol $a$ by selecting $a\in\Sigma$ where $NEXT_{a}$ holds at the last position, we generate a string $\#x\#y$ where all bigrams in $\#y$ had already occurred in $\#x$, a simple version of the in-context Markov chains studied by (Edelman et al.,, ).

##### Special Case: Unique Copying

In the special case of an input prefix where each symbol occurs at most once in $x$, the generation procedure defined above will copy $x$, and (assume we stop at $\#$) resulting in an overall string of the form $\#x\#x\#$. This is essentially the RASP-L construction of unique copying noted by .

##### Necessity of Positional Relations

Intuitively, an induction head circuit requires positional information; indeed, we observe length generalization in unique copying with APE but not with NoPE (Figure [1](#S5.F1)).
Formally, we can prove as follows that the predicate $NEXT_{a}$ defined above for each $a\in\Sigma$, while definable in ${\textbf{C-RASP}[\text{local}]}$, is not definable in $\textbf{C-RASP}[\emptyset]$.
Consider $\Sigma=\{a,b\}$; then the predicate $NEXT_{b}$ can be used to define the (disjoint) union of the languages $\Sigma^{*}ab\Sigma^{*}a$, $\Sigma^{*}bb\Sigma^{*}b$. As the first one is definable in $\textbf{C-RASP}[\emptyset]$(^4^44It is sufficient to check whether $a$ and $b$ both are present and whether one $b$ has a $a$ in its preceding context; as $\Sigma=\{a,b\}$, this is equivalent to $ab$ being a substring) and the union is disjoint, the second would be definable if $NEXT_{b}$ is.
This contradicts the fact that $\Sigma^{*}bb\Sigma^{*}\not\in\textbf{C-RASP}[\emptyset]$, because $\Sigma^{*}bb\Sigma^{*}\not\in\widehat{MAJ}_{2}[<]$ (Lemma 6.11 in Krebs, ()) and the inclusion $\textbf{C-RASP}[\emptyset]\subseteq\widehat{MAJ}_{2}[<]$ (see Section [C.3.1](#A3.SS3.SSS1)).

#### C.2.3 ( a ​ a ) ∗ superscript 𝑎 𝑎 (aa)^{*}

The following function that checks the parity of a position mod $2$ is a periodic function.

$$ $\phi(i):=i\equiv 0\mod 2$ $$

So the following program recognizes $(aa)^{*}$

The Boolean value of the last operation in the last position of the string is the accepting value. This is true if the string is of even length and contains only $a$’s.
Overall, we have constructed a program in ${\textbf{C-RASP}[\text{periodic},\text{local}]}$.

### C.3 Expressibility of Regular Languages in C-RASP [ periodic , local ]

###### Lemma 29 (Restated from Lemma 11 ) .

Consider the alphabet $\Sigma=\{a,b,e\}$.

- 1.
$PARITY:=b^{*}(ab^{*}ab^{*})^{*}\not\in{\textbf{C-RASP}[\text{periodic},\text{local}]}$
- 2.
$(aa)^{*}\in\textbf{C-RASP}[\text{periodic},\text{local}]$ and $(aa)^{*}\not\in\textbf{C-RASP}[\emptyset]$
- 3.
$(a|b|e)^{*}be^{*}\not\in\textbf{C-RASP}[\text{periodic},\text{local}]$
- 4.
$L\in\textbf{C-RASP}[\emptyset]$ for piecewise testable $L$

###### Proof.

1–3 are shown in Lemmas [36](#theorem36) (for 3.), [38](#theorem38) (for 2.), [41](#theorem41) (for 1.), and Appendix [C.2.3](#A3.SS2.SSS3) (for 2.).
4. is shown in Appendix [C.1.5](#A3.SS1.SSS5).
∎

###### Lemma 29 (Restated from Lemma 11 ) .

###### Proof.

#### C.3.1 Link to Majority Logic

In understanding the expressiveness of C-RASP, we draw on an established body of work on logics using MAJORITY quantifiers.
; Strobl, () show that the expressiveness of transformers is upper-bounded by uniform $\textbf{TC}^{0}$, which can be defined as the logic $\textbf{FOM}[\text{BIT}]$. This logic is defined in terms of MAJORITY quantifiers and various predicates.
C-RASP[periodic,local] can be viewed as a highly restricted fragment of this logic.
Specifically, it is contained in $\widehat{MAJ}_{2}[<,+1,Mod]$, which was studied by Krebs, (); Behle et al., (; ); results about that logic help understand the expressiveness of C-RASP:

###### Definition 30 .

$\widehat{MAJ}_{2}[<,+1,Mod]$ is the logic defined by the constructs

- 1.
The construct
$\displaystyle\widehat{Maj}\;x\left\langle\phi_{1},\ldots,\phi_{c}\right\rangle$
- 2.
The predicates $Q_{a}(x)$ for $a\in\Sigma$
- 3.
Numerical predicates for $q,j\in\mathbb{N}$: $Mod_{m,r}(x)$, $Succ(y,x)$
- 4.
Boolean connectives
- 5.
First-order quantifiers(^5^55These can be simulated by majority quantifiers with two variables by Proposition 5.5 in Krebs, (), which is based on Corollary 3.3 in Lange, (). Nonetheless, as the simulation is unobvious, they are useful for writing formulas in $\widehat{MAJ}_{2}[<,+1,MOD]$.)

such that only two variables (say, $x$ and $y$) can appear within a formula. We define the semantics, when $x\in\Sigma^{*}$, by defining

- 1.
for the majority quantifier:
$\displaystyle w\models\widehat{Maj}\;x\left\langle\phi_{1},\ldots,\phi_{c}\right\rangle\Leftrightarrow$
$\displaystyle 0<\sum_{i=1}^{n}\sum_{j=1}^{c}\begin{cases}1&\text{if }w|_{x=i}\models\phi_{j}\\
-1&\text{else}\end{cases}$
- 2.
for the predicates:
$\displaystyle w|_{x=i}\models Q_{a}(x)\quad$
$\displaystyle\Leftrightarrow\quad$
$\displaystyle w_{i}=a$
$\displaystyle w|_{x=i}\models Mod_{m,r}(i)\quad$
$\displaystyle\Leftrightarrow\quad$
$\displaystyle i\equiv r\pmod{m}$
$\displaystyle w|_{x=i,y=j}\models Succ(j,i)\quad$
$\displaystyle\Leftrightarrow\quad$
$\displaystyle j+1=i$

Semantics of Boolean connectives and first-order quantifiers follow the standard definition.

A language $\mathcal{L}\subseteq\Sigma^{*}$ is *definable* in $\widehat{MAJ}_{2}[<,+1,Mod]$ if there is a formula $\phi$ without free variables such that $w\in\mathcal{L}$ if and only if $w\models\phi$.

The logic $\widehat{MAJ}_{2}[<]$ results by omitting the numerical predicates defined under (3).

It is straightforward to convert C-RASP programs into formulas of $\widehat{MAJ}_{2}[<,+1,\text{MOD}]$. As we shall see later in Section [C.3.3](#A3.SS3.SSS3), the inclusion is strict because $PARITY$ is expressible even in $\widehat{MAJ}_{2}[<]$.

###### Proposition 31 .

$\textbf{C-RASP}[\text{periodic},\text{local}]\subseteq\widehat{MAJ}_{2}[<,+1,MOD]$

We make two remarks about the corollaries of the result:

###### Remark 32 .

First, the proof, simply by omitting the positional relations, also yields a corresponding inclusion without positional relations: $\textbf{C-RASP}[\emptyset]\subseteq\widehat{MAJ}_{2}[<]$.

Second, the result implies that $\textbf{C-RASP}[\text{periodic},\text{local}]$ defines a subclass of $\textbf{TC}^{0}$, in fact, all $\textbf{C-RASP}[\text{periodic},\text{local}]$ programs translate into uniform $\textbf{TC}^{0}$ circuits with *a linear number of gates* by results in Krebs, (, Theorem 4.33 and Figure 4.4).
The inclusion is strict, e.g., PARITY has a linear-size $\textbf{TC}^{0}$ circuit but is not definable by $\textbf{C-RASP}[\text{periodic},\text{local}]$, as we show below.

###### Proof of Proposition 31 .

First, every periodic positional function $\phi(i)$ is a Boolean function $Mod_{m,r}(i)\iff i=r\mod m$. For local functions $\psi(i,j)$, it suffices to only consider functions of the form $\psi(i,j)\iff j=i+c$ for $c\in\mathbb{Z}$. This is because the counting operation $C(i):=\textsc{{\#}}\left[j\leq i,|i-j|\leq c\right]\;P(j)$ is equivalent to $\hat{C}(i)$ where

$$ $\hat{C}(i):=\textsc{{\#}}\left[j\leq i,j=i-c\lor j=i-(c-1)\lor\ldots\lor j=i+c\right]\;P(j)$ $$

And it is possible to further reduce this by distributing the disjunctions over many counting operations so that each one only contains a single disjunct as positional function. It helps that a predicate $\text{fst}(i)$ is definable in $\widehat{MAJ}_{2}[<,+1,\text{MOD}]$ which is true iff $i=0$. For instance $\text{fst}(i):=\widehat{Maj}_{j}\langle j\leq i,\top\rangle$

For each Boolean C-RASP operation $P(i)$, there exists a $\widehat{MAJ}_{2}[<,+1,\text{MOD}]$ formula $\hat{P}(i)$ with one free variable that is equivalent. By induction, all cases are straightforward except for comparison operations.

For comparison operations, we will first show a formula that is equivalent for all nonempty strings. Accounting for the empty string is easy, depending on the constants in the comparison. WLOG we are able to rewrite the formula (not in standard C-RASP notation) as the following, where $\alpha_{k},\beta\in\mathbb{Z}$

$$ $\left(\displaystyle\sum_{k\leq K}\alpha_{k}\cdot\textsc{{\#}}\left[j\leq i\right]\;P_{k}(j)\right)+\left(\displaystyle\sum_{m\leq M}\alpha_{m}\cdot\textsc{{\#}}\left[j\leq i,j=i+c_{m}\right]\;P_{m}(j)\right)+\beta>0$ $$

We’ve grouped the uniform counting operations that have $\psi=\top$ together. Then using a case disjunction, we can rewrite it all the local counting operations as the following (using $\mathbb{I}[\phi]$ as notational convenience to turn $\phi(j)\in\{\bot,\top\}$ to the corresponding value in $\{0,1\}$):

$$ $\bigvee_{\tau\in\{0,1\}^{M}\colon\mathbb{I}[P_{m}(j-c_{m})]=\tau_{m}}\left(\displaystyle\sum_{k\leq K}\alpha_{k}\cdot\textsc{{\#}}\left[j\leq i\right]\;P_{k}(j)\right)+\left(\displaystyle\sum_{m\leq M}\alpha_{m}\tau_{m}\right)+\beta>0$ $$

We can see that for nonempty strings within each case, the additive constant $\beta$ can be reformulated as $\left(\beta+\sum_{m\leq M}\alpha_{m}\tau_{m}\right)\cdot\textsc{{\#}}\left[j\leq i\right]\;\text{fst}(j)$, and we can just add it to the summation using $\alpha_{k+1}=\left(\beta+\sum_{m\leq M}\alpha_{m}\tau_{m}\right)$. Then, it is possible to define formulas for each case disjunction using a series of existential quantifiers and $succ(j,i)$. For instance:

$$ $\phi(j-2)\equiv\exists i.\left(Succ(i,j)\land\exists j.\left(Succ(j,i)\land\phi(j)\right)\right)$ $$

This means it now suffices to focus on the sum of counting terms and simulate that using a $\widehat{Maj}$ formula. For $k\leq(K+1)$, if $\alpha_{k}>0$ consider the list of formulas

$$ $L_{k}:=[\underbrace{\hat{P}_{k}(j),\hat{P}_{k}(j),\ldots,\hat{P}_{k}(j)}_{\text{$\alpha_{k}$ many}},\underbrace{\top,\top,\ldots,\top}_{\text{$\alpha_{k}$ many}}]$ $$

Intuitively, the $\widehat{Maj}\;j$ quantifier can only check if the total count is greater than half the possible positions, so to check if a count is $>0$ we need to pad the quantifier with a bunch of trivially true formulas to ensure the total count is at least half by default. And if $\alpha_{k}<0$ we use

$$ $L_{k}:=[\underbrace{\lnot\hat{P}_{k}(j),\lnot\hat{P}_{k}(j),\ldots,\lnot\hat{P}_{k}(j))}_{\text{$\alpha_{k}$ many}},\underbrace{\bot,\bot,\ldots,\bot}_{\text{$\alpha_{k}$ many}}]$ $$

Let $L=L_{1}++L_{2}++\ldots++L_{K+1}$ be the concatenation of all these lists, and let $\varphi_{1},\varphi_{2},\ldots,\varphi_{|L|}$ list out the formulas in $L$. Then we claim the following formula will compute the correct value for nonempty strings.

$$ $\phi_{1}(i):=\widehat{Maj}\;j\langle\varphi_{1},\varphi_{2},\ldots,\varphi_{|L|}\rangle$ $$

For empty strings, if $\left(\beta+\sum_{m\leq M}\alpha_{m}\tau_{m}\right)>0$ then define $\phi_{0}(i):=\lnot\widehat{Maj}\;j\langle\top\rangle\land\top$. Otherwise, use $\phi_{0}(i):=\lnot\widehat{Maj}\;j\langle\top\rangle\land\bot$. Then we can define

$$ $\hat{P}(i):=\phi_{0}(i)\lor\phi_{1}(i)$ $$

And verifying the correctness of this is straightforward.
∎

###### Definition 30 .

###### Proposition 31 .

###### Remark 32 .

###### Proof of Proposition 31 .

#### C.3.2 Inexpressibility of Σ ∗ ​ b ​ e ∗ superscript Σ 𝑏 superscript 𝑒 \Sigma^{*}be^{*}

Krebs, (); Behle et al., (; ) used infinite groups to establish results about the expressiveness of $\widehat{MAJ}_{2}[<]$; by Corollary [31](#theorem31), these results entail results on $\textbf{C-RASP}[\text{periodic},\text{local}]$.
In particular, Lemma 6.11 in Krebs, () shows that $L_{bb}\not\in\widehat{MAJ}_{2}[<]$; this result turns out to have profound consequences for C-RASP expressiveness.

###### Definition 33 .

Let $\Sigma=\{a,b,e\}$.
Define $L_{bb}:=\Sigma^{*}be^{*}b\Sigma^{*}$.

###### Lemma 34 .

Let $\varphi$ be a $\widehat{MAJ}_{2}[<,+1,MOD]$ formula. There exists a morphism $h(\sigma)=e^{s}\sigma e^{s-1}$ and a $\widehat{MAJ}_{2}[<]$ formula $\psi$ such that for every $w\in\Sigma^{*}$, $h(w)\vDash\phi\iff w\vDash\psi$

###### Proof.

Let $M$ be the least common multiple of all occurring moduli in $\varphi$. Let $C$ be the maximum nesting depth of $Succ(x,y)$ predicates (which must be bounded by the quantifier depth of $\varphi$). Intuitively, we can think of $C$ as the largest number where a subformula $\varphi(x+C)$ occurs in $\varphi$. Let $s=MC$, and define the morphism $h(\sigma)=e^{s}\sigma e^{s-1}$. Here we will use the notation $\phi^{c}(x)$ that is true at position $x$ in $w$ whenever $\phi$ is true at position $x+c$ in $h(w)$, for $c\in[-s,s-1]$.

We will show that for every formula $\varphi(x)$ of $\widehat{MAJ}_{2}[<,+1,MOD]$ with at most one free variable, we can define $\varphi^{-s}(x),\varphi^{-(s-1)}(x)\ldots\varphi^{(s-1)}(x)$ such that for all $i\in[0,|w|-1]$ and $c\in[-s,s-1]$

$$ $h(w)\vDash\varphi(i+c)\iff w\vDash\varphi^{c}(i)$ $$

Intuitively, what this does is it takes every interval of $[x-s,x+(s-1)]$ around each position in $h(w)$ and stores it vertically at that position in $w$. We will induct on the complexity of $\varphi$. If $\varphi(x)$ is $Q_{e}(x)$, then $\varphi^{0}(x):=Q_{e}(x)$, and then $\varphi^{c}(x)=\top$ for every other $c\neq 0$, since the morphism $h$ pads neutral symbols $e$ in $h(w)$ between every symbol from $w$. If $\varphi(x)$ is $Q_{\sigma}(x)$ for $\sigma\neq e$, we have that $\varphi^{0}:=Q_{\sigma}(x)$ and $\varphi^{+c}(x)=\bot$ for every other $c\neq 0$. If $\varphi(x)$ is $Mod_{m,r}(x)$, the $\varphi^{c}$ can also be “hardcoded” similarly, as every position in $h(w)$ that has a symbol from $w$ is going to be $=0\mod s$.

Boolean formulas are also straightforward. The only hard case is if we have a formula $\varphi(x)=\widehat{MAJ}\;y\langle\varphi_{1}(x,y),\ldots,\varphi_{k}(x,y)\rangle$. We can think of $\varphi$ specifying the constraint

$$ $\left(\displaystyle\sum_{i\leq k}\#\;y\left[{\varphi_{i}(x,y)}\right]\right)>k\cdot\frac{|w|}{2}$ $$

The idea here is to rewrite $\psi_{i}(x,y)$ in terms of its unary formulas (which we can apply the inductive hypothesis to) and then split $h(w)$ into some intervals, upon which evaluating $\varphi^{+c}(x)$ will be simpler. First we can rewrite each $\varphi_{i}(x,y)$ as

$$ $F_{i}(\alpha_{1}(x),\ldots,\alpha_{q}(x),\beta_{1}(y),\ldots,\beta_{r}(y),\chi_{1}(x,y),\ldots,\chi_{p}(x,y))$ $$

Where $F_{i}$ is a Boolean function, the $\alpha$ are unary in $x$, the $\beta$ are unary in $y$, and the $\chi(x,y)$ are inequalities of $x$ and $y$, possibly with $+1$’s, of the form $x\leq y+1$, for example. To save space, we will abbreviate the above expression by grouping the $\alpha,\beta,\chi$ formulas together notationally $F_{i}(\overline{\chi}_{i}(x,y),\overline{\alpha_{i}}(x),\overline{\beta_{i}}(y))$.

The $\chi$ formulas are not unary, but we can “eliminate” the $\chi$ terms by casework over intervals of the string. We will show this by example for a summation with only one $\#\;y$ term. This argument works identically if we had many of $\#\;y$ terms, but it would add notational clutter. So if we had a formula $\varphi(x)=\widehat{MAJ}\;y\langle\varphi_{1}(x,y)\rangle$ we could think of it as in the form

$$ $\varphi(x)=\#\;y\left[F(\overline{\chi}(x,y),\overline{\alpha}(x),\overline{\beta}(y)\right]\geq\frac{|w|}{2}$ $$

Then we can construct the formula $\varphi^{c}(x)$ for $c\in[-s,s-1]$ by using the following partition of intervals of the string. Let $\Xi$ be the set of inequalities

$$ $\displaystyle\Xi=\{y<x-s,y=x-s,\ldots,y=x,\ldots,y=x+(s-1),y>x+(s-1)\}$ $$

And we define some notation. For $\xi\in\Xi$, let $\chi^{\xi}(x,y)\in\{\top,\bot\}$ evaluate $\chi$ in the case $\xi$ holds. For instance if $\chi(x,y)$ is $y<x+1$, then $\chi^{y>x+2}(x,y)=\bot$. Since the intervals defined by $\xi\in\Xi$ disjoint and cover the entirety of the string, every $\chi$ can be evaluated in this manner. Then, we can essentially compute the sum in each interval, and only precision is needed in the interval $[x-s,x+(s-1)]$, so $\varphi(x)$ is equivalent to

$$ $\displaystyle\#\;y\left[y<x\land F(\overline{\chi}^{y<x-s}(x,y),\overline{\alpha}^{c}(x),\overline{\beta}^{-s}(y)\right]$ $\displaystyle+$ $\displaystyle\#\;y\left[y<x\land F(\overline{\chi}^{y<x-s}(x,y),\overline{\alpha}^{c}(x),\overline{\beta}^{-(s-1)}(y)\right]$ $\displaystyle\quad\vdots$ $\displaystyle+$ $\displaystyle\#\;y\left[y<x\land F(\overline{\chi}^{y<x-s}(x,y),\overline{\alpha}^{c}(x),\overline{\beta}^{+(s-1)}(y)\right]$ $\displaystyle+$ $\displaystyle\#\;y\left[y<x\land F(\overline{\chi}^{y<x-s}(x,y),\overline{\alpha}^{c}(x),\overline{\beta}^{s}(y)\right]$ $\displaystyle+$ $\displaystyle\#\;y\left[x=y\land F(\overline{\chi}^{y=x-s}(x,y),\overline{\alpha}^{c}(x),\overline{\beta}^{-s}(y)\right]$ $\displaystyle+$ $\displaystyle\#\;y\left[x=y\land F(\overline{\chi}^{y=x-(s-1)}(x,y),\overline{\alpha}^{c}(x),\overline{\beta}^{-(s-1)}(y)\right]$ $\displaystyle\quad\vdots$ $\displaystyle+$ $\displaystyle\#\;y\left[x=y\land F(\overline{\chi}^{y=x+(s-1)}(x,y),\overline{\alpha}^{c}(x),\overline{\beta}^{(s-1)}(y)\right]$ $\displaystyle+$ $\displaystyle\#\;y\left[x=y\land F(\overline{\chi}^{y=x+s}(x,y),\overline{\alpha}^{c}(x),\overline{\beta}^{s}(y)\right]$ $\displaystyle+$ $\displaystyle\#\;y\left[x<y\land F(\overline{\chi}^{x+s<y}(x,y),\overline{\alpha}^{c}(x),\overline{\beta}^{-s}(y)\right]$ $\displaystyle+$ $\displaystyle\#\;y\left[x<y\land F(\overline{\chi}^{x+s<y}(x,y),\overline{\alpha}^{c}(x),\overline{\beta}^{-(s-1)}(y)\right]$ $\displaystyle\quad\vdots$ $\displaystyle+$ $\displaystyle\#\;y\left[x<y\land F(\overline{\chi}^{x+s<y}(x,y),\overline{\alpha}^{c}(x),\overline{\beta}^{(s-1)}(y)\right]$ $\displaystyle+$ $\displaystyle\#\;y\left[x<y\land F(\overline{\chi}^{x+s<y}(x,y),\overline{\alpha}^{c}(x),\overline{\beta}^{s}(y)\right]$ $\displaystyle\geq\frac{|w|}{2}$ $$

By the inductive hypothesis, all $\alpha^{c}$ and $\beta^{c}$ are definable solely in terms of $\widehat{MAJ}_{2}[<]$, so the entire formula is equivalent to a $\widehat{MAJ}\;y$ formula that quantifies over all the bracketed formulas above, as well as equally many trivially true formulas, as described more clearly in Proposition [31](#theorem31). As mentioned before, since this argument also applies to a summation of $\#\;y$ terms, this completes the proof. Then for any $\varphi(x)$ in $\widehat{MAJ}_{2}[<,+1,MOD]$, after performing the above translation the resulting formula $\varphi^{0}(x)$ is our desired formula in $\widehat{MAJ}_{2}[<]$.
∎

###### Lemma 35 .

$L_{bb}\not\in\widehat{MAJ}_{2}[<,+1,MOD]$

###### Proof.

Assume for sake of contradiction that $L_{bb}$ is definable by a formula $\varphi$ of $\widehat{MAJ}_{2}[<,+1,MOD]$. Let $h$ and $\psi$ be as guaranteed by the above lemma. Then for $w\in\Sigma^{*}$, $w\in L_{bb}\iff h(w)\in L_{bb}$. This means $\psi$ defines $L_{bb}$ which contradicts Lemma 6.11 in Krebs, (), which has shown that $L_{bb}\not\in\widehat{MAJ}_{2}[<]$.
∎

###### Lemma 36 .

For $\Sigma=\{a,b,e\}$, it holds that

$$ $\Sigma^{*}be^{*}\not\in\textbf{C-RASP}[\text{periodic},\text{local}]$ (4) $$

###### Proof.

To get a contradiction, note that a $\textbf{C-RASP}[\text{periodic},\text{local}]$ program $\Phi$ for $\Sigma^{*}be^{*}$ could be used to construct one for $L_{bb}$, as:

$$ $\displaystyle C_{1}(i)$ $\displaystyle:=\textsc{{\#}}\left[j\leq i,j=i-1\right]\;\Phi(j)$ $\displaystyle PREV_{\Phi}(i)$ $\displaystyle:=C_{1}(i)\geq 1$ $\displaystyle C_{2}(i)$ $\displaystyle:=\textsc{{\#}}\left[j\leq i\right]\;Q_{b}(j)\land PREV_{\Phi}(j)$ $\displaystyle L_{bb}$ $\displaystyle:=C_{2}(i)\geq 1$ $$

∎

###### Definition 33 .

###### Lemma 34 .

###### Proof.

###### Lemma 35 .

###### Proof.

###### Lemma 36 .

###### Proof.

#### C.3.3 Inexpressibility of PARITY

First, let the *depth* of a C-RASP operation be the maximum depth of nesting of counting operations in it. For instance if $C(i):=\textsc{{\#}}\left[j\leq i\right]\;P(j)$, the depth of the $C$ is depth of $P(i)$ plus one. None of the other operations are greater than the depth of its dependencies. We will induct on program depth for the following proof:

###### Lemma 37 .

Let $\Sigma=\{a\}$. For any $\textbf{C-RASP}[\emptyset]$ program $P$ there exists an $n$ such that for all $w$ where $|w|\geq n$, either all such $w$ are accepted by $P$ or all are rejected

###### Proof.

If $P$ is depth 0 0, it is equivalent to either $Q_{a}(i)$ or $\lnot Q_{a}(i)$, which either rejects every string or accepts every string.

Otherwise, let all C-RASP programs of depth $k$ give constant output for strings above length $n$, and then consider a program $P$ of depth $k+1$. $P$ will be equivalent to a Boolean combination of linear constraints. We will see that each linear constraint becomes constant for strings above a certain length. Consider any linear constraints over $X$ many counts $C_{x}$ of depth $k$:

$$ $L(i):=\left(\displaystyle\sum_{x\leq X}\alpha_{x}\textsc{{\#}}\left[j\leq i\right]\;C_{x}(j)\right)\geq c$ $$

For string of length $i\geq n$, this is equivalent to

$$ $L(i):=\left(\displaystyle\sum_{x\leq X}\alpha_{x}\left((i-n)\mathbb{I}[C_{x}(n)]+\textsc{{\#}}\left[j\leq n\right]\;C_{x}(j)\right)\right)\geq c$ $$

Where $\mathbb{I}[C_{x}(n)]$ denotes the truth value of $C_{x}(n)\in\{0,1\}$. Rearrange this to

$$ $L(i):=\left(\displaystyle(i-n)\sum_{x\leq X}\alpha_{x}\left(\mathbb{I}[C_{x}(n)]\right)\right)+\left(\displaystyle\sum_{x\leq X}\alpha_{x}\left(\textsc{{\#}}\left[j\leq n\right]\;C_{x}(j)\right)\right)\geq c$ $$

The sums $c_{1}=\sum_{x\leq X}\alpha_{x}\left(\mathbb{I}[C_{x}(n)]\right)$ and $c_{2}=\sum_{x\leq X}\alpha_{x}\left(\textsc{{\#}}\left[j\leq n\right]\;C_{x}(j)\right)$ are constants depending on the formula and $n$.

$$ $(i-n)c_{1}+c_{2}\geq c$ $$

Depending on if $c_{1},c_{2}$ are positive or negative, we either derive a lower bound $m$ after which the linear constraint $L(i)$ is always true, or always false for $i\geq m$. Since any formula of depth $k+1$ is a Boolean combination of these linear constraints, we take the max of all the $m$’s from them, and any string larger than this will always be accepted or rejected by $P$.
∎

###### Lemma 38 .

$(aa)^{*}\not\in\textbf{C-RASP}[\emptyset]$

That is, no $\textbf{C-RASP}[\emptyset]$ program can determine if a general string has an even length.
The same proof applies to testing whether the string length is a multiple of any other fixed integer.

###### Proof.

Using the previous lemma, for every $\textbf{C-RASP}[\emptyset]$ program there exists an $n$ such that the program accepts $(aa)^{n}$ iff it accepts $(aa)^{n}a$. So no program can recognize $(aa)^{*}$.
∎

We will use this to show that no $\textbf{C-RASP}[\text{periodic},\text{local}]$ program cannot recognize $PARITY$, as the extra positional operations do not give sufficient expressive power. We start with an observation that simplifies the proof

###### Proposition 39 .

As syntactic sugar we allow $j<i$ as a mask in C-RASP counting operations.

###### Proof.

Consider the counting operation $C(i):=\textsc{{\#}}\left[j\leq i\right]\;P(j)$. We can define the program

$$ $\displaystyle I(i)$ $\displaystyle:=P(i)\;\mathbf{?}\;1\;\textbf{:}\;0$ $\displaystyle C(i)$ $\displaystyle:=\textsc{{\#}}\left[j\leq i\right]\;P(j)$ $\displaystyle C^{\prime}(i)$ $\displaystyle:=C(i)-I(i)$ $$

And essentially, this operation will compute the count

$$ $C^{\prime}(i):=\textsc{{\#}}\left[j<i\right]\;P(j)$ $$

∎

###### Lemma 40 .

Let $P$ be a $\textbf{C-RASP}[\text{periodic},\text{local}]$ program over $\Sigma=\{a,b\}$. There is some $s>0$ and a morphism $h(a)=b^{s}ab^{s-1}$ and a $\textbf{C-RASP}[\emptyset]$ program $\hat{P}$ over $\Sigma=\{a\}$ such that for all $w\in a^{*}$, if $P$ accepts $h(w)$ iff $\hat{P}$ accepts $w$.

###### Proof.

Choose $s$ to be the least multiple of all moduli occurring in $P$ that is also greater than all the $|c|$ in local functions $j=i+c$.
For every operation $P(i)$ of $P$, we will define $\hat{P}^{c}(i)$ for $c\in[-s,s-1]$ such that $\hat{P}^{c}(i)$ when run on $w$ is equivalent to $P(s+i(2s)+c)$ when run on $h(w)$.

If $P(i)$ is $Q_{a}(i)$ or $Q_{b}(i)$, it is straightforward, as $\hat{P}^{c}(i)$ is true iff $c=0$. Modular predicates are also capable of being “hardcoded”, as positions in $w$ are always $0\mod s$ in $h(w)$. All other kinds of operations are also straightforward using the inductive hypothesis. The only ones that need care are counting operations. First, consider a counting operation without positional functions:

$$ $C(i):=\textsc{{\#}}\left[j\leq i\right]\;A(j)$ $$

We can define each $\hat{C}^{c}(i)$ using a program like the following. The idea is that the entire window of $[j-s,j+(s-1)]$ around each $j<i$ can be counted up completely, but around $i$ we only consider the interval $[-s,i+c]$:

$$ $\displaystyle C_{-s}(i)$ $\displaystyle:=\textsc{{\#}}\left[j<i\right]\;\hat{A}^{-s}(j)$ $\displaystyle C_{-(s-1)}(i)$ $\displaystyle:=\textsc{{\#}}\left[j<i\right]\;\hat{A}^{-(s-1)}(j)$ $\displaystyle\vdots$ $\displaystyle C_{(s-1)}(i)$ $\displaystyle:=\textsc{{\#}}\left[j<i\right]\;\hat{A}^{(s-1)}(j)$ $\displaystyle I_{-s}(i)$ $\displaystyle:=\hat{A}^{-s}(j)\;\mathbf{?}\;1\;\textbf{:}\;0$ $\displaystyle I_{-(s-1)}(i)$ $\displaystyle:=\hat{A}^{-(s-1)}(j)\;\mathbf{?}\;1\;\textbf{:}\;0$ $\displaystyle\vdots$ $\displaystyle I_{c}(i)$ $\displaystyle:=\hat{A}^{c}(j)\;\mathbf{?}\;1\;\textbf{:}\;0$ $\displaystyle\hat{C}^{c}(i)$ $\displaystyle:=\displaystyle\sum_{t\in[-s,s-1]}C_{t}(i)+\sum_{t\in[-s,c]}I_{t}(i)$ $$

Otherwise, if we have a counting operation that involves a local positional function

$$ $C(i):=\textsc{{\#}}\left[j\leq i,j=i+d\right]\;A(j)$ $$

Then the operation returns either the count $1$ or 0 0 and we can just use

$$ $\displaystyle\hat{C}^{c}(i)$ $\displaystyle:=(c=d\land\hat{A}^{c}(i))\;\mathbf{?}\;1\;\textbf{:}\;0$ $$

Since $d$ will not exceed $\pm s$, $\hat{A}^{d}$ exists. Using these constructions we can see that $(b^{s}ab^{s-1})^{n}$ is accepted by operation $P(i)$ in $P$ iff $a^{n}$ is accepted by the constructed operation $\hat{P}^{(s-1)}(i)$.
∎

###### Lemma 41 .

$PARITY\not\in\textbf{C-RASP}[\text{periodic},\text{local}]$

###### Proof.

If such a program existed, it would be able to distinguish between $(b^{s}ab^{s-1})^{2n}$ and $(b^{s}ab^{s-1})^{2n+1}$ for all $n$ (using the $s$ guaranteed by the previous lemma). However, this implies the existence of a $\textbf{C-RASP}[\emptyset]$ program over $\Sigma=\{a\}$ that recognizes $(aa)^{*}$. This contradicts Lemma [38](#theorem38).
∎

###### Lemma 37 .

###### Proof.

###### Lemma 38 .

###### Proof.

###### Proposition 39 .

###### Proof.

###### Lemma 40 .

###### Proof.

###### Lemma 41 .

###### Proof.

## Appendix D Discussion of Design Choices

### D.1 MLP Activation Functions

Our analysis allows ReLU and the Heaviside function as activation functions in MLPs.
ReLU is a standard choice in theoretical studies of neural networks and transformers (e.g. Bhattamishra et al.,, ; Sanford et al.,, ).
Modern LLMs also use other functions such as SwiGLU (Shazeer,, ), but universal approximation theorems guarantee that ReLU networks can approximate smooth functions on bounded domains well.
While the choice of ReLU is not necessarily key to our results, it is important that the number of active units provides a meaningful upper bound on the complexity of the function expressed. Our results would continue to go through if $\phi$ is an arbitrary activation function but operates at $p$-bit precision.

We also allow the Heaviside function as a second activation function. Heaviside allows exactly performing threshold computations at arbitrary input lengths, which is relevant to simulating C-RASP at arbitrary input lengths.
This includes simple problems such as MAJORITY, on which transformers empirically do well.
Real-world transformers generally do not include this function, though ReLU MLPs can approximate it arbitrarily closely.
Theoretically modeling length generalization on such functions with only ReLU MLPs is an interesting possible extension of our theory.

### D.2 Fixed Precision

As described in Section [2](#S2), we assume that attention logits and the exponentials inside softmax are rounded to $p$ fractional bits of precision before further processing.
This allows us to cluster keys and queries into finite numbers of clusters, and compute all activations at logarithmic precision, used for proving the logarithmic communication complexity bound (Theorem [12](#theorem12)). We note that logarithmic precision of the intermediate activations is also key to upper bounds of transformers in terms of $\textbf{TC}^{0}$ shown by .

We also assume that the parameters in transformers and Limit Transformers are expressed at fixed precision (Definitions [4](#theorem4) and [2](#theorem2)), and penalize the precision $p$ used of representing the parameters as part of the regularizer used in our inference procedure (Definition [5](#theorem5)).
Indeed, penalizing unbounded precision of parameter values is necessary to enable full identification of a transformer algorithm from behavior at finite lengths.
For any real number $\alpha\geq 0$, a one-layer transformer with real-valued parameters can express the function

$$ $\displaystyle F_{\alpha}(x)=\begin{cases}1&\text{ if }\alpha\cdot\#_{1}(x)\geq\#_{0}(x)\\ 0&else\end{cases}$ $$

For any two distinct $\alpha,\beta$, the functions $F_{\alpha}$ and $F_{\beta}$ are distinct on sufficiently long inputs (though, when $\alpha$ and $\beta$ are close, very long inputs will be needed to distinguish them).
Thus, there are *uncountably* many distinct functions implemented by transformers; however, their distinction relies on infinite precision.
In an infinite precision setup, one cannot hope to identify algorithms implemented by transformers from finite data, no matter the input length and the regularization applied to the model size.
In contrast, when parameters are representable in finite precision (as in real computers), the number of distinct algorithms expressed by Limit Transformers is countable, and ultimate identification from long inputs is possible when the precision required for representing the parameters is penalized.

### D.3 Layer Norm

Real-world transformers use Layer Norm or RMSNorm, whereby activations ${\bm{y}}_{i}^{(l)}$ are rescaled to have norm or standard deviation $\sqrt{d}$.
Layer norm can be incorporated into the translation to Limit Transformers (Lemma [52](#theorem52)) by recording terms of the form

$$ ${{\bm{v}}}^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}\left(\prod_{S\in\mathcal{S}_{2}}S\right){{\bm{w}}}^{T}$ (5) $$

when ${{\bm{v}}}\in\mathcal{VO}_{l_{1}}$
and ${{\bm{w}}}\in\mathcal{VO}_{l_{2}}$
and $\mathcal{S}_{1},\mathcal{S}_{2}\in\mathcal{P}$.
We can record these products in further dimensions of $\widehat{{\bm{y}}_{i}^{(l)}}$, so that $\|{\bm{y}}_{i}^{(l)}\|_{2}^{2}$ is recoverable from $\widehat{{\bm{y}}_{i}^{(l)}}$. The simplest approach is then to modify the definition of Limit Transformers by normalizing $\widehat{{\bm{y}}_{i}^{(l)}}$ based on this recovered norm.

## Appendix E Additional Details for Experiments

### E.1 Regular Languages from the Bhattamishra et al 2020 Benchmark

#### E.1.1 Language Definitions

Descriptions follow Bhattamishra et al., ().

Tomita Grammars. Definitions are shown in Table [1](#A5.T1).

$\bm{D_{n}}$ are defined on the alphabet $\Sigma=\{a,b\}$ by the recursion $D_{n}=(aD_{n-1}b)^{*}$.

PARITY. PARITY is $b^{*}(ab^{*}ab^{*})^{*}$. It is contained in the set of algorithmic tasks.

Others. Other languages: $(aa)^{*}$, $(aaaa)^{*}$ and $(abab)^{*}$ (not star-free), $aa^{*}bb^{*}cc^{*}dd^{*}ee^{*}$, $\{ab\}^{*}d\{b,c\}^{*}$, and $\{0,1,2\}^{*}02^{*}$ (star-free).

**Table 1: Tomita Grammars (originally due to Tomita,, ), following Bhattamishra et al., ().**
| Grammar | Star-Free | Definition |
| --- | --- | --- |
| 1 | Yes | 1* |
| 2 | Yes | (10)* |
| 3 | No | strings without odd-length strings of ones followed by odd-length strings of zeros (i.e., no $01^{2n+1}0^{2m+1}1$ substrings) |
| 4 | Yes | strings without any 000’s substrings |
| 5 | No | strings of even length with an even number of 1’s |
| 6 | No | strings where number of 0’s - number of 1’s is divisible by 3 |
| 7 | Yes | 0*1*0*1 |

#### E.1.2 C-RASP Expressiveness

All languages in the benchmark are in $\textbf{TC}^{0}$, and all are expressible in principle by transformers (Liu et al.,, ).
We were able to provably settle the C-RASP expressiveness for all languages, with results shown in Table [2](#A5.T2).

**Table 2: The finite-state languages in the benchmark from Bhattamishra et al., (), with the numbering from Figure [1](#S5.F1), C-RASP expressiveness properties, and three established notions of complexity (star-freeness, dot depth, and membership in the circuit complexity class $\textbf{AC}^{0}$). In the C-RASP columns, “yes” means we found a C-RASP program; “no” means we proved that no C-RASP program can exist. Note that $\{0,1,2\}^{*}02^{*}$ is equivalent to the language $\Sigma^{*}be^{*}$ from Lemma [11](#theorem11). Note also that we discuss PARITY in the algorithmic benchmark, as it is included in . See discussion in Appendix [E.1.2](#A5.SS1.SSS2). See Figure [2](#A5.F2) for a version of Figure [1](#S5.F1) (right) with languages labeled.**
|  |  | C-RASP expressiveness |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| # | Language | $[\emptyset]$ | [periodic, local] | Star-Free? | Dot-Depth | $\textbf{AC}^{0}$? |  |
| 1 | Tomita 1 | yes | yes | yes | 1 | yes |  |
| 2 | Tomita 2 | yes | yes | yes | 1 | yes |  |
| 3 | Tomita 3 | no | no | no | – | yes |  |
| 4 | Tomita 4 | no | yes | yes | 1 | yes |  |
| 5 | Tomita 5 | no | no | no | – | no |  |
| 6 | Tomita 6 | no | no | no | – | no |  |
| 7 | Tomita 7 | yes | yes | yes | 1 | yes |  |
| 8 | $D_{2}$ | yes | yes | yes | 2 | yes |  |
| 9 | $D_{3}$ | yes | yes | yes | 3 | yes |  |
| 10 | $D_{4}$ | yes | yes | yes | 4 | yes |  |
| 11 | $D_{12}$ | yes | yes | yes | 12 | yes |  |
| – | PARITY | no | no | no | – | no |  |
| 12 | $(aa)^{*}$ | no | yes | no | – | yes |  |
| 13 | $(aaaa)^{*}$ | no | yes | no | – | yes |  |
| 14 | $(abab)^{*}$ | no | yes | no | – | yes |  |
| 15 | $aa^{*}bb^{*}cc^{*}dd^{*}ee^{*}$ | yes | yes | yes | 1 | yes |  |
| 16 | $\{a,b\}^{*}d\{b,c\}^{*}$ | yes | yes | yes | 1 | yes |  |
| 17 | $\{0,1,2\}^{*}02^{*}$ | no | no | yes | 2 | yes |  |

We compare C-RASP expressiveness with the standard notions of complexity of finite-state languages considered by Bhattamishra et al., (): whether languages are star-free, and (among the star-free ones) their dot-depth.
While many star-free languages in the sample show length-generalization, star-freeness does not overall account for the observed behavior (Figure [4](#A5.F4)).
Within the star-free languages, a standard complexity metric is dot-depth (Figure [2](#A5.T2)); this again does not accurately predict length-generalization: it succeeds for a language with dot depth 12 but fails for a language with dot depth 2.
We also considered the circuit complexity of regular languages (Barrington et al.,, ).
All regular languages included in the sample are in the class $\textbf{TC}^{0}$; most are also in $\textbf{AC}^{0}$, a smaller class sometimes compared to transformers (Hao et al.,, ; Barcelo et al.,, ).
Transformers show poor length generalization on the non-$\textbf{AC}^{0}$ regular languages(^6^66By the results of Barrington et al., (), regular languages outside of $\textbf{AC}^{0}$ are all, informally speaking, at least as PARITY, and indeed they provably are not in $\textbf{C-RASP}[\text{periodic},\text{local}]$.), but also fail on various languages that are in $\textbf{AC}^{0}$.
On algorithmic problems, transformers succeed on some non-$\textbf{AC}^{0}$ problems such as Majority (Table [3](#A5.T3)).
Overall, C-RASP expressiveness is much more successful than previously considered notions of complexity in accounting for empirical length generalization behavior of transformers.

##### Proof Sketches for Membership Claims

We sketch proofs for all C-RASP expressiveness claims in Table [2](#A5.T2).
We first note that $\textbf{C-RASP}[\text{periodic},\text{local}]$ is closed under the inverse images of morphisms where each symbol is mapped to a string of the same length. That is:

###### Lemma 42 .

$\textbf{C-RASP}[\text{periodic},\text{local}]$ is closed under the inverse images of morphisms where each symbol is mapped to a string of the same length. That is, if $h:\Sigma_{1}\rightarrow\Sigma_{2}^{n}$ (for some fixed $n$) is extended to a map $\Sigma_{2}^{*}\rightarrow\Sigma_{1}^{*}$ and $\mathcal{L}\in\textbf{C-RASP}[\text{periodic},\text{local}]$, then $h^{-1}(\mathcal{L})\in\textbf{C-RASP}[\text{periodic},\text{local}]$.

###### Proof.

Let $h$ be such a morphism. We we take any $\textbf{C-RASP}[\text{periodic},\text{local}]$ program operation $P(i)$ that operates on $h(w)$ into an equivalent program which operates on $w$

For $P(i)=Q_{a}(i)$, we can use $\hat{P}^{0}(i):=Q_{h(a)_{0}}(i),\hat{P}^{1}(i):=Q_{h(a)_{1}}(i)$, and so on.

For $P(i)=Mod_{m,r}(i)$, then for $c\leq n$ we can define (relying on the fact that $m,n$ are fixed)

$$ $\hat{P}^{c}(i):=\displaystyle\bigvee_{k<m}\left(Mod_{m,k}(i)\rightarrow Mod_{m,(ck\mod m)}(i)\right)$ $$

For $P(i)=\textsc{{\#}}\left[j\leq i,\psi(i,j)\right]\;V(j)$, where $\psi(i,j)=\top$, then like before we can define $\hat{P}^{c}(i)$ as

$$ $\displaystyle C^{0}(i)$ $\displaystyle:=\textsc{{\#}}\left[j<i\right]\;\hat{V}^{0}(j)$ $\displaystyle C^{1}(i)$ $\displaystyle:=\textsc{{\#}}\left[j<i\right]\;\hat{V}^{1}(j)$ $\displaystyle\vdots$ $\displaystyle C^{n-1}(i)$ $\displaystyle:=\textsc{{\#}}\left[j<i\right]\;\hat{V}^{n-1}(j)$ $\displaystyle I^{0}(i)$ $\displaystyle:=\hat{A}^{-s}(j)\;\mathbf{?}\;1\;\textbf{:}\;0$ $\displaystyle I^{1}(i)$ $\displaystyle:=\hat{A}^{-(s-1)}(j)\;\mathbf{?}\;1\;\textbf{:}\;0$ $\displaystyle\vdots$ $\displaystyle I^{c}(i)$ $\displaystyle:=\hat{A}^{c}(j)\;\mathbf{?}\;1\;\textbf{:}\;0$ $\displaystyle\hat{P}^{c}(i)$ $\displaystyle:=\displaystyle\sum_{t\in[0,n-1]}C^{t}(i)+\sum_{t\in[0,c]}I^{t}(i)$ $$

If $\psi(i,j):=j=i-d$, then it takes a little bit more care, as the single position $j$ upon which to check $V(j)$ may occur far behind the current position. First, via a similar argument to Proposition [39](#theorem39), it is possible to simulate the counting operation $\textsc{{\#}}\left[j\leq i-k,\psi(i,j)\right]\;V(j)$ for any constant $c\in\mathbb{N}$. We can do this by counting $\textsc{{\#}}\left[j\leq i\right]\;j=i-k^{\prime}\land\psi(i,j){V(j)}$ for each $k^{\prime}\leq k$, since $j=i-k^{\prime}\land\psi(i,j)$ remains a local function, and then subtracting that from the original count over $j\leq i$. Then, for $c\leq d$

$$ $\displaystyle C^{0}(i)$ $\displaystyle:=\textsc{{\#}}\left[j\leq\left\lfloor\frac{d}{n}\right\rfloor\right]\;(0=(n-(d\mod n)))\land\hat{V}^{0}(j)$ $\displaystyle C^{1}(i)$ $\displaystyle:=\textsc{{\#}}\left[j\leq\left\lfloor\frac{d}{n}\right\rfloor\right]\;(1=(n-(d\mod n)))\land\hat{V}^{1}(j)$ $\displaystyle\vdots$ $\displaystyle C^{n-1}(i)$ $\displaystyle:=\textsc{{\#}}\left[j\leq\left\lfloor\frac{d}{n}\right\rfloor\right]\;(n-1=(n-(d\mod n)))\land\hat{V}^{n-1}(j)$ $\displaystyle\hat{P}^{c}(i)$ $\displaystyle:=\displaystyle\sum_{t\in[0,n-1]}C^{t}(i)$ $$

For $c>d$, the position to check occurs beyond $i$, so we use:

$$ $\displaystyle I^{0}(i)$ $\displaystyle:=c-d=0\land\hat{V}^{0}(i)\;\mathbf{?}\;1\;\textbf{:}\;0$ $\displaystyle I^{1}(i)$ $\displaystyle:=c-d=1\land\hat{V}^{0}(i)\;\mathbf{?}\;1\;\textbf{:}\;0$ $\displaystyle\vdots$ $\displaystyle I^{c}(i)$ $\displaystyle:=c-d=c\land\hat{V}^{c}(i)\;\mathbf{?}\;1\;\textbf{:}\;0$ $\displaystyle\hat{P}^{c}(i)$ $\displaystyle:=\displaystyle\sum_{t\in[0,c]}I^{t}(i)$ $$

All other cases are straightforward, and it can be verified that these constructions are correct.
∎

###### Lemma 42 .

###### Proof.

##### Tomita 1 ∈ C-RASP ​ [ ∅ ] absent C-RASP delimited-[] \in\textbf{C-RASP}[\emptyset]

A C-RASP program can detect the presence of a symbol other than $1$ and flag a violation.

##### Tomita 2 ∈ C-RASP ​ [ ∅ ] absent C-RASP delimited-[] \in\textbf{C-RASP}[\emptyset]

A C-RASP program expresses: At each position, either the current symbol is a 0 and the count of ones and zeros is balanced, or the current symbol is a 1 and the count of ones is one more than the count of zeros.

##### Tomita 3 ∉ C-RASP ​ [ ∅ ] absent C-RASP delimited-[] \not\in\textbf{C-RASP}[\emptyset]

For a given string of the form $010^{K}$, the outputs will converge as $K\rightarrow\infty$ by the same argument as for $(aa)^{*}$ in Lemma [38](#theorem38). Hence, Tomita 3 $\not\in\textbf{C-RASP}[\emptyset]$.

##### Tomita 3 ∉ C-RASP ​ [ periodic , local ] absent C-RASP periodic local \not\in\textbf{C-RASP}[\text{periodic},\text{local}]

Informally, the only way periodic and local predicates are likely to help is if the lengths of contiguous blocks of zeros and ones were bounded (local), or the parity of the lengths of $1$ and 0 0 substrings were globally linked to the parity of the positions of transitions $10$ and $01$ (periodic), but neither is the case.
Sketching a formal proof, assume a $\textbf{C-RASP}[\text{periodic},\text{local}]$ program is given for Tomita 3.
First, we eliminate the periodic predicates by labeling every symbol with the position modulo $s$, where $s$ is a multiple of 2 and all moduli appearing in periodic functions; giving an extended alphabet $1_{1},\dots,1_{s};0_{1},\dots,0_{s}$.
For sufficiently large $c$ that is a co-prime with $s$, we can then also eliminate local functions by merging an adjacent block of length $c$ around every transition between ones and zeros into a single symbol $\Lambda$; indexed by the first symbol inside the block and whether the transition happens at the $floor(s/2)$-th (second part has even length) or $ceil(s/2)$-th (second part has odd length) position in the block.
The resulting language, over an extended alphabet, is recognized by a $\textbf{C-RASP}[\emptyset]$ program capable of determining whether a string of the form

$$ $\Lambda_{1_{1},\dots}\Lambda_{0_{(1+c)\%s},\dots}\Lambda_{1_{(1+2c)\%s},\dots}\dots$ (6) $$

contains a substring of the form

$$ $\Lambda_{0_{\dots},even}\Lambda_{1_{\dots},even}\Lambda_{0_{\dots},even}\ \ \ \ \ \ \ \ \text{ or }\ \ \ \ \ \ \ \ \Lambda_{0_{\dots},odd}\Lambda_{1_{\dots},odd}\Lambda_{0_{\dots},odd}$ (7) $$

This is impossible for the same reasons that $\Sigma^{*}bbb\Sigma^{*}$ (Tomita-4) is not in ${\textbf{C-RASP}[\emptyset]}$.(^7^77Formally, Theorems 6.10 and 6.12 in Krebs, () show that the regular languages in $\widehat{MAJ}_{2}[<]$ are contained in ${\bf DA}*{\bf G}$. The second component in this product can capture the first subscript of each $\Lambda$, but not the second. Since the language $\Sigma^{*}aaa\Sigma^{*}\cup\Sigma^{*}bbb\Sigma^{*}$ over $\Sigma=\{a,b\}$ is not in ${\bf DA}$ (shown, e.g., via Theorem 2c in Tesson and Thérien, (), which would entail syntactic congruence of $(aabb)^{\omega}bb(aabb)^{\omega}$ and $(aabb)^{\omega}$), the claimed ${\textbf{C-RASP}[\emptyset]}$ program cannot exist.)

##### Tomita 4 ∉ C-RASP ​ [ ∅ ] absent C-RASP delimited-[] \not\in\textbf{C-RASP}[\emptyset]

By Lemma 6.11 in Krebs, () (discussed in Appendix [C.3.2](#A3.SS3.SSS2)), $\Sigma^{*}bb\Sigma^{*}\not\in\widehat{MAJ}_{2}[<]$.
In analogy, $\Sigma^{*}bbb\Sigma^{*}\not\in\widehat{MAJ}_{2}[<]$.

##### Tomita 4 ∈ C-RASP ​ [ periodic , local ] absent C-RASP periodic local \in\textbf{C-RASP}[\text{periodic},\text{local}]

A C-RASP program tests whether there is a position with a 0 0 where the preceding position also holds a 0 0 and the position preceding that also holds a 0 0.

##### Tomita 5 ∉ C-RASP ​ [ periodic , local ] absent C-RASP periodic local \not\in\textbf{C-RASP}[\text{periodic},\text{local}]

This language is the intersection of PARITY with the strings of even length. $\textbf{C-RASP}[\text{periodic},\text{local}]$ inexpressiveness follows from the same arguments as for PARITY (Lemma [41](#theorem41)).

##### Tomita 6 ∉ C-RASP ​ [ periodic , local ] absent C-RASP periodic local \not\in\textbf{C-RASP}[\text{periodic},\text{local}]

Consider first the language $\mathcal{L}_{3}$ where the number of 1’s is divisible by 3.
This is not in $\textbf{C-RASP}[\text{periodic},\text{local}]$ in analogy to PARITY.
Now consider the length-preserving morphism
$h(1)=001$,
$h(0)=000$.
Then $h(w)\in\mathcal{L}_{Tomita\ 6}\Leftrightarrow w\in\mathcal{L}_{3}$.

##### Tomita 7 ∈ C-RASP ​ [ ∅ ] absent C-RASP delimited-[] \in\textbf{C-RASP}[\emptyset]

Tomita 7 is equivalent to $\{\epsilon\}\cup a^{+}\cup b^{+}\cup a^{+}b^{+}\cup b^{+}a^{+}\cup a^{+}b^{+}a^{+}\cup a^{+}b^{+}a^{+}b^{+}$. It can be shown this is equivalent to $\Sigma^{*}\setminus\Sigma^{*}b\Sigma^{*}a\Sigma^{*}b\Sigma^{*}a\Sigma^{*}$, and this was constructed in section [C.1.5](#A3.SS1.SSS5).
Interestingly, directly implementing this $\textbf{C-RASP}[\emptyset]$ construction in a transformer appears to require at least four layers; this is in contrast to the $\textbf{C-RASP}[\text{periodic},\text{local}]$ construction discussed next, where two layers are sufficient.(^8^88The first layer collects bigrams, the second layer compares the count of each bigram to the count of SOS tokens (which is known to be one) to count the bigrams.)

##### Tomita 7 ∈ C-RASP ​ [ periodic , local ] absent C-RASP periodic local \in\textbf{C-RASP}[\text{periodic},\text{local}]

A C-RASP program can count the number of positions with the bigrams 01, 10 to detect a violation.

##### D n ∈ C-RASP ​ [ ∅ ] subscript 𝐷 𝑛 C-RASP delimited-[] D_{n}\in\textbf{C-RASP}[\emptyset]

Similar to Tomita 2.

##### PARITY ∉ C-RASP ​ [ periodic , local ] absent C-RASP periodic local \not\in\textbf{C-RASP}[\text{periodic},\text{local}]

See Lemma [41](#theorem41).

##### ( a ​ a ) ∗ ∉ C-RASP ​ [ ∅ ] superscript 𝑎 𝑎 C-RASP delimited-[] (aa)^{*}\not\in\textbf{C-RASP}[\emptyset]

See Lemma [38](#theorem38).

##### ( a ​ a ) ∗ ∈ C-RASP ​ [ periodic , local ] superscript 𝑎 𝑎 C-RASP periodic local (aa)^{*}\in\textbf{C-RASP}[\text{periodic},\text{local}]

See Example [C.2.3](#A3.SS2.SSS3).

##### ( a a a a ) ∗ ∉ C-RASP [ ∅ ] , ∈ C-RASP [ periodic , local ] (aaaa)^{*}\not\in\textbf{C-RASP}[\emptyset],\in\textbf{C-RASP}[\text{periodic},\text{local}]

Analogous to $(aa)^{*}$.

##### ( a b a b ) ∗ ∉ C-RASP [ ∅ ] , ∈ C-RASP [ periodic , local ] (abab)^{*}\not\in\textbf{C-RASP}[\emptyset],\in\textbf{C-RASP}[\text{periodic},\text{local}]

Analogous to $(aa)^{*}$.

##### a ​ a ∗ ​ b ​ b ∗ ​ c ​ c ∗ ​ d ​ d ∗ ​ e ​ e ∗ ∈ C-RASP ​ [ ∅ ] 𝑎 superscript 𝑎 𝑏 superscript 𝑏 𝑐 superscript 𝑐 𝑑 superscript 𝑑 𝑒 superscript 𝑒 C-RASP delimited-[] aa^{*}bb^{*}cc^{*}dd^{*}ee^{*}\in\textbf{C-RASP}[\emptyset]

A C-RASP program indicates, first, the presence of “a”, “b”, “c“, “d”, “e”, and, second, that every “a” is preceded only by “a”; every “b” is preceded only by “a” or “b”; every “c” is preceded only by “a”, “b”, “c”; and analogously for “d”, “e”.

##### { a , b } ∗ ​ d ​ { b , c } ∗ ∈ C-RASP ​ [ ∅ ] superscript 𝑎 𝑏 𝑑 superscript 𝑏 𝑐 C-RASP delimited-[] \{a,b\}^{*}d\{b,c\}^{*}\in\textbf{C-RASP}[\emptyset]

There is a single $d$; $a$ can only appear before it; $c$ can only appear after it.

##### { 0 , 1 , 2 } ∗ ​ 02 ∗ ∉ C-RASP ​ [ periodic , local ] superscript 0 1 2 superscript 02 C-RASP periodic local \{0,1,2\}^{*}02^{*}\not\in\textbf{C-RASP}[\text{periodic},\text{local}]

See Lemma [36](#theorem36). Note that this is equivalent to the language $\Sigma^{*}be^{*}$ over the alphabet $\Sigma=\{a,b,e\}$.

### E.2 Algorithmic Tasks

#### E.2.1 Task Definitions for Algorithmic Problems

The tasks are generally from , except for Binary Majority Interleave. Here, we define each formally.

##### Binary Majority.

The binary majority problem identifies the most frequent bit in a sequence of random bits. An example is
{NiceTabular}*8c[hvlines]
SOS & 0 1 ... 0 SEP 1 EOS
The part
{NiceTabular}*4c[hvlines]
0 & 1 ... 0
is the sequence of random bits. We define LEN to be the length of this part. We constrain the sequences such that the number of 0s and 1s are always not equal. The model is trained with the language modeling loss on the part
{NiceTabular}*2c[hvlines]
1 & EOS
, in other words, it is only trained to predict the most frequent bit and EOS token. The minimum length $l_{min}$ of this task is 1.

##### Binary Majority Interleave.

The sequences in this problem are created by interleaving multiple binary majority (see above) inputs while avoiding repeating special tokens (e.g., SOS). We use 3 binary majority sequences to compose one sequence in this task.
Formally speaking, given 3 binary sequences of the same length, $x^{1}_{1},\cdots x^{1}_{n}$, $x^{2}_{1},\cdots x^{2}_{n}$, and $x^{3}_{1},\cdots x^{3}_{n}$, and their corresponding labels (most frequent bits) $y^{1}$, $y^{2}$, $y^{3}$, the interleaved input is SOS $x^{1}_{1},x^{2}_{1},x^{3}_{1},x^{1}_{2},x^{2}_{2},x^{3}_{2},\cdots,x^{1}_{n},x^{2}_{n},x^{3}_{n}$. SEP $y^{1}$, $y^{2}$, $y^{3}$ EOS.

An example is
{NiceTabular}*13c[hvlines]
SOS& 1 0 1 1 0 0 ... SEP 1 0 0 EOS
Len in this problem refers to length between SOS and SEP (excluding). The model is trained with the language modeling loss on the part
{NiceTabular}*4c[hvlines]
1 & 0 0 EOS
and $l_{min}=3$.

##### Majority.

This problem is similar to the binary majority problem except that the vocabulary is bigger. An example is
{NiceTabular}*8c[hvlines]
SOS& c b a b SEP b EOS
,
where
{NiceTabular}*4c[hvlines]
c & b a b
is a sequence of random tokens, each of which is sampled independently from an alphabet of 26 symbols. The Len is defined as the length of this part. We constrain the sequences such that there is always a unique answer. The model is trained with the language modeling loss on the part
{NiceTabular}*2c[hvlines]
b & EOS
. $l_{min}=1$.

##### Sort.

In sort problem, the model outputs a sorted version of the given sequence. An example is
{NiceTabular}*11c[hvlines]
SOS& 14 23 6 9 SEP 6 9 14 23 EOS
where
{NiceTabular}*4c[hvlines]
14 & 23 6 9
is a sequence of unique numbers. Len in this problem refers to the length of this part. The model is trained with the language modeling loss on the part
{NiceTabular}*5c[hvlines]
6 & 9 14 23 EOS
. In this problem, $l_{min}=1$. The total vocabulary size of tokens except for special tokens is equal to the maximum testing length, i.e., 150.

##### Copy (unique).

In this problem, the model outputs the same sequence as the given sequence, which consists of unique tokens. An example is
{NiceTabular}*11c[hvlines]
SOS& 14 23 6 9 SEP 14 23 6 9 EOS
where the first
{NiceTabular}*4c[hvlines]
14 & 23 6 9
is a sequence of unique numbers. Len in this problem refers to length this part. The model is trained with the language modeling loss on the second part
{NiceTabular}*5c[hvlines]
14 & 23 6 9 EOS
. In this problem, $l_{min}=1$. The total vocabulary size of tokens except for special tokens is equal to the maximum testing length, i.e., 150.

##### Copy (repeat).

In this problem, the model outputs the same sequence as the given sequence, which can contain repeated tokens. An example is
{NiceTabular}*9c[hvlines]
SOS& b a b SEP b a b EOS
where the first
{NiceTabular}*3c[hvlines]
b & a b
is a sequence of random symbols. As in , we use an alphabet of only 2 symbols. Len in this problem refers to length this part. Each symbols is sampled independently and uniformly. The model is trained with the language modeling loss on the second part
{NiceTabular}*4c[hvlines]
b & a b EOS
. In this problem, $l_{min}=1$.

##### Parity.

In the parity problem, the model recognizes whether the given sequence contains an even number of 1s and outputs a corresponding token. An example is
{NiceTabular}*9c[hvlines]
SOS& 1 0 0 1 0 SEP e EOS
The bits before SEP and after SOSare a random sequence of bits. Len in this problem refers to the length of this part. The token between SEP and EOS is the label, it can be either “e” or “o”, meaning even or odd number of 1s. The model is trained with the loss on the part
{NiceTabular}*2c[hvlines]
e & EOS
. $l_{min}=0$ for this problem. The bits are randomly sampled in a way such that the number of 1s is distributed uniformly given a fixed Len.

##### Addition.

In the addition problem, the model does binary addition. An example is
{NiceTabular}*12c[hvlines]
SOS& 1 0 1 + 1 0 = 1 1 1 EOS
The two operands are sampled randomly. Len in this problem refers to the total length of them, including “+” and “=”. The model is trained with the loss on the part
{NiceTabular}*4c[hvlines]
1 & 1 1 EOS
. $l_{min}=4$ for this problem. Note that we do not pad zeros in the front of operands to make them of equal length. The length of the first operand is sampled uniformly in [1, Len-2], and the remaining length is for the second operand. After determining the lengths, random bits are sampled uniformly.

#### E.2.2 Limit Transformers and C-RASP expressiveness on algorithmic tasks

**Table 3: Expressiveness properties of algorithmic tasks as defined in Appendix [E.2.1](#A5.SS2.SSS1) and discussed in Appendix [E.2.2](#A5.SS2.SSS2). In the C-RASP columns, “yes” means we found a C-RASP program; “no” means we proved that no C-RASP program can exist; “none found” means that we found no program despite best efforts. All problems are expressible in $\textbf{TC}^{0}$, the tightest known upper-bound on the expressiveness of transformers. We also show membership in the circuit complexity class $\textbf{AC}^{0}$, a smaller class sometimes compared to transformers (Hao et al.,, ; Barcelo et al.,, ); it is not predictive of length generalization either here or in the regular languages benchmark (e.g., Addition is in $\textbf{AC}^{0}$ but Majority is not).**
|  | $\textbf{C-RASP}[\emptyset]$ | $\textbf{C-RASP}[\text{periodic},\text{local}]$ | $\textbf{AC}^{0}$? |
| --- | --- | --- | --- |
| Binary Majority | yes | yes | no |
| Binary Majority Interleave | none found | yes | no |
| Majority | yes | yes | no |
| Sort | yes | yes | yes |
| Copy (unique) | no | yes | yes |
| Copy (repeat) | no | no | yes |
| Parity | no | no | no |
| Addition | no | no | yes |

See Table [3](#A5.T3).
We provide proof sketches.

##### Binary Majority ∈ C-RASP ​ [ ∅ ] absent C-RASP delimited-[] \in\textbf{C-RASP}[\emptyset]

A single count operation is sufficient.

##### Binary Majority Interleave and C-RASP ​ [ ∅ ] C-RASP delimited-[] \textbf{C-RASP}[\emptyset]

We did not find a $\textbf{C-RASP}[\emptyset]$ program, though we do not have a rigorous proof of nonexistence.
Note that, by Lemma [38](#theorem38), even the (seemingly easier) task of determining whether a given input is well-formed (input length is a multiple of the number of different majority sequences) cannot be solved by $\textbf{C-RASP}[\emptyset]$.

##### Binary Majority Interleave ∈ C-RASP ​ [ periodic , local ] absent C-RASP periodic local \in\textbf{C-RASP}[\text{periodic},\text{local}]

Periodic functions can be used to separately implement each count operation.

##### Majority ∈ C-RASP ​ [ ∅ ] absent C-RASP delimited-[] \in\textbf{C-RASP}[\emptyset]

Similar to Binary Majority.

##### Copy (unique) ∈ C-RASP ​ [ periodic , local ] absent C-RASP periodic local \in\textbf{C-RASP}[\text{periodic},\text{local}]

If character (or $n$-gram) repetition is prevented, then the sequence length is bounded by the alphabet, so that the space of possible inputs becomes finite, seemingly precluding the asymptotic analysis done in Theorem [7](#theorem7).
To overcome this (apparent) challenge, we consider two formalizations of this task as operating on unbounded-length inputs.

First, as explained in (and relatedly by Jelassi et al., ()), the unique copying task can be realized with an induction head circuit (Section [4.1](#S4.SS1) and Appendix [C.2.2](#A3.SS2.SSS2)).
More specifically, each position first records whether SEP has already appeared.
An induction circuit then predicts new tokens in proportion to how frequently they have previously followed appearances of the current token before SEP (Section [4.1](#S4.SS1)).
Copying without repetition is a special case where each token occurs at most once, so the output of $f$ in ([10](#S4.E10)) is always 0 or 1.
We show in Appendix [C.2.2](#A3.SS2.SSS2) that the induction head construction from Section [4.1](#S4.SS1) is expressible in $\textbf{C-RASP}[\text{periodic},\text{local}]$ but not in $\textbf{C-RASP}[\emptyset]$.

We also considered a second formalization of the task, in terms of *repeated* copying, where, given an input such as SOS a c b SEP, the model repeatedly copies the string, always predicting the next character, leading to an unbounded sequence SOS a c b SEP a c b SEP a c b SEP.... This turns the copying task into a function $f\in\mathcal{F}(\Sigma)$ that operates on unboundedly long sequences, outputting next-token predictions at each position.

##### Copy (repeat) ∉ C-RASP ​ [ periodic , local ] absent C-RASP periodic local \not\in\textbf{C-RASP}[\text{periodic},\text{local}]

One proof proceeds via communication complexity: By Corollary [13](#theorem13), copying of general strings is not expressible by Limit Transformers and hence not in $\textbf{C-RASP}[\text{periodic},\text{local}]$. While valid, this proof does not make transparent why length generalization is much easier if repetition is avoided. A different approach, not using communication complexity and crucially using the *presence of repetition* proceeds from the fact that over the alphabet $\Sigma\supseteq\{a,b,e\}^{*}$, the language $\Sigma^{*}be^{*}b\Sigma^{*}\not\in\textbf{C-RASP}[\text{periodic},\text{local}]$ (Lemma [35](#theorem35)), and uses it to deduce that, given an input of the form $vbe^{k}bw\#vbe^{k}$ ($v,w\in\Sigma^{*}$, $k$ large), no C-RASP[periodic,local] program can reliably determine whether a $b$ should follow.

##### Sort ∈ C-RASP ​ [ ∅ ] absent C-RASP delimited-[] \in\textbf{C-RASP}[\emptyset]

As explained in , this can be realized by selecting the smallest number in the input that is larger than the last output symbol. This algorithm does not require local or periodic positional information.

Note that, as in COPY (unique), the input length is bounded by the alphabet size in this case, but we can view it as a task defined with unbounded length by the same trick as for COPY (unique), whereby an initial sequence such as SOS a c b SEP is repeatedly sorted, leading to an unbounded sequence SOS a c b SEP a b c SEP a b c SEP....

##### Parity ∉ C-RASP ​ [ periodic , local ] absent C-RASP periodic local \not\in\textbf{C-RASP}[\text{periodic},\text{local}]

See Lemma [41](#theorem41).

##### Addition ∉ C-RASP ​ [ periodic , local ] absent C-RASP periodic local \not\in\textbf{C-RASP}[\text{periodic},\text{local}]

Addition is at least as hard as copying because the special case of adding zero to a number amounts to copying (Corollary [13](#theorem13)).

### E.3 Details of Experimental Setup

As mentioned in the main paper, at train time, we add random offsets to position indices so that all position embeddings are trained. The offsets are sampled uniformly at random in the range $[0,N-|x|]$ (see Section [2](#S2)). Like , we sample independent training batches on the fly instead of using a finite-size training set. In contrast, each test set contains 2000 samples that are sampled at the beginning of each experiment.

For the problems where we train models with language modeling loss, the length of inputs is sampled uniformly from minimum up to maximum length in the specified range. This is true for training data and all test sets. As mentioned before, we also use predictive modeling. At each step, the model outputs a label indicating the set of possible next characters, including EOS. The models are trained on a whole sequence of tokens. In decoder-only models, standard predictive modeling approaches are less straightforward. Therefore, we assess predictions by combining the input and output spaces. For every position in the sequence, we evaluate the predicted character by comparing the output space (where each embedding represents a subset of possible next tokens) against the expected value.

We train decoder-only transformer from scratch, using implementations from Hugging Face Transformers(^9^99[https://huggingface.co/docs/transformers/en/model_doc/gpt2#transformers.GPT2LMHeadModel](https://huggingface.co/docs/transformers/en/model_doc/gpt2#transformers.GPT2LMHeadModel)). We train models for maximum 30k steps with a batch size of 64. We stop training early once the model’s accuracy reaches 100% on the in-distribution test set (the one in range $[l_{min},50]$. The model is trained with a dropout rate of 0.0. We use AdamW, with a weight decay rate of 0.01.

In preliminary experiments, we found that different model architectures, while achieving 100% accuracy on in-distribution data, may perform differently on out-of-distribution data. To draw a conclusion about how the model performs on a problem in general, we determine the hyperparameters as follows: We consider configurations of {1, 2, 4} layers, {1, 2, 4} heads and model dimension of {16, 64, 256}, and learning rate of {0.001, 0.0001}. We sweep all the configurations by iterating over every combination and choose the one that achieves the highest accuracy on $[51,100]$ among those configurations whose accuracy on $[l_{min},50]$ is 100%. When there are multiple such options, e.g., their accuracy on $[51,100]$ is 100%, the one with the simplest architecture is selected (when estimating complexity, we assume the following priority: number of layers $>$ number of heads $>$ model dimension). The final hyperparameters we used for each task are shown in Table [6](#A5.T6) and [7](#A5.T7).

When no configuration from the search space defined above can achieve accuracy of 100% on $[l_{min},50]$, e.g., in the case of ADDITION, we use an extra configuration, where the number of layers is 12, number of heads is 12, model dimension is 768, learning rate is 1e-4 or 3e-5 (if 1e-4 does not work), and a bigger maximum number of iterations, 60k, we also use the first 3k steps as warm-up steps.

After we determine the hyperparameter configuration, we run the experiments with multiple random seeds and report the average accuracy of 5 successful runs (those runs where the model achieves 100% accuracy on in-distribution data). We do no select successful runs in cases where we use the biggest architecture (the 12-layer configuration), because we find in many cases the accuracy on in-distribution data stops at around 99%.

The random baseline plotted for the algorithmic tasks (Figure [1](#S5.F1), left) is computed using a 2-layer MLP with a token embedding layer; hence, the model predicts the next token solely based on the current token. It is trained with the same hyperparameters as transformers, the learning rate is 1e-3.

**Table 4: Experimental Hyperparameters for testing NoPE on the Regular Languages**
| Problem | Model Size | LR | Max Steps |
| --- | --- | --- | --- |
| Tomita-1, 2 | 1 layer; 1 head; 16 dim | 1e-3 | 30k |
| $D_{2}$, $D_{3}$, $D_{4}$, $D_{12}$ | 1 layer; 4 head; 128 dim | 1e-4 | 30k |
| Tomita-4, 7 | 4 layer; 2 head; 64 dim | 1e-3 | 30k |
| $\{a,b\}^{*}d\{b,c\}^{*}$,<br>$aa^{*}bb^{*}cc^{*}dd^{*}ee^{*}$ | 6 layer ; 4 head; 64 dim | 1e-4 | 30k |
| $(aa)^{*}$, $(aaaa)^{*}$, $(abab)^{*}$ | 6 layer; 4 head; 256 dim | 1e-4 | 60k |
| Tomita-3, 5, 6 | 6 layer; 4 head; 256 dim | 1e-4 | 60k |
| $\{0,1,2\}^{*}02*$, Parity | 6 layer; 4 head; 256 dim | 1e-4 | 60k |

**Table 5: Experimental Hyperparameters for testing APE on the Regular Languages.**
| Problem | Model Size | LR | Max Steps |
| --- | --- | --- | --- |
| Tomita-1, 2 | 1 layer; 1 head; 16 dim | 1e-3 | 30k |
| $D_{2}$, $D_{3}$, $D_{4}$, $D_{12}$ | 1 layer; 4 head; 128 dim | 1e-4 | 30k |
| Tomita-4, 7 | 4 layer; 2 head; 128 dim | 1e-3 | 30k |
| $\{a,b\}^{*}d\{b,c\}^{*}$,<br>$aa^{*}bb^{*}cc^{*}dd^{*}ee^{*}$ | 4 layer ; 4 head; 64 dim | 1e-4 | 30k |
| $(aa)^{*}$, $(aaaa)^{*}$, $(abab)^{*}$, Parity | 4 layer; 4 head; 128 dim | 1e-4 | 40k |
| $\{0,1,2\}^{*}02*$, Tomita-3, 5, 6 | 6 layer; 4 head; 128 dim | 1e-3 | 30k |

**Table 6: Experimental hyperparameters for testing APE on each problem. In the last column, numbers in parenthesis mean the warm-up steps, which is 0 when there is no number in parenthesis.**
| Problem | Model Size | LR | Max Steps |
| --- | --- | --- | --- |
| Binary Majority | 1 layer; 1 head; 16 dim | 1e-3 | 30k |
| Binary Majority Interleave | 2 layer; 4 head; 256 dim | 1e-4 | 30k |
| Majority | 1 layer; 2 head; 256 dim | 1e-3 | 30k |
| Sort | 1 layer; 2 head; 256 dim | 1e-4 | 30k |
| Copy (unique) | 2 layer; 1 head; 64 dim | 1e-3 | 30k |
| Copy (repeat) | 4 layer; 4 head; 256 dim | 1e-3 | 30k |
| Parity | 4 layer; 2 head; 256 dim | 1e-4 | 30k |
| Addition | 12 layer; 12 head; 768 dim | 1e-4 | 60k (3k) |

**Table 7: Experimental hyperparameters for testing NoPE on each problem. In the last column, numbers in parenthesis mean the warm-up steps, which is 0 when there is no number in parenthesis.**
| Problem | Model Size | LR | Max Steps |
| --- | --- | --- | --- |
| Binary Majority | 1 layer; 1 head; 16 dim | 1e-3 | 30k |
| Binary Majority Interleave | 12 layer; 12 head; 768 dim | 1e-4 | 60k (3k) |
| Majority | 1 layer; 1 head; 64 dim | 1e-3 | 30k |
| Sort | 1 layer; 1 head; 256 dim | 1e-3 | 30k |
| Copy (unique) | 4 layer; 4 head; 256 dim | 1e-3 | 30k |
| Copy (repeat) | 4 layer; 4 head; 256 dim | 1e-3 | 30k |
| Parity | 12 layer; 12 head; 768 dim | 3e-5 | 60k (3k) |
| Addition | 12 layer; 12 head; 768 dim | 1e-4 | 60k (3k) |

Figure: Figure 2: Detailed results for regular languages with language names, corresponding to the right part of Figure [1](#S5.F1) but with individual languages labeled.
Refer to caption: /html/2410.02140/assets/x2.png

Figure: Figure 3: Membership in the circuit complexity class $\textbf{AC}^{0}$ does not predict transformers’ length generalization on algorithmic problems (top) or regular languages (bottom). Prior work has often linked the expressiveness of transformers to circuit complexity (e.g. Hahn,, ; Hao et al.,, ; ; Strobl,, ; Barcelo et al.,, ). All tasks included in our experiments are in the class $\textbf{TC}^{0}$, the tightest known upper bound on transformers’ expressiveness. A well-known circuit complexity class within $\textbf{TC}^{0}$ is $\textbf{AC}^{0}$, known to upper-bound the power of certain hard-attention models of transformers (Hao et al.,, ; Barcelo et al.,, ), which may raise hopes that it helps understand transformers’ practical abilities. However, membership in this class does not predict transformers’ length generalization behavior. On the algorithmic problems, there is no apparent correlation at all; majority-type problems, which the attention mechanism can easily implement, are not in $\textbf{AC}^{0}$, but problems with super-logarithmic communication complexity such as copying and addition (Corollary [13](#theorem13)) are contained. On the regular languages, $\textbf{AC}^{0}$ exactly covers the class $\textbf{FO}[reg]$. This class can be proven to include all regular languages in C-RASP, but it also includes various languages that transformers length-generalize poorly on, such as Tomita-3. A natural subclass, obtained by restricting the size of $\textbf{AC}^{0}$ circuits to a linear number of wires, yields the class $\textbf{FO}_{2}[Reg]$ (Cadilhac and Paperman,, ), which does not match transformers’ behavior well either, e.g. it includes $\{0,1,2\}^{*}02^{*}$ (bottom right, equals $\Sigma^{*}be^{*}$ from Lemma [11](#theorem11)) but does not include D-12. Taken together, established circuit complexity classes do not account for Transformers’ length generalization behavior. Compare to C-RASP results in Figures [1](#S5.F1) and [2](#A5.F2).
Refer to caption: /html/2410.02140/assets/x3.png

Figure: Figure 4: (1) Comparing length-generalization with a standard notion of the complexity of finite-state languages: Star-free languages (green) do not require modular counting (McNaughton and Papert,, ), have simpler algebraic representations in terms of group-free monoids (Schützenberger,, ), are easily represented by modern state-space models (Sarrof et al.,, ), and match the expressiveness of a formal model of hard attention Transformers (Yang et al.,, ). However, they do not consistently lead to length generalization in transformers, which on the other hand length-generalize on some non-star-free languages such as $(aa)^{*}$. The expressiveness of C-RASP correctly accounts for the observed behavior. (2) Within the star-free languages, a standard complexity metric is dot-depth, with increased dot-depth indicating increased complexity (non-star-free languages are plotted in gray color). Dot-depth does not predict length generalization, which succeeds on some languages at dot depths 1 and 12 and fails at some languages at intermediate depth. See Figure [3](#A5.F3) for further discussion regarding another existing notion of complexity, circuit complexity, also much less successful than C-RASP expressiveness at predicting length generalization. Compare to C-RASP results in Figures [1](#S5.F1) and [2](#A5.F2).
Refer to caption: /html/2410.02140/assets/x5.png

## Appendix F Translating between Transformers and Limit Transformers

Here, we formally introduce the product parameterization, formally state the hypothesis class, and state and prove the technical lemmas establishing the correspondence between ordinary transformers and limit transformers.

### F.1 Product Parameterization

This parameterization is defined as follows:

###### Definition 43 (Product Parameterization) .

For $l=1,\dots,L$, set

$$ $\displaystyle\mathcal{VO}_{0}=$ $\displaystyle\{{\bm{p}}_{i}:i\}\cup\{{\bm{E}}_{\sigma}:\sigma\}$ $\displaystyle\mathcal{VO}_{l}=$ $\displaystyle\{({\bm{B}}_{l})_{\cdot,s}:s=1,\dots,d\}$ $\displaystyle\mathcal{VI}_{0}=$ $\displaystyle\emptyset$ $\displaystyle\mathcal{VI}_{l}=$ $\displaystyle\{({\bm{A}}_{l})_{s,\cdot}:s=1,\dots,d;{\bm{U}}_{\sigma}:\sigma\in\Sigma\}$ $\displaystyle\mathcal{VO}=$ $\displaystyle\bigcup_{l=0,1,\dots,L}\mathcal{VO}_{l}$ $\displaystyle\mathcal{VI}=$ $\displaystyle\bigcup_{l=0,1,\dots,L}\mathcal{VI}_{l}$ $\displaystyle\mathcal{P}=$ $\displaystyle\{\{V_{l_{1},h_{1}},\dots,V_{l_{k},h_{k}}\}\ \ \ :\ \ \ 0\leq k\leq L;\ \ \ \ l_{1}<\dots<l_{k};\ \ \ \ 1\leq h_{i}\leq H\}$ $$

Given a transformer $T$ with $N(T)<\infty$, define:

$$ $\displaystyle\alpha_{l,h,\mathcal{S}_{1},\mathcal{S}_{2},{{\bm{v}}},{{\bm{w}}}}:=$ $\displaystyle{{\bm{v}}}^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right){{\bm{w}}}\in\mathbb{R}$ $\displaystyle\text{ for }1\leq l\leq L;\ \ \ 1\leq h\leq H;\ \ \ {{\bm{v}}}\in\mathcal{VO};\ \ \ {{\bm{w}}}\in\mathcal{VO};\ \ \ \mathcal{S}_{1},\mathcal{S}_{2}\in\mathcal{P}$ $\displaystyle\beta_{\mathcal{S},{{\bm{v}}},{{\bm{w}}}}:=$ $\displaystyle{{\bm{v}}}^{T}\left(\prod_{S\in\mathcal{S}}S\right){{\bm{w}}}\in\mathbb{R}$ $\displaystyle\text{ for }{{\bm{v}}}\in\mathcal{VI}_{l_{1}};\ \ \ \ {{\bm{w}}}\in\mathcal{VO}_{l_{2}};\ \ \ \mathcal{S}\in\mathcal{P}$ $$

where the matrix product over a set $\mathcal{S}\in\mathcal{P}$

$$ $\prod_{S\in\mathcal{S}}S$ (8) $$

is computed in descending order of layers; with the $S$ associated with the lowest layer at the right. For instance,

$$ $\prod_{S\in\{V_{1,h},V_{3,h^{\prime}},V_{4,h^{\prime\prime}}\}}S=V_{4,h^{\prime\prime}}V_{3,h^{\prime}}V_{1,h}$ (9) $$

###### Remark 44 .

Here, we exemplify the Product Parameterization (Definition [43](#theorem43)).

$$ $\displaystyle\alpha_{1,h,\emptyset,\emptyset,{\bm{p}}_{i},{\bm{E}}_{\sigma}}=$ $\displaystyle{\bm{p}}_{i}^{T}{\bm{K}}_{1,h}^{T}{\bm{Q}}_{1,h}{\bm{E}}_{\sigma}$ $\displaystyle\alpha_{2,h,\{{\bm{V}}_{1,h^{\prime}}\},\emptyset,{\bm{p}}_{i},{\bm{p}}_{j}}=$ $\displaystyle{\bm{p}}_{i}^{T}{\bm{V}}_{1,h^{\prime}}^{T}{\bm{K}}_{2,h}^{T}{\bm{Q}}_{2,h}{\bm{p}}_{j}$ $\displaystyle\alpha_{3,h,\{{\bm{V}}_{2,h^{\prime}}{\bm{V}}_{1,h^{\prime\prime}}\},\{{\bm{V}}_{1,h^{\prime\prime\prime}}\},{\bm{E}}_{\sigma},{\bm{E}}_{\tau}}=$ $\displaystyle{\bm{E}}_{\sigma}^{T}{\bm{V}}_{1,h^{\prime\prime}}^{T}{\bm{V}}_{2,h^{\prime}}^{T}{\bm{K}}_{2,h}^{T}{\bm{Q}}_{2,h}{\bm{V}}_{1,h^{\prime\prime\prime}}{\bm{E}}_{\tau}$ $\displaystyle\beta_{\emptyset,({\bm{A}}_{1})_{s,\cdot},{\bm{p}}_{i}}=$ $\displaystyle({\bm{A}}_{1})_{s,\cdot}^{T}{\bm{p}}_{i}$ $\displaystyle\beta_{\{{\bm{V}}_{1,h}\},({\bm{A}}_{3})_{s,\cdot},{\bm{E}}_{\sigma}}=$ $\displaystyle({\bm{A}}_{3})_{s,\cdot}^{T}{\bm{V}}_{1,h}{\bm{E}}_{\sigma}$ $\displaystyle\beta_{\{{\bm{V}}_{3,h^{\prime}},{\bm{V}}_{1,h}\},{\bm{U}}_{\tau},{\bm{E}}_{\sigma}}=$ $\displaystyle{\bm{U}}_{\tau}^{T}{\bm{V}}_{3,h^{\prime}}{\bm{V}}_{1,h}{\bm{E}}_{\sigma}$ $\displaystyle\beta_{\{{\bm{V}}_{3,h^{\prime}},{\bm{V}}_{2,h}\},{\bm{U}}_{\tau},({\bm{B}}_{1})_{\cdot,s}}=$ $\displaystyle{\bm{U}}_{\tau}^{T}{\bm{V}}_{3,h^{\prime}}{\bm{V}}_{2,h}({\bm{B}}_{1})_{\cdot,s}$ $$

###### Remark 45 .

For ease of notation, we have not restricted the layers from which different vector parameters are taken in the definition of $\alpha$ and $\beta$; hence, they will also include products that are not relevant to actual computations, such as

$$ ${{\bm{p}}_{i}}^{T}{\bm{V}}_{2,h}^{T}{\bm{K}}_{1,h^{\prime}}^{T}{\bm{Q}}_{1,h^{\prime\prime}}{\bm{V}}_{3,h^{\prime\prime\prime}}{\bm{p}}_{j}$ (10) $$

where a vector of the form $V_{3,h^{\prime\prime\prime}}{\bm{p}}_{j}$ cannot actually feed into the computation of queries in the first layer.
This is simply for simplicity of notation; such products will not impact results, though one could explicitly exclude them if one wants to obtain tighter quantitative bounds on the parameter count of Limit Transformers in Lemma [52](#theorem52).

###### Definition 43 (Product Parameterization) .

###### Remark 44 .

###### Remark 45 .

### F.2 Formal Definition of Hypothesis Class

###### Definition 46 (Hypothesis Class, corresponds to Definition 4 ) .

Let $p\in\mathbb{N}$ be fixed.
For each $n=1,2,3,\dots$, define the hypothesis class $\Theta_{n}$ as the set of transformers $T$ (as defined in Section [2](#S2)) where

- 1.
$N(T_{n})=n$
- 2.
each parameter vector and matrix of $T$ is represented at $p$ bits of precision
- 3.
each product function (Definition [43](#theorem43)) involving positional encodings is translation-invariant.
That is, every product function involving exactly one positional encoding is constant across positions, and for every $1\leq i,j,i+\Delta,j+\Delta\leq n$,
$\displaystyle\alpha_{l,h,\mathcal{S}_{1},\mathcal{S}_{2},{\bm{p}}_{i},{\bm{p}}_{j}}=\alpha_{l,h,\mathcal{S}_{1},\mathcal{S}_{2},{\bm{p}}_{i+\Delta},{\bm{p}}_{j+\Delta}}$
for all $l,h,\mathcal{S}_{1},\mathcal{S}_{2}$ making these objects well-defined.

###### Definition 46 (Hypothesis Class, corresponds to Definition 4 ) .

### F.3 From Limit Transformers to Transformers

###### Lemma 47 .

Let $T_{\infty}$ be a Limit Transformer satisfying Periodic and Local.
Then there are transformers $T_{1},T_{2},\dots$ ($T_{i}\in\Theta_{i}$) such that, for all $i\in\mathbb{N},x\in\mathfrak{S},o\in\{0,\dots,i-|x|\}$:

$$ $T_{i}(x,o)\equiv T_{\infty}(x,0)\ \ \ \ \text{when }\ \ o+|x|\leq i$ (11) $$

and

$$ $\sup_{i}{\mathcal{R}}(T_{i})<\infty$ (12) $$

###### Proof.

We use hats to indicate the parameters of the constructed transformers.
Fix $N\in\mathbb{N}$; we construct $T_{N}$.

Let $\Delta$ be the periodicity of ${\bm{p}}_{i}$.
The construction sets $\hat{L}=L+2$ and $\hat{H}=\max\{1,H,\Delta\}$.
Each activation has $\hat{d}:=d+N+3\Delta+2$ dimensions, which can be partitioned into six regions:

$$ $\left(\begin{matrix}d\text{ dimensions}\text{ (Region I: main region)}\\ N\text{ dimensions}\text{ (Region II: position region)}\\ \Delta\text{ dimensions}\text{ (Region III: periodic region I)}\\ \Delta\text{ dimensions}\text{ (Region IV: periodic Region II)}\\ 1\text{ dimension}\text{ (Region V: {{SOS}} Region)}\\ \Delta+1\text{ dimensions}\text{ (Region VI: Copied {{SOS}} Region)}\\ \end{matrix}\right)$ (13) $$

Region I directly emulates the computations of the Limit Transformer.
Region II carries absolute positional information, used for simulating the positional functions $\phi_{l,h}$.

We define the token and positional encodings as

$$ $\displaystyle\hat{{\bm{p}}_{i}}=$ $\displaystyle\left(\begin{matrix}0\in\mathbb{R}^{d}\\ {\bm{e}}_{i}\in\mathbb{R}^{N}\\ {\bm{e}}_{(i\%\Delta)+1}\in\mathbb{R}^{\Delta}\\ 0\in\mathbb{R}^{\Delta}\\ 0\in\mathbb{R}\\ 0\in\mathbb{R}^{\Delta+1}\end{matrix}\right)$ $\displaystyle\hat{{\bm{E}}}_{\sigma}=$ $\displaystyle\left(\begin{matrix}{\bm{E}}_{\sigma}\in\mathbb{R}^{d}\\ 0\in\mathbb{R}^{N}\\ 0\in\mathbb{R}^{\Delta}\\ 0\in\mathbb{R}^{\Delta}\\ 1_{\sigma=\$}\in\mathbb{R}\\ 0\in\mathbb{R}^{\Delta+1}\end{matrix}\right)$ $$

where ${\bm{e}}_{i}$ is the $i$-th unit vector.
Region I holds token information.
Region II holds exact position information.
Region III holds modular position information.
Region V indicates whether the token is start-of-sequence (SOS) or not.
Content will be written to Regions IV and VI by attention components.

###### Lemma 47 .

###### Proof.

##### Intuition

At first sight, a simple intuitive translation just uses Regions I and II, placing all parameters of $T_{\infty}$ into Region I, and taking one-hot vectors ${\bm{e}}_{i}$ for Region II to encode exact positional information, so that the functions $\phi_{l,h}$ can be implemented by products $\hat{{\bm{p}}}_{i}\hat{{\bm{K}}}_{T}\hat{{\bm{Q}}}\hat{{\bm{p}}}_{j}$. One would thus use the simpler encoding $\tilde{{\bm{E}}}_{\sigma}:=[{\bm{E}}_{\sigma},0]$ and $\tilde{p}_{i}:=[{\bm{p}}_{i},{\bm{e}}_{i}]$.
Such a translation would be able to reproduce the input-output behavior of $T_{\infty}$.
However, it would fall short in two ways:
First, the positional encodings ${\bm{p}}_{i}$ can give rise to patterns whereby ${\bm{p}}_{i}^{T}K^{T}Q{\bm{p}}_{j}$ is periodic in $j-i$ when $j-i$ is large, and thus bounded away from zero at unboundedly many distances $j-i$, making ([8](#S3.E8)) unbounded.
We will avoid this by routing modular positional information through a value matrix ${\bm{V}}_{1,1}$ before making it available to attention computations; intuitively, this is possible because the vectors ${\bm{p}}_{i}$ have a bounded-dimensional span.
Modular positional information starts out in Region III of $\hat{{\bm{p}}}_{i}$, and is copied by ${\bm{V}}_{1,1}$ into Region IV; no ${\bm{K}}^{T}{\bm{Q}}$ matrix in the construction will directly address Region III.
Second, transformers in $\Theta_{N}$ must satisfy the requirement that all product functions are translation-invariant; such a requirement need not be implemented by $T_{\infty}$ (e.g., MLPs could respond differently to different ${\bm{p}}_{i}$’s), and thus also not by the simple translation sketched.
We overcome this by adding $\Delta$ different attention heads, each assigned to some $k\in\{0,\dots,\Delta-1\}$, each of which primarily attends to the SOS symbol (based on Region V), with a stronger weight in the $k$-th head falling on SOS if the distance between the query position and the SOS position is congruent to $k$ modulo $\Delta$ (based on Region IV).
These attention weights are written, via the value matrix, to Region VI.
An MLP then compares each of these attention weights to the weights resulting from uniform attention, thereby determining the head giving rise to the highest attention weight, and places the matching encoding ${\bm{p}}_{i}$ into Region I.
This construction maintains translation-invariance of all product functions; most importantly, it makes all $\beta$ functions involving positional encodings equal to zero:
The MLP reads from Region VI, whose entries are linear combinations of entries from Region V, which is zero in all $\hat{p}_{i}$.
Crucially, the dependence of the positional encoding ${\bm{p}}_{i}$ written to Region I on the original positional information in $\hat{p}_{i}$ is mediated entirely through attention weights, which do not enter the definition of the product functions.
Once these computations are completed, Region I matches the activations in $T_{\infty}$ at offset 0, and a direct simulation of $T_{\infty}$ can proceed based on Regions I and II.
Taken together, we expand the intuitive simple construction to make rigorous the intuition that bounded-rank positional information can be utilized by transformers even under the constraints that ([8](#S3.E8)) be bounded and that product functions be translation-invariant.

##### Layer 1: Copying Periodic Positional Information to Region IV

In the lowest layer, each position attends to itself and moves the periodic positional information from Region III to Region IV.
Formally:

$$ $\displaystyle{\bm{K}}_{1,1}={\bm{Q}}_{1,1}=\left(\begin{matrix}0&0&0&0&0&0\\ 0&\Omega\cdot{\bm{I}}_{N\times N}&0&0&0&0\\ 0&0&0&0&0&0\\ 0&0&0&0&0&0\\ 0&0&0&0&0&0\\ 0&0&0&0&0&0\end{matrix}\right)$ $\displaystyle{\bm{V}}_{1,1}=\left(\begin{matrix}0&0&0&0&0&0\\ 0&0&0&0&0&0\\ 0&0&0&0&0&0\\ 0&0&{\bm{I}}_{\Delta\times\Delta}&0&0&0\\ 0&0&0&0&0&0\\ 0&0&0&0&0&0\\ \end{matrix}\right)$ $$

for some $\Omega>0$ to be chosen later.
After attention (the MLP does nothing, ${\bm{A}}_{1},{\bm{B}}_{1},{\bm{b}}_{1}\equiv 0$), the output will be

$$ $\hat{{\bm{y}}}_{i}^{(1)}=\left(\begin{matrix}{\bm{E}}_{\sigma}\\ {\bm{e}}_{i}\\ {\bm{e}}_{(i\%\Delta)+1}\\ (1-\epsilon){\bm{e}}_{(i\%\Delta)+1}\\ 1_{x_{i}=\$}\\ 0\end{matrix}\right)$ (14) $$

where $\epsilon<0.1$ when $\Omega$ is sufficiently large.
Region III will not be addressed by any further downstream matrices or vectors.
The idea behind this operation is to ensure no $\hat{{\bm{K}}}^{T}\hat{{\bm{Q}}}$ matrix will directly have to read from Region III – rather, any dependence of attention logits on modular positional information is mediated by the ${\bm{V}}_{1,1}$ matrix. This allows us to keep ([8](#S3.E8)) bounded even in the presence of such dependence, as we detail below.
Note that this strategy importantly relies on $\Delta\leq\mathcal{R}_{\infty}(T_{\infty})$, so that $rank({\bm{V}}_{1,1})$ is bounded independently of N.
Intuitively, modular positional information (unlike the full positional information encoded in Region II) can be routed through bounded-rank components, which enables keeping ([8](#S3.E8)) bounded.

##### Layer 2: Determining Position Relative to SOS

We now add a second layer, in which we attend with $\Delta+1$ different heads, where the $s$-th head tests whether the distance to SOS is congruent to $s$ modulo $\Delta$, and the $\Delta+1$-st head attends uniformly.
By determining which head attends most strongly to SOS, we can read out the relative position with an MLP without breaking the translation invariance of product functions.

For $h=1,\dots,\Delta$, let ${\bm{Q}}_{2,h}$ be such that

$$ ${\bm{Q}}_{2,h}\left(\begin{matrix}\dots\\ \dots\\ \dots\\ {\bm{e}}_{i}\\ \dots\\ \dots\end{matrix}\right)=\left(\begin{matrix}0\\ 0\\ 0\\ {\bm{e}}_{i+h\%\Delta}\\ 0\\ 0\end{matrix}\right)$ (15) $$

Let ${\bm{K}}_{2,h}$ ($h=1,\dots,\Delta$) be the identity matrix restricted to Regions IV and V.
Further, let

$$ ${\bm{Q}}_{2,\Delta+1}={\bm{K}}_{2,\Delta+1}\equiv 0$ (16) $$

Then, for $h=1,\dots,\Delta$,

$$ $\hat{y}_{i}^{(1)}\hat{K}_{2,h}^{T}\hat{Q}_{2,h}\hat{y}_{j}^{(1)}=1_{i-j\equiv h\operatorname{(mod)}\Delta}$ (17) $$

and

$$ $\hat{y}_{i}^{(1)}{\bm{K}}_{2,\Delta+1}^{T}{\bm{Q}}_{2,\Delta+1}\hat{y}_{j}^{(1)}=0$ (18) $$

Intuitively, heads $1,\dots,\Delta$ attend preferentially to positions at a given distance modulo $\Delta$; head $\Delta+1$ attends everywhere.
Define ${\bm{V}}_{2,h}$ for $h=1,\dots,\Delta+1$ by

$$ ${\bm{V}}_{2,h}\left(\begin{matrix}\dots\\ \dots\\ \dots\\ \dots\\ z\\ \dots\end{matrix}\right)=\left(\begin{matrix}0\\ 0\\ 0\\ 0\\ 0\\ z\cdot{\bm{e}}_{h}\end{matrix}\right)$ (19) $$

As the only vector parameter with an entry in Region V is the token embedding for the SOS token, the outcome of this attention block effectively writes the attention falling on the SOS token for each of the $\Delta+1$ attention heads.
We then use a Heaviside MLP with the number of hidden units bounded in $\Delta$ to determine for which $s=1,\dots\Delta$ it holds that entry $s$ in Region VI has a greater (as opposed to smaller) entry than entry $\Delta+1$.
The MLP, via the $B$ matrix, then writes ${\bm{p}}_{s}$ to Region I.
A special case occurs at SOS, where Region V is 1 (it is 0 everywhere else); here, the $\Delta$ MLP units described above are disabled (a 1 in Region V causes a large negative number to be added to their inputs) and the MLP instead writes ${\bm{p}}_{1}$ to Region I.
Overall, $\Delta+1$ MLP units with Heaviside activation are sufficient for this construction.
Overall, after the MLP, the $i$-th position in the string has ${\bm{p}}_{((i-1)\%\Delta)+1}$ added to Region I.
Overall,

$$ $\hat{{\bm{y}}}_{i}^{(2)}=\left(\begin{matrix}{\bm{E}}_{x_{i}}+{\bm{p}}_{i-o+1}\\ {\bm{e}}_{i+o}\\ \dots\\ \dots\\ \dots\\ \dots\end{matrix}\right)=\left(\begin{matrix}{\bm{y}}_{i}^{(0)}\\ {\bm{e}}_{i+o}\\ \dots\\ \dots\\ \dots\\ \dots\end{matrix}\right)$ (20) $$

where $o$ is the offset, and the second equality holds if the Limit Transformer is run at $o=0$.
In layers $2,\dots,\hat{L}$, only Regions I and II will receive any consideration.

##### Higher Layers: Emulating T ∞ subscript 𝑇 T_{\infty}

Next, for $l\geq 1$,
define $\hat{{\bm{K}}}_{l+2,h}^{T}\hat{{\bm{Q}}}_{l+2,h}\in\mathbb{R}^{\hat{d}\times\hat{d}}$ as

$$ $\hat{{\bm{K}}}_{l+2,h}^{T}\hat{{\bm{Q}}}_{l+2,h}=\left(\begin{matrix}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}&0&0&0&0&0\\ 0&W_{l,h}&0&0&0&0\\ 0&0&0&0&0&0\\ 0&0&0&0&0&0\\ 0&0&0&0&0&0\\ 0&0&0&0&0&0\\ \end{matrix}\right)$ (21) $$

where $i\leq j$:

$$ $(W_{l,h})_{i,j}=\phi_{l,h}(i,j)$ (22) $$

to satisfy

$$ ${\bm{p}}_{i}^{T}W_{l,h}{\bm{p}}_{j}=\phi_{l,h}(i,j)$ (23) $$

Then $W$ is a sum of matrices each of which has the value $\phi_{l,h}(i-j,j)$ for some $i$ on an (off-)diagonal and zeros elsewhere; hence

$$ $\|W_{l,h}\|_{2}\leq\sum_{i}|\phi_{l,h}(1,i)|$ (24) $$

Overall, $\|\hat{K}_{l+1,h}^{T}\hat{Q}_{l+1,h}\|$ can be bounded, independently of $N$, in terms of $\|{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}\|$ and $\|\phi_{l,h}\|_{1}$.
As $\phi_{l,h}$ is local, this 1-norm is finite.

We construct all other parameter matrices and vectors by placing the parameter from the Limit Transformer into the Region I and leaving all other regions zero.
Now, by induction, the first $d$ dimensions of any activation will match those of the Limit Transformer, but shifted by two layers:

$$ $\hat{y}_{i}^{(l+2)}=\left(\begin{matrix}{\bm{y}}_{i}^{(l)}\\ {\bm{e}}_{i+o}\\ \dots\\ \dots\\ \dots\\ \dots\end{matrix}\right)$ (25) $$

As the $U$ matrix only reads from Region I, the output will also be the same as for the Limit Transformer.

For the excess heads, the matrices are just set to 0 – in the first or the higher layers – depending on whether $H$ or $\Delta$ is larger than the other.

##### Bounding Norms and Ranks

At $l\geq 2$, now the ranks of ${\bm{V}}_{l,h}$ and the norms of ${\bm{A}}_{l},{\bm{B}}_{l},U$ and the $\ell_{2}$ norms of $e,b,c$ will be the same they were in the Limit Transformer.
The increases from the first and second layer are bounded in terms of $\Delta$ and hence $\mathcal{R}_{\infty}(T_{\infty})$

##### Verifying Boundedness of ( 8 )

By construction, $\hat{{\bm{p}}}_{i}^{T}\hat{{\bm{K}}}_{1,h}^{T}\hat{{\bm{Q}}}_{1,h}\hat{{\bm{p}}}_{j}=\delta_{ij}\delta_{h1}$ and $\hat{{\bm{p}}}_{i}^{T}\hat{{\bm{K}}}_{2,h}^{T}\hat{{\bm{Q}}}_{2,h}\hat{{\bm{p}}}_{j}\equiv 0$.
For the higher layers, the boundedness follows because $T_{\infty}$ satisfies Local.

##### Verifying Translation Invariance

We need to verify that all product functions are translation-invariant.
Each $\hat{p}_{i}$ contains entries in Regions II and III.
In the first layer, we have $\alpha_{1,1,\emptyset,\emptyset,{\bm{p}}_{i},{\bm{p}}_{j}}=\delta_{ij}$, hence, these are translation-invariant.
In the second layer, we have

$$ $\displaystyle\alpha_{2,s,\emptyset,\emptyset,{\bm{p}}_{i},{\bm{p}}_{j}}=$ $\displaystyle\left(\begin{matrix}0\\ {\bm{e}}_{i}\\ {\bm{e}}_{(i\%\Delta)+1}\\ 0\\ 0\\ 0\end{matrix}\right)^{T}{\bm{K}}_{2,s}^{T}{\bm{Q}}_{2,s}\left(\begin{matrix}0\\ {\bm{e}}_{i}\\ {\bm{e}}_{(j\%\Delta)+1}\\ 0\\ 0\\ 0\end{matrix}\right)=0$ (26) $\displaystyle\alpha_{2,s,\{{\bm{V}}_{1}\},\emptyset,{\bm{p}}_{i},{\bm{p}}_{j}}=$ $\displaystyle 0$ (27) $\displaystyle\alpha_{2,s,\{{\bm{V}}_{1}\},\{{\bm{V}}_{1}\},{\bm{p}}_{i},{\bm{p}}_{j}}=$ $\displaystyle\left(\begin{matrix}0\\ {\bm{e}}_{i}\\ {\bm{e}}_{(i\%\Delta)+1}\\ 0\\ 0\\ 0\end{matrix}\right)^{T}{\bm{V}}_{1,1}^{T}{\bm{K}}_{2,s}^{T}{\bm{Q}}_{2,s}{\bm{V}}_{1,1}\left(\begin{matrix}0\\ {\bm{e}}_{i}\\ {\bm{e}}_{(j\%\Delta)+1}\\ 0\\ 0\\ 0\end{matrix}\right)$ (28) $\displaystyle=$ $\displaystyle\left(\begin{matrix}0\\ 0\\ 0\\ {\bm{e}}_{(i\%\Delta)+1}\\ 0\\ 0\end{matrix}\right)^{T}{\bm{K}}_{2,s}^{T}{\bm{Q}}_{2,s}\left(\begin{matrix}0\\ 0\\ 0\\ {\bm{e}}_{(j\%\Delta)+1}\\ 0\\ 0\end{matrix}\right)$ (29) $\displaystyle=$ $\displaystyle 1_{j-i\equiv s(\operatorname{mod}\Delta)}$ (30) $$

These are all translation-invariant.
All $\alpha_{2+l,h,\emptyset,\emptyset,{\bm{p}}_{i},{\bm{p}}_{j}}$ equal a function $\phi_{l,h}(i,j)$ and hence are translation invariant.
No ${\bm{V}}$ matrix ever reads from Region II.
Overall, all $\alpha$ products are translation-invariant.
In higher layers, ${\bm{K}}^{T}{\bm{Q}}$ matrices read from Regions I and II. The positional encodings $\hat{{\bm{p}}}_{i}$ write to Region II but not – not even when mediated directly through value matrices – Region I, and the products are translation-invariant because the functions $\phi_{l,h}$ are.

Consider

$$ $\displaystyle\beta_{\emptyset,({\bm{A}}_{1})_{s,\cdot},{\bm{p}}_{i}}=0$ $$

because the first layer MLP does nothing.
Second,

$$ $\displaystyle\beta_{\emptyset,({\bm{A}}_{2})_{s,\cdot},{\bm{p}}_{i}}=0$ $$

because the second layer MLP only reads from Region VI, and none of ${\bm{p}}_{i},{\bm{V}}_{1}{\bm{p}}_{i}$ contain any entries in Region VI.
Also,

$$ $\displaystyle\beta_{\mathcal{S},{\bm{U}}_{\sigma},{\bm{p}}_{i}}=0$ $\displaystyle\beta_{\{{\bm{V}}_{1}\},{\bm{U}}_{\sigma},{\bm{p}}_{i}}=0$ $\displaystyle\beta_{\{{\bm{V}}_{2}{\bm{V}}_{1}\},{\bm{U}}_{\sigma},{\bm{p}}_{i}}=0$ $\displaystyle\beta_{\{{\bm{V}}_{2}\},{\bm{U}}_{\sigma},{\bm{p}}_{i}}=0$ $$

since none of ${\bm{p}}_{i}$, ${\bm{V}}_{1}{\bm{p}}_{i}$, ${\bm{V}}_{2}{\bm{V}}_{1}{\bm{p}}_{i},{\bm{V}}_{2}{\bm{p}}_{i}$ contain any entries in Region I, where ${\bm{U}}_{\sigma}$ has its entries.

Since ${\bm{V}}_{2,s}$ all read only from Region V, whose entries have no connection to $\hat{p}_{i}$, we also have:

$$ $\displaystyle\beta_{\{{\bm{V}}_{2},\dots\},({\bm{A}}_{l})_{s,\cdot},{\bm{p}}_{i}}=0$ $\displaystyle\beta_{\{{\bm{V}}_{2},\dots\},{\bm{U}}_{\sigma},{\bm{p}}_{i}}=0$ $\displaystyle\beta_{\{{\bm{V}}_{2},{\bm{V}}_{1},\dots\},({\bm{A}}_{l})_{s,\cdot},{\bm{p}}_{i}}=0$ $\displaystyle\beta_{\{{\bm{V}}_{2},{\bm{V}}_{1},\dots\},{\bm{U}}_{\sigma},{\bm{p}}_{i}}=0$ $$

Overall, all $\beta$ products involving ${\bm{p}}_{i}$ are translation-invariant.

∎

### F.4 From Transformers to Limit Transformers

We first establish various smaller lemmas.
The first lemma informally says that, when $f(i,j):={\bm{p}}_{i}^{T}{\bm{A}}{\bm{p}}_{j}$ is translation-invariant in $(i,j)$, and ${\bm{A}}$ has bounded rank, then $f(i,j)$ is periodic. This lemma is key to the prominent role of the Periodic property in Theorem [7](#theorem7).

###### Lemma 48 .

Let $p\in\mathbb{N}$, and let $N\in\mathbb{N}$.
Let ${\bm{p}}_{1},\dots,{\bm{p}}_{N}\in\mathbb{R}^{k}$ such that $\|{\bm{p}}_{i}\|_{2}<C$; let ${\bm{A}}\in\mathbb{R}^{k\times k}$.
Let $f(i,j):={\bm{p}}_{i}^{T}{\bm{A}}{\bm{p}}_{j}$ be translation invariant, in the sense that

$$ $\forall 0\leq i\leq j:\forall M\geq 0:j+M\leq N\Rightarrow f(i,j)=f(i+M,j+M)$ (32) $$

Further assume that, for $i\leq j$, $f(i,j)$ can be expressed with $p$ fractional bits.(^10^1010In particular, this is satisfied if ${\bm{p}}_{i}$, ${\bm{A}}$ are each expressed at some fixed precision $q$, where $p\geq 3q$.)
Define for $n\geq 0$

$$ $G(n):=f(1,1+n)$ (33) $$

Then there is $\Delta\in\mathbb{N}$ upper-bounded in terms of $\operatorname{rank}({\bm{A}})$, $p$, $C$, and $\|{\bm{A}}\|$ (but not $N$) such that

$$ $\forall n:n+\Delta<N\Rightarrow G(n)=G(n+\Delta)$ (34) $$

###### Proof.

Let $\rho:=\operatorname{rank}({\bm{A}})$; it is $>0$ without loss of generality.
We write the singular value decomposition of ${\bm{A}}$ as

$$ ${\bm{A}}={\bm{U}}^{T}\Sigma{\bm{V}}$ (35) $$

where $\Sigma\in\mathbb{R}^{\rho\times\rho}$, ${\bm{U}},{\bm{V}}\in\mathbb{R}^{\rho\times d}$, where $\|{\bm{U}}\|,\|{\bm{V}}\|\leq 1$.
Then we can write

$$ ${\bm{p}}_{i}^{T}{\bm{A}}{\bm{p}}_{j}={\bm{p}}_{i}^{T}{\bm{U}}^{T}\Sigma{\bm{V}}{\bm{p}}_{j}=\left(\begin{matrix}{\bm{U}}{\bm{p}}_{i}\\ {\bm{V}}{\bm{p}}_{i}\end{matrix}\right)^{T}\left(\begin{matrix}{\bm{I}}_{\rho\times\rho}&0_{\rho\times\rho}\end{matrix}\right)^{T}\Sigma\left(\begin{matrix}0_{\rho\times\rho}&{\bm{I}}_{\rho\times\rho}\end{matrix}\right)\left(\begin{matrix}{\bm{U}}{\bm{p}}_{j}\\ {\bm{V}}{\bm{p}}_{j}\end{matrix}\right)$ (36) $$

We will henceforth replace ${\bm{A}}$ by $\left(\begin{matrix}{\bm{I}}_{\rho\times\rho}&0_{\rho\times\rho}\end{matrix}\right)^{T}\Sigma\left(\begin{matrix}0_{\rho\times\rho}&{\bm{I}}_{\rho\times\rho}\end{matrix}\right)\in\mathbb{R}^{\rho\times\rho}$ and ${\bm{p}}_{i}$ by $\left(\begin{matrix}{\bm{U}}{\bm{p}}_{i}\\
{\bm{V}}{\bm{p}}_{i}\end{matrix}\right)\in\mathbb{R}^{2\operatorname{rank}(A)}$; note that this increases the norms of these objects at most by a multiplicative constant while preserving the products ${\bm{p}}_{i}^{T}{\bm{A}}{\bm{p}}_{j}$.

As $\|{\bm{p}}_{i}\|_{2}<C$ for all $i$, we find that, when $N$ is sufficiently large, for each $\epsilon>0$, there are ${\bm{p}}_{i},{\bm{p}}_{j}$ ($i<j$) such that:

$$ $\|{\bm{p}}_{i}-{\bm{p}}_{j}\|_{2}\leq\epsilon$ (37) $$

Take $\Delta:=j-i$.
The minimum required distance $\Delta$ can be upper bounded by considering the $\epsilon$-packing number of $\{v:\|v\|\leq C\}$ and applying the pigeonhole principle.
Hence, overall, $\Delta$ can be upper-bounded in terms of $\epsilon$, $\operatorname{rank}({\bm{A}})$, and $C$. Importantly, this bound is independent of $N$.
Hence, $\forall k\in\{i+\Delta,\dots,N\}$:

$$ $\displaystyle=|G(k-i)-G(k-i-\Delta)|$ $\displaystyle=|f(i,k)-f(i+\Delta,k)|$ $\displaystyle=|f(i,k)-f(i+(j-i),k)|$ $\displaystyle=|{\bm{p}}_{i}^{T}{\bm{A}}{\bm{p}}_{k}-{\bm{p}}_{j}^{T}{\bm{A}}{\bm{p}}_{k}|$ $\displaystyle\leq\|{\bm{p}}_{i}-{\bm{p}}_{j}\|_{2}\|{\bm{A}}{\bm{p}}_{k}\|_{2}$ $\displaystyle\leq\epsilon\cdot\|{\bm{A}}{\bm{p}}_{k}\|_{2}$ $$

Equivalently, using the substitution $l:=k-i-\Delta$, we have, for any $l\in\{l,\dots,N-\Delta\}$:

$$ $|G(l+\Delta)-G(l)|\leq\epsilon\cdot\|{\bm{A}}{\bm{p}}_{k}\|_{2}$ (38) $$

Take $\epsilon=\frac{2^{-p}}{4C\|A\|}$; then

$$ $G(l)=G(l+\Delta)$ (39) $$

due to the assumption about fixed-precision outputs.

∎

###### Lemma 49 .

Assume each parameter in a transformer is represented at $p$ bits of precision.
Then each product function is exactly represented at $(4+2L)p$ bits of precision.

###### Proof.

Each product function consists at most of two vectors, a key and query matrix, and up to $2L$ value matrices. This results in a sum of numbers that each are a product of up to $4+2L$ numbers that each are individual parameters. As each number is represented at $p$ bits of precision, each product is represented at $(4+2L)p$ bits of precision.
∎

It will be useful to define a complexity metric applicable to Limit Transformers:

###### Definition 50 .

For a Limit Transformer $T$, define $\mathcal{R}_{\infty}(T)$ as the sum of

- 1.
$L+H+d$
- 2.
the precision $p$ used for expressing the parameters (Definition [2](#theorem2)), and the precision $p$ used for rounding attention logits and the output of $\exp(\cdot)$ (Section [2](#S2)).
- 3.
the maximum $\ell^{\infty}$ norm of all parameter vectors and matrices (including positional encodings)
- 4.
the minimal periodicity $\Delta$ of the positional encodings(^11^1111Formally, $1$ plus the supremum of the set of $\Delta$’s for which $p_{i}\not\equiv{\bm{p}}_{i+\Delta}$.)
- 5.
$\max_{l,h}\sum_{i=1}^{\infty}|\phi_{l,h}(1,1+i)|^{2}$ (short: $\max_{l,h}\|\phi_{l,h}\|_{2}^{2}$).

###### Proposition 51 .

Let $A\in\mathbb{R}_{+}$. Let $U$ be the set of Limit Transformers $T$ such that $\mathcal{R}_{\infty}(T)\leq A$.
Then the set of parameter settings in $U$, other than the $\phi_{l,h}$ functions, is finite.

###### Proof.

Immediate.
∎

We now state the key lemma translating ordinary transformers to Limit Transformers using the product parameterization:

###### Lemma 52 .

Let $T\in\Theta_{n}$, at $p$ bits of precision. Let the alphabet $\Sigma$ be fixed.(^12^1212The bound on $R_{\infty}(T_{\infty})$ depends on it, but during the inference procedure, the alphabet is assumed fixed.)
Then there is a Limit Transformer $T_{\infty}$ such that $T\equiv T_{\infty}$ at length $\leq n$ and

$$ $R_{\infty}(T_{\infty})\leq F\left({\mathcal{R}}(T)\right)$ (40) $$

for some universal function $F:\mathbb{R}_{+}\rightarrow\mathbb{R}_{+}$
and

$$ ${\bm{p}}_{i}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{p}}_{j}=\phi_{l,h}(i,j)$ (41) $$

for the ${\bm{p}}_{i},{\bm{K}}_{l,h},{\bm{Q}}_{l,h}$ parameters of $T$; for each $l,h$.
In particular, $T_{\infty}$ satisfies Periodic and each $\phi_{l,h}$ is translation-invariant.

We prove the lemma in the remainder of the section.

###### Lemma 48 .

###### Proof.

###### Lemma 49 .

###### Proof.

###### Definition 50 .

###### Proposition 51 .

###### Proof.

###### Lemma 52 .

#### F.4.1 Proving Lemma 52 (I): Preliminaries

We discuss various preliminaries, before presenting the construction, explaining its intuition, and explaining how its soundness is formally proven.

##### Basic Idea

We will construct $T_{\infty}$ so that the entries in every parameter are 0, 1, or one of the product functions from Definition [43](#theorem43).
This will automatically ensure that its parameters are represented at fixed precision $p$ bounded in terms of $\mathcal{R}(T)$, and with each entry bounded in terms of the spectral norms of parameter matrices and the $\ell_{2}$ norms of parameter vectors, hence, also bounded in terms of $\mathcal{R}(T)$.
Importantly, we will use the definition of $\mathcal{R}(T)$ and the definition of $\Theta_{n}$ to restrict attention to a number of product functions that are bounded only in terms of $\mathcal{R}(T)$, independently of $n$.

##### Bounding Active MLP Units

First, given $\|{\bm{A}}_{l}\|_{F}<\infty$, the number of nonzero entries is bounded as $\leq\frac{\|{\bm{A}}_{l}\|_{F}^{2}}{2^{-2p}}$, which is bounded in terms of $\mathcal{R}(T)$.
Similarly, the number of nonzero entries in ${\bm{b}}_{l}$ is bounded as $\leq\frac{\|b\|_{2}^{2}}{2^{-2p}}$, and similarly for ${\bm{B}}_{l}$.
Let $d_{MLP}\leq d$ be, across $l=1,\dots,L$ the maximum of the maximum number of nonzero entries in ${\bm{b}}_{l}$, and of the maximum number of nonzero rows in ${\bm{A}}_{l}$. Without loss of generality, by reordering rows in ${\bm{A}}_{l}$ and columns in ${\bm{B}}_{l}$, we may assume that, in each layer, these entries are in the first $d_{MLP}$ dimensions.
Then $d_{MLP}$ is upper-bounded in terms of these (or $d$, if the bounds exceed $d$); this is bounded in terms of $\mathcal{R}(T)$ and independent of $n$.
In each layer, only $\leq d_{MLP}$ of the MLP units have nonzero input weights ${\bm{A}}_{l}$, output weights ${\bm{B}}_{l}$, or biases ${\bm{b}}_{l}$.
Removing product functions belonging to the inactive units, we set:

$$ $\displaystyle\widehat{\mathcal{VI}_{l}}:=$ $\displaystyle\mathcal{VI}_{l}-\{({\bm{A}}_{l})_{s,\cdot}:s=d_{MLP}+1,\dots,d\}$ $\displaystyle\widehat{\mathcal{VI}}:=$ $\displaystyle\bigcup_{l=1}^{L}\widehat{\mathcal{VI}_{l}}$ $$

Then, the size of $\widehat{\mathcal{VI}}$ is bounded in terms of $\mathcal{R}(T)$.

##### Periodicity of Bounded-Rank Positional Functions

By Lemma [48](#theorem48) all products ${\bm{p}}_{i}^{T}\dots\bm{p}_{j}$ where the intervening material has bounded rank are periodic in $j-i$ with period bounded in terms of the rank of the intervening material, and hence $\mathcal{R}(T)$.
Let $\Delta$ be the least common multiple of the periods obtained from Lemma [48](#theorem48) across the (finitely many) different products of the form $\beta_{l,h,\mathcal{S}_{1},\mathcal{S}_{2},{\bm{p}}_{i},{\bm{p}}_{j}}$ where $\mathcal{S}_{1}\cup\mathcal{S}_{2}\neq\emptyset$.
Define

$$ $\displaystyle\widehat{\mathcal{VO}_{0}}=$ $\displaystyle\{p_{i}:i=1,\dots,\Delta\}\cup\{{\bm{E}}_{\sigma}:\sigma\}$ $\displaystyle\widehat{\mathcal{VO}_{l}}=$ $\displaystyle\{({\bm{B}}_{l})_{\cdot,s}:s=1,\dots,d_{MLP}\},\ \ \ \ l=1,\dots,L$ $\displaystyle\widehat{\mathcal{VO}}=$ $\displaystyle\widehat{\mathcal{VO}_{0}}\cup\bigcup_{l=1}^{T}\widehat{\mathcal{VO}_{l}}$ $$

Then, the size of $\widehat{\mathcal{VO}}$ is bounded in terms of $\mathcal{R}(T)$.

#### F.4.2 Proving Lemma 52 (II): Construction of T ∞ subscript 𝑇 T_{\infty}

##### Translation in Terms of Regions

We use hats (i.e., $\widehat{\cdot}$) to mark the parameters and activations of the Limit Transformer, distinguishing those from the parameters and activations of the original transformer $T$.

Each $d$-dimensional parameter vector and activation (residual streams and attention outputs) is translated to a vector consisting of three regions, each having a fixed number of dimensions bounded in terms of $\mathcal{R}(T)$,
That is, each vector parameter or activation ${\bm{v}}$ (e.g., ${\bm{p}}_{i}$, ${\bm{E}}_{\sigma}$, ${\bm{y}}_{i}^{(l)}$) is translated to a parameter or activation $\widehat{{\bm{v}}}$ (e.g., $\widehat{{\bm{p}}_{i}}$, $\widehat{{\bm{E}}_{\sigma}}$, $\widehat{{\bm{y}}_{i}^{(l)}}$) vector consisting of the following three regions:

$$ $\widehat{{\bm{v}}}=\left(\begin{matrix}\Gamma_{\mathcal{S},{{\bm{w}}}}(\widehat{{\bm{v}}}):\mathcal{S}\in\mathcal{P};{{\bm{w}}}\in\widehat{\mathcal{VI}_{l_{2}}}\\ \Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{w}}_{1},{\bm{w}}_{2}}({\bm{v}}):1\leq l\leq L;1\leq h\leq H;\mathcal{S}_{1}\subseteq\mathcal{T}_{1};\mathcal{T}_{1},\mathcal{T}_{2}\in\mathcal{P},{\bm{w}}_{1},{\bm{w}}_{2}\in\widehat{\mathcal{VO}}\\ \Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{w}}_{1},{\bm{w}}_{2}}({\bm{v}}):1\leq l\leq L;1\leq h\leq H;\mathcal{S}_{2}\subseteq\mathcal{T}_{2};\mathcal{T}_{1},\mathcal{T}_{2}\in\mathcal{P},{\bm{w}}_{1},{\bm{w}}_{2}\in\widehat{\mathcal{VO}}\\ \end{matrix}\right)$ (42) $$

##### Intuition of the Construction

The first region, denoted $\Gamma_{\mathcal{S},{{\bm{w}}}}(\widehat{{\bm{v}}})$, has one entry for every choice of $\mathcal{S}\in\mathcal{P};{{\bm{w}}}\in\widehat{\mathcal{VI}_{l_{2}}}$. Intuitively, the entry $\Gamma_{\mathcal{S},{{\bm{w}}}}(\widehat{{\bm{v}}})$ describes the outcome of applying all value matrices in $\mathcal{S}$ and then finally the vector ${\bm{w}}^{T}$:

$$ $\Gamma_{\mathcal{S},{{\bm{w}}}}(\widehat{{\bm{v}}})={\bm{w}}^{T}\left(\prod_{S\in\mathcal{S}}S\right){\bm{v}}\in\mathbb{R}$ (43) $$

(Recall Definition [43](#theorem43) for the notation $\prod_{S\in\mathcal{S}}S$.)
The second and third regions each have one entry for every choice of $1\leq l\leq L,\mathcal{S}_{2}\subseteq\mathcal{T}_{2};\mathcal{T}_{1},\mathcal{T}_{2}\in\mathcal{P},{\bm{w}}_{1},{\bm{w}}_{2}\in\widehat{\mathcal{VO}}$.
These regions contain the information necessary for computing attention logits. Intuitively, $\mathcal{T}_{1}$, $\mathcal{T}_{2}$ describe the value matrices through which ${\bm{w}}_{1}$ and ${\bm{w}}_{2}$, respectively, pass before the computation of attention logits in layer $l$. For parameter vectors ${\bm{w}}_{1},{\bm{w}}_{2}$ (e.g., token embeddings or columns of a ${\bm{B}}_{l}$ matrix – positional encodings are somewhat special), we simply expect (note the duplicated arguments $\mathcal{T}_{1}$, $\mathcal{T}_{2}$ – these will be explained in the next paragraph):

$$ $\Lambda_{\mathcal{T}_{1},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{w}}_{1},{\bm{w}}_{2}}({\bm{w}}_{1})\Omega_{\mathcal{T}_{2},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{w}}_{1},{\bm{w}}_{2}}({\bm{w}}_{2})={\bm{w}}_{1}^{T}\left(\prod_{S\in\mathcal{T}_{1}}S\right)^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}\left(\prod_{S\in\mathcal{T}_{2}}S\right){\bm{w}}_{2}$ (44) $$

Thus, $\Lambda$ can be viewed as holding key parameters, whereas $\Omega$ can be viewed as holding query parameters, for the contribution that the pair of ${\bm{w}}_{1},{\bm{w}}_{2}$ makes to attention logits in layer $l$, after passing through the value matrices in $\mathcal{T}_{1}$, $\mathcal{T}_{2}$, respectively.
As a convention, at the level of parameter vectors, the $\Lambda$ component will hold the attention logit contribution (the RHS of this equation), whereas the $\Omega$ component will just hold zeros and ones.
At the level of intermediate activations ${\bm{v}}$ (${\bm{y}}_{i}^{(l)}$ or ${\bm{Y}}_{i}^{(l)}$), the situation is slightly more complex: here,

$$ $\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{w}}_{1},{\bm{w}}_{2}}(\widehat{{\bm{y}}_{i}^{(l)}})$ (45) $$

denotes the contribution to attention logits for head $h$ at layer $k$ arising from multiples of $\left(\prod_{S\in\mathcal{T}_{1}}S\right)\widehat{{\bm{w}}_{1}}$ in an activation $\widehat{{\bm{y}}_{i}^{(k-1)}}$ interacting with multiples of $\left(\prod_{S\in\mathcal{T}_{2}}S\right)\widehat{{\bm{w}}_{2}}$ in an activation $\widehat{{\bm{y}}_{j}^{(k-1)}}$; a similar idea applies to $\Omega_{\dots}$. However, additional care is needed to ensure that only contributions from value matrices are counted that were actually passed through. The additional argument $\mathcal{S}_{1}$ serves as a “to-do-list”: it records which of the value matrices in $\mathcal{T}_{1}$ still have to be traversed; whenever an activation passes through a value matrix ${\bm{V}}_{l,h^{\prime}}$, the value matrix $\widehat{{\bm{V}}_{l,h^{\prime}}}$ of the Limit Transformer moves entries from $\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{w}}_{1},{\bm{w}}_{2}}$ to $\Lambda_{\mathcal{S}_{1}-\{{\bm{V}}_{l,h^{\prime}}\},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{w}}_{1},{\bm{w}}_{2}}$ – effectively removing itself from the “to-do-list”. The same princple applies to $\Omega$, which maintains a to-do-list $\mathcal{S}_{2}$ for $\mathcal{T}_{2}$.
In the end, only those components where the to-do-lists are empty (formally, $\mathcal{S}_{1}=\mathcal{S}_{2}=\emptyset$) will enter attention logit computations of the Limit Transformer:

$$ $\phi_{k,h}(i,j)+\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1},\mathcal{T}_{2}}\Lambda_{\emptyset,\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{y}}_{i}^{(l)}})\cdot\Omega_{\emptyset,\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{y}}_{j}^{(l)}})=({\bm{y}}_{i}^{(l)})^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}{\bm{y}}_{j}^{(l)}$ (46) $$

where the sum runs over all ${\bm{v}},{\bm{w}}\in\widehat{\mathcal{VO}}$, and $\mathcal{T}_{1},\mathcal{T}_{2}$ runs over all sets of value matrices from layers $\leq l$.

In the remainder of the proof, we present a detailed formal construction implementing this intuition.
We first define, for each vector ${{\bm{v}}}\in\{{\bm{p}}_{i},{\bm{E}}_{\sigma},{\bm{b}}_{l}:i,l,\sigma\}$ its translation $\widehat{{\bm{v}}}$; throughout, we will define each of the three regions.

##### Vector Parameters 𝒃 l ∈ 𝒱 ​ 𝒪 subscript 𝒃 𝑙 𝒱 𝒪 {{\bm{b}}_{l}}\in\mathcal{VO}

We take $(\widehat{{\bm{b}}_{l}})_{s}:=({\bm{b}}_{l})_{s}$ for $s=1,\dots,d_{MLP}$.

##### Vector Parameters 𝑬 σ ∈ 𝒱 ​ 𝒪 subscript 𝑬 𝜎 𝒱 𝒪 {{\bm{E}}_{\sigma}}\in\mathcal{VO}

The first region provides products with other vectors appearing at higher layers (rows/columns of the ${\bm{A}}_{l}$ matrices and the unembedding matrix). The second and third regions provide products leading up to keys and values.

$$ $\displaystyle\Gamma_{\mathcal{S},{{\bm{w}}}}(\widehat{{\bm{E}}_{\sigma}}):=$ $\displaystyle\begin{cases}\beta_{\mathcal{S},{{\bm{w}}},{{\bm{E}}_{\sigma}}}&\text{if }\mathcal{S}\in\mathcal{P},{{\bm{w}}}\in\widehat{\mathcal{VI}}\\ 0&\text{else}\end{cases}$ $\displaystyle\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{w}}_{1},{\bm{w}}_{2}}(\widehat{{\bm{E}}_{\sigma}}):=$ $\displaystyle\begin{cases}\alpha_{l,h,\mathcal{T}_{1},\mathcal{T}_{2},{\bm{w}}_{1},{\bm{w}}_{2}}&\text{if }{\bm{E}}_{\sigma}={\bm{w}}_{1},\mathcal{S}_{1}=\mathcal{T}_{1}\\ 0&\text{else}\\ \end{cases}$ $\displaystyle\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{w}}_{1},{\bm{w}}_{2}}(\widehat{{\bm{E}}_{\sigma}}):=$ $\displaystyle\begin{cases}1&\text{if }{\bm{E}}_{\sigma}={\bm{w}}_{2},\mathcal{S}_{2}=\mathcal{T}_{2}\\ 0&\text{else}\\ \end{cases}$ $$

##### Vector Parameters: 𝒑 i ^ ^ subscript 𝒑 𝑖 \widehat{{\bm{p}}_{i}}

For ${{\bm{v}}}={\bm{p}}_{i}$, the construction is analogous, however, we zero out the entries for $\Lambda_{\emptyset,\emptyset,l,h,{\bm{p}}_{i},{\bm{p}}_{j}}$ and $\Omega_{\emptyset,\emptyset,l,h,{\bm{p}}_{i},{\bm{p}}_{j}}$, as these will be taken care of by the $\phi_{l,h}$ functions. Formally:

$$ $\displaystyle\Gamma_{\mathcal{S},{{\bm{w}}}}(\widehat{{\bm{p}}_{i}}):=$ $\displaystyle\begin{cases}\beta_{\mathcal{S},{{\bm{w}}},{{\bm{p}}_{i}}}&\text{if }\mathcal{S}\in\mathcal{P},{{\bm{w}}}\in\widehat{\mathcal{VI}}\\ 0&\text{else}\end{cases}$ $\displaystyle\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{w}}_{1},{\bm{w}}_{2}}(\widehat{{\bm{p}}_{i}}):=$ $\displaystyle\begin{cases}\alpha_{l,h,\mathcal{T}_{1},\mathcal{T}_{2},{\bm{w}}_{1},{\bm{w}}_{2}}&\text{if }{\bm{p}}_{i\%\Delta}={\bm{w}}_{1};\left({\bm{w}}_{2}\not\in\{{\bm{p}}_{j}:j\}\vee\mathcal{T}_{1}\cup\mathcal{T}_{2}\neq\emptyset\right),\\ &\ \ \ \ \ \mathcal{S}_{1}=\mathcal{T}_{1}\\ 0&\text{else}\\ \end{cases}$ $\displaystyle\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{w}}_{1},{\bm{w}}_{2}}(\widehat{{\bm{p}}_{i}}):=$ $\displaystyle\begin{cases}1&\text{if }{\bm{p}}_{i\%\Delta}={\bm{w}}_{2};\left({\bm{w}}_{1}\not\in\{{\bm{p}}_{j}:j\}\vee\mathcal{T}_{1}\cup\mathcal{T}_{2}\neq\emptyset\right),\\ &\ \ \ \ \ \ \mathcal{S}_{2}=\mathcal{T}_{2}\\ 0&\text{else}\\ \end{cases}$ $$

We need to establish that $T_{\infty}$ satisfies Periodic with the period $\Delta$ given above.
First, $\beta_{\mathcal{S},{\bm{v}},{\bm{p}}_{i}}$ is independent of $i$ by translation-invariance, thus trivially periodic in $i$.
Second, the $\Lambda$ and $\Omega$ entries are periodic in $i$ with period $\Delta$ by construction.

##### Matrix Parameters: 𝑽 l , h ^ ^ subscript 𝑽 𝑙 ℎ \widehat{{\bm{V}}_{l,h}}

Each entry in the $\widehat{{\bm{V}}_{l,h}}$ matrix is zero or one.
We define it implicitly, in terms of its action on the three different regions:

$$ $\displaystyle\Gamma_{\mathcal{S},{{\bm{w}}}}(\widehat{{\bm{V}}_{l,h}}(\widehat{{\bm{y}}_{l}^{(l)}}))=$ $\displaystyle\begin{cases}\Gamma_{\mathcal{S}\cup\{{\bm{V}}_{l,h}\},{{\bm{w}}}}(\widehat{{\bm{y}}_{l}^{(l)}})&{\bm{V}}_{l,h}\not\in\mathcal{S}\\ 0&\text{else}\end{cases}$ $\displaystyle\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{w}}_{1},{\bm{w}}_{2}}(\widehat{{\bm{V}}_{l,h}}(\widehat{{\bm{y}}_{l}^{(l)}}))=$ $\displaystyle\begin{cases}\Lambda_{\mathcal{S}_{1}\cup\{{\bm{V}}_{l,h}\},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{w}}_{1},{\bm{w}}_{2}}(\widehat{{\bm{y}}_{l}^{(l)}})&{\bm{V}}_{l,h}\not\in\mathcal{S}_{1},{\bm{V}}_{l,h}\in\mathcal{T}_{1}\\ 0&\text{else}\end{cases}$ $\displaystyle\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{w}}_{1},{\bm{w}}_{2}}(\widehat{{\bm{V}}_{l,h}}(\widehat{{\bm{y}}_{l}^{(l)}}))=$ $\displaystyle\begin{cases}\Omega_{\mathcal{S}_{2}\cup\{{\bm{V}}_{l,h}\},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{w}}_{1},{\bm{w}}_{2}}(\widehat{{\bm{y}}_{l}^{(l)}})&{\bm{V}}_{l,h}\not\in\mathcal{S}_{2},{\bm{V}}_{l,h}\in\mathcal{T}_{2}\\ 0&\text{else}\end{cases}$ $$

##### Matrix Parameters: 𝑨 l ^ ^ subscript 𝑨 𝑙 \widehat{{\bm{A}}_{l}} , 𝑩 l ^ ^ subscript 𝑩 𝑙 \widehat{{\bm{B}}_{l}}

Let $s\in\{1,\dots,d_{MLP}\}$.
For the $s$-th unit in the MLP at layer $l$, we first
define the $s$-th row of $\widehat{{\bm{A}}_{l}}$ by setting

$$ $\displaystyle(\widehat{{\bm{A}}_{l}})_{s,\cdot}\cdot\widehat{{\bm{Y}}_{i}^{(l)}}=\Gamma_{\emptyset,({\bm{A}}_{l})_{s,\cdot}}\left(\widehat{{\bm{Y}}_{i}^{(l)}}\right)\in\mathbb{R}$ $$

and define the $s$-th column of $\widehat{{\bm{B}}_{l}}$ as follows – writing $\hat{X}\in\mathbb{R}$ for the $s$-th hidden unit activation:

$$ $\displaystyle\Gamma_{\mathcal{S},{{\bm{w}}}}(\widehat{{\bm{B}}}_{s,\cdot}\hat{X}):=$ $\displaystyle\hat{X}\cdot\beta_{\mathcal{S},{{\bm{w}}},({\bm{B}}_{l})_{\cdot,s}}$ $\displaystyle\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{w}}_{1},{\bm{w}}_{2}}(\widehat{{\bm{B}}_{l}}_{\cdot,s}\hat{X}):=$ $\displaystyle\hat{X}\cdot\begin{cases}\alpha_{l_{1},h,l_{2},\mathcal{T}_{1},\mathcal{T}_{2},({\bm{B}}_{l})_{\cdot,s},{\bm{w}}_{2}}&\text{if }{\bm{w}}_{1}=({\bm{B}}_{l})_{\cdot,s},\mathcal{S}_{1}=\mathcal{T}_{1}\\ 0&\text{else}\\ \end{cases}$ $\displaystyle\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{w}}_{1},{\bm{w}}_{2}}(\widehat{{\bm{B}}_{l}}_{\cdot,s}\hat{X}):=$ $\displaystyle\hat{X}\cdot\begin{cases}1&\text{if }{\bm{w}}_{2}=({\bm{B}}_{l})_{\cdot,s},\mathcal{S}_{2}=\mathcal{T}_{2}\\ 0&\text{else}\\ \end{cases}$ $$

This defines the $\widehat{{\bm{A}}_{l}}$ and $\widehat{{\bm{B}}_{l}}$ matrix parameters, and we can write, letting $\psi_{l,s}$ denote the activation function (ReLU or Heaviside) applying to the $s$-th hidden MLP unit:

$$ $\widehat{{\bm{y}}_{i}^{(l)}}=\widehat{{\bm{Y}}_{i}^{(l)}}+\sum_{s=1}^{d_{MLP}}(\widehat{{\bm{B}}}_{l})_{\cdot,s}\cdot\psi_{l,s}\left((\widehat{{\bm{A}}}_{l})_{s,\cdot}(\widehat{{\bm{Y}}_{i}^{(l)}})+({\widehat{{\bm{b}}_{l}}})_{s}\right)$ (47) $$

or equivalently

$$ $\widehat{{\bm{y}}_{i}^{(l)}}=\widehat{{\bm{Y}}_{i}^{(l)}}+\widehat{{\bm{B}}_{l}}\cdot\phi_{l}(\widehat{{\bm{A}}_{l}}\cdot\widehat{{\bm{Y}}_{i}^{(l)}}+\widehat{{\bm{b}}_{l}})$ (48) $$

matching the formulation of MLPs for our model of transformers (Equation [4](#S2.E4)).

Note that the hidden dimension of the MLP in the Limit Transformer is now $d_{MLP}$, which will be smaller than $\widehat{d}$. We thus pad the remaining rows/columns of $\widehat{{\bm{A}}_{l}},\widehat{{\bm{B}}_{l}}$, and the remaining entries of $\widehat{{\bm{b}}_{l}}$ with zeros.

##### A partial order on sets of value matrices

For $\mathcal{S},\mathcal{T}\in\mathcal{P}$, we write $\mathcal{T}\geq_{l}\mathcal{S}$ to denote that

- 1.
$\mathcal{T}\supseteq\mathcal{S}$
- 2.
$\forall l^{\prime}\in\{1,\dots,L\}:\left[\left({\bm{V}}_{l^{\prime},h^{\prime}}\in\mathcal{S}\right)\Rightarrow l^{\prime}>l\right]$
- 3.
$\forall l^{\prime}\in\{1,\dots,L\}:\left[\left({\bm{V}}_{l^{\prime},h^{\prime}}\in\mathcal{T}-\mathcal{S}\right)\Rightarrow l\geq l^{\prime}\right]$

Intuitively, “$\mathcal{T}\geq_{l}\mathcal{S}$” says that “among the value matrices in $\mathcal{T}$, the activation has already passed through all value matrices at layer $l$ and below”.
For example:

$$ $\displaystyle\{V_{1,h},V_{2,h^{\prime}},V_{3,h^{\prime\prime}},V_{5,^{\prime\prime\prime}}\}\geq_{2}\{V_{3,h^{\prime\prime}},V_{5,^{\prime\prime\prime}}\}$ $\displaystyle\{V_{1,h},V_{2,h^{\prime}},V_{3,h^{\prime\prime}},V_{5,^{\prime\prime\prime}}\}\not\geq_{2}\{V_{4,h^{\prime\prime\prime}}\}$ $\displaystyle\{V_{1,h},V_{2,h^{\prime}},V_{3,h^{\prime\prime}},V_{5,^{\prime\prime\prime}}\}\not\geq_{1}\{V_{3,h^{\prime\prime}},V_{5,^{\prime\prime\prime}}\}$ $\displaystyle\{V_{1,h},V_{2,h^{\prime}},V_{3,h^{\prime\prime}},V_{5,^{\prime\prime\prime}}\}\not\geq_{3}\{V_{3,h^{\prime\prime}},V_{5,^{\prime\prime\prime}}\}$ $$

##### Matrix Parameters: 𝑲 l , h T ​ 𝑸 l , h ^ ^ superscript subscript 𝑲 𝑙 ℎ 𝑇 subscript 𝑸 𝑙 ℎ \widehat{{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}}

We again define them implicitly in terms of regions; this can be realized using matrices $\widehat{{\bm{K}}_{l,h}^{T}},\widehat{{\bm{Q}}_{l,h}}$ where all entries are 0 or 1.
Importantly, we sum only those entries where the “to-do-lists” $\mathcal{S}_{1},\mathcal{S}_{2}$ are empty, and the sets $\mathcal{T}_{1},\mathcal{T}_{2}$ only contain value matrices at layers $\leq l$:

$$ $\displaystyle(\widehat{{\bm{y}}_{i}^{(l)}})^{T}\widehat{{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}}\widehat{{\bm{y}}_{j}^{(l)}}=$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\emptyset,\mathcal{T}_{2}\geq_{l}\emptyset}\Lambda_{\emptyset,\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{y}}_{i}^{(l)}})\cdot\Omega_{\emptyset,\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{y}}_{j}^{(l)}})$ (49) $$

##### Matrix Parameters: 𝑼 𝑼 {\bm{U}}

The ${\bm{U}}$ matrix is translated as follows:

$$ $\widehat{{\bm{U}}}_{\sigma}^{T}\widehat{{\bm{y}}_{i}^{(L)}}=\Gamma_{\emptyset,{\bm{U}}_{\sigma}}(\widehat{{\bm{y}}_{i}^{(L)}})$ (50) $$

##### Positional Function

Define for $l=1,\dots,L$ and $h=1,\dots,H$, when $1\leq i\leq j\leq N(T)$:

$$ $\phi_{l,h}(i,j)={\bm{p}}_{i}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{p}}_{j}$ (51) $$

As $T\in\Theta_{n}$, $\phi_{l,h}(i,j)$ only depends on $j-i$.

##### Bounding ℛ ∞ ​ ( T ∞ ) subscript ℛ subscript 𝑇 \mathcal{R}_{\infty}(T_{\infty})

First, we showed above that $\widehat{d}$ is upper-bounded in $\mathcal{R}(T)$.
Second, all parameters are represented at precision bounded in terms of $\mathcal{R}(T)$: those parameters that are taken from product functions have precision $\leq 4Lp$ bits; those involving the SVDs of $K^{T}Q$ matrices also have bounded precision.
Third, the $\ell^{\infty}$ norm of all parameter vectors is bounded in terms of $\mathcal{R}(T)$ by construction.
Fourth, $\Delta$ is bounded in terms of $\mathcal{R}(T)$ as discussed above.
The boundedness of the fifth term is immediate.

##### Summary

We have constructed a Limit Transformer $T_{\infty}$ such that

$$ $R_{\infty}(T_{\infty})\leq F\left({\mathcal{R}}(T)\right)$ (52) $$

for some universal function $F:\mathbb{R}_{+}\rightarrow\mathbb{R}_{+}$
and

$$ ${\bm{p}}_{i}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{p}}_{j}=\phi_{l,h}(i,j)$ (53) $$

for the ${\bm{p}}_{i},{\bm{K}}_{l,h},{\bm{Q}}_{l,h}$ parameters of $T$; for each $l,h$.
By assumption on $T$, each $\phi_{l,h}$ is translation-invariant.
We have also constructed $\widehat{{\bm{p}}_{i}}$ with period $\Delta$, so that $T_{\infty}$ satisfies Periodic.

#### F.4.3 Proving Lemma 52 (III): Proving Correctness

In order to conclude Lemma [52](#theorem52), it remains to establish the correctness of the translation; that is, $T\equiv T_{\infty}$ at length $\leq N(T)$.
To do this, it suffices to show that both transformers provide the same next-token predictions for each $i=1,\dots,N(T)$:

$$ $\boxed{\widehat{{\bm{U}}}_{\sigma}^{T}\widehat{{\bm{y}}_{i}^{(L)}}=\Gamma_{\emptyset,{\bm{U}}_{\sigma}}(\widehat{{\bm{y}}_{i}^{(L)}})={\bm{U}}_{\sigma}^{T}{\bm{y}}_{i}^{(L)}}$ (54) $$

Informally, proving this requires showing that the attention logits and MLP activations in $T_{\infty}$ match those in $T$; the result then follows from the linearity of $\Gamma_{\emptyset,{\bm{U}}_{\sigma}}$ and the way $\Gamma_{\dots}$ is defined for the vector parameters and how value matrices $\widehat{{\bm{V}}_{l,h}}$ move information.
Formally, we prove the correctness of the translation inductively, by showing the following equalities.
Recall (from Definition [43](#theorem43)) that, when $\mathcal{S}\in\mathcal{P}$ is a set of value matrices, we write $\prod_{S\in\mathcal{S}}S$ for the product of these matrices, ordered by layers, with the matrix associated with the lowest layer at the right.
Then

###### Lemma 53 (Preservation of Products by Translation) .

For layer $l\in\{0,1,\dots,L\}$, for any $\mathcal{S},\mathcal{S}_{1},\mathcal{S}_{2}\in\mathcal{P}$
for any $k>l$, for any $m\geq l$,
and for any ${\bm{w}}\in\widehat{\mathcal{VI}}$, by induction over the layers $l$:

$$ $\boxed{\begin{aligned} (A)\ \ \ \ \ \ &\text{Preservation of Products with Vector Parameters}\\ &\Gamma_{\mathcal{S},{\bm{w}}}(\widehat{{\bm{y}}_{i}^{(l)}})={\bm{w}}^{T}\left(\prod_{S\in\mathcal{S}}S\right){\bm{y}}_{i}^{(l)}\\ &\text{if }\forall l^{\prime}\in\{1,\dots,L\}:\left[\left({\bm{V}}_{l^{\prime},h^{\prime}}\in\mathcal{S}\right)\Rightarrow l^{\prime}>l\right]\\ (B)\ \ \ \ \ \ &\text{Preservation of Attention Logits (I)}\\ &\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{y}}_{i}^{(l)}})\cdot\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{y}}_{j}^{(l)}})\\ &=({\bm{y}}_{i}^{(l)})^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right){\bm{y}}_{j}^{(l)}\\ &\text{if }\mathcal{S}_{1}\cup\mathcal{S}_{2}\neq\emptyset\\ (C)\ \ \ \ \ \ &\text{Preservation of Attention Logits (II)}\\ &\phi_{k,h}(i,j)+\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\emptyset,\mathcal{T}_{2}\geq_{l}\emptyset}\Lambda_{\emptyset,\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{y}}_{i}^{(l)}})\cdot\Omega_{\emptyset,\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{y}}_{j}^{(l)}})\\ &=({\bm{y}}_{i}^{(l)})^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}{\bm{y}}_{j}^{(l)}\\ (D)\ \ \ \ \ \ &\text{Preservation of Attention Logits (III)}\\ &\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}((\widehat{{\bm{B}}_{m}})_{\cdot,s})\cdot\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{y}}_{j}^{(l)}})\\ &=({\bm{B}}_{m})_{\cdot,s}^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right){\bm{y}}_{j}^{(l)}\\ (E)\ \ \ \ \ \ &\text{Preservation of Attention Logits (IV)}\\ &\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{y}}_{i}^{(l)}})\cdot\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}((\widehat{{\bm{B}}_{m}})_{\cdot,s})\\ &=({\bm{y}}_{i}^{(l)})^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right)({\bm{B}}_{m})_{\cdot,s}\\ \end{aligned}}$ (55) $$

and analogous statements with the post-MLP activations ${\bm{y}}_{i}^{(l)},\widehat{{\bm{y}}_{i}^{(l)}}$ replaced by the pre-MLP activations ${\bm{Y}}_{i}^{(l)},\widehat{{\bm{Y}}}_{i}^{(l)}$.

From (A), we in particular obtain the correctness of the translation by noting that next-token predictions are replicated:

$$ $\widehat{{\bm{U}}}_{\sigma}^{T}\widehat{{\bm{y}}_{i}^{(L)}}=\Gamma_{\emptyset,{\bm{U}}_{\sigma}}(\widehat{{\bm{y}}_{i}^{(L)}})={\bm{U}}_{\sigma}^{T}{\bm{y}}_{i}^{(L)}$ (56) $$

###### Proof of Lemma 53 .

The formal proof proceeds by induction over $l$.
It is conceptually straightforward, consisting of expanding definitions and taking care of the special treatment of positional encodings.
We show it in considerable detail to build intuition.
For the inductive base, at $l=0$, where ${\bm{y}}_{i}^{(0)}$ is a sum of a word embedding and a positional embedding, the claims are immediate from the definitions.
For expository purposes, and for building intuition for the more complex inductive step, we show them in more detail.
Starting from (for simplicity, we are taking the offset to be zero here):

$$ $\displaystyle{{\bm{y}}_{i}^{(0)}}={{\bm{E}}_{x_{i}}}+{{\bm{p}}_{i}}$ (57) $\displaystyle\widehat{{\bm{y}}_{i}^{(0)}}=\widehat{{\bm{E}}_{x_{i}}}+\widehat{{\bm{p}}_{i}}$ $$

we first, for (A), write

$$ $\displaystyle\Gamma_{\mathcal{S},{\bm{w}}}(\widehat{{\bm{E}}_{x_{i}}}+\widehat{{\bm{p}}_{i}})$ $\displaystyle=\Gamma_{\mathcal{S},{\bm{w}}}(\widehat{{\bm{E}}_{x_{i}}})+\Gamma_{\mathcal{S},{\bm{w}}}(\widehat{{\bm{p}}_{i}})$ $\displaystyle=\beta_{\mathcal{S},{\bm{w}},{\bm{E}}_{x_{i}}}+\beta_{\mathcal{S},{\bm{w}},{\bm{p}}_{i\%\Delta}}$ $\displaystyle={\bm{w}}^{T}\left(\prod_{S\in\mathcal{S}}S\right){\bm{y}}_{i}^{(0)}$ $$

proving case (A) of the inductive base.
Second, for (B) and (C), write using the linearity of $\Lambda_{\dots},\Omega_{\dots}$:

$$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{0}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{0}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{E}}_{x_{i}}}+\widehat{{\bm{p}}_{i}})\cdot\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{E}}_{x_{i}}}+\widehat{{\bm{p}}_{i}})$ $\displaystyle=$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{0}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{0}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{E}}_{x_{i}}})\cdot\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{E}}_{x_{i}}})$ $\displaystyle+\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{0}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{0}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{E}}_{x_{i}}})\cdot\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{p}}_{i}})$ $\displaystyle+\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{0}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{0}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{p}}_{i}})\cdot\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{E}}_{x_{i}}})$ $\displaystyle+\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{0}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{0}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{p}}_{i}})\cdot\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{p}}_{i}})$ $$

The only way of satisfying $\mathcal{T}\geq_{0}\mathcal{S}$ is for $\mathcal{T}$ to equal $\mathcal{S}$. After plugging in the definitions, the sums collapse due to the indicator terms in the definition of the token and positional encodings, and we obtain after simplifying:

$$ $\displaystyle=$ $\displaystyle\alpha_{k,h,\mathcal{S}_{1},\mathcal{S}_{2},{{\bm{E}}_{x_{i}}},{{\bm{E}}_{x_{j}}}}$ $\displaystyle+\alpha_{\mathcal{S}_{1},\mathcal{S}_{1},\mathcal{S}_{2},k,h,{{\bm{E}}_{x_{i}}},{\bm{p}}_{j\%\Delta}}$ $\displaystyle+\alpha_{\mathcal{S}_{1},\mathcal{S}_{1},\mathcal{S}_{2},k,h,{{\bm{p}}_{i\%\Delta}},{\bm{E}}_{x_{j}}}$ $\displaystyle+\alpha_{\mathcal{S}_{1},\mathcal{S}_{1},\mathcal{S}_{2},k,h,{{\bm{p}}_{i\%\Delta}},{\bm{p}}_{j\%\Delta}}\cdot 1_{\mathcal{S}_{1}\cup\mathcal{S}_{2}\neq\emptyset}$ $$

By translation-invariance, the second and third term are independent of the positional encoding arguments. By our choice of $\Delta$ at the beginning of the proof, the fourth term equals $\alpha_{\mathcal{S}_{1},\mathcal{S}_{1},\mathcal{S}_{2},k,h,{{\bm{p}}_{i}},{\bm{p}}_{j}}\cdot 1_{\mathcal{S}_{1}\cup\mathcal{S}_{2}\neq\emptyset}$, as this is periodic in $(i,j)$ with periodicity $\Delta$.
We can thus rewrite as

$$ $\displaystyle=$ $\displaystyle\alpha_{k,h,\mathcal{S}_{1},\mathcal{S}_{2},{{\bm{E}}_{x_{i}}},{{\bm{E}}_{x_{j}}}}$ $\displaystyle+\alpha_{\mathcal{S}_{1},\mathcal{S}_{1},\mathcal{S}_{2},k,h,{{\bm{E}}_{x_{i}}},{\bm{p}}_{j}}$ $\displaystyle+\alpha_{\mathcal{S}_{1},\mathcal{S}_{1},\mathcal{S}_{2},k,h,{{\bm{p}}_{i}},{\bm{E}}_{x_{j}}}$ $\displaystyle+\alpha_{\mathcal{S}_{1},\mathcal{S}_{1},\mathcal{S}_{2},k,h,{{\bm{p}}_{i}},{\bm{p}}_{j}}\cdot 1_{\mathcal{S}_{1}\cup\mathcal{S}_{2}\neq\emptyset}$ $$

Applying the definition of $\alpha_{\dots}$, the above equals

$$ $\displaystyle=$ $\displaystyle\begin{cases}({\bm{y}}_{i}^{(0)})^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}({\bm{y}}_{j}^{(0)})-\phi_{k,h}(i,j)&\mathcal{S}_{1}\cup\mathcal{S}_{2}=\emptyset\\ ({\bm{y}}_{i}^{(0)})^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right)({\bm{y}}_{j}^{(0)})&\mathcal{S}_{1}\cup\mathcal{S}_{2}\neq\emptyset\\ \end{cases}$ $$

This establishes cases (B) and (C) of the inductive base.
The proof of cases (D) and (E) in the inductive base is analogous.

For the inductive step, the intuition is that each activation ${\bm{y}}_{i}^{(l)}$ is a linear combination of vector parameters, with different sets of value matrices acting on those:

$$ ${\bm{y}}_{i}^{(l)}=\sum_{{\bm{v}}\in\mathcal{VO}}\sum_{\mathcal{S}\in\mathcal{P}}\lambda_{{\bm{v}},i,l,\mathcal{S}}\left(\prod_{S\in\mathcal{S}}S\right){\bm{v}}$ (58) $$

where the coefficients are determined by attention weights and the activations of MLP hidden units.
Importantly, the attention weights and MLP activations turn out to be the same in the Limit Transformer as in the original transformer, provided we can prove that attention and MLPs are faithfully simulated (which indeed follows from cases A and C of the inductive claim).
Hence, the same decomposition is valid in the Limit Transformer:

$$ $\widehat{{\bm{y}}_{i}^{(l)}}=\sum_{{\bm{v}}\in\mathcal{VO}}\sum_{\mathcal{S}\in\mathcal{P}}\lambda_{{\bm{v}},i,l,\mathcal{S}}\left(\prod_{S\in\mathcal{S}}\widehat{S}\right)\widehat{{\bm{v}}}$ (59) $$

with the same $\lambda_{{\bm{v}},i,l,\mathcal{S}}$ coefficients as in the original transformer.
Then, case (A) of the inductive claim follows intuitively by the calculation:

$$ $\displaystyle{\bm{w}}^{T}{\bm{y}}_{i}^{(l)}=$ $\displaystyle\sum_{{\bm{v}}\in\mathcal{VO}}\sum_{\mathcal{S}\in\mathcal{P}}\lambda_{{\bm{v}},i,l,\mathcal{S}}{\bm{w}}^{T}\left(\prod_{S\in\mathcal{S}}S\right){\bm{v}}$ $\displaystyle=$ $\displaystyle\sum_{{\bm{v}}\in\mathcal{VO}}\sum_{\mathcal{S}\in\mathcal{P}}\lambda_{{\bm{v}},i,l,\mathcal{S}}\widehat{{\bm{w}}}^{T}\left(\prod_{S\in\mathcal{S}}\widehat{S}\right)\widehat{{\bm{v}}}$ $\displaystyle=$ $\displaystyle\widehat{{\bm{w}}}^{T}\sum_{{\bm{v}}\in\mathcal{VO}}\sum_{\mathcal{S}\in\mathcal{P}}\lambda_{{\bm{v}},i,l,\mathcal{S}}\left(\prod_{S\in\mathcal{S}}\widehat{S}\right)\widehat{{\bm{v}}}$ $\displaystyle=$ $\displaystyle\widehat{{\bm{w}}}^{T}\widehat{{\bm{y}}_{i}^{(l)}}$ $$

which is warranted provided that, when ${\bm{b}}\in\mathcal{VO}$ and ${\bm{w}}\in\mathcal{VI}$, we have that $\widehat{{\bm{w}}}^{T}\left(\prod_{S\in\mathcal{S}}\widehat{S}\right)\widehat{{\bm{v}}}$ equals ${\bm{w}}^{T}\left(\prod_{S\in\mathcal{S}}S\right){\bm{v}}$ – this is ensured because of the way the vector parameters $\widehat{{\bm{v}}}$ and the value matrices $\widehat{{\bm{V}}_{l,h}}$ are defined.
The same idea establishes cases (D–E).
A similar, though somewhat more complex (due to the *bilinear* nature of attention) calculation establishes cases (B–C).
Formalizing this reasoning essentially amounts to inductively proving cases (A–E); it will not be necessary to keep track of an explicit decomposition using $\lambda_{\dots}$ coefficients; rather, one can mechanically verify these conditions inductively by plugging in definitions and applying the inductive hypothesis.

Formally proving the inductive step consists in mechanically expanding definitions and applying the inductive hypothesis. First, (C) applied to layer $l-1$ entails that the attention logits for attention heads operating in layer $l$ match those of the original transformer.
We start by establishing the inductive step for the pre-MLP activations ${\bm{Y}}_{i}^{(l)}$.
Starting from:

$$ $\displaystyle{{\bm{Y}}_{i}^{(l)}}=$ $\displaystyle{{\bm{y}}_{i}^{(l-1)}}+\sum_{h=1}^{H}\sum_{j=1}^{i}\tilde{a}_{i,j}^{(l,h)}{{\bm{V}}_{l,h}}{{\bm{y}}_{j}^{(l-1)}}$ (60) $\displaystyle\widehat{{\bm{Y}}_{i}^{(l)}}=$ $\displaystyle\widehat{{\bm{y}}_{i}^{(l-1)}}+\sum_{h=1}^{H}\sum_{j=1}^{i}\tilde{a}_{i,j}^{(l,h)}\widehat{{\bm{V}}_{l,h}}\widehat{{\bm{y}}_{j}^{(l-1)}}$ $$

we show the inductive step first for (A) in the case of the pre-MLP activation ${\bm{Y}}_{i}^{(l)}$. For $\mathcal{S}$ satisfying

$$ $\forall l^{\prime}\in\{1,\dots,L\}:\left[\left({\bm{V}}_{l^{\prime},h^{\prime}}\in\mathcal{S}\right)\Rightarrow l^{\prime}>l\right]$ (61) $$

we consider

$$ $\displaystyle\Gamma_{\mathcal{S},{\bm{w}}}\left(\widehat{{\bm{Y}}_{i}^{(l)}}\right)=$ $\displaystyle\Gamma_{\mathcal{S},{\bm{w}}}\left(\widehat{{\bm{y}}_{i}^{(l-1)}}\right)+\sum_{h=1}^{H}\sum_{j=1}^{i}\tilde{a}_{i,j}^{(l,h)}\Gamma_{\mathcal{S},{\bm{w}}}\left(\widehat{{\bm{V}}}_{l,h}\widehat{{\bm{y}}_{j}^{(l-1)}}\right)$ $\displaystyle=$ $\displaystyle\Gamma_{\mathcal{S},{\bm{w}}}\left(\widehat{{\bm{y}}_{i}^{(l-1)}}\right)+\sum_{h=1}^{H}\sum_{j=1}^{i}\tilde{a}_{i,j}^{(l,h)}\Gamma_{\mathcal{S}\cup\{{\bm{V}}_{l,h}\},{\bm{w}}}\left(\widehat{{\bm{y}}_{j}^{(l-1)}}\right)$ $$

where $\tilde{a}_{i,j}$ denotes attention weights.
The claim here now follows from the inductive hypothesis for (A):

$$ $\displaystyle=$ $\displaystyle{\bm{w}}^{T}\left(\prod_{S\in\mathcal{S}}S\right){{\bm{y}}_{i}^{(l-1)}}+\sum_{h=1}^{H}\sum_{j=1}^{i}\tilde{a}_{i,j}^{(l,h)}{\bm{w}}^{T}\left(\prod_{S\in\mathcal{S}}S\right){\bm{V}}_{l,h}{{\bm{y}}_{j}^{(l-1)}}$ $\displaystyle=$ $\displaystyle{\bm{w}}^{T}\left(\prod_{S\in\mathcal{S}}S\right){{\bm{Y}}_{i}^{(l)}}$ $$

where the last step used ([60](#A6.E60)).
Next, we consider (B) and (C).
First, for (B), assuming $\mathcal{T}_{1}\cup\mathcal{T}_{2}\neq\emptyset$, we first find using ([60](#A6.E60)) and the linearity of $\Lambda_{\dots}$ and $\Omega_{\dots}$ (portions changed marked in blue):

$$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1},\mathcal{T}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{Y}}_{i}^{(l)}})\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{Y}}_{j}^{(l)}})$ $\displaystyle=$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1},\mathcal{T}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left({\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\widehat{{\bm{y}}_{i}^{(l-1)}}+\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{i}\tilde{a}_{iw}^{(l,h^{\prime})}\widehat{{\bm{V}}_{l,h^{\prime}}}\widehat{{\bm{y}}_{w}^{(l-1)}}}\right)$ $\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \ \ \cdot\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left({\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\widehat{{\bm{y}}_{j}^{(l-1)}}+\sum_{h^{\prime\prime}=1}^{H}\sum_{w^{\prime}=1}^{j}\tilde{a}_{jw^{\prime}}^{(l,h^{\prime\prime})}\widehat{{\bm{V}}_{l,h^{\prime\prime}}}\widehat{{\bm{y}}_{w^{\prime}}^{(l-1)}}}\right)$ $\displaystyle=$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1},\mathcal{T}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{i}^{(l-1)}}\right)\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{j}^{(l-1)}}\right)$ $\displaystyle+\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1},\mathcal{T}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{i}^{(l-1)}}\right)\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{j}\tilde{a}_{jw}^{(l,h^{\prime})}\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{V}}_{l,h^{\prime}}}\widehat{{\bm{y}}_{w}^{(l-1)}}\right)$ $\displaystyle+\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1},\mathcal{T}_{2}}\sum_{h^{\prime\prime}=1}^{H}\sum_{w=1}^{i}\tilde{a}_{iw}^{(l,h^{\prime\prime})}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{V}}_{l,h^{\prime\prime}}}\widehat{{\bm{y}}_{w}^{(l-1)}}\right)\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{j}^{(l-1)}}\right)$ $\displaystyle+\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1},\mathcal{T}_{2}}\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{i}\tilde{a}_{iw}^{(l,h^{\prime})}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{V}}_{l,h^{\prime}}}\widehat{{\bm{y}}_{w}^{(l-1)}}\right)$ $\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \cdot\sum_{h^{\prime\prime}=1}^{H}\sum_{w^{\prime}=1}^{j}\tilde{a}_{jw^{\prime}}^{(l,h^{\prime\prime})}\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{V}}_{l,h^{\prime\prime}}}\widehat{{\bm{y}}_{w^{\prime}}^{(l-1)}}\right)$ $$

Now the definition of $\widehat{{\bm{V}}}_{l,h}$ allows us to rewrite this as:

$$ $\displaystyle=$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{i}^{(l-1)}}\right)\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{j}^{(l-1)}}\right)$ $\displaystyle+\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{i}^{(l-1)}}\right)\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{j}\tilde{a}_{jw}^{(l,h^{\prime})}\Omega_{{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\mathcal{S}_{2}\cup\{{\bm{V}}_{l,h^{\prime}}\}},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left({\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\widehat{{\bm{y}}_{w}^{(l-1)}}}\right)$ $\displaystyle+\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{i}\tilde{a}_{iw}^{(l,h^{\prime})}\Lambda_{{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\mathcal{S}_{1}\cup\{{\bm{V}}_{l,h^{\prime}}\}},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left({\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\widehat{{\bm{y}}_{w}^{(l-1)}}}\right)\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{j}^{(l-1)}}\right)$ $\displaystyle+\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\sum_{h^{\prime}=1}^{H}\sum_{j=1}^{i}\tilde{a}_{i,j}^{(l,h)}\Lambda_{{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\mathcal{S}_{1}\cup\{{\bm{V}}_{l,h^{\prime}}\}},\mathcal{T}_{1},\mathcal{T}_{2},l,h^{\prime},{\bm{v}},{\bm{w}}}\left({\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\widehat{{\bm{y}}_{j}^{(l-1)}}}\right)$ $\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \cdot\sum_{h^{\prime\prime}=1}^{H}\sum_{w^{\prime}=1}^{j}\tilde{a}_{jw^{\prime}}^{(l,h^{\prime\prime})}\Omega_{{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\mathcal{S}_{2}\cup\{{\bm{V}}_{l,h^{\prime\prime}}\}},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left({{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\widehat{{\bm{y}}_{w^{\prime}}^{(l-1)}}}}\right)$ $$

In order to directly apply the inductive hypothesis, we rearrange the summations:

$$ $\displaystyle=$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{i}^{(l-1)}}\right)\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{j}^{(l-1)}}\right)$ $\displaystyle+{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{j}\tilde{a}_{jw}^{(l,h^{\prime})}\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{i}^{(l-1)}}\right)\Omega_{\mathcal{S}_{2}\cup\{{\bm{V}}_{l,h^{\prime}}\},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{w}^{(l-1)}}\right)$ $\displaystyle+{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{i}\tilde{a}_{iw}^{(l,h^{\prime})}\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}}\Lambda_{\mathcal{S}_{1}\cup\{{\bm{V}}_{l,h^{\prime}}\},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{w}^{(l-1)}}\right)\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{j}^{(l-1)}}\right)$ $\displaystyle+{{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{i}\tilde{a}_{iw}^{(l,h^{\prime})}\sum_{h^{\prime\prime}=1}^{H}\sum_{w^{\prime}=1}^{j}\tilde{a}_{jw^{\prime}}^{(l,h^{\prime\prime})}\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}}}\Lambda_{\mathcal{S}_{1}\cup\{{\bm{V}}_{l,h^{\prime}}\},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{w}^{(l-1)}}\right)$ $\displaystyle\ \ \ \ \ \ \ \ \ \ \ \cdot\Omega_{\mathcal{S}_{2}\cup\{{\bm{V}}_{l,h^{\prime\prime}}\},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{w^{\prime}}^{(l-1)}}\right)$ $$

Note that, as above, $\mathcal{S}_{1}$, $\mathcal{S}_{2}$ are fixed and the sums run over $\mathcal{T}_{1},\mathcal{T}_{2}$.
We can rewrite the above as:

$$ $\displaystyle=$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\mathcal{T}_{1}\geq_{l-1}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l-1}\mathcal{S}_{2}}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{i}^{(l-1)}}\right)\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{j}^{(l-1)}}\right)$ $\displaystyle+\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{j}\tilde{a}_{jw}^{(l,h^{\prime})}\sum_{{\bm{v}},{\bm{w}},{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\mathcal{T}_{1}\geq_{l-1}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l-1}\mathcal{S}_{2}\cup\{{\bm{V}}_{l,h^{\prime}}\}}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{i}^{(l-1)}}\right)$ $\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \cdot\Omega_{\mathcal{S}_{2}\cup\{{\bm{V}}_{l,h}\},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{w}^{(l-1)}}\right)$ $\displaystyle+\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{i}\tilde{a}_{i,j}^{(l,h)}\sum_{{\bm{v}},{\bm{w}},{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\mathcal{T}_{1}\geq_{l-1}\mathcal{S}_{1}\cup\{{\bm{V}}_{l,h^{\prime}}\},\mathcal{T}_{2}\geq_{l-1}\mathcal{S}_{2}}}\Lambda_{\mathcal{S}_{1}\cup\{{\bm{V}}_{l,h^{\prime}}\},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{j}^{(l-1)}}\right)$ $\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \cdot\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{j}^{(l-1)}}\right)$ $\displaystyle+\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{i}\tilde{a}_{iw}^{(l,h^{\prime})}\sum_{h^{\prime\prime}=1}^{H}\sum_{w^{\prime}=1}^{j}\tilde{a}_{jw^{\prime}}^{(l,h^{\prime\prime})}\sum_{\begin{subarray}{c}{\bm{v}},{\bm{w}},{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\mathcal{T}_{1}\geq_{l-1}\mathcal{S}_{1}\cup\{{\bm{V}}_{l,h^{\prime\prime}}\}},\\ {\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\mathcal{T}_{2}\geq_{l-1}\mathcal{S}_{2}\cup\{{\bm{V}}_{l,h^{\prime}}\}}\end{subarray}}\Lambda_{\mathcal{S}_{1}\cup\{{\bm{V}}_{l,h^{\prime}}\},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{w}^{(l-1)}}\right)$ $\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \cdot\Omega_{\mathcal{S}_{2}\cup\{{\bm{V}}_{l,h^{\prime\prime}}\},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{w^{\prime}}^{(l-1)}}\right)$ $$

We are now ready to apply the inductive hypothesis: Directly plugging the inductive hypothesis for (B) into the second through fourth terms gives us:

$$ $\displaystyle=$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l-1}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l-1}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{i}^{(l-1)}}\right)\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},l,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{j}^{(l-1)}}\right)$ $\displaystyle+{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{j}\tilde{a}_{jw}^{(l,h^{\prime})}({\bm{y}}_{i}^{(l-1)})^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right)^{T}{\bm{V}}_{l,h}{\bm{y}}_{w}^{(l-1)}}$ $\displaystyle+{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{i}\tilde{a}_{iw}^{(l,h^{\prime})}{\bm{y}}_{j}^{(l-1)}{\bm{V}}_{l,h^{\prime}}^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right){\bm{y}}_{j}^{(l-1)}}$ $\displaystyle+{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{i}\tilde{a}_{iw}^{(l,h^{\prime})}\sum_{h^{\prime\prime}=1}^{H}\sum_{w^{\prime}=1}^{j}\tilde{a}_{jw^{\prime}}^{(l,h^{\prime\prime})}({\bm{y}}_{w}^{(l-1)})^{T}{\bm{V}}_{l,h^{\prime}}^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right){\bm{V}}_{l,h^{\prime\prime}}{\bm{y}}_{w^{\prime}}^{(l-1)}}$ $$

We now distinguish two cases, for proving (B) and (C).
The first one is that $\mathcal{S}_{1}=\mathcal{S}_{2}=\emptyset$.
In this case, by case (C) of the inductive hypothesis:

$$ $\displaystyle=$ $\displaystyle({\bm{y}}_{i}^{(l-1)})^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}{\bm{y}}_{j}^{(l-1)}-\phi_{k,h}(i,j)$ $\displaystyle+\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{j}\tilde{a}_{jw}^{(l,h^{\prime})}({\bm{y}}_{i}^{(l-1)})^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}{\bm{V}}_{l,h}{\bm{y}}_{w}^{(l-1)}$ $\displaystyle+\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{i}\tilde{a}_{iw}^{(l,h^{\prime})}{\bm{y}}_{j}^{(l-1)}{\bm{V}}_{l,h^{\prime}}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{y}}_{j}^{(l-1)}$ $\displaystyle+\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{i}\tilde{a}_{iw}^{(l,h^{\prime})}\sum_{h^{\prime\prime}=1}^{H}\sum_{w^{\prime}=1}^{j}\tilde{a}_{jw^{\prime}}^{(l,h^{\prime\prime})}({\bm{y}}_{w}^{(l-1)})^{T}{\bm{V}}_{l,h^{\prime}}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{V}}_{l,h^{\prime\prime}}{\bm{y}}_{w^{\prime}}^{(l-1)}$ $$

Now, applying ([60](#A6.E60)) again, we rearrange to sums to obtain the conclusion

$$ $=({\bm{Y}}_{i}^{(l)})^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}{\bm{Y}}_{j}^{(l)}-\phi_{k,h}(i,j)\\$ (62) $$

proving (upon rearranging) the inductive step for (C) in the case of ${\bm{Y}}_{i}^{(l)}$.
In the second case, $\mathcal{S}_{1}\cup\mathcal{S}_{2}\neq\emptyset$; here, we use case (B) of the inductive hypothesis to instead rewrite as

$$ $\displaystyle=$ $\displaystyle{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}({\bm{y}}_{i}^{(l-1)})^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right)({\bm{y}}_{j}^{(l-1)})}$ $\displaystyle+\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{j}\tilde{a}_{jw}^{(l,h^{\prime})}({\bm{y}}_{i}^{(l-1)})^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right){\bm{V}}_{l,h^{\prime}}{\bm{y}}_{w}^{(l-1)}$ $\displaystyle+\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{i}\tilde{a}_{iw}^{(l,h^{\prime})}{\bm{y}}_{j}^{(l-1)}{\bm{V}}_{l,h^{\prime}}^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right){\bm{y}}_{j}^{(l-1)}$ $\displaystyle+\sum_{h^{\prime}=1}^{H}\sum_{w=1}^{i}\tilde{a}_{iw}^{(l,h^{\prime})}\sum_{h^{\prime\prime}=1}^{H}\sum_{w^{\prime}=1}^{j}\tilde{a}_{jw^{\prime}}^{(l,h^{\prime\prime})}({\bm{y}}_{w}^{(l-1)})^{T}{\bm{V}}_{l,h^{\prime}}^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right){\bm{V}}_{l,h^{\prime\prime}}{\bm{y}}_{w^{\prime}}^{(l-1)}$ $$

Now, applying ([60](#A6.E60)) again, we rearrange to sums to obtain the conclusion

$$ $=({\bm{Y}}_{i}^{(l)})^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right){\bm{Y}}_{j}^{(l)}\\$ (63) $$

This proves the inductive step for (B) in the case of ${\bm{Y}}_{i}^{(l)}$.
We next address the inductive step for (D) in the case of the pre-MLP activation:

$$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{B}}_{m}})\cdot\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}(\widehat{{\bm{Y}}_{j}^{(l)}})$ $\displaystyle=$ $\displaystyle({\bm{B}}_{m})^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right){\bm{Y}}_{j}^{(l)}$ $$

By unfolding $\widehat{{\bm{Y}}_{j}^{(l)}}$ using ([60](#A6.E60)) and using the linearity of $\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}$, the claim follows directly from the inductive hypothesis for (D).
The same reasoning applies to (E).
Overall, we have proven the inductive step (A–E) for the pre-MLP activations ${\bm{Y}}_{i}^{(l)}$.

We now need to show that the inductive step also holds for the post-MLP activations.
Recall that the MLP acts as

$$ $\displaystyle{{\bm{y}}_{i}^{(l)}}={{\bm{Y}}_{i}^{(l)}}+\sum_{s=1}^{d_{MLP}}({{\bm{B}}}_{l})_{\cdot,s}\cdot\psi_{l,s}\left(({{\bm{A}}}_{l})_{s,\cdot}({{\bm{Y}}_{i}^{(l)}})+({{{\bm{b}}_{l}}})_{s}\right)$ (64) $\displaystyle\widehat{{\bm{y}}_{i}^{(l)}}=\widehat{{\bm{Y}}_{i}^{(l)}}+\sum_{s=1}^{d_{MLP}}(\widehat{{\bm{B}}}_{l})_{s,\cdot}\cdot\psi_{l,s}\left((\widehat{{\bm{A}}}_{l})_{s,\cdot}(\widehat{{\bm{Y}}_{i}^{(l)}})+({\widehat{{\bm{b}}_{l}}})_{s}\right)$ $$

The proof proceeds by expanding this equation and reducing the claim to the already-proven inductive step for pre-MLP activations (for handling the direct contribution from the pre-MLP activation), and cases (D) and (E) (for handling the contributions of the MLP units).
First, we note that, for each $l,s$, by the case (A) of the inductive hypothesis and by the definition of $\widehat{{\bm{b}}_{l}}$,

$$ $\psi_{l,s}\left(({{\bm{A}}}_{l})_{s,\cdot}({{\bm{Y}}_{i}^{(l)}})+({{{\bm{b}}_{l}}})_{s}\right)=\psi_{l,s}\left((\widehat{{\bm{A}}}_{l})_{s,\cdot}(\widehat{{\bm{Y}}_{i}^{(l)}})+({\widehat{{\bm{b}}_{l}}})_{s}\right)$ (65) $$

We will abbreviate this number as $\Xi_{l,s,i}\in\mathbb{R}$.
We now prove the case (A) of the inductive step for the post-MLP activation:

$$ $\displaystyle\Gamma_{\mathcal{S},{\bm{w}}}(\widehat{{\bm{y}}_{i}^{(l)}})=$ $\displaystyle\Gamma_{\mathcal{S},{\bm{w}}}(\widehat{{\bm{Y}}_{i}^{(l)}}+\sum_{s=1}^{d_{MLP}}(\widehat{{\bm{B}}}_{l})_{s,\cdot}\cdot\Xi_{l,s,i})$ $\displaystyle=$ $\displaystyle\Gamma_{\mathcal{S},{\bm{w}}}(\widehat{{\bm{Y}}_{i}^{(l)}})+\sum_{s=1}^{d_{MLP}}\Gamma_{\mathcal{S},{\bm{w}}}((\widehat{{\bm{B}}}_{l})_{s,\cdot})\cdot\Xi_{l,s,i}$ $\displaystyle=$ $\displaystyle{\bm{w}}^{T}\left(\prod_{S\in\mathcal{S}}S\right){{\bm{Y}}_{i}^{(l)}}+\sum_{s=1}^{d_{MLP}}{\bm{w}}^{T}\left(\prod_{S\in\mathcal{S}}S\right)({{\bm{B}}}_{l})_{\cdot,s}\cdot\Xi_{l,s,i}$ $\displaystyle=$ $\displaystyle{\bm{w}}^{T}\left(\prod_{S\in\mathcal{S}}S\right){\bm{y}}_{i}^{(l)}$ $$

To prove cases (B) and (C) for the post-MLP activations ${\bm{y}}_{i}^{(l)}$, we now consider

$$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{i}^{(l)}}\right)\cdot\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{y}}_{j}^{(l)}}\right)$ $\displaystyle=$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{Y}}_{i}^{(l)}}+\sum_{s=1}^{d_{MLP}}\Xi_{l,s,i}\cdot(\widehat{{\bm{B}}}_{l})_{\cdot,s}\right)$ $\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \cdot\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{Y}}_{j}^{(l)}}+\sum_{t=1}^{d_{MLP}}\Xi_{l,t,j}\cdot(\widehat{{\bm{B}}}_{l})_{\cdot,t}\right)$ $\displaystyle=$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{Y}}_{i}^{(l)}}\right)\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{Y}}_{j}^{(l)}}\right)$ $\displaystyle+\sum_{s=1}^{d_{MLP}}\Xi_{l,s,j}\cdot\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{Y}}_{i}^{(l)}}\right)\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left((\widehat{{\bm{B}}}_{l})_{\cdot,s}\right)$ $\displaystyle+\sum_{s=1}^{d_{MLP}}\Xi_{l,s,i}\cdot\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left((\widehat{{\bm{B}}}_{l})_{\cdot,s}\right)\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{Y}}_{j}^{(l)}}\right)$ $\displaystyle+\sum_{s=1}^{d_{MLP}}\Xi_{l,s,i}\cdot\sum_{t=1}^{d_{MLP}}\Xi_{l,t,j}\cdot\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left((\widehat{{\bm{B}}}_{l})_{\cdot,s}\right)$ $\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \cdot\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left((\widehat{{\bm{B}}}_{l})_{\cdot,t}\right)$ $$

We apply the inductive step for the pre-MLP activation in cases (D), and (E) to rewrite the second and third term, and apply the definition of $\widehat{{\bm{B}}_{l}}$ to rewrite the fourth term:

$$ $\displaystyle=$ $\displaystyle\sum_{{\bm{v}},{\bm{w}},\mathcal{T}_{1}\geq_{l}\mathcal{S}_{1},\mathcal{T}_{2}\geq_{l}\mathcal{S}_{2}}\Lambda_{\mathcal{S}_{1},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{Y}}_{i}^{(l)}}\right)\Omega_{\mathcal{S}_{2},\mathcal{T}_{1},\mathcal{T}_{2},k,h,{\bm{v}},{\bm{w}}}\left(\widehat{{\bm{Y}}_{j}^{(l)}}\right)$ $\displaystyle+\sum_{s=1}^{d_{MLP}}\Xi_{l,s,j}\cdot{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}({\bm{Y}}_{i}^{(l)})^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right)({\bm{B}}_{l})_{s,\cdot}}$ $\displaystyle+\sum_{s=1}^{d_{MLP}}\Xi_{l,s,i}\cdot{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}({\bm{B}}_{l})_{\cdot,s}^{T}\left(\prod_{S\in\mathcal{S}_{1}}S\right)^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}\left(\prod_{S\in\mathcal{S}_{2}}S\right){\bm{Y}}_{j}^{(l)}}$ $\displaystyle+\sum_{s=1}^{d_{MLP}}\Xi_{l,s,i}\cdot\sum_{t=1}^{d_{MLP}}\Xi_{l,t,j}\cdot{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}({\bm{B}}_{l})_{\cdot,s}^{T}(\prod_{S\in\mathcal{S}_{1}}S)^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}(\prod_{S\in\mathcal{S}_{2}}S)({\bm{B}}_{l})_{\cdot,t}}$ $$

In the case where $\mathcal{S}_{1}=\mathcal{S}_{2}=\emptyset$, we obtain using case (C) of the inductive hypothesis for the pre-MLP activation:

$$ $\displaystyle=$ $\displaystyle({\bm{Y}}_{i}^{(l)})^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}{\bm{Y}}_{j}^{(l)}-\phi_{k,h}(i,j)$ $\displaystyle+\sum_{s=1}^{d_{MLP}}\Xi_{l,s,j}\cdot{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}({\bm{Y}}_{i}^{(l)})^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}({\bm{B}}_{l})_{s,\cdot}}$ $\displaystyle+\sum_{s=1}^{d_{MLP}}\Xi_{l,s,i}\cdot{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}({\bm{B}}_{l})_{\cdot,s}^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}{\bm{Y}}_{j}^{(l)}}$ $\displaystyle+\sum_{s=1}^{d_{MLP}}\Xi_{l,s,i}\cdot\sum_{t=1}^{d_{MLP}}\Xi_{l,t,j}\cdot{\color[rgb]{0,0,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.69921875}({\bm{B}}_{l})_{\cdot,s}^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}({\bm{B}}_{l})_{\cdot,t}}$ $$

Using ([64](#A6.E64)) and the definition of $\Xi_{l,s,i}$, this rewrites to

$$ $\displaystyle=$ $\displaystyle({\bm{y}}_{i}^{(l)})^{T}{\bm{K}}_{k,h}^{T}{\bm{Q}}_{k,h}{\bm{y}}_{j}^{(l)}-\phi_{k,h}(i,j)$ $$

from which case (B) of the inductive hypothesis follows by rearranging.
If instead $\mathcal{S}_{1}\cup\mathcal{S}_{2}\neq\emptyset$, the same reasoning leads to case (C) of the inductive hypothesis.
Analogous reasoning establishes cases (D) and (E) for the post-MLP activations.
Overall, we have established the inductive step for cases (A–E) for the post-MLP activations.
∎

###### Lemma 53 (Preservation of Products by Translation) .

###### Proof of Lemma 53 .

## Appendix G Additional Supporting Results

### G.1 Regularizer at Initialization

Here, we provide evidence that the additional regularizer ([8](#S3.E8)) is bounded independently of $N$ under plausible initializations of parameters.
Recall

$$ $(\ref{eq:additional-penalty})=\sum_{l=1}^{L}\sum_{h=1}^{H}\sum_{j=1}^{N(T)}\left|{\bm{p}}_{1}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{p}}_{j}\right|^{2}$ (66) $$

Intuitively, and as formalized in Proposition [54](#theorem54), when independently initializing the positional encodings ${\bm{p}}_{i}$, their inner products as mediated through ${\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}$ will tend to be small. As long as the width grows linearly with $N$, the aggregate value of ([8](#S3.E8)) will tend to be bounded independently of $N$.
Note that ([8](#S3.E8)) only includes products involving position $1$, which due to translation invariance for $T\in\Theta_{n}$ places a bound on all products.
As standard training does not enforce translation invariance of the products ${\bm{p}}_{i}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{p}}_{j}$, one may also be interested in a variant that takes all pairs of positions into account, to the extent that they can enter causal attention:

$$ $\frac{1}{N(T)}\sum_{l=1}^{L}\sum_{h=1}^{H}\sum_{j=1}^{N(T)}\sum_{i=1}^{j}\left|{\bm{p}}_{i}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{p}}_{j}\right|^{2}$ (67) $$

Here, the same conclusion holds.
We describe it formally, at the example of the second variant, in Proposition [54](#theorem54).

###### Proposition 54 .

Assume $d=\Theta(N)$. Assume the entries of each ${\bm{p}}_{1},\dots,{\bm{p}}_{N}\in\mathbb{R}^{d}$ and ${\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}\in\mathbb{R}^{d\times d}$ ($l=1,\dots,L$; $h=1,\dots,H$) are initialized i.i.d. from $\mathcal{N}(0,\frac{1}{d})$. The number of layers $L$ and heads $H$ are constant with respect to $N$. Then

$$ $\mathbb{E}\left(\frac{1}{N}\sum_{l=1}^{L}\sum_{h=1}^{H}\sum_{1\leq i\leq j\leq N}|{\bm{p}}_{i}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}_{l,h}{\bm{p}}_{j}|^{2}\right)=O(1)$ (68) $$

###### Proof.

We begin by showing that the expectation of each term in the sum is $O(1/d)$ and hence the sum is bounded by a constant. There are two cases for the expectation of terms: (i) The first is $i\neq j$ when the vectors ${\bm{p}}_{i}$ and ${\bm{p}}_{j}$ are independent and the second is $i=j$ when ${\bm{p}}_{i}$ and ${\bm{p}}_{j}$ are dependent.

For this section, let ${\bm{K}},{\bm{Q}}$ denote the matrices ${\bm{K}}_{l,h}{\bm{Q}}_{l,h}$ for any fixed $l,h$. Let ${\bm{K}}_{ij}$ and ${\bm{Q}}_{ij}$ denote the entry of the corresponding matrices $i$th column and $j$th row. Let $A={\bm{K}}^{T}{\bm{Q}}\in\mathbb{R}^{d\times d}$. Note that the expectation of any entry of $A$,

$$ $\mathbb{E}[{\bm{A}}_{ij}]=\mathbb{E}[{\bm{K}}_{i}^{T}{\bm{Q}}_{j}]=\mathbb{E}[\sum_{k=1}^{d}{\bm{K}}_{i,k}{\bm{Q}}_{j,k}]=0.$ $$

Further,

$$ $\mathbb{E}[{\bm{A}}_{ij}^{2}]=\mathbb{E}[(\sum_{k=1}^{d}{\bm{K}}_{i,k}{\bm{Q}}_{j,k})^{2}]=\mathbb{E}[\sum_{k=1}^{d}{\bm{K}}_{i,k}^{2}{\bm{Q}}_{j,k}^{2}]+2\mathbb{E}[\sum_{m=1}^{d-1}\sum_{n=m+1}^{d}{\bm{K}}_{i,m}{\bm{Q}}_{j,m}{\bm{K}}_{i,n}{\bm{Q}}_{j,n}]$ $$

$$ $=\mathbb{E}[\sum_{k=1}^{d}{\bm{K}}_{i,k}^{2}{\bm{Q}}_{j,k}^{2}]=d\sigma^{4}=\sigma^{2}$ $$

For products of two different entries ${\bm{A}}_{i,j}{\bm{A}}_{mn}$, note that $\mathbb{E}[{\bm{A}}_{i,j}{\bm{A}}_{m,n}]=\mathbb{E}[\sum_{u=1}^{d}\sum_{v=1}^{d}{\bm{K}}_{i,u}{\bm{Q}}_{j,u}{\bm{K}}_{m,v}{\bm{Q}}_{n,v})]$ which is 0 0 when $i\neq m$ or $j\neq n$.

For each term $|{\bm{p}}_{i}^{T}A{\bm{p}}_{j}|^{2}$, we have

$$ $\displaystyle\mathbb{E}[|{\bm{p}}_{i}^{T}AQ{\bm{p}}_{j}|^{2}]=$ $\displaystyle\mathbb{E}[(\sum_{u=1}^{d}\sum_{v=1}^{d}{\bm{p}}_{i,u}{\bm{A}}_{u,v}{\bm{p}}_{j,v})^{2}]$ $\displaystyle=$ $\displaystyle\mathbb{E}[\sum_{u=1}^{d}\sum_{v=1}^{d}({\bm{p}}_{i,u}{\bm{A}}_{u,v}{\bm{p}}_{j,v})^{2}]+2\mathbb{E}[\sum_{u,v\neq m,n}{\bm{p}}_{i,u}{\bm{p}}_{i,v}{\bm{A}}_{u,v}{\bm{A}}_{m,n}{\bm{p}}_{j,v}{\bm{p}}_{j,n}]$ $\displaystyle=$ $\displaystyle\mathbb{E}[\sum_{u=1}^{d}\sum_{v=1}^{d}{\bm{p}}_{i,u}^{2}{\bm{A}}_{u,v}^{2}{\bm{p}}_{j,v}^{2}]$ $$

since all terms of the form $\mathbb{E}[{\bm{p}}_{i,u}{\bm{p}}_{i,v}{\bm{A}}_{u,v}{\bm{A}}_{m,n}{\bm{p}}_{j,v}{\bm{p}}_{j,n}]$ are 0 0 due to independence of ${\bm{A}}$ and ${\bm{p}}$.

For $i\neq j$, we have

$$ $\mathbb{E}[\sum_{u=1}^{d}\sum_{v=1}^{d}{\bm{p}}_{i,u}^{2}{\bm{A}}_{u,v}^{2}{\bm{p}}_{j,v}^{2}]=\sum_{u=1}^{d}\sum_{v=1}^{d}\mathbb{E}[{\bm{p}}_{i,u}^{2}]\mathbb{E}[{\bm{A}}_{u,v}^{2}]\mathbb{E}[{\bm{p}}_{j,v}^{2}]=d^{2}\sigma^{6}=\frac{1}{d}.$ $$

For $i=j$, we have

$$ $\mathbb{E}[\sum_{u=1}^{d}\sum_{v=1}^{d}{\bm{p}}_{i,u}^{2}{\bm{A}}_{u,v}^{2}{\bm{p}}_{i,v}^{2}]=\mathbb{E}[\sum_{u=1}^{d}{\bm{p}}_{i,u}^{4}{\bm{A}}_{u,v}^{2}]+\mathbb{E}[\sum_{u=1}^{d}\sum_{v\neq u}{\bm{p}}_{i,u}^{2}{\bm{A}}_{u,v}^{2}{\bm{p}}_{i,v}^{2}]$ $$

$$ $=d(3\sigma^{6})+(d^{2}-d)\sigma^{6}<\frac{3}{d}.$ $$

Since $d=\Theta(N)$ and each term is less than $\frac{3}{d}$, we have that the sum in Eq.[68](#A7.E68) is $O(1)$.

∎

###### Proposition 54 .

###### Proof.

### G.2 Empirical Length Generalization of Positional Functions

Here, we show empirically that, when directly fitting parameters so that a product ${\bm{p}}_{i}{\bm{K}}^{T}{\bm{Q}}{\bm{p}}_{j}$ reproduces some function $\phi(\cdot,\cdot)$ at smaller distances, it will length generalize when these are local or periodic but under different conditions matching the role of local and periodic functions in our theory.
Specifically, we show that they length-generalize well at large $d$ when they are Local; whereas, when $d$ is smaller, length generalization works well when they are Periodic.
Length generalization is poor on functions that are neither local nor periodic.

##### Experimental Setup

We randomly initialize 200 position embeddings $\{{\bm{p}}_{i}\in\mathbb{R}^{d}:1\leq i<201\}$, as well as query and key matrices, ${\bm{Q}},{\bm{K}}\in\mathbb{R}^{d\times d}$. We experiment with $d=\{32,256\}$. We optimize the mean square error (MSE) between ${\bm{p}}_{i}^{T}{\bm{K}}^{T}{\bm{Q}}{\bm{p}}_{j}$ and $\phi(\cdot,\cdot)$ on length of 50, and test on length $\{50,100,150\}$. Concretely, during training, we sample random offsets $o$ from [0, 150] and take the sequence of ${\bm{p}}_{1+o},\dots,{\bm{p}}_{50+o}$ to compute the loss. When testing on length $n$, we compute the average loss over all offsets in [0, 200-n]. We ignore the loss on those entries where $j<i$ to mimic causal masking. The $\phi(\cdot,\cdot)$ we use in experiments (except for the one combined from two $\phi(\cdot,\cdot)$) only takes two values, 0 when the condition is false and $2\log 50$ when the condition is true. We thus use different conditions to describe different $\phi(\cdot,\cdot)$. For example, we use $\phi:\texttt{j=i-c}$ to denote the following function:

$$ $\phi(j,i)=\left\{\begin{array}[]{ll}2\log 50&j=i-c\\ 0&else\\ \end{array}\right.$ (69) $$

where $c$ is a constant number.

The embeddings and weight matrices are trained with Adam optimizer, using a batch size of 64, a learning rate of 1e-3, for 15k steps. Additionally, we add mean squared weights (i.e., squared Frobenius norm divided by the number of elements) to the loss to mimic the training regularizer, with a coefficient of 0.01.

Results are in Figure [5](#A7.F5) and [6](#A7.F6). Note that in both figures, the y-axis uses a logarithmic scale above 1.0 and stays linear scale below 1.0. In the last column of [6](#A7.F6), “combined” denotes functions that combine two functions as follows: $\phi=\phi_{1}+\phi_{2}$, where $\phi_{1}:\texttt{j=i-c}$ and $\phi_{2}:$ (i-j)=$c_{2}$ mod $c_{1}$.

Figure: Figure 5: Appendix [G.2](#A7.SS2): MSE loss in fitting (length $=50$) and generalizing (higher lengths) functions $\phi(\cdot,\cdot)$ with products ${\bm{p}}_{j}^{T}{\bm{K}}^{T}{\bm{Q}}{\bm{p}}_{i}$. We show local functions testing if $j=i-c$ (left), if $j>i-c$ (center), periodic functions testing whether $i-j\equiv c_{2}\ (\text{mod}c_{1})$ (right). We show results at small (top, $d=32$) and high (bottom, $d=256$) dimensionality. Local functions length-generalize well when dimensionality is high (bottom left, bottom center); generalization is more successful with functions concentrated on few pairs (bottom left is nonzero at only one value of $j-i$; bottom center is nonzero at $c$ different values of $j-i$). Periodic functions length-generalize well when dimensionality is low (top right). The results match the distinct roles played by local and periodic functions in our theoretical constructions: Periodic functions are mediated by bounded-rank products (Lemma [48](#theorem48)), local functions are mediated by the products ${\bm{p}}^{T}{\bm{K}}_{l,h}^{T}{\bm{Q}}{\bm{p}}$.
Refer to caption: /html/2410.02140/assets/images/nice-func-vis.png

Figure: Figure 6: Appendix [G.2](#A7.SS2): MSE loss in fitting (length $=50$) and generalizing (higher lengths) functions $\phi(\cdot,\cdot)$ with products ${\bm{p}}_{j}^{T}{\bm{K}}^{T}{\bm{Q}}{\bm{p}}_{i}$. We show functions that are neither local nor periodic, which test if $j<i-c$ (left) if $i-j$ is a prime number (center), and a function created by adding a local and a periodic function (right). We show results at small (top, $d=32$) and high (bottom, $d=256$) dimensionality. Compared with results in Figure [5](#A7.F5), we can see that such functions, neither local nor periodic, length-generalize poorly.
Refer to caption: /html/2410.02140/assets/images/non-nice-func-vis.png

### G.3 Bound for Encodings Norm in Terms of Function Complexity

Recall that our regularizer includes a penalty ([8](#S3.E8)) on attention dot products.
Here, we discuss a conjecture:

###### Conjecture 55 .

The term ([8](#S3.E8)) can be removed from the regularizer while maintaining a (potentially weaker) length generalization guarantee for Limit Transformers.

To provide a heuristic argument for this, assume that for each upper bound N on the input length, we have a configuration of positional encodings and the matrix ${\bm{A}}$, such that the following property holds: For any indices $N\geq j>i>0$, let

$$ ${\bm{p}}_{i}^{T}{\bm{A}}{\bm{p}}_{j}=F(j-i)$ (70) $$

where $F:\mathbb{N}\rightarrow\mathbb{R}$ is a function that maps to numbers representable in $p$-bit precision. The function $F$ and the precision $p$ are chosen globally, across the different N’s.
Boundedness of $\mathcal{R}(T_{n})$ across $n$ entails $\|{\bm{p}}_{i}\|_{2},\|{\bm{A}}\|<C$, for $C$ a global constant.
We also know that $\sup_{x\in\mathbb{N}}|F(x)|<\infty$.
We conjecture that one can use these assumptions, and Lemma [56](#theorem56), to prove that F cannot be “too complicated”. Specifically, we conjecture that $F$ will be ultimately periodic: when $x$ exceeds some threshold, $F(x+\Delta)=F(x)$ for some period $\Delta$.
For, if $F$ is not ultimately periodic, we hope to construct a matrix $G$ whose nuclear norm can be made arbitrarily large, so large as to give a superconstant lower bound on $\|{\bm{A}}\|$ – which is a contradiction.
First note that Lemma [56](#theorem56) even holds if ${\bm{G}}={\bm{Y}}^{T}{\bm{A}}{\bm{X}}$ where ${\bm{X}},{\bm{Y}}$ are two different matrices with $n$ unit-norm columns.
That is, we can consider a matrix ${\bm{G}}_{ij}={\bm{p}}_{x_{i}}^{T}{\bm{A}}{\bm{p}}_{{\bm{y}}_{j}}$ where we conjecture that one can choose ${\bm{x}}_{1},...,{\bm{x}}_{n}$ and ${\bm{y}}_{1},...,{\bm{y}}_{n}$ to give an arbitrarily large lower bound on $\|{\bm{A}}\|$, under the assumption that $F$ is not ultimately periodic.
Here, it is important that $F$ maps to fixed-precision output; otherwise, one could get functions that have irrational periods and thus are not periodic when restricted to $\mathbb{N}$.

###### Lemma 56 .

Let $x_{1},\dots,x_{n}\in\mathbb{R}^{d}$ be vectors with $\|x_{i}\|_{2}=1$, and let $A\in\mathbb{R}^{d\times d}$ arbitrary. Let $G\in\mathbb{R}^{n\times n}$ such that $G_{ij}=x_{i}^{T}Ax_{j}$.
Then

$$ $\|A\|\geq\frac{\|G\|_{*}}{n}$ (71) $$

where $\|A\|$ denotes the spectral norm.

###### Proof.

First, note that for any matrix $B=U\Sigma V$, we have

$$ $\|B\|_{*}=tr(\Sigma)=tr(\Sigma VV^{T}U^{T}U)=tr(U\Sigma VV^{T}U^{T})=tr(BV^{T}U^{T})$ (72) $$

where $V^{T}U^{T}$ has each singular value bounded by 1 (in fact, it’s orthogonal).
In other words, $\|B\|_{*}=tr(BM)$ where $M$ is an orthogonal matrix.

For any two real-valued and possibly non-square matrices, we have

$$ $Tr(A^{T}B)\leq\|A\|_{F}\|B\|_{F}$ (73) $$

and by submultiplicativity, we have

$$ $\|AB\|_{F}\leq\|A\|\|B\|_{F}$ (74) $$

Using Eq. [72](#A7.E72) and the above two properties, it follows that,

$$ $Tr(X^{T}AXM)=Tr(AXMX^{T})\leq\|X^{T}A^{T}\|_{F}\|MX^{T}\|_{F}$ $$

$$ $\leq\|AX\|_{F}\|MX^{T}\|_{F}\leq\|A\|\|X\|_{F}\|M\|\|X\|_{F}\leq n\|A\|.$ $$

∎

###### Conjecture 55 .

###### Lemma 56 .

###### Proof.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 6
presentation: 2
questions: * Could the authors more directly relate their theorem to widely used transformer models? For example, which specific conditions do not hold in these models, and how does this align with the empirical observations in Section 5?
* The FAQ in Appendix A actually was quite useful providing valuable insights and gives readers motivation. Could the authors consider integrating some of these discussions more naturally into the introduction or relevant sections?
soundness: 3
strengths: * Length generalization is a very important problem that is being explored from various perspectives, though primarily through algorithmic and empirical studies, with relatively few theoretical analyses. This work takes a different approach, offering distinct advantages.
* It's encouraging to see that, under certain conditions (as shown in Theorem 7), they identify cases where length generalization is achievable.
* Provides several formalizations (e.g. Limit Transformer) and proofs on this topic that would be valuable for future research.
summary: This work studies the challenge of enabling transformers to generalize to sequences longer than those seen during training. This capability is often inconsistent across tasks and lacking strong theoretical understanding. The authors present a new theoretical framework to examine length generalization in causal-attention transformers with learnable positional encodings. Using this framework, they analyze the conditions under which transformers can generalize to longer sequences under certain conditions. The theoretical findings are further validated through empirical tests on a set of tasks.
weaknesses: * The paper’s current writing seems to focus more on demonstrating how to achieve length generalization under specific conditions than on examining the limitations of existing models. For example, it centers on proving Theorem 7 by introducing various modeling assumptions, such as the Limit Transformer and specific inference conditions (like the proposed regularizer). This approach differs somewhat from the expectations set in the introduction, which suggests a more interpretive analysis of current models.
* The practical impact of this paper feels somewhat limited. It would be valuable if the authors could identify practical applications that leverage their findings.
* C-RASP and formal languages are quite distinct from traditional text corpora or other common scenarios. Even though the aim is to theorize the formalism in this field, there seems to be a significant gap between the formalism presented in this paper and that of other widely used settings.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: - In Figure 1, why do Transformers with NoPE fail even on in-distribution samples (Bin 1) for Copy Unique and Addition? Doesn’t this contradict the observations in [2], which argue that NoPE can length-generalize for these tasks to some extent?

[2] Kazemnejad, Amirhossein, et al. "The impact of positional encoding on length generalization in transformers."
soundness: 4
strengths: - The paper addresses an important question in the literature: which tasks can achieve length generalize and which cannot. Prior work on length generalization has mainly focused on improving empirical performance, with fewer studies providing a theoretical understanding. While [1] introduces RASP-L to identify tasks that can achieve length generalize, their argument still remains conjectural. This study goes one step further than the previous RASP-L conjecture, offering a theoretical framework to determine whether certain tasks will provably achieve length generalization or not. Therefore, I believe this paper makes a significant contribution to the literature.
- The FAQ section in the appendix effectively conveys the paper's intuition and enhances the reader’s understanding.
- Overall, the paper is well-structured and provides a detailed analysis to present their findings.

[1] Zhou, Hattie, et al. "What algorithms can transformers learn? a study in length generalization."
summary: The paper studies the length generalization problem in decoder-only Transformers, which refers to the inability of models to deal with longer samples than encountered during the training phase. The paper aims to identify which tasks can achieve length generalization. To this end, the authors introduce Limit Transformer, a theoretical model designed to generalize across varying sequence lengths. Importantly, the authors prove that under an idealized inference procedure, any tasks that can be expressed by Limit Transformer, satisfying Periodic and Local constraints, can provably achieve length generalization. Furthermore, the authors show that any C-RASP programs can be expressed by Limit Transformer, leading to the conclusion that such tasks can also generalize to longer inputs. On the other hand, the paper presents that copying and addition cannot be generalized as these tasks do not exhibit logarithmic communication complexity. Finally, the paper provides experimental results, demonstrating that tasks expressible by C-RASP programs (binary majority, sort, copy unique) easily length-generalize while tasks outside C-RASP (copy repeat, parity, addition) fail to achieve length generalization.
weaknesses: I don’t see any significant weaknesses in the paper, but there are a few minor limitations, which I don’t consider critical issues in evaluating this paper.

- The scope of the paper is limited to algorithmic tasks and does not cover length generalization problems in natural language processing tasks.
- As explained in the paper, a key assumption in the framework is the idealized inference procedure, which assumes that we can obtain Transformers that are fitted to reproduce a target function while minimizing a specific regularizer $R(T)$, and thus the current framework is not fully end-to-end. Introducing an analysis of training dynamics to replace this assumption would be a promising direction, achieving a truly end-to-end, complete argument.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Have you tried the performance of RPE although the current theory does not cover it yet? In [1], NOPE is shown to inherently learn some kind of relative positional encoding. Does RPE behave similarly to NOPE in certain tasks? 
2. The model size provided in the appendix is highly dependent on the specific tasks (different tasks use different model depths, embedding space sizes, etc). Do you observe the sensitivity of the generalization performance in terms of model sizes? Or are there other reasons for this?

[1] The Impact of Positional Encoding on Length Generalization in Transformers
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, Siva Reddy
soundness: 3
strengths: 1. The paper provides theoretical analysis of length generalization in transformers, including both sufficient conditions for generalization and communication complexity bounds showing certain functions cannot exhibit length generalization. This addresses an important open problem in theory. Also, it is the first to include length generalization theories with positional encodings to my best knowledge.
2. The theoretical framework successfully predicts empirical length generalization behavior across diverse tasks. The C-RASP formalism provides an interpretable way to determine whether a function should exhibit length generalization, making the theory practically useful.
3. The experimental evaluation is thorough with both algorithmic tasks and formal languages.
summary: This paper develops a theoretical framework for transformer length generalization, introducing "Limit Transformers" and proving generalization guarantees for functions satisfying PERIODIC and LOCAL properties. They characterize functions that are identifiable with APE and NOPE, and validate the prediction from theory with experiments on algorithmic and formal language tasks.
weaknesses: 1. The main contribution compared to the previous C-RASP work and the RASP-L work is to extend the framwork to include positional encodings. However, the positional encodings considered are APE and NOPE, while the more frequently used encodings in practice are relative positional encodings. Also see question 1.
2. The limitation of the proposed theoretical framework is discussed, but it could benefit from more empirical evidence, like where the theory predicts length generalization but empirical performance is poor, or vice versa.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: **Improvements:**
(I consider all of them optional suggestions, not strict requirements)
 1. Maybe give the reader a better sense of where this is going early in the paper. It should be clear early on that the paper constructs a function class for which generalization can be proven in the theoretical limit, but the theory does not answer how this relates to training actual transformers of fixed context length via SGD (i.e., there may be functions that can theoretically be proven to be length-generalizable, but this may not work in practice). Also state that this function class is likely not complete (i.e., there may be functions where transformers can length-generalize that lie outside this function class).
2. I really liked lines 255-264 in terms of clarifying the paper. Maybe the same information can be qualitatively given early in the paper to prime the reader.
3. Definition 2 can be a bit misleading - it informally may suggest that a Limit Transformer is basically “just a normal transformer with infinite context”. This needs to be clarified. The text already mentions that the Limit Transformer is a *theoretical* construct (maybe consider calling it a Limit Transformer Process or similar, to make sure that the object is not confused with a concrete architecture). The finite precision argument in Def. 2 is theoretically ok, but in practice it just says that the precision is an arbitrarily high natural number (with no upper bound given), which is not implementable “like a standard transformer”. Also state here or earlier in the paper that the Limit Transformer cannot be trained via SGD, but uses a theoretical procedure (Def. 6) that cannot be practically implemented. And finally, the functions $\phi_{l,h}$, whose complexity is not bounded as far as I can tell, do a lot of the heavy lifting and are not just marginal additions to a standard transformer.

**Minor questions**

1. L200-202: This is a very interesting requirement. Together with the requirement for periodicity and locality I am reminded of the pumping lemma for regular languages. But if I understand correctly (some experiments, Fig. 10, and discussion following L 486), C-RASP defines a subset of $TC^0$ and it covers some but not all regular languages and some simple non-regular languages.
2. Showing that C-RASP with a small extension can provably be expressed by a Limit Transformer is very interesting. Can C-RASP potentially still be extended without violating this equivalence, or is the current set of operations (likely) complete?
3. What is the relation of Theorem 7 to standard learnability / language identifiability results (language identifiability is generally not possible from positive examples only)?
soundness: 3
strengths: * Very timely question. While frontier models show many surprising and unprecedented capabilities, they fail catastrophically on some really simple problems in length generalization. Understanding the underlying reasons is crucial for Safety and Reliability, and may also pave the way to address these issues in future-generation architectures.
* The expressivity result stating that all C-RASP programs can be expressed by a Limit Transformer, and are thus identifiable in theory via the inference process in Def. 6, is a strong result that bridges the theory to very concrete and empirically testable hypotheses.
* Empirical results show that despite a very elaborate construction of a limit process and complex regularizer, the theoretical predictions have actual practical consequences on standard transformers trained in a standard fashion (that differs significantly from the inference process in Def. 6)
summary: **Update after rebuttal:**
The authors have clarified my questions and made improvements to the presentation of the main manuscript (though it remains a very long paper of course, with some important parts in the appendix, hence I will leave my 'Presentation' rating on 2). I am now more confident that the work is important and is ready to be presented and discussed with the wider ML community. I would now raise my score to a 7, but ICLR does not allow this score this year. I do think that some weaknesses remain, and that the paper does not quite hit an 8. But for the sake of expressing a clear opinion for the decision-making process, I am raising to an 8 (as the only available option), but will keep my confidence low to indirectly indicate that the score is a bit inflated.

---

This paper formally defines a function class, implicitly via the construction of the Limit Transformer, and shows that length generalization with transformers (of arbitrary context length) is provably guaranteed within this function class (this does not include SGD training though). The main part of the proof is the use of an “inference procedure” (Definition 6), which, informally, iterates through all transformers of increasing context size and eliminates the ones that do not fit the data. By construction, the set of functions that these transformers can implement (with increasing long context) is finite, such that eventually, at some finite context size, only the transformer that generalizes correctly to arbitrary length remains (all others have been ruled out by the data at this point; to be precise a complexity regularizer is also required to make the choice unique). This is the main argument in Theorem 7.2 (part of the main result). Keeping the parameters of transformers with increasing context width finite is central to the whole construction, and is reflected in the notions of PERIODIC and LOCAL of the Limit Transformer (informally: the functions implemented via attention only operate on a finite/small context window, and are translation invariant). Finally, the paper shows that a version of C-RASP (with either learned absolute positional encodings or no positional encodings) can be proven to be expressible via Limit Transformers, leading to the conjecture that length generalization with practical transformers is strongly related to whether a solution can be expressed in C-RASP or not. A small set of relevant experiments supports this conjecture well, including a (potential) explanation why some relatively simple regular languages cause trouble with length generalization (for which no C-RASP implementation provably exists).
weaknesses: * The paper is very extensive (72 pages, 62 are appendix, with 3887 lines in total; given the conference timelines and workload I could not review the appendix). The main paper is thus more a summary of the appendix than a standalone paper. While it makes no sense to have the main results and theorems without the proofs, maybe splitting the publication into a journal- / long-format theory paper and a separate more extensive empirical verification for a mainstream ML conference could be better.
* The construction of the Limit Transformer is quite elaborate (or rather the construction to go to the infinite context limit with non-exploding parameter sets / maintaining a finite function class) and the inference procedure in Def. 6 is completely impractical. Without any empirical results I would have been very skeptical about the practical relevance, and to be fully convinced I would like to see further results (though I believe Fig. 1 is significant and very promising). It is a bit unclear whether the theory had to be this complex and the connection to C-RASP dropped out as a lucky coincidence (which is how the paper is currently written), or whether putting C-RASP on a theoretical footing was the original goal that demanded this level of complexity.
* The inference procedure in Def. 6 seems a bit crude (though it does the job theoretically; same applies to the regularizer which is composed of 8 different complexity terms). The standard approach (e.g., in algorithmic information theory / Solomonoff Induction) is to not try to identify the correct function at some point $N_0$ but bound the overall number of mistakes (which is finite for any finite-length program). $N_0$ would be the point where the “last” mistake happens, which is generally unknown, and usually having bounds and generalization guarantees in terms of numbers of (remaining) mistakes are much tighter and shrink much faster with increasing number of observations. This is probably a question for future work, but it is unclear whether the restriction of using Def. 6 (having to identify the correct function) is implicitly limiting the function class where generalization is possible, and whether this class could be extended by focusing on a theoretical scheme that relies on bounding the mistakes (number of mistakes, and/or their cumulative magnitude).

**Verdict:**
Overall I am a bit ambivalent about the paper. On one hand it tackles a very timely and important problem by starting to make good progress from a theoretical angle (rather than adding even more contradicting experiments). On the other hand the current theory and presentation are quite complex and extensive. If the theory had been published in a journal / long-format paper before, and this paper would solely focus on empirically testing the conjecture that C-RASP expressiveness predicts length generalization, then a 10-page conference format seems like a great fit. Similarly, without the empirical results, I would have strongly doubted the practical relevance of the theory. But, the empirical results in Fig. 1 look very promising; though at this point it is unclear whether any results that contradict the main claims can easily be found or not. I do believe that the ML community needs more exposure to good theory, particularly at conferences, though I am not sure that this paper is the best example (due to its excessive length). I am also quite certain that this paper will spark quite a bit of follow-up work to make the theory simpler and/or more complete (expand the function class), and that publishing it will stir the community to conduct more empirical tests of the theory (which will make overall faster progress than asking the authors to perform more experiments). I am therefore currently slightly in favor of accepting the paper, though I would not be upset if others argue that ICLR may be the wrong venue. My confidence is currently on the low end - I did not have time to go through the extensive appendix, and there are a few bits and pieces of the main paper where I am not fully sure how they work out / will be proven. I am very happy to reconsider my opinion based on the other reviews and authors’ responses, and to make my criticism concrete, I leave some suggestions for improvement in the Questions section below.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Please see weakness section, where I list both the weaknesses and questions.
soundness: 2
strengths: 1. The authors tackle a hard problem, i.e., providing a formal justification of RASP conjecture for multi-block transformer models. This is both a hard problem and an important one. 
2. The authors have been creative in several aspects of the paper -- i) the regularization constraints that have been imposed seem particularly important to the inference procedure's success, ii) the construction of the limit transformer, iii) the tight connection between the limit transformer and the C-RASP language from Yang and Chiang.
3. The main body of the paper is nicely written and does a nice job of getting to the main results quite fast.
summary: In the recent years, with the surge of LLMs and transformers, there has been an increasing interest to formally understand the generalization failures of transformer. A notorious and common failure mode that is often mentioned in the literature pertains to length generalization, where the model is trained on sequences of certain length, and then is tested to longer sequences. Many works have shown the failures of modern architectures on a range of tasks (most popular ones being parity) and others have come up with strategies to fix the failures (e.g., changes in positional embeddings). In recent work, Zhou et al., authors proposed an insightful conjecture, the RASP-L conjecture, that aims to dilineates the tasks that are easy for transformers to learn and length generalize on, and then tasks that continue to be hard for transformers. While the conjecture was empirically backed, theoretical justifications for the same have been lacking. In this work, the authors aim to formalize and provide a theoretical justification to the RASP-L conjecture. 

The authors study two types of positional encodings -- no positional encodings, and absolute positional encodings. When dealing with absolute positional encodings the number of parameters grow with length of the input. To address this challenge, the authors introduce a new object, the limit transformer. This transformer encapsulates the behavior of transformers on longer and longer sequences into a single object. The authors define an idealized inference procedure, which searches for the transformer that minimizes the risk along with a regularization constraint. The regularization constraint is a special one, i.e., it is not the standard constraint based on purely say the l2 norm of the weights. With these constraints in place, the authors show that if the ideal function f is expressable in a limit transformer that satisfies two properties -- Local and Periodic, then the transformer is able to learn the function f and length generalize on it. In the second half of the paper, the authors show that for every program C-RASP[phi, Psi) with local and periodic phi and Psi respectively, there exists a limit transformer that accepts the same set of strings that P accepts. The authors also provide communication complexity based arguments to explain the limits of limit transformers. Finally, the authors conduct experiments to match the predictions of the theory.
weaknesses: There are quite a few concerns that I have for various parts of the paper. My current score is a reflection of these weaknesses. I would be happy to change my score if the authors can provide satisfactory explanations and no other major concerns appear in the course of discussion. 

1. **No Positional Encoding vs. Absolute Positional Encoding**:  In the current work, the authors went through great detail to construct limit transformer with the idea that limit transformer can encapsulate longer and longer transformers into one limiting object. The first thing that bothers me is that if we take no positional encodings then we do not need this object. The authors do not explain the need of limiting transformer in the context of NoPE as there is no growth in number of parameters anymore that limit transformer needs to cater to. 
The second thing that bothers me is that from an expressivity point of view, the NoPE based transformer can express APE up to a large length. So why can't we use
Theorem 1  (https://proceedings.neurips.cc/paper_files/paper/2023/file/4e85362c02172c0c6567ce593122d31c-Paper-Conference.pdf) from this work. This work essentially is arguing that NoPE can approximate APE (or RoPE). 

2. **About invariance to offsets and regularization** 
     a) The authors introduce the constraint that the transformer should be invariant to offsets, i.e., if the problem appears at different locations in the context window then the solution should not change. This constraint is not explicitly enforced.  
     b)  The authors also have an idealized inference procedure, where the regularizer has been introduced for the purpose of theory. The authors argue that there is an implicit bias towards small values of it at initialization. I don't quite see how. Also, this regularizer is also not explicitly enforced. 
  Since there is quite some gap between the theory and expmts, what do you think is the explanation for this gap? 

3. **Regarding the definition of limit transformers**
     a) The limit transformer is introduced in Definition 2. The term y_i(l) in equation (6), is it the same as how it was defined in equation 4. If so, then does it already have positional encoding in it like in equation (1). If so, then how does the function phi that is introduced additionally absorb the terms that involve inner product of two different positional encodings. It feels if y_i^(0) already had positional encodings in it then won't the first term in equation 6 already take care of the stuff. 

   b) Is the rest of the construction of limit transformer same as standard transformer and the only difference is equation (6)? I ask this because following equation (6), we are not told what happens to attention logits. 

   c) In point 2 in the definition u say that the positional encodings p_i are encoded in finite precision. If that is the case, then when we increase the length to arbitrary large values, the positional encodings start overlapping and we only have finitely many positional encodings.    This does not address the increase in the parameter count issue that the author state was the very reason to define the limit transformer. If we are happy with finitely many positional encodings, then why not just do some periodic encodings in the standard transformer? Also, if we are happy to do everything with finitely many positonal encodings then this goes back to my first concern on NoPE, we can operate with NoPE, express finitely many positional encodings, and simplify the whole story right? 

 
4. **Concerning periodic and local in definition 3.** In Definition 3, you state that phi_l,h is translation-invariant and local. You also stated that phi_l,h expresses the inner product involving positional encodings. This translates into a constraint on the positional encodings. This creates confusions. The authors should be more clear on this whole connection in the main body. 

5. **About the hypothesis class definition 4** In definition 4, you say that each product function involving position encoding is translation invariant. You also say that each function involving exactly one product function is translation invariant. These constraints seem very restrictive. Is there a reason to believe that imposition of these constraints is not over simplifying the problem somehow? Are these constraints implying offset invariant condition? I think offset invariance on its own was reasonable but this seems not very digestable. I would appreciate if authors gave more insights into why these constraints are not unreasonable? Also, some numerical insights into what it means to enforce these constraints? This goes back to my point 2. If you see point 2, I state that there is gap between theory and expmts. In this case, the theory would require some of these constraints, which how do u really enforce? Perhaps these constraints are strong sufficient conditions for length generalization? and far from necessary? Is offset invariance a necessary condition btw for length generalization?

6. **On the regularizer in definition 5** 
    a) After definition 5, you state that the idea of this regularizer is to discourage attention between far-away positions that do not appear together during training, which could hamper length generalization. At what point do you use this insight in the proofs. For instance in Lemma  17 how does it come up? I don't quite see it. Also, since all positions are equally penalized in this regularizer, why would things far off be more penalized? Also, the justification based on initialization making the regularizer small is not fully clear. 

   b) If we use NoPE positional encoding, then p_i is set to zero for all i. As a result, I don't quite understand the role of the regularizer anymore. Since the regularizer is supposed to penalize far away positions, those positions don't seem distinguisable under the regularizer as p_i is set to zero. Further, if p_i is set to zero, then what is the role of the phi function in equation (6), and eventually why do we need the limit transformer? 

7. **On phi function** In the line 260, you indicate that for phi function, we need to only care abt it its values such as phi(1,1), phi(1,2),..phi(1,tau). The values above the diagonal are taken care of by translation invariance, but what abt the values below the diagonal, i.e., phi(2,1)..I don't think you assume symmetry, do you? 

8. **Minor remark on line 326/327** Shouldn't the Q_a in the RHS be Q_a(j) and not Q_a(i)?

9. **Notation remark on Theorem 9**. In an unfortunate use of notation, u call Psi function local and Phi function periodic. Earlier u had used phi for local in the definition 3. 

10. Since the results from the work hold for NoPE positional embeddings. From Theorem 2 in https://proceedings.neurips.cc/paper_files/paper/2023/file/4e85362c02172c0c6567ce593122d31c-Paper-Conference.pdf, the results should extend to relative positional encodings too?

11. Currently the results require a large N_0, which practically speaking can be very large. The authors do mention this limitation. While I am not expecting a bound of any sort in this work, I want to understand the consequences of the results better. The current machinery in this work (if correct), seems to indicate that allowing for a very large N_0 and some strong constraints (periodic, local on hypothesis class), length generalization is achievable for a large N_0 seen during training. If this N_0 from the theory is quite large, then would you say that research in this should try to explain why can transformers do it with a much smaller N_0 than theory predicts? I want to understand the hunch of the authors here. If one tries to bound N_0, then would we run the risk of vaccuous bounds?


12. **Concerns on the proof of Lemma 17**:  
    a) In line 976 you say there are only a finitely many settings traversed by the limit transformer \tilde{T}_i. Why is this the case? Can you properly justify?

      b) In equation 9, you say that we select an R(Tn) that is less than 1/n + inf (R(T)). In line 981, you argue that R(Tn) should converge because inf R(Tn) is bounded and monotonically increasing. This argument only tells that the RHS in equation (9) converges, which is the upper bound. Why does it imply that the LHS converges? You crucially use the existence of this limit in equation (14).

     c) In line 995, you say that lim D_v_i(v_i) is D_0. Why does this limit exist? 

     d) Below equation 17, you state "As this function is monotonically increasing, and as phi_{l,h} has bounded precision, there must be t_infty..." Why does this hold true? I don't quite follow. 

      e) In line 1010-1017, you construct a sequence of T'n satisfying certain properties. Why does this have to exist? For instance why is  D_{vi(n)}(n) = D_{infty}(n) true?

      f) The phi_l,h(i,j) in equation (10) should also bear the index n as it would be different for each limit transformer. This makes the rest of the stuff bit confusing.  For instance, you define D_n(tau) in line 984. Why would it be the case that tau can be larger than n in the summation? Since positional embeddings for that transformer would only be defined up to n right? 

      g) I do not follow the inequality in line 1044?  

      h) In line 1049-1052, you say that set of functions traversed becomes stationary, what do u precisely mean here? 

13. In line 1121 to 1127, how does the C logN fall out. Do you mean that since you partition stuff into N positions and that takes log N bits to compute, we get C log N?


14. I realize that there was one question I had forgotten to add in the above list. In the Definition 4 of your hypothesis class, I do not see any constraint on positional encodings being periodic. Does periodicity fall out as a consequence of the offset independent constraint you add later in the definition? What confuses me quite a bit is that while your transformer is not periodic but your limit transformer is periodic. How is it that a transformer with non-periodic positional encodings with an ever growing context window is captured by another transformer like object with periodic positional encodings? And relatedly why not just have periodic positional encodings on the original transformer to begin with instead of the very strong offset independence conditions in definition 4.

Meta-review: Summary:
The paper introduces a theoretical framework analyzing length generalization in causal transformers with learnable absolute positional encodings. The authors introduce "Limit Transformer" to handle growing parameters and show that C-RASP programs (under specified periodic and local constraints) can be translated into Limit Transformers, providing a concrete class of problems where length generalization is guaranteed. The theoretical findings are supported by empirical experiments demonstrating that tasks expressible by C-RASP programs succeed in length generalization, while others, such as copying with repetitions and n-digit addition, fail.

Strengths:
- Addresses an important open problem in transformer theory - length generalization capabilities
- Provides rigorous theoretical analysis with concrete proofs and guarantees
- Formalizes and proves aspects of the previously conjectural RASP-L hypothesis
- Strong empirical validation across diverse tasks showing alignment between theory and practice

Weakness:
- Treatment is limited to absolute positional encodings rather than more popular schemes
- The theoretical setup makes some idealized assumptions that do not reflect practice
- Focus is primarily on algorithmic tasks rather than natural language

Decision:
All the reviewers were in consensus that the paper represents an important step forward in understanding transformer capabilities and limitations, providing both theoretical insights and practical implications for future work. It merits publication at ICLR 2025.

Decision: Accept (Poster)