Why are Sensitive Functions Hard for Transformers? (2024)

Abstract: Empirical studies have identified a range of learnability biases and limitations of transformers, such as a persistent difficulty in learning to compute simple formal languages such as PARITY, and a bias towards low-degree functions. However, theoretical understanding remains limited, with existing expressiveness theory either overpredicting or underpredicting realistic learning abilities. We prove that, under the transformer architecture, the loss landscape is constrained by the inputspace sensitivity: Transformers whose output is sensitive to many parts of the input string inhabit isolated points in parameter space, leading to a low-sensitivity bias in generalization. We show theoretically and empirically that this theory unifies a broad array of empirical observations about the learning abilities and biases of transformers, such as their generalization bias towards low sensitivity and low degree, and difficulty in length generalization for PARITY. This shows that understanding transformers’ inductive biases requires studying not just their in-principle expressivity, but also their loss landscape.

heading: 1 Introduction
text: Given dramatic advances in machine learning applications powered by transformer models, there has been substantial interest in understanding which functions are easier or harder to learn and represent using transformers. Empirical research on both formal languages and synthetic functions has uncovered an intriguing array of learning biases, but theoretical understanding is lacking. For instance, Abbe et al. (2023) experimentally argued that heldout generalization is biased towards low-degree polynomials and Bhattamishra et al. (2023) provided empirical evidence that transformers prefer to represent functions of low sensitivity, that is, functions that do not strongly depend on many input bits. Perhaps the most prominent example of such learning biases is a consistent difficulty in learning the PARITY function, mapping bitstrings to their parity. This function is extremely sensitive, in the sense that flipping any bit flips the string’s parity. Empirical studies have consistently found that training transformers
to compute parities is difficult, and that solutions for shorter inputs do not generalize to longer inputs (e.g. Bhattamishra et al., 2020; Chiang and Cholak, 2022; Delétang et al., 2023; Ruoss et al., 2023). This stands in stark contrast to previously-popular reccurent models which easily fit PARITY with correct length generalization (Bhattamishra et al., 2020).
While a substantial amount of theoetical work has considered both the learnability (e.g. Edelman et al., 2022; Ahn et al., 2023) and the expressiveness of transformers (e.g. Yun et al., 2019; Hahn, 2020; Yao et al., 2021; Hao et al., 2022; Merrill et al., 2022; Merrill and Sabharwal, 2023b; Chiang et al., 2023; Strobl et al., 2023; Strobl, 2023; Angluin et al., 2023), existing theoretical studies do not consistently explain such learning biases. Hahn (2020) proved that, under two formal models of self-attention, no transformer can express PARITY at all input lengths. However, various other formal results showed that slightly relaxed assumptions about the transformer architecture resolved such expressiveness limitations. Most notably, Chiang and Cholak (2022) found that layer norm, by breaking the Lipschitz assumption used in Hahn (2020)’s Theorem 2, allows expressing PARITY in principle. Simultaneously, they empirically confirmed that such a
14973
solution could not be practically found via (S)GD training. Various other formal models of transformers (e.g. Weiss et al., 2021; Merrill and Sabharwal, 2023b,c; Strobl, 2023) can also express PARITY despite its empirical difficulty. As already concluded by Chiang and Cholak (2022), these findings highlight a disconnect between expressive capacity and learnability: not all functions which transformers may express in principle are also learnt efficiently. Evidently, existing expressiveness theory for transformers is not able to consistently account for the practical learnability of problems under gradient descent.
Some prior work has studied the learnability of problems for transformers. For example, Edelman et al. (2022) bound the statistical capacity of the transformer architecture, showing that on those functions that transformers prefer to represent, they can generalize with good sample efficiency. Notably, they found that sparse parities could indeed be learned well by transformers. However, this result does not prove that PARITY, or other highly sensitive functions, are hard to learn, as that technique does not provide a direct characterization of which functions transformers prefer to represent. Other work has studied simplified setups such as linear attention (e.g. Ahn et al., 2023) or individual attention layers (e.g. Sanford et al., 2023).
Here, we provide results that have direct bearing on the learnability of PARITY and other sensitive functions, characterizing the loss landscape of transformers in terms of input-space sensitivity. We formally prove that, for the transformer architecture, parameter settings achieving high sensitivity in input space are necessarily brittle, so that close neighbors in parameter space will usually define different (typically much less sensitive) functions when inputs are long. As a consequence, transformers fitting high-sensitivity functions must inhabit very steep minima. We argue that this explains both difficulty in training and length generalization for PARITY (observed by Bhattamishra et al. (2020); Delétang et al. (2023); Ruoss et al. (2023)), and a low-sensitivity and low-degree bias in random initialization and generalization (observed by Abbe et al. (2023); Bhattamishra et al. (2023)).
Expressiveness theory does not account for learnability. While unique hard attention provably cannot represent PARITY (Hahn, 2020; Hao et al., 2022; Angluin et al., 2023), more realistic upper bounds accounting for soft attention (Weiss et al., 2021; Merrill and Sabharwal, 2023b,c; Strobl, 2023; Chiang and Cholak, 2022) leave the hardness of sensitive functions unexplained. Not only does PARITY have transformers (Chiang and Cholak, 2022), but it can also be easily represented in formalisms that have been suggested to meaningfully upper-bound the abilities of various formal models of soft-attention:1
1Zhou et al. (2023) suggest that PARITY may not be representable in the RASP-L model, though the expressiveness of RASP-L is not well understood.
Fact 1 (Existing theory overpredicting abilities). Simple representations for PARITY, valid across all input lengths, exist in RASP (Weiss et al., 2021), uniform circuits with majority gates (Merrill and Sabharwal, 2023c; Strobl, 2023), and FO[M] (Merrill and Sabharwal, 2023b).
We prove this in Appendix A. Thus, existing expressiveness bounds do not account for the difficulty that transformers encounter in learning sensitive functions, in particular given that previously-popular recurrent models do not encounter this difficulty. Another family of results consists of Lipschitzness bounds (Hahn, 2020; Li et al., 2023), which bound the influence that any individual input bit has on the output of a transformer. These turn out to underpredict the abilities of transformers:
Fact 2 (Existing theory underpredicting abilities). By results of Hahn (2020); Li et al. (2023), the following holds: Consider a transformer without layer norm. If x,x′ ∈ {±1}n differ only in the i-th bit, then at any other position j ̸= i, the output of a transformer differs only by O( 1n ).
This accounts for the difficulty of learning PARITY. But the bound suggests even simple sparse functions, such as FIRST (the language 1(0|1)∗) to be difficult, but transformers learn these well (Bhattamishra et al., 2023; Edelman et al., 2022). Indeed, Chiang and Cholak (2022) note that the bound is overcome by layer norm or input-length-dependent scaling of attention logits, which enable modeling of sparse functions.
We will show that the observed low-sensitivity bias can be understood in terms of the loss landscape: while transformers can express highly sensitive functions, such transformers are isolated in parameter space, and minima interpolating a sensitive function are very sharp. Indeed, we prove that tiny perturbations of a highly sensitive transformer tend to define, when inputs are sufficiently long, very different functions with much lower sensitivity.

heading: 2 Model of Transformers
text: We will focus on boolean functions. Following the conventions in the Analysis of Boolean Functions literature (O’Donnell, 2014) in modeling bitstrings as elements of {−1,1}n, we assume the alphabet Σ = {−1,1}, with word embeddings e(−1),e(+1) ∈ Rd . There further are positional encodings p1, p2, p3, · · · ∈ Rd . At the zero-th layer, token and positional encodings are added: y(0)i := e(xi)+ pi (i = 1, . . . ,n), where x ∈ {±1}n is the input string.
A transformer has a fixed number L of layers; the activations y(k)i ∈Rd at position i of the k-th layer (k = 1, . . . ,L) are defined as follows. Each layer has a set of H attention heads; we first compute attention scores for the h-th head:
a(k,h)i, j =(Kk,hy (k−1) j ) T Qk,hy (k−1) i
â(k,h)i, j = exp(a(k,h)i, j )
∑s a (k,h) i,s
where Kk,h (“key”), Qk,h (“query”) are ∈ Rd×d . The activation of the head is computed by weighting according to attention weights â(k,h)i, j , and applying a linear transformation V (“value”):
b(k)i,h = n
∑ j=1
â(k,h)i, j Vk,hy (k−1) j (1)
The per-position activations are then computed as
Y (k)i := f MLP ( y(k−1)i + H
∑ h=1 b(k)i,h
) (2)
where f MLP is a one-layer MLP with a skip-connection. Transformers additionally implement layer norm (Ba et al., 2016):
LayerNorm(y) := y−mean(y)√
σ2(y)+ ε (3)
where ε ≥ 0 is a hyperparameter ensuring numerical stability, and σ2(·) denotes the variance. By design, ∥LayerNorm(y)∥2 ≤ √ d, with equality at ε= 0. Transformer variants differ in where exactly layer norm is applied (e.g. Takase et al., 2022); we here assume for notational simplicity that layer norm applies after the MLP, but the details are irrelevant to our results, provided layer norm applies at least once. We thus set:
y(k)i := LayerNorm ( Y (k)i )
(4)
Of key importance will be the the normalization factor:
N(k)i := 1√
σ2(Y (k)i )+ ε (5)
Our theoretical results will link N(k)i both to input-space sensitivity and parameter-space sharpness: We will find that large values of N(k)i can increase expressive capacity, but at the price of increased brittleness.
Finally, we assume that predictions are made by T := vTout · y (L) n for some parameter vout ∈ Rd . Throughout, we will add the input string x ∈ {±1}n as an argument when needed for disambiguation, e.g., writing T (x) for the overall prediction made on x.

heading: 3 Average Sensitivity
text: Our results are centered around average sensitivity, a simple but foundational complexity metric for functions on the Boolean cube (e.g. Kahn et al., 1988; De Wolf, 2008; O’Donnell, 2014):
Definition 3. For a bitstring x ∈ {±1}n and a function f : {±1}n → R, we write
s(x, f ) := 1 4
n
∑ i=1
| f (x)− f (x⊕i)|2 (6)
where x ⊕i
denotes the bitstring x with the i-th bit flipped. The average sensitivity for inputs of length n is
asn( f ) := 1 2n ∑x∈{±1}n s(x, f ) (7)
If f maps to {±1}, then s(x, f ) is the number of Hamming neighbors of x on which f flips. This definition of asn( f ) corresponds to the “total influence” from O’Donnell (2014, Def. 2.27). We explicitly define average sensitivity relative to input length n, as we will investigate the behavior of transformers performing a single function f across varying input lengths. The use of squared distances, rather than simple absolute distances, ensures that results about asn( f ) transfer to results about degree profiles (Abbe et al., 2023), which we will later investigate (Eq. 17).
Average sensitivity is a general complexity metric with wide-ranging applications in theoretical computer science (e.g. Jukna, 2012). It is closely linked to the Fourier analysis on the Boolean cube (De Wolf, 2008), and is an average-case version of a family of sensitivity measures, closely related to other natural metrics such as decision tree depth and polynomial degree (Hatami et al., 2010). Both average sensitivity itself (Bhattamishra et al., 2023) and the Fourier structure (Abbe et al., 2023) have been empirically linked to transformers’ generalization behavior. We will ground these empirical findings by relating average sensitivity to loss landscapes for the transformer architecture.
Example Functions Our theoretical results will apply to general functions on the Boolean cube. In order to ground these, we will illustrate transformers’ lowsensitivity bias at the example of a few natural functions which have played a role in the theoretical literature of transformers or are otherwise illustrative of variability in average sensitivity.
PARITY indicates whether the number of ones in a bitstring is even (output 1) or odd (output -1); over the input space {±1}n and output space {±1}, it can be formally defined as the map x 7→ ∏ni=1 xi. As flipping any input bit flips the output, asn( f ) = n. As described above, this function can in principle be represented by transformers, but has empirically been found to be very hard to learn.
MAJORITY maps x ∈ {±1}n to 1 if #{i : xi = 1}> n/2, and −1 else. Transformers show good length generalization (Merrill et al., 2022; Zhou et al., 2023). It is known that asn( f ) = Θ( √ n) (O’Donnell, 2014, Ex. 2.22). However, s( f ,x) = n whenever the ones and zeros in x are almost fully balanced.
FIRST maps x to its first bit, x1. As only the first bit matters, asn( f ) = 1. It is a simple example of a sparse function; more generally, a k-PARITY is a restriction of the PARITY function to only k inputs, where k is a constant. Transformers learn such sparse functions well (Bhattamishra et al., 2023; Edelman et al., 2022).
MEAN maps x 7→ 1n ∑ni=1 xi ∈ [−1,1]. We have asn( f ) = 1n .
The PARITY function, when varying the number of bits considered, is a universal basis for Boolean functions, in the sense that any function {±1}n →R can be represented as a linear combination of parities applying to different subsets of {x1, . . . ,xn}. Functions are more sensitive when parities applying to larger subsets appear in this representation. We will investigate this connection further below.

heading: 4 Lower Bounds for Sensitive Functions
text: We first prove that representing sensitive functions with transformers requires large parameter norms and, when inputs get longer and longer, highly unbounded normalization factors N(k)i in layer norm (5). We start from the global Lipschitzness bounds developed by Hahn (2020); Edelman et al. (2022); Li et al. (2023), but obtain more fine-grained average-case and highprobability bounds. These will then form the basis of our characterization of loss landscape sharpness around sensitive transformers. Our bounds will include a constant C that is the product of
exp ( 4d max
h
L
∑ i=2
∥KTi,hQi,h∥2 )
(8)
and a term polynomial in H, d, the spectral norms of all parameter matrices appearing in the transformer, and the maximum norm of any positional or word embedding. See (32) in the Appendix for formal definition.
Existing Lipschitzness bounds (Fact 2) imply a bound along the lines of2
s( f ,x)≤ C exp(4d maxh ∥KT1,hQ1,h∥2)
εL/2 (9)
uniformly for each x ∈ {±1}n. Li et al. (2023) noted that the exponential dependency on the spectral norm of the key-query matrix is unavoidable for a global Lipschitzness bound. Our first result is that this dependency can be eliminated at the input layer for the vast majority of inputs, at the cost of a logarithmic factor, leading to a bound of the form (assuming ε > 0 in (3))
s( f ,x)≤C logn εL/2
(10)
for 1−Hn−2 of inputs. We show this using a concentration bound argument, combining a Chernoff bound applying to each attention weight individually, with a union bound over all attention weights (Appendix, Lemma 11). Next, we address the role of layer norm. Chiang and Cholak (2022) showed that, at ε → 0 in (3), layer norm enables transformers to represent PARITY. Lipschitzness bounds in terms of ε (10) cease being meaningful in this limit. We thus study the layernorm induced blowup in more detail. In each layer, we
2Lipschitzness bounds by Edelman et al. (2022) are not directly applicable here, as these authors consider Lipschitzness in the parameter space, not the effect of changes in the input space as Theorem 3. See also Appendix, Remark 10.
consider the maximum blowup, given an input string x ∈ {±1}n:
τ(k)(x) := max w=1,...,n {1+N(k)w (x)} (11)
The addition of 1 is for technical reasons (Appendix, Lemma 16); it has little impact in the cases relevant to our results, which will be when τ(k)(x) = ω(1). We then write Blowup(x) := ∏Lk=1 τ(k)(x), an upper bound on the product of the successive layer-norm-induced blowups when going from the input to the output. When ε = 0 in (3), Blowup(x) can be arbitrarily large. Foreshadowing Theorem 6, we will find that large values of Blowup(x) create very sharp minima.
Our first theorem localizes the layer norm blowup to the Hamming neighborhoods of sensitive inputs:
Theorem 4 (Local Bounds on Layer Norm Blowup). Consider a transformer with layer norm at arbitrary ε ≥ 0. With probability 1− Hn2 over the choice of x ∈{±1}n, we have
s( f ,x) C √ n logn ≤ Blowup(x)2 + 1 n
n
∑ i=1
Blowup(x⊕i)2 (12)
The proof is in Appendix B.5. This permits us to state a bound on average sensitivity, in terms of the average layer norm blowup:
Corollary 5 (First Main Result). Consider a transformer with layer norm at arbitrary ε ≥ 0. Then
C ·E[Blowup(x)2]≥ asn( f )√ n logn − H n
(13)
where the expectation is over the uniform distribution over x ∈ {±1}n.
The proof is in Appendix B.6. Note that Hn is small when n is large, and the bound is dominated by asn( f )√n logn . This result thus shows a tradeoff between parameters and LN blowup: at least one of them needs to be large to represent a sensitive function. When the sensitivity depends on the input length and grows faster than√
n logn, this tradeoff changes with the input length, requiring larger parameters or larger layer norm blowup as the input length increases.
Let us investigate the implications for the functions introduced in Section above. For PARITY, asn( f ) = n, and (13) predicts
C ·E[Blowup(x)2] = Ω ( √
n logn
) = ω(1) (14)
showing that, for fixed parameters, the layer norm blowup needs to increase as the input length increases. For the other functions, the bound is O(1): For FIRST, s( f ,x) = asn( f ) = 1, and the RHS of (13) is O(1). Indeed, sparse functions can be modeled well by a family of transformers where the logits in the input layer scale with logn (Edelman et al., 2022; Chiang and Cholak, 2022). Unlike the prior Lipschitzness bounds (9), these
scaled logits do not contribute to C – our new bound is thus consistent with the ease with which transformers learn sparse functions. For MEAN, s( f ,x)∼ 1n2 ; again no blowup is predicted. For MAJORITY, as asn( f ) ∼√
n, no nontrivial average blowup is predicted. However, s( f ,x) = n whenever the ones and zeros in x are almost fully balanced; on such strings or their Hamming neighbors, (12) predicts Blowup = Ω(n1/4).

heading: 5 Sensitive Transformers are Brittle
text: Leveraging Corollary 5, we now show that transformers expressing sensitive functions must be very sensitive to small perturbations of model parameters. That is, sensitivity in input space entails sensitivity in parameter space. An important consequence is that, for any highly sensitive function, any interpolating minima will be very sharp. This property is nontrivial, as seen from the fact that it disappears when adding a scratchpad (see below).
Given a parameter vector θ defining the transformer Tθ, the average direction sharpness is (e.g. Wen et al., 2022; Jiang et al., 2020; Foret et al., 2020)
Lρ,n(T ) := Ex∈{±1}nE∥∆∥2=ρ(Tθ+∆(x)−Tθ(x))2 (15)
where the second expectation is over the radius-ρ sphere in the space of parameter vectors. Lρ,n quantifies the change in Tθ when perturbing θ by a size-ρ vector in a random direction. Lower bounds on Lρ,n (Theorem 6) immediately entail lower bounds on other common sharpness measures, such as worst-case sharpness, and (in the limit ρ → 0) the trace of the loss function’s Hessian at θ.
In defining the parameter vector θ in (15), we exclude positional encodings, both because they are often frozen, and because their number depends on n, hindering fair comparison across n.
Our next result lower-bounds Lρ,n in terms of the average sensitivity, provided the transformer is sufficiently wide in relation to its depth:
Theorem 6 (Second Main Result). Let Tθ be a transformer where d > 12L. Assume Tθ(x)∈ [−1,1] for each x. Then:
lim ρ→0 liminf n→∞ Lρ,n(T )≥ liminf n→∞ asn(Tθ) 2n
−Lexp(−Ω(d)) (16)
Here, “Ω(d)” scales positively with d. If Tθ has Boolean outputs (Tθ(x) ∈ {±1}), then the factor “2” can be eliminated from (16).
Informally, this theorem says that, when asn(Tθ)∼ n, then even tiny perturbations to the parameters will, in expectation, lead to a substantial change in predictions on long inputs. This means that, as the input gets longer, the Hessian of the mean-squared loss at the minimizer fitting a sensitive function has unboundedly large entries.
See Appendix C for the proof. The key idea of the proof is that, if Tθ is very sensitive in input space, small
perturbations to the parameters usually lead to a large drop in sensitivity when n is large, because they lead to large changes in the layer-norm induced blowup that is needed to represent high-sensitivity functions. As a consequence, transformers computing sensitive functions are isolated in parameter space.
The theorem applies when d is substantially larger than L, as is indeed true of typical transformers (e.g., LLaMa 7B has d = 4096 and L = 32). The convergence of the limits is slower when the parameternorms, as summarized by C, are larger, as larger parameters can increase sensitivity. However, remarkably, for any fixed transformer, C becomes irrelevant for Lρ,n in the limit where n → ∞ (16).
While we stated Theorem 6 for an individual transformer, the same statement holds for families of transformers where weights may depend on n, as long as C remains bounded. A consequence is that scaling attention logits with logn (used to represent sparse functions by Edelman et al. (2022); Chiang and Cholak (2022)) will, at least in the input layer, not mitigate the difficulty of sensitive functions.

heading: 6 Implications
text: We discuss how Theorem 6 unifies a range of diverse empirical findings about the behavior of transformers.
Difficulty of PARITY For PARITY, Lρ,n converges to 1 for large d, showing that arbitrarily small perturbations to a transformer computing PARITY will lead to a high loss for sufficiently long inputs. Previously, Chiang and Cholak (2022) noted that, for their handconstructed transformer representing PARITY, small changes to a specific parameter led to a large increase in loss, suggesting that this made the solution impossible to reach with SGD. Theorem 6 shows that this phenomenon is unavoidable for any transformer representing a high-sensitivity function. For functions with sublinear average sensitivity, Theorem 6 entails no nontrivial lower bound on sharpness, and no such phenomenon is predicted.
Random Initialization A key step in Theorem 6 is to show that s(Tθ+∆,x) is bounded with very high probability over the choice of ∆ (Appendix, Eq. (70)); this immediately entails that high-sensitivity transformers can only inhabit a small volume in parameter space. This explains why randomly initialized transformers empirically show low average sensitivity, more so than recurrent networks (Bhattamishra et al., 2023).
Length Generalization An important corollary is that, for a sensitive function, length generalization requires exact match to the minimum: the slightest deviation from the exact minimum will, in expectation, lead to failure when inputs get sufficient long, even if the minimum itself represents a length-generalizing solution. This provides, for the first time, a rigorous explanation why, despite the in-principle existence of lengthgeneralizing transformers, transformers struggle with
length generalization for PARITY (e.g. Bhattamishra et al., 2020).
Generalization on Unseen Data Another corollary is that, in expectation, the training loss landscape around an interpolating minimum places a constraint on a function’s overall sensitivity, even if not a single pair of Hamming neighbors are in the training set. This is because (16) remains true, on average across randomly selected training sets, if replacing the expectation over the full input space with an expectation over the training set in (15). This means that, for long inputs, flat minima of the training loss generalize with bounded sensitivity. To the extent that gradient-based training tends to find flatter minima (e.g. Damian et al., 2023), this provides a theoretical justification for the empirical result that transformers’ generalization behavior, when trained on a Boolean function on some training subset of {±1}n, shows a strong bias towards low average sensitivity (Bhattamishra et al., 2023).
Abbe et al. (2023) proposed a min-degree bias, that is, a generalization bias towards functions that are linear combinations of functions that each depend on only a few inputs. Any function f : {±1}n → R can be uniquely written as a linear combination of the multilinear monomials χP(x) := ∏i∈P xi, where P ⊆ {1, . . . ,n}: f (x) = ∑P λPχP(x). The coefficients λP define the Fourier-Walsh transform of f . For χP, both its degree as a polynomial, and its average sensitivity, are |P|. The degree profile, as defined by Abbe et al. (2023), is the tuple (d1, ...,dn) where di = ∑P:|P|=i |λP|2. Minimization of degree profile then refers to setting as many of the later entries to zero, and minimizing the size of the last nonzero entry. Abbe et al. (2023) proved inductive biases towards functions with a low degree profile for a random features model and diagonal networks. Evidence in the case of transformers was limited to empirical data on three example functions. Our results entail prove an bound on degree profiles for the full transformer architecture, because average sensitivity is a summary statistic of the degree profile (O’Donnell, 2014):
asn( f ) = ∑ P⊆{1,...,n}
λ2P|P|= n
∑ i=0 i ·di (17)
Hence, functions with degree profile assigning substantial weight to degrees of order ∼ n are brittle in parameter space, corresponding to very sharp minima.
Intermediate Steps Reduce Sensitivity PARITY can be solved well with a scratchpad (Anil et al., 2022; Liu et al., 2023). Existing theoretical accounts of the benefit of intermediate steps for transformers’ expressive capacity (e.g. Merrill and Sabharwal, 2023a; Feng et al., 2023) do not account for the benefit of intermediate steps for PARITY-like problems: While the theoretical models of transformer expressivenes used in these studies do predict versions with intermediate steps to be easy, they do not predict that computing PARITY in
a single step would be hard, due to Fact 1. The concept of average sensitivity provides a simple explanation for the benefit of intermediate steps. Formally, we can consider the problem of simulating a finite automaton with state set X either translating to the final state tn in one go (standard), or to autoregressively translate it into a sequence of states t1, ..., tn (scratchpad). Then (proof in Appendix D):
Theorem 7. Simulating an automaton with scratchpad has sensitivity O(1) for each autoregressive step.

heading: 7 Experiments

heading: 7.1 Setup
text: We conducted experiments to test the predictions made by our theory, specifically assessing predictions regarding loss landscapes and sharpness of the minima. In all experiments, we use the transformer encoder architecture, using the default implementation in PyTorch (Paszke et al., 2019). Each model is trained to fit a function f on a specific sequence length n. Each training input x is generated uniformly from {±1}n, and each input bit, treated as a separate token, is embedded using learned token and positional encodings.3
The representation of the last token is passed to a linear layer to generate the prediction Tθ(x). Parameters θ of the transformer are optimized for MSE loss between Tθ(x) and f (x) using AdamW (Loshchilov and Hutter, 2017). For full details on hyperparameters and training setup, refer to Appendix E.1.
In implementation, we assumed versions of the functions outputting to {0,1}; we rescaled sharpness values accordingly for comparability with the theory.
We analyzed models using the following metrics:
1. Parameter Norm. We compute the L2 norm of the entire model’s parameter vector, excluding positional encoding matrices. We discard the norm of positional encodings, so that the norms of the models trained for different sequence lengths are comparable.
2. LayerNorm Blowup. This metric is computed by computing the maximum normalization factor (5) across the entire layer in each application of layer norm, and take the product over all applications of layer norm. This essentially corresponds to Blowup.4
3. Sharpness. In order to avoid committing to any specific ρ, we sample ∆ in (15) not from the radius-ρ-sphere, but from a mean-zero Gaussian with STD ρ = 0.02. We estimate using Ns,p perturbations and Ns,b input strings x. This provides
3Code is available at https://github.com/ lacoco-lab/sensitivity-hardness.
4In the theory, Blowup has an additional 1+... in each factor for technical reasons. This difference is immaterial, as we are interested in situations where Blowup = ω(1).
results equivalent to Lρ,n in the ρ → 0 asymptotic, while avoiding committing to any specific ρ.
As we are interested in properties of models that compute given functions, runs that did not converge (evaluation MSE higher than 10−3) were discarded.
7.2 Results Higher Sensitivity Implies Sharper Minima. In this experiment, we train transformers to fit f ∈ {PARITY,MAJORITY,FIRST,MEAN} on sequence lengths from 4 to 30. For each function and sequence length, we retrain the model 10 times from different random initializations.
For PARITY, sharpness stably increases with the input length (Figure 1). For the other functions, whose sensitivity grows more slowly with n, (a) the absolute value of sharpness is orders of magnitude lower than for PARITY; (b) there is little increase with n. More results are shown in Appendix E.2.
Tradeoff between Weight Norm and Layer Norm Blowup. At any fixed input length n, high sensitivity can be achieved by a combination of large weights and a large LayerNorm blowup. By Theorem 5, the product of C and squared blowup is bounded from below with some value Bn( f ). Hence, the product of√
C and blowup is bounded with √
Bn( f ), and the sum of 12 logC and logBlowup is bounded with 1 2 logBn( f ). However, C depends exponentially on parameters, and thus we expect the parameter norm to trade off with the logarithm of the layer norm blowup. Moreover, for PARITY the value of Bn( f ) increases with n, and therefore sum of parameter norms and the logarithm of the blowup should also increase with n.
5000 2500 0 2500 5000 7500 10000 12500 15000 Steps after convergence
0.0
0.1
0.2
0.3
Lo ss
Loss
0
1
2
Sh ar
pn es
s
Loss Sharpness

heading: The Limits of Transformers’ Generalization. As
text: Scratchpad Eliminates Sharpness. By Theorem 7, sensitivity of each autoregressive step when computing PARITY with scratchpad is O(1). Hence, Theorem 6 provides no nontrivial lower bound for Lρ,n(T ). We trained an Encoder-Decoder Transformer, predicting PARITY of i-th substring on i-th autoregressive step: ti = PARITY(x1:i) = xi ⊕ ti−1 (t0 = 0). The visual dependency between sharpness and length of input for PARITY with a scratchpad is shown in Figure 11. Even for length around 300, sharpness is low and there is little increase with input length. Thus, decrease in sensitivity due to the scratchpad can explain why prior work (Anil et al., 2022) found that PARITY is easy for Transformers with scratchpad.
LayerNorm Blowup Enables Learning Figure 3 represents the evolution of loss and sharpness of Transformer models trained for PARITY with input length 25. The results are averaged across 39 converged runs. Similar curves for parameter norm and LayerNorm blowup are presented in Appendix 12.
A dramatic increase in sharpness occurs at exactly the time when loss falls to 0. This suggests the presence of a steep minimum in the loss landscape, and fitting PARITY requires the optimization procedure to find this minimum. Figure 12 (Appendix) shows further details of this process. During learning, there is a small but sharp increase in the blowup which makes the high sensitivity of the model possible, increasing the left-hand side of the inequality in Corollary 5. Fol-
lowing that, non-zero weight decay drives the parameter norm down, which – by the theoretically predicted tradeoff between blowup and parameter norm – is accompanied by an exponential increase in blowup.

heading: 8 Discussion
text: We have provided a rigorous theoretical explanation of the inductive bias towards low sensitivity observed in empirical research on transformers. Theorem 6 describes a fundamental inductive bias of the transformer architecture: for long inputs, fitting sensitive functions is only possible in sharp minima of the loss. This holds without assumptions often made in previous work about expressive capacity, such as noninfinite precision (e.g. Merrill and Sabharwal, 2023b; Angluin et al., 2023), hard attention (e.g. Hahn, 2020), or Lipschitz-continuous variants of layer norm (Edelman et al., 2022). We speculate that Theorem 6 reflects a fundamental limitation of parallelized differentiable computing with bounded depth. Our results show that it is overcome by scaling the number of computation steps with the input length. While we focused on functions outputting a single label; an interesting direction for future research is the extension of this theory to sequence-to-sequence transductions, for which some empirical data has been reported (e.g. Zhou et al., 2023), but theoretical understanding remains wide open.
Due to the relation between average sensitivity and Fourier analysis on the Boolean cube (Equation 17), a low-sensitivity bias can be viewed as a sequence modeling analogue of a spectral bias towards low frequencies observed in other neural architectures (e.g. Rahaman et al., 2019; Fridovich-Keil et al., 2022).
We note that, while our results show that sensitive transformers are very brittle, these results do not by themselves have implications for real-world generalization, as a low-sensitivity bias need not always be beneficial in real-world setups. Indeed, the relationship between sharpness and real-world generalization is not straightforward (e.g. Andriushchenko et al., 2023; Kaur et al., 2023). Our theory suggests that transformers generalize well to the extent that real-world data has bounded sensitivity (e.g. Hahn et al., 2021).

heading: 9 Conclusion
text: We have proven that, under the transformer architecture, high sensitivity in input space can only be achieved in very sharp minima. Empirical results confirm the predictions of the theory. Taken together, our results explain a diverse set of empirical observations about transformers not explained by previous theoretical work. They suggest shifting theoretical research from in-principle expressiveness considerations to studying quantitative bounds and the shape of the loss landscape in order to understand the abilities of transformers.

heading: Limitations
text: A limitation of our results is that the theoretical results are asymptotic, providing statements about the limit of very long input sequences. Providing more quantitative bounds that tightly characterize finite-length behavior is an interesting problem for future research.
A second limitation is that we only target functions outputting a single value. Operationalizing sensitivitylike metrics for sequence-to-sequence functions may be required in order to expand the theory to such functions.
Third, our results apply to transformer encoders. It remains open if transformer decoders, with causal attention masking, face a different set of limitations than we have shown here in the absence of masking.
Fourth, our theoretical results concern the loss landscape, not the training dynamics itself. Further technical advances may be needed to directly prove corresponding results for training dynamics.

heading: Acknowledgments
text: We thank Lena Strobl, Dana Angluin, and David Chiang for useful discussion. We thank David Chiang, Paul Lintilhac, and Yuval Pinter for spotting various errors in a previous version, and the anonymous ARR reviewers for useful feedback.

heading: A Simple Constructions for PARITY
text: While it is well-known that PARITY can be expressed using uniform circuits with majority gates (Håstad, 1986), it may not be obvious just how simple the construction can be, and how it can easily embed into formalisms like RASP. In order to highlight the existence of very simple constructions for PARITY in various formalisms, and to make discussion here self-contained, we provide a simple proof. The key is the following insight:
Lemma 8 (Folklore). A bit string is odd if and only if (†) for the (strict) majority of positions i holding a 1, it holds that strictly more than 1/2 of 1s appear no later than at i.
This fact appears to be well known in the circuit complexity community, though we are not aware of the original published reference. We provide a proof for self-containedness.
Proof. As only positions holding a one play a role, it is sufficient to consider strings of the form x ∈ 1∗. First, consider the case where x = 12n:
Index 1 2 . . . n n+1 . . . 2n A: # of 1s no later than at i 1 2 . . . n n+1 . . . 2n
B: # of 1s later than at i 2n-1 2n-2 . . . n n-1 . . . 0 A>B? no no . . . no yes . . . yes
Here, A > B holds at exactly n positions, less than the strict majority of positions. On the other hand, when x = 12n+1:
Index 1 2 . . . n n+1 . . . 2n 2n+1 A: # of 1s no later than at i 1 2 . . . n n+1 . . . 2n 2n+1
B: # of 1s later than at i 2n 2n-1 . . . n+1 n . . . 1 0 A>B? no no . . . no yes . . . yes yes
Here, A > B holds at exactly n+1 positions, a strict majority of positions.
Proof of Fact 1. The property (†) from Lemma 8 is straightforwardly formalized in RASP. To formalize it in FO[M] (Merrill and Sabharwal, 2023b), we note that FO[M] permits defining constructs of the form “for the majority of positions x satisfying φ(x), it holds that ψ(x)”, for any formula ψ. FO[M] formulas have a known translation to (highly uniform) majority circuits. This concludes the proof of Fact 1.
Lemma 8 shows that, in terms of expressive capacity, PARITY is tightly linked to MAJORITY. While the components of (†), MAJORITY and position comparison, are easily learned by transformers, the composite function is hard to learn. This observation suggests that function classes satisfying typical closure properties, as satisfied by typical logic and circuit classes, cannot model which functions transformers learn easily. On the other hand, average sensitivity elegantly explains this difference: The expression (†) uses two nested “majority” operations, each introducing an average sensitivity of √ n, multiplying to the asymptotically larger sensitivity n of the composite function. This result does not exclude a role of minimizing description length in formalisms such as RASP as a factor impacting transformers’ inductive biases (as suggested empirically by Zhou et al. (2023)), but suggests that description length and expressive capacity might be unable to account for important aspects of transformers’ inductive biases, including the low-sensitivity bias and difficulty learning PARITY.

heading: B Proof of Theorem 4 and Corollary 5
text: For a bitstring x ∈ {±1}n, we write Ii[ f ](x) to denote the absolute influence of the i-th bit:
Ii[ f ](x) := ∥∥ f (x)− f (x⊕i) ∥∥ 2 = √ ∑ s | f (x)s − f (x⊕i)s|2 (18)
which simplifies to the absolute value | f (x)− f (x⊕i)| if f (x) ∈ R. We will suppress the argument x where it is contextually given. When f outputs a scalar, then we have by (6):
s( f ,x) = 1 4
n
∑ i=1 Ii[ f ](x)2 (19)
We begin by noting
∥ f (x)− f (x⊕q)∥22 ≤ ∥ f (x)− f (x⊕q)∥2 ·maxx ∥ f (x)− f (x ⊕q)∥2 ≤ 2max x ∥ f (x)∥2 · ∥ f (x)− f (x⊕q)∥2 (20)
and hence
1 2 Iq[ f ]2(x) = 1 2 ∥ f (x)− f (x⊕q)∥22 ≤ (maxx ∥ f (x)∥2)∥ f (x)− f (x ⊕q)∥2 = (max x′ ∥ f (x′)∥2) Iq[ f ](x) (21)

heading: B.1 Bounding the Sensitivity of the Attention Heads
text: The first step will be bound the sensitivity of the outputs of individual attention heads. At a high level, such bounds are also part of the existing pointwise Lipschitzness bounds (Hahn, 2020; Edelman et al., 2022; Li et al., 2023). Our key innovation in this part of the proof is to replace a pointwise bound with a high-probability bound that, at the input layer, is independent of the key-query matrix. The key novel part is Lemma 11. In order to make the proof self-contained, we also include proofs for the other steps.
We will refer to the following bounds, for suitably defined constants 1 ≤CV ,CP,C(k)A ,CMLP,L f MLP ,C(k),C < ∞:
∥Vk,h∥2 ≤CV (22)( max
w ∥y(k)w ∥2
) ≤ √
d when k ≥ 1 (23) (
max w
∥y(0)w ∥2 ) ≤CP (24)
a(k,h)i j ≤C (k) A := d ·maxh ∥K T k,hQk,h∥spectral if k > 1 (25)
CMLP := sup x:∥x∥2≤(H+1)CV min{CP, √ d} ∥ f MLP(x)∥2 (26)
L f MLP := sup ∥x∥2,∥x′∥2≤(H+1) √ d
∥ f MLP(x)− f MLP(x′)∥2 ∥x− x′∥2
(27)
C(0) :=150L f MLPC 2 VCPH (28)
C(1) =2 (
max w
∥Y (1)w ∥2 ) ·L f MLP ≤ 2CMLP(H +1)CVCP ·L f MLP (29)
C(k) :=12L f MLP(H +1)∥Vk,h∥14 √ d exp(4C(k)A ) (for k > 1) (30)
∥Y (1)j ∥2 ≤(2)CMLP(H +1)CVCP (31)
C :=25 √ dH(1+∥vout∥) ( k
∏ l=0
C(k) )
(32)
We first bound the influence of any bit on an attention head’s activation in terms of its influence on the attention values and the activations at the preceding layer:
Lemma 9 (Sensitivity of an Attention Head). We have
Iq[b (k) j,h]≤CV
n
∑ w=1
âk,hj,w Iq[y (k−1) w ]+CV
( max
w ∥y(k−1)w ∥2
) n ∑
w=1 Iq[â
k,h j,w] (33)
The main task will be to bound ∑nw=1 Iq[â k,h j,w]. We separately bound this for k = 1 and k > 1.
Remark 10. We note that the conceptually similar Lemma 4.3 in Edelman et al. (2022) might suggest a seemingly stronger bound where ∑w Iq[â k,h j,w] is replaced (using their Lemma A.6) by the maximum influence on the logits: maxw=1,...,N Iq[a k,h j,w] using the fact that softmax has a Lipschitz constant ≤ 1. However, s( f ,x) requires summing over q (not relevant to Edelman et al. (2022)5), and the expression ∑q maxw=1,...,N Iq[a k,h j,w] can easily be Θ(n), for instance, if a1,hj,w ≡ 1xw=1. Importantly, at the lowest layer, we find sublinear bounds for ∑q ∑w Iq[â k,h j,w] that, at k = 1, involve no bound on the logits ak,hj,w at all.
Proof of Lemma 9. Using the definition of b:
b(k)i,h = n
∑ j=1
â(k,h)i, j Vk,hy (k−1) j (34)
5They considered Lipschitzness in the parameter space, with parameters that grow at most sublinearly with n. In contrast, we investigate sensitivity in the input space.
and the triangle inequality, we find:
Iq[b (k) j,h] = ∥∥∥∥∥ n
∑ j=1
â(k,h)i, j (x) ·Vk,h · y (k−1) j (x)−
n
∑ j=1
â(k,h)i, j (x ⊕q) ·Vk,h · y(k−1)j (x⊕q) ∥∥∥∥∥ 2
≤ n
∑ j=1
∥∥∥â(k,h)i, j (x)Vk,hy (k−1) j (x)− â (k,h) i, j (x ⊕q)Vk,hy (k−1) j (x ⊕q) ∥∥∥
2
≤ n
∑ j=1
|â(k,h)i, j (x)| ∥∥∥Vk,hy(k−1)j (x)−Vk,hy (k−1) j (x ⊕q) ∥∥∥ 2 + ∣∣∣â(k,h)i, j (x)− â (k,h) i, j (x ⊕q) ∣∣∣ ∥∥∥Vk,hy(k−1)j (x) ∥∥∥ 2
≤CV n
∑ w=1
âk,hj,w Iq[y (k−1) w ]+CV
n
∑ w=1
∥y(k−1)w ∥2 Iq[âk,hj,w]
≤CV n
∑ w=1
âk,hj,w Iq[y (k−1) w ]+CV
( max
w ∥y(k−1)w ∥2
) n ∑
w=1 Iq[â
k,h j,w]
Bounding the Sensitivity of Attention (First Layer) First, we bound the sensitivity of the attention distribution in the first layer. Here, it will be crucial to provide high-probability bounds independent of ∥KT1,hQ1,h∥, unlike the pointwise Lipschitzness bounds of Hahn (2020); Li et al. (2023). The resulting Lemma 11 is one of the key innovations compared to the pointwise Lipschitzness bounds from prior work, and the main technical innovation of Section B.1.
Lemma 11. Let δ > 0. With probability ≥ 1− H nδ−2 over the choice of x, the following holds: For each i = 1, . . . ,n, h = 1, . . . ,H,
n
∑ q=1
n
∑ j=1
Iq[â (1,h) i, j ]≤(8+32δ) logn+10 (35)
Proof of Lemma 11. Let xi denote the token at position i (-1 or 1), and e(xi) the corresponding word embedding. Throughout, we will suppress the index h = 1, . . . ,H for notational simplicity. If e(xi) is the word embedding for the token xi and pi is the i-th positional embedding, then:
a(1)i j = ( pTi + e(xi) T ) ·QT K · (p j + e(x j))
= ( pTi e(xi) T )(Id Id )T QT K ( Id Id )( p j e(x j) ) = ( pTi e(xi) T ) ( A B C W )( p j
e(x j)
)
= pTi Ap j︸ ︷︷ ︸ A(i j) + pTi C︸︷︷︸ Ci e(x j)+ e(xi)T Bp j︸︷︷︸ B j +e(xi)TWe(x j)
for appropriate matrices A,B,C,W . Note that this decomposition holds for any linear combination of Note that adding positional and word embeddings is a special case, where Q and K can each be written as products of two matrices, where the second one has the form ( Id Id ) . Then
â(1,h)i, j = exp(qi(xi)k j(x j))
∑nw=1 exp(qi(xi)kw(xw))
= exp(Cie(x j)+ e(xi)T B j + e(x j)We(xi)+A(i j))
∑nw=1 exp(Qie(xw)+ e(xi)T Bw + e(xw)We(xi)+A(iw))
We can write, taking x̂i ∈ {0,1} to be the indicator that xi = 16:
exp(e(xi)T Bw +A(iw)+Cie(xw)+ e(xi)TWe(xw))
6I.e., x̂i = 1 of xi = 1 and else 0.
=exp ( e(xi)T Bw +A(iw) ) · exp ( Cie(xw)+ e(xi)TWe(xw) )
=exp(e(xi)T Bw +A(iw)+Cie(−1)+ e(xi)TWe(−1))︸ ︷︷ ︸ γiw exp(Cie(xw)+ e(xi)TWe(xw)−Cie(−1)− e(xi)TWe(−1))︸ ︷︷ ︸ 1+x̂wρi
where
ρi =exp(Cie(1)+ e(xi)TWe(1)−Cie(−1)− e(xi)TWe(−1))−1
Next, we may assume w.l.o.g. that ρi is nonnegative, by, if ρi otherwise were negative, renaming the input bits across the entire input string for a fixed i. So we have
â(1,h)i, j = γi j(1+ x̂ jρi)
∑w γiw(1+ x̂wρi)
We split the summation over j and q into three cases: q ̸= i, j; q = j,q ̸= i; q = i: n
∑ j=1
n
∑ q=1 = ∑ j,q:q̸=i, j + ∑ j,q:q= j,q ̸=i + ∑ j,q:q=i
(36)
First, for q ̸= i, j:
Iq[â (1,h) i, j ] = γi j(1+ x̂ jρi) γiq +∑w̸=q γiw(1+ x̂wρi) − γi j(1+ x̂ jρi) γiq(1+ρi)+∑w ̸=q γiw(1+ x̂wρi)
= γi j(1+ x̂ jρi)γiqρi
(γiq +∑w̸=q γiw(1+ x̂wρi))(γiq(1+ρi)+∑w ̸=q γiw(1+ x̂wρi))
≤ γi j(1+ x̂ jρi)γiq(1+ρi) (γiq +∑w̸=q γiw(1+ x̂wρi))(γiq(1+ρi)+∑w ̸=q γiw(1+ x̂wρi))
= γi j(1+ x̂ jρi) (γiq +∑w̸=q γiw(1+ x̂wρi)) γiq(1+ρi) (γiq(1+ρi)+∑w ̸=q γiw(1+ x̂wρi))
=â(1,h)i, j (x : x̂q = 0)â (1,h) i,q (x : x̂q = 1)
where â(1,h)i, j (x : x̂q = 0) is â (1,h) i, j evaluated at the input x = x1 . . .xq−1(−1)xq+1 . . .xn, and similarly for â (1,h) i,q (x : x̂q = 1). We may assume w.l.o.g., by appropriate reordering given a fixed i, that γi1 ≥ γi2 ≥ γi3 ≥ . . . . Then,
∑ q ∑ j ̸=q
Iq[â (1,h) i, j ]≤∑
q ∑ j ̸=q
â(1,h)i, j (x : x̂q = 0)â (1,h) i,q (x : x̂q = 1)
=∑ q â(1,h)i,q (x : x̂q = 1) ∑ j ̸=q â(1,h)i, j (x : x̂q = 0)
≤∑ q â(1,h)i,q (x : x̂q = 1)
=∑ q γiq(1+ρi) γiq(1+ρi)+∑w̸=q γiw(1+ x̂wρi)
≤(∗) ∑ q>16δ logn γiq(1+ρi) γiq(1+ρi)+ γiq ∑w<q(1+ 14 ρi) + ∑ q≤16δ logn γiq(1+ρi) γiq(1+ρi)
≤ ∑ q>16δ logn (1+ρi) (1+ρi)+∑w<q(1+ 14 ρi) +16δ logn
≤ ∑ q>16δ logn (1+ρi) (1+ρi)+q( 1+ρi4 ) +16δ logn
= ∑ q>16δ logn
4 4+q +16δ logn
≤4 ∑ q>0 1 q +16δ logn
≤(4+16δ) logn+4
where (∗) holds with overwhelming probability over the choice of x, say, 1− 1 nδ (multiplicative Chernoff bound for binomial random variables), for any individual q, and 1− 1
nδ−1 for the whole expression by a union bound over
all q. We further used ∑ni=1 1 i ≤ logn+1.
Second, consider the case where q = j, q ̸= i:
∑ q= j;q ̸=i
I j[â (1,h) i, j ] = ∑
q= j;q̸=i;q>16δ logn I j[â
(1,h) i, j ]+ ∑
q= j;q̸=i;q≤16δ logn I j[â
(1,h) i, j ]
≤ ∑ q= j;q̸=i;q>16δ logn I j[â (1,h) i, j ]+16δ logn
Then:
∑ q= j;q̸=i;q>16δ logn
I j[â (1,h) i, j ] = ∑
q= j;q̸=i
γi j(1+1ρi) γi j(1+ρi)+∑w ̸= j γiw(1+ x̂wρi) − γi j γi j +∑w ̸= j γiw(1+ x̂wρi)
= ∑ q= j;q̸=i;q>16δ logn γi jρi ∑w ̸= j γiw(1+ x̂wρi) (γi j +∑w̸= j γiw(1+ x̂wρi))(γi j +∑w ̸= j γiw(1+ x̂wρi)+ γi jρi +∑w ̸= j γiw(1+ x̂wρi))
≤ ∑ q= j;q̸=i;q>16δ logn
γi jρi · ( γi j +∑w ̸= j γiw(1+ x̂wρi)+ γi jρi +∑w ̸= j γiw(1+ x̂wρi) )
(γi j +∑w̸= j γiw(1+ x̂wρi))(γi j +∑w ̸= j γiw(1+ x̂wρi)+ γi jρi +∑w ̸= j γiw(1+ x̂wρi))
= ∑ q= j;q̸=i;q>16δ logn γi jρi (γi j +∑w̸= j γiw(1+ x̂wρi))
≤ ∑ q= j;q̸=i;q>16δ logn γi jρi (γi j +∑w< j γiw(1+ x̂wρi))
≤ ∑ q= j;q̸=i;q>16δ logn γi jρi (γi j + γi j ∑w< j(1+ x̂wρi))
≤(∗) ∑ q= j;q̸=i γi jρi (γi j + γi j j 1+ρi4 ))
= ∑ q= j;q̸=i ρi (1+ j 1+ρi4 ))
≤ ∑ q= j;q̸=i 4ρi 4+ j+ jρi
≤ ∑ q= j;q̸=i 4ρi j+ jρi
= 4ρi 1+ρi ∑q= j;q̸=i 1 j
≤4logn+4
where again (∗) holds with probability 1− 1 nδ−1 by a Chernoff bound for each j, and a union bound across all j. Importantly, the third inequality used the fact that γiw ≥ γi j when w < j. So overall
∑ q= j;q ̸=i
I j[â (1,h) i, j ]≤ 4logn+4+16δ logn
with probability 1− 1 nδ−1 . Third, for the case q = i:
∑ j ̸=q
Iq[â (1,h) q, j ] = ∑ j ̸=q |â(1,h)q, j (x)− â (1,h) q, j (x ⊕q)|
≤ ∑ j ̸=q |â(1,h)q, j (x)|+ ∑ j ̸=q |â(1,h)q, j (x⊕q)|
≤2
Overall, we get
n
∑ q=1
n
∑ j=1
Iq[â (1,h) i, j ]≤(8+32δ) logn+10 (37)
for each i,h, with probability 1− H nδ−2 , via a union bound.
Remark 12. To illustrate why the statement can only hold with high probability, rather than pointwise at every x, we consider a one-layer transformer where the first layer at each position computes the OR function, accomplished by setting
a(1)i j = { logn if x j = 1 − logn if x j = 0
(38)
That is, any occurrence of a 1 draws attention to it, allowing a subsequent MLP to test whether a 1 appears in the string. Consider the input x = 0n. Here, for any Hamming neighbor x′ = x⊕q:
â(1,h)i, j =    n n+ n−1n = 11+1/n−1/n2 ≥ 1− 1 n if j = q
1/n n+ n−1n = 1n2+n−1 ≤ 1 n2 else
(39)
Hence,
Iq[â (1,h) i, j ]≥ { 1− 2n if j = q 1 n − 1n2 else
(40)
and
n
∑ q=1
n
∑ j=1
Iq[â (1,h) i, j ]≥
n
∑ q=1
Iq[â (1,h) i,q ]≥ n−2 (41)
The bound (35) thus cannot hold for all input strings x. However, the lemma shows that it holds for the vast majority of inputs. The intuitive reason is that the vast majority of strings x ∈ {0,1}n have a much less skewed distribution of ones and zeros. For a string where a substantial number of both ones and zeros appear, the influence of any individual input bit is low; the proof of Lemma 11 makes this idea rigorous using a simple concentration bound argument. This intuition roughly matches the sensitivity properties of the OR function: s( fOR,0n) = n, but asn( fOR) = o(1), because for the vast majority of strings, flipping any bit cannot flip the OR output. Lemma 11 generalizes this idea to arbitrary assignments of attention logits, and to the continuous output of the softmax operation, as opposed to discrete Boolean functions such as OR.
Bounding the Sensitivity of Attention: Higher Layers At higher layers, the logits ai j may be highly correlated across j, impeding the use of concentration bounds. In fact, for our purposes here, a pointwise Lipschitzness bound as in Hahn (2020); Li et al. (2023) will be sufficient here (roughly corresponding to Lemmas B.1, B.2 of Li et al. (2023)). While no novelty is claimed for Lemma 13, we include its proof for completeness.
Lemma 13. For k > 1 and q ∈ {1, . . . ,n},
Iq[â (k,h) i, j ]≤ 2exp(4C (k) A ) ( Iq[y (k−1) i ]+ 1 n n ∑ j=1 Iq[y (k−1) j ] ) (42)
Remark 14. Note that this bound depends on ∥KT Q∥spectral , unlike the first layer, where we found a bound independent of ∥KT Q∥spectral . We do not know whether the exponential dependency on ∥KT Q∥spectral is optimal. However, some dependency on ∥KT Q∥spectral is unavoidable. To see why, consider a transformer where y(L)n computes a function close to PARITYn at ε = 1 (Chiang and Cholak, 2022), and y (L) 1 computes its negation. Then adding another layer that, by rescaling logits with a factor of n2, attends to y(L)1 or y (n) N depending on the input’s parity, can compute a function very close to PARITY, if the two activations also provide some disambiguating positional information. This holds even for high-probability or on-average bounds. However, Lemma 11 shows that such dependency can be eliminated at the lowest layer.
Proof. Fixing k,h, i, we write
cu =exp ( a(k,h)iu (x) ) ≤(25) exp(C(k)A ) (43) du =exp ( a(k,h)iu (x) ) − exp ( a(k,h)iu (x ⊕q) ) ≤(25) 2exp(C(k)A ) (44)
Then:
∣∣∣â(k,h)i,u (x)− â (k,h) i,u (x ⊕q) ∣∣∣= ∣∣∣∣∣ cu ∑ny=1 cy − cu +du ∑ny=1 cy +dy ∣∣∣∣∣
= ∣∣∣∣∣ cu(∑ny=1 cy +dy)− (cu +du)∑ny=1 cy
(∑ny=1 cy)(∑ n y=1 cy +dy)
∣∣∣∣∣
= ∣∣∣∣∣ cu ∑ny=1 dy −du ∑ny=1 cy (∑ny=1 cy)(∑ n y=1 cy +dy) ∣∣∣∣∣ ≤ cu ∑ny=1 Iq[cy]+ Iq[cu]∑ n y=1 cy
n2 exp(−2C(k)A )
Now using Iq[cu] = Iq [ exp ( a(k,h)iu (x) )]
≤exp(C(k)A ) Iq [ a(k,h)iu (x) ] ≤exp(C(k)A )∥KTk,hQk,h∥ ( Iq [ y(k−1)i (x) ] + Iq [ y(k−1)u (x) ]) (45)
By x ≤ exp(x), we can bound ∥KTk,hQk,h∥2 ≤ exp(C (k) A ). Now taking a sum over u yields:
∑ u
∣∣∣â(k,h)i,u (x)− â (k,h) i,u (x ⊕q) ∣∣∣≤ n
∑ u=1
cu ∑ny=1 Iq[cy]+ Iq[cu]∑ n y=1 cy
n2 exp(−2C(k)A )
≤(45,43) exp(2C(k)A ) 2n2 Iq
[ y(k−1)i (x) ] +2n∑nu=1 Iq [ y(k−1)u (x) ]
n2 exp(−2C(k−1)A )
≤2exp(4C(k−1)A ) ( Iq[y (k−1) i ]+ 1 n n ∑ j=1 Iq[y (k−1) j ] )
Bounding the Sensitivity of the Attention Heads We now put together Lemmas 9,13 in order to bound the sensitivity of any individual attention head’s output.
Lemma 15. For k > 1, the following holds for each j and h:
Iq[b (k) j,h]≤ 4 √ d exp(4C(k)A ) [ Iq[y (k−1) j ]+ 1 n n ∑ w=1 Iq[y (k−1) w ] ] (46)
Proof. In the higher layers (k > 1):
Iq[b (k) j,h]≤(33)
n
∑ w=1
âk,hj,w Iq[y (k−1) w ]+
( max
w ∥y(k−1)w ∥
) n ∑
w=1 Iq[â
k,h j,w]
≤(42) exp(2C (k) A )
n
n
∑ w=1
Iq[y (k−1) w ]+ ( max
w
∥∥∥y(k−1)w ∥∥∥ ) 2exp(4C(k)A ) ( Iq[y (k−1) j ]+ 1 n n ∑ j=1 Iq[y (k−1) j ] )
= exp(2C(k)A )+
√ d2exp(4C(k)A )
n
n
∑ w=1
Iq[y (k−1) w ]+ √ d2exp(4C(k)A ) ( Iq[y (k−1) j ] )
≤4 √
d exp(4C(k)A ) [ Iq[y (k−1) j ]+ 1 n n ∑ w=1 Iq[y (k−1) w ] ]
where the second step uses the fact that
âk,hj,w ≤(43) exp(2C(k)A )
n , (47)
and the third step uses the fact that the outcome of layer norm has an L2 norm bounded by √ d.
B.2 Impact of Layer Norm
Next, we investigate the effect that layer norm has on sensitivity. We note that layer norm was not fully included in any of the pointwise Lipschitzness bounds (Hahn, 2020; Edelman et al., 2022; Li et al., 2023): Hahn (2020); Li et al. (2023) do not seem to mention layer norm, and Edelman et al. (2022) assumes a Lipschitz-continuous proxy, projection on the unit ball. We find that standard layer norm indeed can increase sensitivity substantially, which will be key to our overall results: Lemma 16. Let Y (x) ∈ Rd , and y(x) := LayerNorm(Y (x)). Define N(x) := √
σ2(Y (x))+ ε. Let C > 1 be such that ∥Y (x)∥2 ≤C. Then
Ii[y]≤ 2 · Ii[Y ] ·C · [ 1+ 1
N(x)
] · [ 1+ 1
N(x⊕i)
] (48)
Proof. First, we note
Ii[(Y −µ(Y ))]≤ Ii[Y ]+ 1 d ∑s Ii[Ys] = (1+ 1 d ) Ii[Y ]≤ 2Ii[Y ]
Thus, at a factor of 2, it is sufficient to consider the case where µ(Y ) = 0, because ∥Y −µ(Y )∥ ≤ ∥Y∥. Then
Ii[y] =∥y(x)− y(x⊕i)∥2
= ∥∥∥∥ Y (x) N(x) − Y (x ⊕i) N(x⊕i) ∥∥∥∥ 2
=
∥∥N(x⊕i)Y (x)−N(x)Y (x⊕i) ∥∥
2 N(x)N(x⊕i)
≤ ∥Y (x)∥2 · |N(x⊕i)−N(x)|+N(x⊕i)
∥∥Y (x)−Y (x⊕i) ∥∥
2 N(x)N(x⊕i)
≤∥Y (x)∥2 · |N(x ⊕i)−N(x)|
N(x)N(x⊕i) +
∥∥Y (x)−Y (x⊕i) ∥∥
2 N(x)
≤C|N(x ⊕i)−N(x)|
N(x)N(x⊕i) +
∥∥Y (x)−Y (x⊕i) ∥∥
2 N(x)
≤ Ii[Y ] · [
C N(x)N(x⊕i) + 1 N(x)
]
= Ii[Y ] · 1 N(x) · [ C N(x⊕i) +1 ] ≤ Ii[Y ] ·C · [ 1+ 1
N(x)
] · [ 1+ 1
N(x⊕i)
]
where C ≥ 1 is a bound on ∥Yi∥2. Here, we used Ii[N] ≤ Ii[Y ] in the last step: Note that Y is mean zero; hence, standard deviation equals the L2 norm, and
√ d Ii[N] = |∥Y (x)∥2 −∥Y (x⊕i)∥2| ≤ ∥Y (x)−Y (x⊕i)∥2 ≤ ∥Y (x)−Y (x⊕i)∥2 = Ii[Y ]

heading: B.3 Layerwise Bounds on Sensitivity
text: Putting together Lemmas 9 and 16, we bound the influence of the i-th bit on the k-th layer in terms of its influence on the k−1-th layer.
Lemma 17. With probability ≥ 1− Hn2 over the choice of x, the following holds for each j:
n
∑ i=1
Ii[Y (1) j ]≤ 150L f MLPC2VCPH logn =C(0) logn (49)
and
Ii[y (1) j ]≤ 2
( max
w ∥Y (1)w ∥2
) ·N(1)j (x) ·N (1) j (x ⊕i) ·L f MLP · Ii[Y (1) j ] (50)
For k > 1:
Ii[y (k) j ]≤
C(k)
3 ·N(k)j (x) ·N (k) j (x
⊕i) · ( n
∑ w=1 (δw, j + 1 n ) Ii[y (k−1) w ]
) (51)
Intuitively, the δw, j contribution reflects the skip connection (residual stream); the 1n contribution reflects the role of soft attention.
Proof. We first obtain:
Is[y (k) j ]≤(48,2,4,26) N (k) k (x) ·N (k) k (x ⊕s) ·CMLP ·L f MLP · ( Is[y (k−1) j ]+ H
∑ q=1 Is[b (k) i,q ]
) (52)
For k > 1, we find, putting together the previous results:
Ii[y (k) j ]≤(48) Ii[Yj] ·N (k) j (x) ·N (k) j (x ⊕i)
≤(52)(L f MLPCMLP · (
max w
∥Y (k)w ∥2 ) · ( Ii[y (k−1) j ]+ H
∑ q=1 Ii[b (k) i,q ]
) ) ·N(k)j (x) ·N (k) j (x ⊕i)
≤(46)(L f MLPCMLP ·CMLP(H +1)CVCP · ( Ii[y (k−1) j ]+H4 √ d exp(4C(k)A ) [ Ii[y (k−1) j ]+ 1 n n ∑ w=1 Ii[y (k−1) w ] ]) ) ·N(k)j (x) ·N (k) j (x ⊕i)
=(30) C(k)
3 ·N(k)j (x) ·N (k) j (x
⊕i) · ( n
∑ w=1 (δw, j + 1 n ) Ii[y (k−1) w ]
)
In the lowest layer, the influence is bounded instead in terms of the influence on the attention weights. With high probability, a logarithmic bound on the summed influences over all bits results. Crucially, the key-value-matrix plays no role here. For k = 1 we find, setting δ = 4 in (35):
Ii[Y (1) j ]≤(2) Ii [ fMLP ( y(0)j + H
∑ h=1 b(1)j,h
)]
≤(27)L f MLP [ Ii [ y(0)j ] + H
∑ h=1
Ii [ b(1)j,h ]]
≤(33)L f MLP [ Ii [ y(0)j ] + H
∑ h=1 CV
[ n
∑ w=1
â1,hj,w Ii[y (0) w ]+CV
( max
w ∥y(0)w ∥2
) n ∑
w=1 Ii[â
1,h j,w]
]]
≤(24)L f MLP [ 2CPδi j +CV H
∑ h=1
[ n
∑ w=1
â1,hj,wCPδiw +CVCP n
∑ w=1 Ii[â 1,h j,w]
]]
≤L f MLPC2VCP [ 2δi j + H
∑ h=1
â1,hj,i + H
∑ h=1
n
∑ w=1 Ii[â 1,h j,w]
]
and hence, by summing over i,
n
∑ i=1
Ii[Y (1) j ]≤L f MLPC2VCP
[ 2+ H
∑ h=1
n
∑ i=1
â1,hj,i + H
∑ h=1
n
∑ i=1
n
∑ w=1 Ii[â 1,h j,w]
]
≤(35)L f MLPC2VCP [2+H +H136logn+10H] ≤150L f MLPC2VCPH logn
with probability ≥ 1− Hn2 . 7

heading: B.4 Relating Influence at First and Last Layers
text: At this point, we have derived bounds on the influence of any bit on any individual activation y(k)j in terms of the parameter norms, the layer-norm-induced blowup, and influence the bit has on the activations at the preceding layer. We now derive bounds linking these quantities to the overall sensitivity s( f ,x) and the average sensitivity asn( f ). In order to do this, we link influences on the top layer to influences at the first layer. Recall
BlowupK(x) = K
∏ k=1 τ(k)(x) (53)
We first find that
Lemma 18.
Ii[y (k) j ]≤C(1)
( k
∏ l=2
C(k) )
Blowupk(x)Blowupk(x ⊕i) ( 1 n n ∑ w=q Ii[Y (1) w ]+ Ii[Y (1) j ] ) (54)
Proof. By induction using (51). First, for k = 2,
Ii[y (2) j ]≤C(2) ·N (2) j (x) ·N (2) j (x
⊕i) · ( n
∑ w=1 (δw, j + 1 n ) Ii[y (1) w ]
)
≤C(2) ·N(2)j (x) ·N (2) j (x
⊕i) · ( n
∑ w=1 (δw, j + 1 n )2 ( max w ∥Y (1)w ∥2 ) ·N(1)w (x) ·N(1)w (x⊕i) ·L f MLP · I[Y (1) w ]
)
≤C(2)2 (
max w
∥Y (1)w ∥2 ) ·L f MLP ·Blowup2(x) ·Blowup2(x⊕i) ·
( n
∑ w=1 (δw, j + 1 n ) · I[Y (1)w ]
)
≤C(1)C(2) ·Blowup2(x) ·Blowup2(x⊕i) · ( n
∑ w=1 (δw, j + 1 n ) · I[Y (1)w ]
)
where
C(1) = 2 (
max w
∥Y (1)w ∥2 ) ·L f MLP
Now, for k > 2, using the induction hypothesis (IH):
Ii[y (k) j ]≤
C(k)
3 ·N(k)j (x) ·N (k) j (x
⊕i) · ( n
∑ w=1 (δw, j + 1 n ) Ii[y (k−1) w ]
)
≤(IH) 1 3
C(1) ( k
∏ l=2
C(k) )
Blowupk(x)Blowupk(x ⊕i) ·
( n
∑ w=1
n
∑ v=1 (δw, j + 1 n )(δv,w + 1 n ) Ii[Y (1) v ]
)
= 1 3
C(1) ( k
∏ l=2
C(k) )
Blowupk(x)Blowupk(x ⊕i) · ( Ii[Y (1) j ]+3 n
∑ v=1 1 n Ii[Y (1) v ]
)
≤C(1) ( k
∏ l=2
C(k) )
Blowupk(x)Blowupk(x ⊕i) · ( Ii[Y (1) j ]+ n
∑ v=1 1 n Ii[Y (1) v ]
)

heading: B.5 Deriving Almost-Everywhere Pointwise Sensitivity Bounds
text: Putting together the previous findings, we are now in a position to link the sensitivity to layer norm blowup and parameter norms. We find that the sensitivity on an input x is bounded in terms of the blowup on x itself and on its Hamming neighbors, up to sublinear factors √ n logn:
Theorem 19 (Repeated from Theorem 4). With probability at least 1− Hn−2 over the choice of x ∈ {±1} n, we have
s( f ,x) C √ n logn ≤ Blowup(x)2 + 1 n ∑i Blowup(x⊕i)2
7The last inequality holds whenever n > 1. If n = 1, the probability bound evaluates to 0, making the statement trivially true.
Proof. The most immediate idea is to sum both sides of (54) across i to get a bound on s( f ,x) = 14 ∑ n i=1 Ii[y (L) n ]2. In fact, it will pay off to average the influence across the input bits, and only later eliminate the 1/n factor. Formally,
1 n s( f ,x) =(19) 1 4n
n
∑ i=1
Ii[vT y (L) n ] 2
≤(18) 1 4n
n
∑ i=1
Ii[y (L) n ] 2∥vout∥22
≤(21) 1 2n
n
∑ i=1
Ii[y (L) n ]∥vout∥22
√ d
≤(54) √
d∥vout∥2 2
( k
∏ l=2 Ck
) Blowup(x)∑
i
1 n
Blowup(x⊕i) [
1 n ∑w Ii[Y (1) w ]+ Ii[Y (1) n ]
]
≤(CSB) √
d∥vout∥2 2
( k
∏ l=2 Ck
) Blowup(x) √ ∑
i
1 n
Blowup(x⊕i)2 [√
∑ i 1 n 1 n ∑w Ii[Y (1) w ]2 +
√ ∑
i
1 n Ii[Y (1) n ]2
]
where the last inequality uses Cauchy-Schwarz-Bunyakovsky (CSB). Now, we find for any w ∈ {1, . . . ,n}:
∑ i
Ii[Y (1) w ] 2 ≤(21) 2 (
max x
∥Y (1)w ∥2 ) ∑
i Ii[Y
(1) w ]≤(49) ( max
x ∥Y (1)w ∥2
) C0 logn (55)
with probability (simultaneously over w) 1− H nδ−2 over the choice of x. Setting δ = 4 and defining and plugging in (55) together with the fact √ x ≤ x when x ≥ 1, we get:
1 n
s( f ,x)≤C Blowup(x) √
logn n
√ ∑
i
1 n Blowup(x⊕i)2
Rearranging, we find
1 C s( f ,x)
√ 1
n logn ≤Blowup(x)
√ ∑
i
1 n Blowup(x⊕i)2
≤Blowup(x)2 +∑ i 1 n Blowup(x⊕i)2
where the last step follows by Young’s Inequality (ab ≤ a2+b22 for a,b ≥ 0). As we chose δ = 4, this is true with probability 1− Hn2 over the choice of x.

heading: B.6 Deriving On-Average Sensitivity Bounds
text: Finally, we can convert the high-probability statement from the preceding lemma, which bounded the layer norm blowup on x itself or its Hamming neighbors, into an on-average bound over the entire input space:
Corollary 20 (Repeated from Corollary 4).
C ·E[Blowup(x)2]≥ asn( f )√ n logn − H n
(56)
Proof. Recall from the preceding lemma that, with probability 1− Hn2 over the choice of x, we have
s( f ,x) C √ n logn ≤ Blowup(x)2 + 1 n ∑i Blowup(x⊕i)2 (57)
We first note the following, for any function φ : {±1}n → R:
E[φ(x)] = 1 n Ex
n
∑ i=1
φ(x⊕i)
since every one of the maps x 7→ x⊕i (i = 1, . . . ,n) is a bijection on {±1}. Hence,
E[φ(x)] = Ex [ 1 2 φ(x)+ 1 2 n ∑ i=1 φ(x⊕i) ]
Taking φ(x) := Blowup(x)2 yields:
E[Blowup(x)2] = Ex [ 1 2 Blowup(x)2 + 1 2 1 n ∑i Blowup(x⊕i)2 ]
(58)
Let A be the set of x satisfying (57); |A| ≥ 2n(1− Hn2 ). Then
E[Blowup(x)2] = 1 2n ∑x∈{±1}n [ 1 2 Blowup(x)2 + 1 2 1 n ∑i Blowup(x⊕i)2 ]
≥ 1 2n ∑x∈A [ 1 2 Blowup(x)2 + 1 2 1 n ∑i Blowup(x⊕i)2 ]
≥ 1 C √ n logn 1 2n ∑x∈A s( f ,x) ≥ 1 C √ n logn 1 2n ∑x∈{±1}n s( f ,x)− 1 C √ n logn 1 2n ∑x∈∈A s( f ,x)
≥ 1 C √ n logn asn( f )− 1 C √ logn H n3/2
Rearranging, we find:
C ·E[Blowup(x)2]≥ 1√ n logn asn( f )− 1√ logn H n3/2 ≥ 1√ n logn asn( f )− H n

heading: C Theorem 6
text: Before proving Theorem 6, we require the following lemma in the analysis of Boolean functions. Informally, the lemma describes how the RMSE between two functions can be lower-bounded in terms of their average sensitivities.
For any function f : {±1}→ R, we write ∥ f∥2 := √ 1 2n ∑x∈{±1}n f (x)2 = Ex[ f (x) 2]. Then the lemma states:
Lemma 21. For any f , f ′ : {±1}n → R, we have for any α > 1: √
E[( f (x)− f ′(x))2]≥ √√√√ (1− 1 α )asn( f )
(n− asn( f )α·∥ f∥22 ) −∥ f∥2
√ α
asn( f ′) asn( f )
(59)
Note that the optimal α depends on f , f ′. The choice α = 2 will be sufficient for our needs: √ E[( f (x)− f ′(x))2]≥
√√√√ asn( f ) 2n− asn( f )∥ f∥22 −∥ f∥2 √ 2 asn( f ′) asn( f )
(60)
Proof. Recall the discussion of the Fourier-Walsh decomposition preceding (17). The proof idea is that, when functions have different average sensitivities, their degree profiles must be quite different, which then entails that they must be substantially distinct in behavior.
Let 1 ≤ λMin ≤ n; we will fix it later. Let Π be the orthogonal projection on the Fourier components with degree at least λMin, formally given as the unique linear map satisfying
ΠχP = { χP if |P| ≥ λMin 0 else
(61)
Then, for any g : {±1}n →R, by (17) and Parseval’s theorem for the Fourier-Walsh transform (O’Donnell, 2014):8
λMin∥Πg∥22 ≤ asn(g)≤ λMin(∥g∥22 −∥Πg∥22)+n∥Πg∥22 (62) 8If d0 . . . ,dn is the degree profile of g, then λMin∥Πg∥22 = ∑i≥λMin λMindi ≤ asn(g) ≤ ∑0<i<λMin λMindi + ∑i≥λMin ndi = λMin(∥g∥22 −∥Πg∥22) + n∥Πg∥22, where the inequalities follow from (17) and the equalities follow from Parseval’s theorem applied to Πg and to g.
Rearranging and inserting f , f ′ for g in the two inequalities, respectively:
asn( f )−λMin∥ f∥22 n−λMin ≤∥Π f∥22
∥Π f ′∥22 ≤ asn( f ) λMin
Hence
∥Π f∥2 −∥Π f ′∥2 ≥ √
asn( f )−λMin∥ f∥22 (n−λMin)
− √
asn( f ′) λMin
Now at λMin = asn( f )α∥ f∥22 , we get
∥Π f∥2 −∥Π f ′∥2 ≥ √√√√ (1− 1 α )asn( f )
(n− asn( f )α∥ f∥22 ) −∥ f∥2
√ α
asn( f ′) asn( f )
Now by the reverse triangle inequality
∥Π f∥2 −∥Π f ′∥2 ≤ ∥|Π f −Π f ′∥2 ≤ ∥ f − f ′∥2 (63)
and the claim follows.
Proof of the Theorem. The proof can be decomposed into the following steps.
Step 1: Decomposing ∆ We have (F ·d+G ·d2) ·L parameters (where F,G are constant). We are assuming that ∆ is from ρS. For each of the L instances layer norm, we consider the part of ∆ corresponding to the immediately preceding bias; we will name this ∆1, . . . ,∆L.
Step 2: Effect of Perturbations on Cθ We will make explicit the dependence of C on θ by writing Cθ. We next note that, for any fixed θ, as ρ → 0, the effect of a small perturbation ∆ on Cθ is bounded, because Cθ is locally Lipschitz in θ.9 Thus, as ρ → 0:
Cθ+∆ =Cθ · (1+O(ρ)) (64)
uniformly across all ∆ with ∥∆∥= ρ, where O(·) includes constants depending on θ, but not ρ, ∆, or n.
Step 2: Lower Bounding Error in Terms of Sensitivity Drop Set
µ := liminf n→∞ asn(Tθ) n
(65)
Necessarily, µ ∈ [0,1]. The claim of the theorem is nontrivial if and only if µ > 0. Lemma (21) shows that the difference between fθ and fθ+∆ can be lower-bounded in terms of their sensitivities. First, we note, as n → ∞, with α = 2,
asn( f )
2n− asn( f )∥ f∥22 = µ 2− µ∥ f∥22 +o(1)≥ µ 2 +o(1) (66)
where we used µ∥ f∥22 ∈ [0,1] up to o(1). Indeed, if the output of ±1, we have ∥ f∥22 = 1, and the term indeed is lower-bounded by µ: asn( f )
2n− asn( f )∥ f∥22 =
µ 2−µ +o(1)≥ µ+o(1) (67)
9This follows from the fact that in (8), for CP, ℓ1, ℓ∞ norms are locally Lipschitz; for CMLP because it can be chosen to be the product of parameter matrix norms in the MLP; for the exponential term because the spectral norm is locally Lipschitz.
showing that the 2 factor can be eliminated in the case of Boolean output. Next, as n → ∞, for α = 2 as above, and using 0 ≤ asn(Tθ)n ≤ ∥Tθ∥22 ≤ 1, we find
E∆E[(Tθ(x)−Tθ+∆(x))2]
≥E∆ (1− 1α )asn(Tθ) (n− asn(Tθ)α∥Tθ(x)∥22 )︸ ︷︷ ︸ independent of ∆ +∥Tθ∥22αE∆ asn(Tθ+∆) asn(Tθ)︸ ︷︷ ︸ ≥0
−2 √√√√ (1− 1 α )asn(Tθ)
(n− asn(Tθ)α∥Tθ(x)∥22 ) ∥Tθ∥2E∆
√ α
asn(Tθ+∆) asn(Tθ)
≥ asn(Tθ) 2n− asn(Tθ)∥Tθ(x)∥22
−2 √√√√ 1
(n− 1 2∥Tθ(x)∥22
) ∥Tθ∥2E∆
√ asn(Tθ+∆)
≥ asn(Tθ) 2n− asn(Tθ)∥Tθ(x)∥22 −2E∆
√√√√ asn(Tθ+∆) (n− 1
2∥Tθ(x)∥22 )
≥(†) asn(Tθ) 2n− asn(Tθ)∥Tθ(x)∥22
−4E∆ √
asn(Tθ+∆) n
= asn(Tθ)
2n− asn(Tθ)∥Tθ(x)∥22 −4E∆
√ Exsn(Tθ+∆,x)
n
≥µ 2 −4 √ E∆Exs(Tθ+∆,x) n +o(1)
(68)
where (†) uses (n− 1 2∥Tθ(x)∥22 )≥ n−1/2 ≥ n/4, and the final step used Jensen’s inequality applied to the concave function
√·. By the reasoning above, in Eq. (67), the factor 2 in the first term can be eliminated when the output is ±1. In order to lower-bound (68), we want to show the following:
Take any ζ > 2Ld . For each x, with probability at least (1−O(n2−(d−1)ζ/L)−Lexp(−Θ(d))) (where O contains constants depending on ρ,d,L, but not depending on x and n), the blowup is simultaneously bounded on x and its radius-1 Hamming ball:
τ(k)θ+∆(X)≤ 1+nζ/L, ∀X ∈ {x,x⊕1,x⊕2, . . . ,x⊕n} (69)
We will prove this below in Step 4. We may then bound τ(k)θ+∆(X) ≤ 2nζ/L for large n. By (12), if (69) is proven, then:
s(Tθ+∆,x) Cθ+∆ √ n logn ≤ Blowup(x)2 + 1 n
n
∑ i=1
Blowup(x⊕i)2 ≤ 4n2ζ
and hence s(Tθ+∆,x)≤ 4n2ζ+.5Cθ+∆ √ logn (70)
and hence s(Tθ+∆,x)
n ≤ 4n2ζ−.5Cθ+∆
√ logn (71)
with probability 1− Hn2 over x and 1−O(Ln 2−(d−1)ζ/L)−Lexp(−Θ(d)) over ∆, where we used a union bound over the L layers. Let us now fix ζ = 3Ld . Then, as n → ∞,
E∆Ex s(Tθ+∆,x)
n ≤4n6L/d−.5Cθ+∆
√ logn+O(n2−3 d−1 d )+
H n2 +Lexp(−Ω(d))+o(1)
Under the hypothesis 12L < d, all terms except for Lexp(−Ω(d)) are o(1) as n → ∞. Inserting this into (68), the claim follows.
Step 4: Effect on Layer Norm It remains to show (69). We proceed inductively from the input layer upwards, examining the effect of perturbing parameters. In the inductive step, we consider a transformer where all the parameters below the bias immediately preceding the i-th layer norm have already been perturbed.
Fix an input string x ∈ {±1}n. Write Y (k)i for the activation computed using the perturbed parameters θ+∆ for all parameters except the bias of interest, for which θ is used here. Thus, after applying the perturbation to that
bias, the activation is Y (k)i +∆i. The aim is to lower-bound the SD of Y (k) i +∆i with high probability over the choice of ∆. Let center(x) := x−mean(x), i.e. the pre-scaling component of layer norm. First,
P ( ∥center(∆i)∥ ≤
ρ 2 √ d D ) ≤ exp(−Ω(d))
by standard concentration arguments, where Ω(d) scales positively with d.10 Let ΠS : Rd → Sd−1 ⊂ Rd be the projection on the sphere around 0 of radius ρ2 √ d D :
ΠS(x) = ρ 2 √ d D x ∥x∥ (72)
Let ΠB : Rd → B ρ 2 √ d D (0) be the projection on the ball around 0 of radius ρ2
√ d D . ΠB is a contraction (referred to
below with (†)11). Conditional on ∥center(∆i)∥ ≥ ρ2 √ d D , we have ΠBcenter(∆i) = ΠScenter(∆i) and its distribution is, by symmetry, uniform on the ball’s surface. We now bound the probability that the SD of the perturbed activation is large:
P ( ∃ j : SD(∆i +Y (k)i )< n−ζ/L )
=P ( ∃ j : ∥center(∆i +Y (k)i )∥< √ dn−ζ/L ) =P ( ∃ j : center(∆i) ∈ Bn−ζ/L ( −center(Y (k)i ) ))
≤ n
∑ j=1
P ( center(∆i) ∈ B√dn−ζ/L ( −center(Y (k)i ) ))
≤P(∥center(∆i)∥ ≤ ρ 2 √ d D )+ n ∑ j=1 P ( center(∆i) ∈ B√dn−ζ/L ( −center(Y (k)i ) )∣∣∣∣∥center(∆i)∥> ρ 2 √ d D )
≤†P(∥center(∆i)∥ ≤ ρ 2 √ d D )+ n ∑ j=1 P ( ΠBcenter(∆i) ∈ B√dn−ζ/L ( −ΠBcenter(Y (k)i ) )∣∣∣∣∥center(∆i)∥> ρ 2 √ d D )
≤exp(−Ω(d))+ n
∑ j=1 P
( ΠBcenter(∆i) ∈ B√dn−ζ/L ( −ΠBcenter(Y (k)i ) )∣∣∣∣∥center(∆i)∥> ρ 2 √ d D )
≤exp(−Ω(d))+ n
∑ j=1 P
( ΠScenter(∆i) ∈ B√dn−ζ/L ( −ΠScenter(Y (k)i ) )∣∣∣∣∥center(∆i)∥> ρ 2 √ d D )
=exp(−Ω(d))+ n
∑ j=1
Area ( B√dn−ζ/L ( −ΠScenter(Y (k)i ) ))
Area (
ρ 2 √ d DS )
where Area(B√dn−ζ/L ( −ΠScenter(Y (k)i ) ) ) is the area of a hypersphere cap of radius √ dn−ζ/L on a hypersphere of
radius ρ2 √ d D , which is at most n
−(d−1)ζ/L times some constant depending on d,D,ρ but not n,x12. The denominator is constant in n. Overall, as n → ∞,
P ( ∃ j : ∥center(∆i +Y (k)i )∥< n−ζ/L ) ≤ exp(−Ω(d))+O(n1−(d−1)ζ/L)
10One way of obtaining this is by parameterizing ∆ in terms of first sampling D independent X1, . . . ,XD ∼ N (0, 1D ), and dividing the resulting vector by its norm, before multiplying by ρ in the end. Before normalization, the squared norms of the entire vector and the norm of the part corresponding to ∆i concentrate with exponential tail bounds in d, around 1 and dD ,
respectively. Furthermore, the mean of ∆i concentrates around 0. Thus, the norm of center(∆i) concentrates around ρ √ d D .
11The contractivity of ΠB is proven, in a different context, as Lemma A.9 in Edelman et al. (2022). 12A nonasymptotic formula for the area of the hypersphere cap is given by Li (2010). A simple derivation of the asymptotic relationship used here is by approximating, when n is large and thus n−ζ/L is small in relation to d, the cap as a disk on the d−1-dimensional tangent space to the hypersphere. This approximation is valid in this limit since the radius of the hypersphere, and thus its curvature, are independent of n.
If the event denoted by this probability is avoided, then under the parameter setting θ+∆,
τk(x)≤ 1+nζ/L (73)
for both x and its Hamming ball of radius 1, with probability (by a union bound over the radius-1 Hamming ball B1(x) around x):
P ( ∀x′ ∈ B1(x) : ∀ j : ∥center(∆i +Y (k)i (x′))∥ ≥ n−ζ/L )
≥1−P ( ∥center(∆i)∥ ≤
ρ 2 √ d D ) − ∑
x′∈B1(x) P
( center(∆i) ∈ Bn−ζ/L ( −center(Y (k)i (x′)) )∣∣∣∣∥center(∆i)∥> ρ 2 √ d D )
≥(1−O(n2−(d−1)ζ/L)− exp(−Θ(d)))
This concludes the proof.
Remark Note that we assumed for notational simplicity that layer norm only applies a single time, following the MLP (4). Depending on the implementation, LN may appear in other cases, e.g. directly following attention (e.g., in Vaswani et al. (2017)). In this case, we can analogously consider the the effect of V +∆V , using the properties of the random matrix ∆V and the effects of the perturbation on the preceding MLP (or the token embeddings). First, we note that ηi := ∑h ∑ j â (k,h) i j y (k−1) j is, after adding ∆ to the parameters involved, very unlikely to have norm o(n1/(ζL)), because of the random perturbation applied to the MLP bias in the preceding layer (or to the token embeddings, if k = 1), provided d ≫ H. Second, considering the spectral properties of the random matrix ∆V , the perturbed head activation y(k−1)i +(V +∆i)ηi is unlikely to have much smaller SD, hence a bound analogous to (73) follows with high probability.

heading: D Theorem 7
text: Proof. Each autoregressive step consists in determining, based on a sequence x1 . . .xNt1 . . . tt the state tt+1 = T (tt ,xt). Assuming log |Σ| and log |X | bits are used to encode xi and ti, respectively, the function thus only depends on log |Σ||X | inputs, and is represented by a polynomial of that degree, independent of N.

heading: E Further Experimental Results

heading: E.1 The Details of Experimental Setup
text: In the experiment presenting the tradeoff between weight norm and LayerNorm blowup, we uniformly sampled weight decay between 0 and 0.4 and learning rate between 0.0001 and 0.0005.
In the scratchpad experiment, we used an encoder-decoder Transformer with 2 attention heads per block, hidden dimension 32 and 2 layers.
In the experiment including training the models for high sequence lengths (Figures 7 and 8) we used hidden dimension 4.
E.2 Results

heading: Future Work
text: Providing more quantitative bounds that tightly characterize finite-length behavior is an interesting problem for future research. A second limitation is that we only target functions outputting a single value. Operationalizing sensitivitylike metrics for sequence-to-sequence functions may be required in order to expand the theory to such functions. Third, our results apply to transformer encoders. It remains open if transformer decoders, with causal attention masking, face a different set of limitations than we have shown here in the absence of masking. Fourth, our theoretical results concern the loss landscape, not the training dynamics itself. Further technical advances may be needed to directly prove corresponding results for training dynamics