Transformers Learn Low Sensitivity Functions: Investigations and Implications
Authors: Bhavya Vasudeva
Deqing Fu
Tianyi Zhou
Elliott Kau
Youqi Huang
Vatsal Sharan
Venue: ICLR 2025 Poster (2025)
Keywords: transformers
sensitivity
grokking
Primary area: interpretability and explainable AI
TL;DR: Transformers have lower sensitivity than alternative architectures, such as LSTMs, MLPs, ConvMixers, and CNNs. Low-sensitivity bias correlates with improved robustness and can serve as a progress measure for grokking.

Abstract: Transformers achieve state-of-the-art accuracy and robustness across many tasks, but an understanding of their inductive biases and how those biases differ from other neural network architectures remains elusive. In this work, we identify the sensitivity of the model to token-wise random perturbations in the input as a unified metric which explains the inductive bias of transformers across different data modalities and distinguishes them from other architectures. We show that transformers have lower sensitivity than MLPs,  CNNs, ConvMixers and LSTMs, across both vision and language tasks. We also show that this low-sensitivity bias has important implications: i) lower sensitivity correlates with improved robustness; it can also be used as an efficient intervention to further improve the robustness of transformers; ii) it corresponds to flatter minima in the loss landscape; and iii) it can serve as a progress measure for grokking. We support these findings with theoretical results showing (weak) spectral bias of transformers in the NTK regime, and improved robustness due to the lower sensitivity.

Published as a conference paper at ICLR 2025
TRANSFORMERS LEARN Low SENSITIVITY FUNCTIONS: INVESTIGATIONS AND IMPLICATIONS
Bhavya Vasudeva', Deging Fu', Tianyi Zhou, Elliott Kau", Youqi Huang', Vatsal Sharan University of Southern California
bvasudev@usc. edu, deqing edu
ABSTRACT
Transformers achieve sfate-of-the-art accuracy and robustness across many tasks, but an understanding of their inductive biases and how those biases differ from other neural network architectures remains elusive. In this work, we identify the sensitivity of the model to token-wise random perturbations in the input as a unified metric which explains the inductive bias of transformers across different data modalities and distinguishes them from other architectures. We show that transformers have lower sensitivity' than MLPs, CNNs, Mixers and LSTMs, across both vision and language tasks. We also show that this low-sensitivity bias has important implications: i) lower sensitivity correlates with improved robustness; it can also be used of transformers; ii) it corresponds to flatter minima in the loss landscape; and iii) it can serve progress measure for grokking. We support these findings with theoretical results showing (weak) spectral bias of transformers in the NTK regime, and improved robustness due the lower sensitivity.
efficient intervention to further improve the robustness
INTRODUCTION
Transformers, originally introduced for language problems (Vaswani et al., 2017), have become universal backbone across machine learning - including applications such as vision (Dosovitskiy et al. 2021) and protein structure prediction (Jumper et al., 2021). Several recent works have also found that not only do transformers achieve better accuracy, but they are also more robust to various corruptions and changes in the data distribution (Shao et al., 2021; Mahmood al., 2021; Bhojanapalli et al., 2021; Paul & Chen, 2022). Despite their practical success, relatively little is understood about what distinguishes transformers from other neural network architectures. If a transformer and an alternative neural network architecture (such as CNN or an LSTM) are trained obtain similar training accuracy on a dataset, then how do the models differ in terms of the functions they learn? Equivalently, what inductive biases do transformers have which distinguish them from other architectures?
Recently, for the setting of Boolean inputs, Bhattamishra et al. (2023b) and Hahn & Rofin (2024) suggest using the notion of sensitivity distinguish transformers from other candidate architectures. The sensitivity of function measures how likely the output is to change for random changes to the input. Bhattamishra et al. (2023b) and Hahn & Rofin (2024) show that transformers are biased to learn functions with low sensitivity on Boolean inputs. Sensitivity has several desirable properties as a notion of inductive bias. It is closely related to the Fourier representation of the function and various other notions of Boolean function complexity such the degree of the function and the size of the smallest decision tree which represents the function (O' Donnell, 2014). Low sensitivity functions correspond to low complexity functions based on all these notions of Boolean function complexity, and hence an inductive bias towards low-sensitivity functions is regarded as an instance of 'simplicity bias' of the model (Valle-Perez et al., 2019; Bhattamishra et al., 2023b). Sensitivity also has deep connections to well-studied notions of inductive biases such as spectral bias (Rahaman et al., 20))b), which bias towards 'simple' functions such as low frequency functions in the Fourier space. Sensitivity has also been found to correlate with better generalization for connecco networks (Novak et al., 2018). (See Appendix C for a detailed discussion of the related work.)
Coo-frrtt authors " Coo-third authors. 'The code is availahle https: http//ggttttttttt/ wtw.ggit..........iieeeeeeemeeee seniiiiis/siiiiiieeiiii tivity...

---

Published as a conference paper at ICLR 2025
Our results. Sensitivity is a promising notion of inductive bias, but has mainly been investigated for Boolean functions so far. Given the numerous modalities of data across which transformers are successful in practice, the goal of our work is to examine if appropriate extensions of the notion of sensitivity for Boolean functions help understand the inductive bias of transformers across varied data modalities and if these notions help explain properties of transformers such their improved robustness. We now provide an overview of the main claims and results of the paper. We begin our investigation with the following question:
What are appropriate notions of sensitivity beyond Boolean data?
To provide a concrete starting point where we can understand the properties of sensitivity with theo- retical analysis, we first consider the Boolean setup and place the low-sensitivity bias of transformers on firmer theoretical foundation in that setting (Section 2). Using prior work on spectral bias in neural networks (Yang & Salman, 2020) we prove that transformers show a low-sensitivity bias on Boolean functions, and also prove that low sensitivity leads to better robustness. We then consider the above question, and propose suitable notion of sensitivity which takes into account the underlying metric space (Section 3). To investigate if sensitivity is a suitable notion beyond the Boolean case, we first consider a synthetie dataset where we can tease apart sensitivity from related notions which can coincide for Boolean functions - such as a preference towards functions that depend on a sparse set of tokens. We show that transformers prefer to learn lew-sensitivity functions (even if they not sparse). Subsequently, we examine if this sensiiiiyy bias is widely present across different tasks:
Does low-sensitivity serve as a unified notion of simplicity across vision and language tasks, and does it distinguish between transformers and other architecturs?
Here, we first conduct experiments on vision datasets. We empirically compare (Vision-)Transfoon Transformers with MLPs, CNNs, and ConvMixers, and observe that transformers have lower sensitivity compared to other candidate architectures (see Section 4). Similarly, we conduct experiments on language tasks and observe that transformers learn predictors with lower sensitivity than LSTM models. Furthermore, transformers tend to have uniform sensitivity to all tokens while LSTMs are more sensitive to more recent tokens (see Section 5). Given this sensitiitiv bias, we next examine its implications:
What are the implications of a bias towards low-sensitivity functions; is it helpful in certain settings?
We study this in three contexts: robustness, properties of the loss landscape, and training dynamics.
1. Lower Sensitivity Correlates with Better Robustness: We show that transformers have lower sensitivity and are more robust corruptions when tested on the CIFAR-IO-C dataset, compared to CNNs (Section 6). We also demonstrate that sensitivity is not only predictive of robustness but also has prescriptive power: We add a regularization term training time to encourage the model to have lower sensitivity. Since sensitivity is efficient to measure empirically, this is easy to accomplish via data augmentation. We find that models explicitly trained to have lower sensitivity yield even better robustness on LIFAR-IO-C. These results show that low sensitivity correlates with the improved robustness of transformers.
2. Lower Sensitivity Correlates with Flatter Minima: We explore the connection between sensi- tivity and a property of the loss landscape that has been found to correlate to good generalization the sharpness of the minima. We compare the sharpness of the minima with and without sensitiv- ity regularization, and our results show that lower sensitivity correlates with flatter minima. This indicates that sensitivity could serve a unified notion for both robustness and generalization. 3. Sensitivity Serves as a Progress Measure for Grokking: We examine if sensitivity can be used to understand the training dynamics of transformers, specifically from the perspective of the phenomenon of grokking where test accuracy abruptly improves long after the training loss or accuracy saturates. We consider modular addition, task where transformers exhibit grokking. We show that sensitivity provides progress measure that decreases even when the training loss does not reduce and is indicative of stages of grokking.
2 SENSITIVITY AND WEAK SPECTRAL BIAS
In this section, we theoretically show that transformers with linear attention exhibit (weak) spectral bias to learn lower-order Fourier coefficients, which in turn implies a bias to learn sensiivity functions. We start with overview of Fourier analysis on the Boolean eube and sensitivity.
2

---

Published as a conference paper at ICLR 2025
Fourier analysis the Boolean cube (O'Donnell, 2014). The space of vealev functions the Boolean cube @'' forms a 2 d'm-dinenioonal space. Any such function can be written as a unique multilinear polynomial. Specifically, the multilinear monomial functions, Xu (x) := (=):===>>
: T4s for EED
each U - [d]. form Fourier basis of the function space (ʃ : OD' - JR,, i.e., their inner products Consequently, any function f -or + R can be written
satisfy E gn- (x)
as f(x) Σ ((U))o (2r), for unique set of coefficients f(U); U S [d]. where [d] = (1, - d).
Sensitivity in Boolean function analysis. Sensitivity is a common complexity measure for Boolean functions. Intuitively, it captures the changes in the output of the function, averaged over the neighbours of particular input. Formally, let of
{+1] 1d denote the Boolean cube in dimension d. The sensitivity of Boolean function f mard {=1} at input æ E codl is given by a(f. = Ei ([) * f(*""),, where [[-] denotes the indicator function and の = (II,...,
-2; 24, I++1....,, Ia) denotes the sequence obtained after flipping the 7h co- ordinate of æ. Note that in the Boolean case, the neighbor of input can be obtained by flipping a bit, we will define a more general notion later which holds for more complex data. The average sensitivity is measured by averaging S(f, x) across all inputs,
S())
= E (((. w~
1 = 2
Σ S(S EOD
Following Bhattamishra et al. (2023b), when comparing inputs of different lengths, we consider the average sensitivity normalized by the input length, S(ʃ) {J((). The sensitivity of function I is known to be related to the degree D(J) of the multilinear polynomial which represents f (Huang, 2019; Hatami al., 201 1), and low-degree functions have lower sensitivity. Specifically, breakthrough result (Huang, 2019) showed that D(f) S Stmmx()) where Smax (f) == max S x).
a EDD
Attention Layer. The output of single-head self-attention layer, parameterized by key, query, value
matrices Wo. WK E ddxd, Wv E R dxdd for input X TTxx with T tokens of d dimension, is ATTN(X; X; Wo, WK, Wv) == ( X W W X WVW,, where X(XWo) W X E RTXT is the attention map with the softmax map (p()) RT R² applied row-wise.
Main Results. Consider any model with at least one self-attention layer, where X is obtained by reshaping x E G3, d Td. Instead of applying the softmax activation, we consider linear attention and apply an identity activation element-wis with a scaling factor of 1/2 The following result shows that the conjugate kernel (CK) or neural tangent kernel (NTK) (see Appendix B for an overview) induced by transformers with linear attention exhibit a weak form of spectral bias, where the eigenvalues do not decrease the degree of the multi-linear monomials increases, separately for even and odd degrees; see Appendix B for the proof.
Proposition 2.1. Let K be the CK NTK of a transformer with linear attention on Boolean cube E For any x, y WD, we can write K(x, y) = V V((xx y)) for some univariate function : - TR.. Further, for every U C [d], XU is an eigenfunction of with eigenvalue
d x= x. ).
Muul
[2" [2]
E w-at
where 1 = (1,..., 1) E 0d and the eigenvalues Pa. k E [d], satisfy
p0 p2
p2k .... M1 1 p3
p2k+1
Note that for given U, the eigenvalue lall only depends on " and Σ, xi by definition, and hence, it is invariant under any permutation of [d]. Larger eigenvalues for lower-order monomials indicate that simpler features are learned faster. Since low sensitivity implies learning low-degree polynomials, Proposition 2.1 also implies a weak form of low sensitivity bias.
We now show connection between lower sensitivity and better robustness. Given sample dp and some p E (0, 1). consider a noisy sample x'. where z¹; = x; with probability p and uniformly random, otherwise. It can be verified that the pair (ac, x²) has correlation p. The following result
3

---

Published as a conference paper at ICLR 2025
shows that if has lower sensitivity (f) then there is lower probability +f of inconsistent predictions on the pair (z, x' x')p; see Appendix B for the proof.
Proposition 2.2. Given p-correlated pair (at, x'), where p (0, 1), and function f: {+1} with [f(x) + f(x') 0.5(1-
maximum sensitivity Smax (f), 0 5 Pr (x,,²')
Together, Propositions 2.1 and 2.2 imply that transformers have low sensitivity and hence, better robustness. In Section 6, we present experimental evidence showing that the low sensitivity of transformers correlates with their improved robustness.
3 MEASURING SENSITIVITY BEYOND BOOLEAN DATA
While sensitivity appears to be a promising metric to understand the inductive biases of transformers. it is only defined for Boolean data. In order investigate the inductive biases in real-world image and language tasks, we need an equivalent metric for digh-dimensional, real-valued data.
We define the sensitivity metric for nigh-dimensional data, which is an analog of Eq. (1) as follows. Definition 3.1. Given model Φ, dataset D and distribution P, sensitivity is computed as:
3(
E R~P
[SIGN(Ø(0) X)) + SIGN(O(0; x Σ
where X
ST is obtained by replacing the token in with æ.
An important consideration here is to define P. While one can replace token with a randomly selected to- ken measure sensitivity, this may not ensure that the new token lies in a neighbourhood of the original token. Capturing how the output changes with local perturbations according to the metric of the underlying space is an important aspect of the sensitivity definition for Boolean functions (as discussed in Gopalan et al.
Prediction Nari-Footed Alhemoo
Netwook
Add
Is Model Semetroe tee
(2016), for e.g.), and appropriate extensions of sensi- tivity beyond Boolean functions should capture this property. For input spaces such as natural images or text embeddings, there is more structure in the tokens, and a randomly selected token can lie far from the original token's neighborhood. Therefore, instead of replacing token with a random token, we inject small perturbations into the token evaluate sensitivity. This allows us to control the size of the neighbourhood by selecting the strength of the noise perturbation.
Preduction: Laysan Allsttos
Nistwor
Neura
Figure 1: Mcasuring Sensitivity in Vi- sion Tasks. A patch is first selected to add Gaussian noise corruptions. Then the orig- inal image and the corrupted image are fed into the dictions. If the predictions are inconsistent, then the neural network is sensitive to this patch. The process is repeated for every patch measure the overall sensitivity. sensitivity by replacing e, with æ as per Definition 3.1, with P as (0, o(00,²21) I). For image data, each token e, corresponds different patches (see Section 4 for further details), while for language data, the tokens correspond to embeddings of sub-words (see Section 5 for more details). Figure 1 illustrates the measurement for images.
neural network to make pre-
Formally, for each token of an input X, let æ
+ be a perturbed token, where & ~~ (O o²I) an isotropic Gaussian with variance 22, We measure
The important characteristics of this metric are that it is unified notion of complexity across vision and language tasks, and as we will see later, distinguishes transformers from various other architectures. For instance, compare the sensitivity measured with token- wise perturbations as mentioned above, with random perturbations across the input in Appendix A.3 and find that the gap in the latter metric is not large the proposed metric.
We also note that while for Boolean data, sensitivity aligns with other notions of complexity, such as sparsity, this may may not be the case in settings with high-dimensional real-valued data. In the following section, we present experimental results for a self-attention model, in a synthetic data setting to demonstrate this. Specifically, we show that in the synthetic dataset, the related notion of
4

---

Published as a conference paper at ICLR 2025
using sparse set of imput tokens may or may not align with low-sensitivity, but the model learns the low-sensitivity function in both cases.
3.1 EXPERIMENTS ON SYNTHETIC DATA
We construct a synthetic dataset to examine the inductive bias of a single-layer self-attention model. We show that in the presence of two solutions with the same predictive power but different sensitivity values, this model learns the low-sensitivity function. We begin by describing the experimental setup and then discuss our results.
Setup. We compose a single-head self-attention layer with linear head U E IR dxd final prediction, and write the full model
to obtain the
Φ ; x) = (U. (0 (UWWWW x XWv),
where 0 == concat Wo, WK. Wr U). We consider this model for the experiments in this section, with all the parameters initialized randomly at small scale. Next, we describe the process to generate the dataset. We first define the vocabulary as follows:
Vocabulary [M 17,y Tf
Examples of genorated samples:
2:
5parse
frequent
irrelevant
000 Test Accurscy Train Accuracy
:00
100
Sentiment antlysts based Examples:
Awful, seful am sourned.
repettive inaeddblyy beautful breathaaiing dinematoggaaht..
dl
Postively
Figure 2: Visualization of the synthetic data gen- eration process (see Section 3.1 for details). For simplicity, we represent each d-dimensional token with a square. Middle row: In each case, given label y, we randomly sample T 11 tokens, with n, tokens from Vy
J00
qparse. [n, + naa/2] tokens from
187
30'
38'
3.8
Vi
frequent ( + na)/2] tokens from Frequee and the remaining tokens from Virelevant- Note that in the first example, since n, 3 and 1, 1,a a predictor that relies (only) on the sparse tokens less sensitive com- pared to the one that relies the frequent tokens. On the other hand, in the second example, since n, = and n. 3, the predictor that relies the frequent tokens is less sensitive. Bottom row: We include two sentiment unalysis-based examples illustrate the synthetic data samples in the second row, using the
Aligmeent
0.5
0.5
Actention neights
109
16a
107
3.7
122
10'
1.00
155
0.75
59
0.50
0.25
5006 12000
15580 28000
2.00
5200
12009
13500 20000
Iterations (t)
Iterations (t)
Figure 3: Train and test dynamics for a single-layer self-attention model (Eq. (3)) using the synthetic data visualized in Fig. 2; see Section 3.1 for details. Left column: the predictor that uses sparse tokens has lower sensitivity (Ex. I in Figure 2). Right col- umn: the predictor that uses frequent tokens has lower sensitivity (Ex. 2 in Figure 2); see Appendix A. for more examples.
same colors as the first two rows.
Table 1:
Comparison of sensitivity values for models that use only sparse or frequent tokens for the settings considered in Figure 3.
[Ma (Ma. ml] 1, Hgg left col. 2, 3 fight.ol..
Using sparse tokens Using frequent tokens 0.2878
0.0339
Definition 3.2 (Synthetic Vocabulary). Consider a vocabulary of A distinct tokens V:= (e,........ ... ea }. where e, E {0, 1 denotes the 7t basis vector. We define smaller subsets of sparse tokens and larger subsets of frequent tokens for each label y = 1 as well as a subset of irrelevant tokens:
v+
sparse (e1), ; V Vsparse
(e2), Virrelevat
(e2m+3,... .., CMM)
4 Pirequent == { (e3, Es, e2m+ 1. V Vreequent = (e+, e....., 22m+2)
Let T denote the sequence length of each data point, ny and n. denote the number of frequent and sparse tokens, respectively, such that n, < n < min,m,, T -n,), and n, be a parameter satisfying na S ny. Next, describe the process of generating the dataset D; see Fig. 2 for example.
5

---

Published as a conference paper at ICLR 2025
Definition 3.3 (Dataset Generation). Consider the vocabulary in Definition 3.2. To generate data
point (X, y). we first sample the label y E }+1} uniformly at random. We divide the indices [T] into three sets Iirequent Isparse and Iivelevant and sample each set follows:
Inrqquent is composed of L(nz + na)/2] tokens uniformly sampled from V
and ny
Irequent
[nn nd)/2] tokens uniformly sampled from ViTy
froquent"
Ispars contains n, tokens uniformly sampled from Viparse
The remaining T 1--1, n T1, tokens in Zimelevant are uniformly sampled from Vinelevant-
To determine if the tokens in V. Vspanse or those in Viequent have a more significant impact on the model's predictions, we adapt the test set generation process by altering the second step in Definition 3.3: we sample the sparse tokens from V sparse instead of V sparse- If this modification leads to noticeable drop in the test accuracy, it suggests that the model relies on the sparse feature(s) for its predictions.
We consider two other metries to examine the role of the attention head and the linear predictor. Define three vectors: Uap == E1 - E2. Vfroq := -e,-> Ei,
Virrel := Σ e, We plot the average
iE Iegure SE Vingaan:
ie Vinskoaan
alignment (cosine similarity) between the rows of UWr and these vectors to see what tokens the prediction head relies on. Similarly, we plot the sum of the softmax scores for the three types of tokens to see which tokens are selected by the attention mechanism.
We set P in Def. 3.1 as the uniform distribution over V for computing sensitivity in our experiments.
Results. Figure 3 shows the train and test dynamics of the model in Eq. (3) using synthetic datasets generated by following the process in Definition 3.3 (details in Table 1). We consider two cases: in the first case (left column), using the sparse token leads to a function with lower sensitivity, whereas in the second case (right column), using the frequent tokens leads to lower sensitivity (see Table I for a comparison of the sensitivity values). We observe that in the first case, the OOD test accuracy drops to O, the aligmment with Usp is close to 1 and the attention weights on the sparse tokens are the highest. These results show that the model relies on the sparse token in this case. On the other hand, in the second case, the test accuracy remains high, the alignment with Vieeg is close to and the attention weights on the frequent tokens are the highest, which shows that the model relies on the frequent tokens. These results show that the model exhibits a low-sensitivity bias. Note that in both cases, the model can learn function that relies on sparse set of inputs (using the sparse tokens), however, it uses these tokens only when doing so leads to lower sensitivity.
4 INVESTIGATIONS ON VISION TASKS
In this section, test whether our notion of sensitivity captures the inductive bias of transformers on vision tasks. We consider Vision Transformers (ViT, Dosovitskiy et al., 2021) which regard images as sequence of patches instead of a tensor of pixels.
Definition 4.1 (Tokenization for Vision Transformers). Let X E R XP X be the image with height nA, width Twg, and number of channels ne- A tokenization of X is a sequence of T image patches {er,-.. ........ . where each token e; represents an image patch of dimension d Th ne/TT.
Sensitivity is measured on the training set because our goal is to understand the simplicity bias of the model training time, to see if it prefers to learn certain simple classes of functions on the training data. Since different models could have different generalization capabilities, the sensitivity on test data might not reflect the model's preference for low-sensitivity functions at training time. Further, since the choice of optimization algorithm could in principle introduce its own bias and our goal to understand the bias of the architecture, we train both the models with the same optimization algorithm, namely SGD; see Fig. 1 in the App. for comparison with Adam.
We consider three datasets in this section (see Appendix D for details), namely CIFAR-10 (Krizhevsky, 2009), ImageNet-1k (Russakovsky et al., 2015) and Fashion-MNIS" (Xiao et al., 2017). We use o² 1, 15 and 5, respectively. We use a variant of the VIT architecture for small-scale datasets proposed in Lee et al. (2021), referred to as ViT-small here onwards; see Appendix D for more details and Appendix A.3 for additional results where we show that varying model depths and number of heads does not affect the sensitivity of ViT models. We also compare the sensitivity of the ViT-small model and a ResNet- 18 (He et al., 2016) CNN on the SVHN dataset with o² = in Appendix A.3, which leads to the same conclusion.

---

Published as a conference paper at ICLR 2025
Transformers learn lower sensitivity functions than CNNs. Figure 4 shows the train accuracies as well the sen- sitivity comparison between two ViTs: the ViT-small model and ViT-simple model (Beyer ct al., 2022), two CNNs: ResNet-18 and DenseNet-121 (Huang et al., 2016), and a ConvMixer model (Trockman & Kolter, 2022a). Note that the train accuracies are comparable for all architectures, which allows for a fair comparison of sensitivity. We observe that the ViTs have significantly lower sen- sitivity compared to the CNNs and the ConyMixer model. At the end of the training, the sensitivity values are 0.3673 for DenseNet-1211 121, 0.0827 for ResNet- 0.0829 for ConvMixer, 0.0050 for ViT-small and 0.0014 for ViT-si
Train accuracy
Sensitivity
DenseNet-121 CNN
0.8
ResNet 18 CNN
0.6
ConvMixer
ViT-small
VIT-simple
Figure 4: Sensitivity CIFAR-10. Comparison of the sensitivity of two CNNs, two ViTs, and ConvMixer trained on the CIFAR-10 dataset, as function of training epochs. For fair comparison, the figure also shows the train accuracies (see App. Fig. for full train dynamics). All models have similar accuracies but the ViTs have sig- nificantly lower sensitivity.
Do Transformers have lower sensitivity than CNNs because these models process inputs dif- ferently? ViTs process inputs sequence of patches whereas CNNs do not, and hence a natural question to ask is if the difference in sensitivity between the two architectures is due to this difference in processing the inputs as opposed to differences in the architecture. To investigate this, we compare ViTs with ConvMixer (Trockman & Kolter, 2022b). Similar to ViTs, ConvMixer processes the input data in patch-wise manner, but has two key differences: it does not use the self-attention mechanism, which is the core component of transformers, and it relies on convolutions for the feedforward part as well. The higher sensitivity of the ConvMixer model indicates that the low sensitivity simplicity bias of the transformers is not because they process inputs patch-wise, but rather result of other components of the architecture.
Do these observations generalize to pre-trained models? To study this, we consider the ImageNet- Ik dataset (Russakovsky et al., 2015). We compare the sensitivity values of pre-trained ConvNext (Liu et al., 2022) and V1T/L-16 (Dosovitskiy et al., 2021) models. For comparable accuracies, V1T/L-16 has a sensitivity of 0.0191, which is lower than that of ConvNext at 0.0342. This shows that the observations on amall-scal models studied in this section transfer large-scale pretrained models.
Transformers learn lower sensitivity functions than MLPs. Next, we consider the Fashion-MNIST dataset and compare the sensitivity of ViT-small, a 3 3-hidden-layer CNN, an MLP with LeakyRel.U activation and an MI.P with sigmoid activation (see Fig. 15 in the Appendix for the training curves). At the end of training, the sensitivity values are 0.0559 for the MLP with LeakyReI.U, 0.0505 for the MLP with sigmoid, 0.0453 for the CNN and 0.0098 for the VIT.
Thus, transformers learn lower sensitivity functions compared to MLPs, ConvMixers, and CNNs.
5 INVESTIGATIONS ON LANGUAGE TASKS
In this section, we investigate the sensitivity of transformers on natural language tasks, where each datapoint is a sequence of tokens. Similar to the comparison of ViTs with MLPs and CNNs in Section 4, we compare RoBERTa (Liu et al., 2019) transformer model with LSTMs (Hochreiter & Schmidhuber, 1997). an alternative regressgeee model, in this section. Recall that we consider transformer with linear attention for the results in Section 2. Aligning with this setup, we also consider a RoBERTa model with ReLU activation in the attention layer (i.e,, replacing (p)) in Eq. (3) with (e). for our experiments.
We use the usual RoBERTa-like tokenization procedure to process inputs for all the models so that they represented <S> c,. .... eT 
 where each ej represents tokens that are usually subwords and <S> represents the classification (CLS) token, T the sequence length, and 
 the separator token.
We denote 60 = 
 and éT+1 = 
. For each token ej, a token embedding (-)()) [MM] [JM]-HB Rd is trained during the process, where M denotes the vocabulary size. For transformers,
also train
separate positional encoder h Rd where N denotes the maximum sequence length. We denote _LSIM = h()) and c cxiabtsiia = h (oatt) + ())o)t)((()) (j) as the embedding tokens of LSTM and RoBERTa, respectively. We omit the superscript for convenience.
To control the relative magnitude of noise, the embeddings e, f LayerNorm(e,) normalized (Ba et al., 2016) before the additive Gaussian corruption. To better control possible
(e)) are first layer-
7

---

Published as a conference paper at ICLR 2025
19TM RoStEETs Einaa) (BaLL))
1XTM
0.6
LITM
LS Radtii to (Rel moa)
Raderra (Sofiaas)
0.3
500 Jang 2o-i 30-13 le Data Points Trained
Data buints Irained
Baken Position
Token Posittio
(a) MRPC
(b) QQP
(a) MRPC
(b) QQP
Figure 5: Sensitivity over Datapoints Trained. Figure 6: Sensitivity over Token Position. On On both datasets, the Transformer- model both datasets, LSTMs are more sensitive to later RoBERTa displays much lower sensitivity com- tokens than early ones, while RoBERTa's sensitiv- pared to LSTMs during the entire training process.
ity, regardless of the activation function, is more uniform across token positions, except for few tivity compared to its Softmax counterpart at later early bumps in early tokens which come from the stages of training.
RoBERTa with ReLU activation has lower sensi-
CLS token <S>
confounders, we limit both LSTM and RoBERTa to having the same number of layers. Both models trained from scratch, without any pretraining on larger corpora, ensure fair comparisons.
We consider two binary classification datasets, MRPC (Dolan & Brockett, 2005) and QQP (lyer et al., 2017) (see Appendix D for details), which relatively easy to learn without pretraining (Kovaleva et al., 2019). Empirically, we set o==15 15 (results with o² =4 in App. A.4 yield similar observations as the results in this section). Similar to Section 4, measure sensitivity on the train set (results on the validation set in App. A.4 yield similar observations). We include results with different depth values for RoBERTa as well as using GPT-2 in App. A.4 and they lead similar conclusions.
Transformers learn lower sensitivity functions than LSTMs. As shown in Figure 5, both RoBERTa models have lower sensitivity than LSTMs on both datasets, regardless of the number of datapoints trained. Even at initialization with random weights. LSTMs are more sensitive. At the end of training, the sensitivity values on the MRPC dataset are 0.15, 0.002 and 0.001 for the LSTM, the RoBERTa model with softmax activation and the RoBERTa with ReLU activation, respectively. On the QQP dataset, LSTM, RoBERTa-softmaa and ROBERTA- ReLU have sensitivity values of 0.09, 0.03 and 0.02, respectively. Interestingly, RoBERTa with ReLU activation also has lower sensitivity than its softmax counterpart. This may be because softmax attention encourages sparsity because of which the model can be more sensitive to particular token; see Ex. 2 in Fig. 2 and bottom row of Fig. 3 for an example where sparsity can lead to higher sensitivity.
LSTMs are more sensitive to later tokens. In Fig. 6, we plot sensitivity over the token positions. We observe that LSTMs exhibit larger sensitivity towards the end of the sequence, i.e. at later token positions. In contrast, transformers are relatively uniform. Similar observations were made by (Fu et al., 2023) for a linear regression setting: LSTMs do more local updates and only remember the most recent observations, whereas transformers preserve global information and have longer memory.
Transformers are sensitive to the CLS token, In Fig. 6, we also observe that the RoBERTa model with softmax activation has frequent bumps in the sensitivity values at early token positions. This is because different sequences have different lengths and while computing sensitivity versus token positions, we align all the sequences to the right. These bumps early token positions indeed correspond to the starting token after the tokenization procedure, the CLS token 
. This aligns with the observation of Jawahar et al. (2019) that the CLS token gathers all global information. Perturbing the CLS token corrupts the aggregation and results in high sensitivity. We also observe that RoBERTa with ReL.U activation seems less sensitive to the CLS token compared to its softmax counterpart.
IMPLICATIONS OF Low SENSITIVITY BIAS
We saw Section 4 that transformers learn lower sensitivity functions than CNNs. In this section, we first compare the test performance of these models the CIFAR- dataset and show that transformers are more robust than CNNs. Next, we add a regularization term while training the transformer, encourage lower sensitivity. The results demonstrate that lower sensitivity leads to improved robustness. We then explore the connection between sensitivity and the flatness of the minima. Our results show that lower sensitivity leads to flatter minima. Finally, examine if
8

---

Published as a conference paper at ICLR 2025
sensitivity can be used to understand the training dynamics of transformers, where we find sensitivity to be a suitable progress measure for certain grokking instances.
6.1 LOWER SENSITIVITY LEADS TO IMPROVED ROBUSTNESS
The CIFAR- 10-C dataset (Hendrycks & Dietterich, 2019) was developed to benchmark the perfor- mance of various NNs on object recognition tasks under common corruptions that are not confusing to humans. Images from the test set of CIFAR-10 are corrupted with 14 types of algorithmically generated corruptions from blur, noise, weather, and digital categories (see Fig. 1 in Hendrycks & Dietterich (2019) for examples).
Fig. 7 compares the performance of two CNNs: ResNet-18 and DenseNet-121 with two ViTs: ViT-small and ViT-simple various corruptions from the CIFAR-1O-C dataset, the end of training. We observe that the ViTs have lower sensitivity and better test performance on almost all corruptions compared to the CNNs, which have higher sen- sitivity. Since the definition of sensitivity involves the addition of noise and ViTs have lower sensitivity, one can expect to be robust to various noise corrup- tions. However, the ViTs also have better test performance on several corruptions from weather and digital categories, which are significantly different from noise corruptions. This is consistent with the observations in Mahmood et al. (2021); Bhojanapalli et al. (2021).
63.5
DenseNat-121 (0.3673)
Resbar-18 (0.0827)
VIT-omall (0.0050)
WiT-smelling (0.0025)
64.5
VIT-aimpee (0.0014)
35.0
VTT-saall-ugg (0. (0.0007)
Doer Text
Geutain the
Glos the
Motio that
Zoom Blr
daumben tedle Nelle
Anpolie Nolka
Stow
Frost
Pog
Pletic fransfur
Contant
Frenise
Figure 7: Comparison of the test accuracies CIFAR-10 and on various corruptions from the CIFAR-IO-C dataset (see Section for details) of various models trained the CIFAR-10 dataset, the last training epoch (see App. Fig. 17 for comparison of the accuracies function of training epochs.). We observe that the Vision Transformer models ViT-small and ViT- exhibit lower sensitivity and higher robustness to corruptions compared to the CNN models DenseNet-122 121 and ResNet- 18. Additionally, encouraging lower sensitivity while training through regulariza- tion ViiT-mall-ree and data augmentation (ViT- SiT-smalll somall-ap -uug) leads to improved robustness (see Section for details).
Next, we conduct an experiment investigate the role of low sensitivity in the robustness of transformers. We add a regularization term while training the model to explicitly encourage it to have lower sensitivity. If this model is more robust, then we can disentangle the role of low sensitivity from the role of the architecture and establish concrete connection between lower sensitivity and improved robustness. To add the regularization, we use the fact that sensitivity can be estimated efficiently via sampling and consider two methods. In the first method (augmentation), the training set by injecting the images with Gaussian noise (mean 0, variance 0.1) while preserving the label, and train the VIT on the augmented training set. In the second method (regularization), add a mean squared error term using the model outputs for the original image and the image with Gaussian noise (mean 0, variance 1) injected into randomly selected patch.
augment
Fig. 7 also shows the test performance of ViT-small trained with augmentation and regularization methods various corruptions from CIFAR-10-C. We observe that ViT's trained with these methods exhibit lower sensitivity compared to vanilla training. This is accompanied by an improved test performance on various corruptions, particularly on the noise and blur categories. As encouraging lower sensitivity improves robustness, the inductive bias of transformers to learn functions of lower sensitivity could explain their better robustness (to common corruptions) compared CNNs.
6.2 LOWER SENSITIVITY LEADS TO FLATTER MINIMA
In this section, we investigate the connection between low sensitivity and flat minima. Consider a linear model (00 x) = 0 æ. Measuring sensitivity involves perturbing the input by some Ax. Prediction on the perturbed input is equivalent to perturbing the weight vector with 40
e AA æ, as |
Φ(0; x + Ax) = 0 (x + Ax) = (0 - Ax = (0; x) + 40 x (0 20;
This draws a natural connection between sensitivity, which is measured with perturbation in the input space, and flatness of minima, which is measured with perturbation in the weight space (Keskar et al.,

---

Published as a conference paper at ICLR 2025
2017). Below, we investigate whether such connection extends to more complex architectures such as transformers. Given model & and train set D, we consider two metrics to measure the flatness of the minimum, based on the model outputs and model predictions, respectively,
000; x) - 2 - E;a)), ShPred == a~ 11((0;0) x) + 5(0 + +; m)), ShOp = m- m...E-NN
E
where f(0:x) 1 (0)( x) 0j. Intuitively, for flatter minima, the model output and hence its prediction would remain relatively invariant to small perturbations in the model parameters.
Table 2 shows a comparison of these metrics for the ViT-small model trained with and without the sensitivity regularization at the end of train- ing. Both metrics indicate that lower sensitivity corresponds flatter minimum. It is widely believed that flatter minima correlate with better generalization (Jiang* et al., 2020; Keskar et al., 2017: Neyshabur et al., 2017), though they may not always be correlated (Andriushchenko et al., 2023). Our results indicate that cow-sensitivity correlates with improved generalization and investigating this connection for other settings can be an interesting direction for future work.
Table 2: Comparison of two sharpness metrics at the end of training the ViT-small model the CIFAR-10 dataset with and without the sensitivity regularization. Lower values correspond to flatter minima; see text for discussion.
Setting
ShOp
ShPred
ViT-small vanilla training ViT-small + sensitivity regularization
39.166
0.5346
9.025
0.3982
6.3 SENSITIVITY AS A PROGRESS MEASURE FOR GROKKING
In this section, we in- vestigate the sensitiv- ity notion could serve a progress measure for grokking (Nanda al., 2023; Chen et al., 2024). We train an one-layer Transformer model on the modular addition task a + b mod 113. When eval- uating sensitivity, we add random Gaussian noise
Tan and That Accusaay over Spechs
Seraltiity orar Epocte
Whight Norm
1000 NCUBB))
Roach 2003
250 2000 22042200220002200
Epoch
250 1000
Saach
2300 300
spoch
1000
(a) Train & Test accuracy
(b) Sensitivity
(c) Weight norm
Figure 8: Sensitivity measures progress on modular addition task + b mod 113 and indicates different stages of grokking.
with o = 0.1 to the number embeddings. As shown in Figure 8, the test accuracy stays low from epoch 500 to 9,800 while the training accuracy saturates. However, sensitivity values continue to decrease smoothly starting from epoch 3000, and hence it provides a measure of the hidden progress (Barak al., 2022) which the model makes stages of grokking. In contrast, the weight norm is not a progress measure since it has the same flat curve as test accuracy during epoch 3,000 9,800.
though the loss does not change, and indicates
As discussed by Nanda et al. (2023), grokking occurs when the model learns to use Fourier features solve the task. A further bump in sensitivity after the grokked phase at epoch 9800 suggests that the model initially learns less robust Fourier features. At this stage, small random noise could slightly disrupt the model's performance. Over time, the Fourier basis becomes more robust. See Appendix A.5 for further discussion and results for more settings.
CONCLUSION
In this work, we investigate how the notion of sensitivity, which has shown promise in understanding inductive biases of Transformers on Boolean functions in prior work, can be extended to more realistic settings involving real-valued data. Our results show that transformers learn functions that have low sensitivity to small token-wise input perturbations, compared to other architectures, across vision and language tasks. We corroborate these observations with theoretical results, showing that transformers exhibit spectral bias and lower sensitivity corresponds to better robustness. We also demonstrate three important implications of this low-sensitivity bias: it correlates with improved robustness, flatter minima in the loss landscape, and serves progress measure that offers insights about the training dynamics. Investigating sensitivity as a progress measure in more settings can be an interesting direction for future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
The authors thank the anonymous reviewers for helpful suggestions and feedback. This work was supported by an NSF CAREER Award CCF-2239265, an Amazon Research Award, an Open Philanthropy research grant, and Google Research Scholar Award. YH was supported by USC CURVE fellowship. The authors acknowledge the use of USC CARC's Discovery cluster and the USC NLP cluster. This work was done in part while BV, DF, TZ, and VS were visiting the Simons Institute for the Theory Computing.
REFERENCES
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. 27
Maksym Andriushchenko. Francesco Croce, Maximilian Mueller, Matthias Hein, and Nicolas Flam- marion. A modern look at the relationship between sharpness and generalization. In Interna- tional Conference Machine Learning, 2023. URL https://api. www.ssisscmsmooooooo.... org/
CorpusID:25684632 10
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems, volume 32, 2019. 28
Devansh Arpit, Stanistaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pp. 233-242..
PMLR, 2017. 27
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. 7
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. arXiv preprint arXiv:2306.0463: 046007 2023. 27
Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. Advances in Neural Information Processing Systems, 35:21750-21764 2022. 10
Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies, 2019. 28
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for imagenet- 1k. 2022. 7
Satwik Bhattamishra, Arkil Patel, Phil Blunsom, and Varun Kanade. Understanding in-context learning in transformers and lims by learning to learn discrete functions, 2023a. 27
Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom. Simplicity bias in transformers and their ability to learn sparse Boolean functions. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics, 2023b. 1, 28
Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit. Understanding robustness of transformers for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10231-10241 2021. 1, 9, 27, 28
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. Curran Associates Inc., Red Hook, NY, USA, 2019. 28
Christopher M. Bishop. Training with noise is equivalent to tikhonov regularization. Neural Computation, 7:108-116, 1995. 28
11

---

Published as a conference paper at ICLR 2025
Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an omntein--ullnbeee like process. In Conference on learning theory, pp- 483-513. PMLR, 2020. 28
Simone Bombari and Marco Mondelli. Towards understanding the word sensitivity of attention layers: A study via random features, 2024. 28
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretehen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. 27
Yuan Cao, Zhiying Fang. Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. In IJCAI, 2021. 28
Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L. Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. In The Twelfth International Conference on Learning Representations, 2024. URL WRRLhttss///peereiiiee net/
forrm?id: MiSPiIKEEL..... 10
Ting-Rui Chiang. On benefit of mask language modeling: Robustness to simplicity bias, 2021. 28
Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory, pp. 1305-1338. PMLR, 2020. 28
Arthur Conmy, Augustine N. Mavor- Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. In Neural Information Processing Systems, 2023. 27
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoe V. Le. Autoaugment: Learning augmentation strategies from data. 2019 IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR), pp. 113-123, 2019. 28
Alexander G. de G. Matthews, Mark Rowland, Jiri Hron, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks, 2018. 26
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL https::/acclantholog org/ II//- 5002.0002200 8, 30
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https: //tptpse///peervviie. net/
1, 6,7
Spencer Frei, Gal Vardi, Peter I. Bartlett, Nathan Srebro, and Wei Hu. Implicit bias in leaky relu networks trained on high-dimensional data. arXiv preprint arXiv:2210.07082. 2022, 28
Deqing Fu, Tian-Qi Chen, Robin Jia, and Vatsal Sharan. Transformers learn higher-order optimization methods for in-context learning: A study with linear models. arXiv, abs/2310. 17086, 2023. 8, 27
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. ArXiv, abs/2208.01066 2022. 27
Adrià Garriga-Alone Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks as shallow gaussian processes. In International Conference on Learning Representations, 2019. URL httpp:://ppeeree net/forrmm.dd-kk11555e 26
12

---

Published as a conference paper at ICLR 2025
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference Learning Representations, 27, 28
2019. URL
Robert Geirhos, Momn-Heriik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1):665-673, 2020. 27
Soumya Suvra Ghosal, Yifei Ming, and Yixuan Li. Are vision transformers robust to spurious correlations?, 2022. 27
Parikshit Gopalan, Noam Nisan, Rocco A Servedio, Kunal Talwar, and Avi Wigderson. Smooth boolean functions are easy: Efficient algorithms for low-sensitivity functions. In Proceedings of the 2016 ACM Conference Innovations in Theoretical Computer Science, pp. 59-70, 2016. 4
Tianyu Guo, Wei Hu, Song Mei, Huan Wang, Caiming Xiong, Silvio Savarese, and Yu Bai. How do transformers learn in-context beyond simple functions? case study on learning with repre- sentations. In The Twelfth International Conference on Learning Representations, 2024. URL 27
https://openreviet net/ fww.mmid=iwwwweaee..
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pp. 107-112, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 111188811//11820000
URL https: /orttss:/aatt http//accaantohogggyoooo org/N18- 2017. 28
Michael Hahn and Mark Rofin. Why are sensitive functions hard for transformers? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 14973-15008, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/2024. acl-long.80k URL https:tp////t http///canntotoooooooooooooooola org www.:yy..rr2022...0m -oo.aa.....oo.00...
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in pre-trained language model. Thirty-sevnnth Conference Neural Information Processing Systems, 2023. URL httph:t//ppeerreiiiiii wwt.:/ormfffdd.deeeommoomo
p4PckNOR8k. 27
Jeff 7. HaoChen, Colin Wei, Jason Lee, and Tengyu Ma. Shape matters: Understanding the implicit bias of the noise covariance, In Conference on Learning Theory, pp. 2315-2357. PMLR, 2021. 28
Michael Hassid, Hao Peng. Daniel Rotem, Jungo Kasai, Ivan Montero, Noah A. Smith, and Roy Schwartz. How much does attention actually attend? questioning the importance of attention in pretrained transformers, 2022. 27
Pooya Hatami, Raghav Kulkarni, and Denis Pankratov. Variations on the sensitivity conjecture. Theory of Computing, pp. 1-27, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Im- age Recognition. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR '16, pp. 770-778. IEEE, June 2016. doi: 10.1109/CVPR.2016.900 URL http://ieeexplore ieee. org/ ptgg/document/78845555 6
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In Internationall Conference Learning Representations, 2019.
9
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, (8):1735-1780, 11 1997. ISSN 0899-7667. doi: 20.116//ncco.1997 1735.. URL https: //doi. http://.1..112/eeeeeeee 1997. 1735.
13

---

Published as a conference paper at ICLR 2025
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: Nngp and ntk for deep attention networks. In Proceedings of the 37th Interaational Conference on Machine Learning, ICML'20. JMgR.org, 2020. 26
Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261-2269, 2016. URL htt.:tpp://ppp..emmiiooooo.
semanticscholaa org :tt///ooouSI:::333330
Hao Huang. Induced subgraphs of hypercubes and a proof of the sensitivity conjecture. Annals of Mathematics, 190(3):949 955, 2019. 3, 27
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola.
The low-rank simplicity bias in deep networks. Trans. Mach. Learn. Res., 2023, 2021. 27, 28
Shankar Iyer, Nikhil Dandekar, and Kornel Csernai.
Quora Dataset URL https://quoradata quora. com/
First
Release: Question Pairs, 2017.
First-Quora Prrst-Qurr--Datast---Bee -awaasttt-elelse--euieee inestion-Pairrs Online. 8, 30
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3651-3651 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 11118663/11/11--1356 URL https: /taps://aclaass http//clcannoooooyoooooooooooool org/P19- 1356.
Ziwei Ji and Matus Telgarsky. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300, 2018. 28
Ziwei Ji and Matus Telgarsky. Directional convergence and alignment deep learning. Advances in Neural Information Processing Systems, 17777-7117 2020. 28
Ziwei Ji and Matus Telgarsky. Characterizing the implicit bias via primal-dual analysis. In Algorithmic Learning l'heory, pp. 772-804. PMIR, 2021. 28
Ziwei Ji, Nathan Srebro, and Matus Telgarsky. Fast margin maximization via dual acceleration. In International Conference on Machine Learning, pp. 4860-4869. PMIR, 2021. 28
Yiding Jiang*, Behnam Neyshabur*, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantas- tic generalization measures and where to find them. In International Conference on Learning Representutions, 2020. URL
ew www.iieweeetttttee wwt. www.oooddddSgIIPDD
10
John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridg- land, Clemens Meyer, Simon A A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David A. Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Se- bastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596:583 589, 2021. URL https: //ppi.pppp.. htt.app....saiiiaammammmm.mmm..e cscholar
semanticschol.at arr wwwg/ogggouue Stt.rpuuID:2359998855 235959867.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017. htttttttt///opeeee//e net/
forum?id=HHloyllyyn 9, 10
P. Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness spurious correlations. ArXiv, hbs/2204.00237, 2022. 28
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi. and Kentaro Inui. Analyzing feed-forward blocks in transformers through the lens of attention maps. In The Twelfth International Conference on Learning Representations, 2024. URL
27
Yiwen Kou, Zixiang Chen, and Quanquan Gu. Implicit bias of gradient descent for two-layer relu and leaky relu networks nerroy-orthogonal data, 2023. 28
14

---

Published as a conference paper at ICLR 2025
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of BERT. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)) pp. 4365-4374, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.1865//v1/D19-114
URL
org/119---9010000 1445. 8, 30
Alex Krizhevsky. Learning multiple layers of features from tiny images. pp. 32-33, 2009. URL https://www. cs. weaitto..dduukiii/eeeeeee
fewtures---eeeeeeee 200u TR. pdd. 6,
Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 2005. 19
Jachoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks as gaussian processes. International Conference on Learning Representations, 2018. URL http://pppeneeeeeeeeee/rooraaaa/ 26
25,
Seung Hoon Lee, Seunghyun Lee, and Byung Cheol Song. Vision transformer for small-size datasets, 2021. 6
Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning, 2021. 28
Zhiyuan Li, Tianhao Wang, Jason D Lee, and Sanjeev Arora. Implicit bias of gradient descent reparametrized models: On equivalence to mirror descent, Advances in Neural Information Processing Systems, 35:34626-34640, 2022. 28
Sungbin Lim, Iddoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In
Neural Information Processing Systems, 2019. 28
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. 7
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11966-11976, 2022. doi: 00.10099CVPP326682222.001166
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations, 2020. 28
Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent two-layer nets: Margin maximization and simplicity bias. In Neural Information Processing Systems, 2021. 28
Kaleel Mahmood, Rigel Mahmood, and Marten Van Dijk. On the robustness of vision transformers to adversarial examples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7838-7847, 2021. 9, 27, 28
Luke Melas-Kyriaze Do you even need attention? stack of feed-forward layers does surprisingly well on imagenet. ArXiv, abs/2105.02723, 2021. URL. https:tss://app semanticscholar org/
CorpusID: 233866661.... 27
Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (cds.), Advances in Neural Information Processing Systems, 2022. URL https::/opeereet www.oooommnnnnhoAAeeee
27
Depen Morwani, Jatin Batra, Prateck Jain, and Praneeth Netrapalli. Simplicity bias in 1-hidden layer neural networks, 2023. 27, 28
Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. The 22nd International Conference Artificial Intelligence and Statistics, pp. 3420-3428. PMLR, 2019. 28
15

---

Published as a conference paper at ICLR 2025
Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of diatr---iistriuttinn generalization. In International Conference on Learning Representations. 2021. URL https::
28
Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin Edelman, Fred Zhang. and Boaz Barak. SGD on Neural Networks Learns Functions of Increasing Complexity. Curran Associates Inc., Red Hook, NY, USA, 2019. 27
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretabilityy. 2023. 10, 24,2! 27
Muzammal Naseer, Kanchana Ranasinghe, Salman Hameed Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. In Neural Infor- mation Processing Systems, 2021. URL httpp://app. semanticscholar. 235125781. 27
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learn- ing and Unsupervised Feature Learning 2011, 2011. URL http://ufldl. stanford. edu/ http:uubeerrriii/211ousssse
pdf.. 21,29
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. CoRR, abs/1412.6614, 2014. URL https://apil semanticscholas Oagg/Coruuul.a 6021932. 27
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring generalization in deep learning. In Neural Information Processing Systems, 2017. URL
https://api. wwwanticscoolar.......oo. www.aaarorrr/ooorrrirar.. /et///opusID::9976600 10
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: empirical study, arXiv preprint arXiv:1802.088600 2018.
Roman Novak, Lechao Xiao, Yasaman Bahri, Jachoon Lee, Greg Yang, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. In International Conference on Learning Representations, 2019. URL https://openreviex. http://erreeieeeeeeeooreeeeeeeee nett/formmiidd:1100 OFF0 26
Ryan O'Donnell Analysis of Boolean Functions. Cambridge University Press, 2014. doi: 10.1017/
CBO9781 139781881.8877 1, 3, 26, 27
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library, 2019. 29
Sayak Paul and Pin-Yu Chen. Vision transformers are robust learners. In Proceedings of the AAAI conference on Artificial Intelligence, volume 36, pp. 2071-2081, 2022. 1, 27, 28
Mohammad Pezeshki, Sekouba Kaba, Yoshua Bengio, Aaron C. Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. In Neural Information Processing Systems, 2020. 28
Mary Phuong and Christoph H Lampert. The inductive bias of re(lu) networks on orthogonally separable data. In International Conference Learning Representations, 2021. URL https: //openreview eww net/ tret/foorumiin htt.//orrmmdddar277X00SS
28
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: General- ization beyond overfitting on small algorithmic datasets, 2022. 27
Philip Quirke and Fazl Barez. Understanding addition in transformers. In International Conference Learning Representations (ICLR), Vienna, Austria, 2024. 27
16

---

Published as a conference paper at ICLR 2025
Maithra Raghu, Thomas Unterthiner, Simon Korbllihh, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL htw.t:t/s//ooooo /tttt::s///peeeveieee. n BarrmiiddGl8FHLEE
27
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5301-5310. PMLR, 09-15 Jun 2019a. URL httph: http:s//ppoceeinggs... mlr. prwss/ /evees//v7/// w9t/rahamannaaaa 9a. html. 27
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference on machine learning, pp. 5301-5310. PMLR, 2019b. I
Sylvestre-Alvis Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Data augmentation can improve robustness. In Neural Information Processing Systems, 2021. 28
Herbert E. Robbins. A stochastic approximation method. Annals of Mathematical Statistics, 22:
400-407, 1951. 28
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115
13):21,-252, 2015, doi: 10.10/7/s 1263-015-0816-y, 6, 29
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL https:
//openreview n www.oooooduunruuuuuuo
28
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33, 2020. 27, 28
Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin- Yu Chen, and Cho-Jui Hsich. On the adversarial robustness of visual transformers. arXiv preprint arXiv:2 /5670, 122), 2021. 1, 27, 28
Lujia Shen, Yuwen Pu, Shouling Ji, Changjiang Li, Xuhong Zhang, Chunpeng Ge, and Ting Wang. Improving the robustness of transformer-based large language models with dynamic attention. arXiv preprint
2023. 28
Zhouxing Shi, Huan Zhang, Kai-Wei Chang, Minlie Huang, and Cho-Jui Hsich. Robustness verifica- tion for transformers. arXiv preprint arXiv:2002.06622 2020. 28
Zhouxing Shi, Yihan Wang, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsich. Fast certified robust training with short warmup. Advances in Neural Information Processing Systems, 34:18335-18349, 2021.
28
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent separable data. The Journal of Machine Learning Research, 19(1): 2878, 2018. 28
Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, and Samet Oymak. Transformers as support vector machines. ArXiv, abs/2308. 16898, 2023a. 27
Davoud Ataee Tarzanagh, Yingcong Li, Xuechen Zhang, and Samet Oymak. Max-margin token selection in attention mechanism, 2023b. 27
Rishabh Tiwari and Pradeep Shenoy. Overcoming simplicity bias in deep networks using a feature sieve, 2023. 28
17

---

Published as a conference paper at ICLR 2025
Asher Trockman and J. Zico Kolter. Patches are all you need? Transactions Machine Learning Research, 2023, 2022a. URL https://api. wwmanticscholar..ommm ott.://cccoooo.ooooooooooossaaaa 2.
Asher Trockman and Zico Kolter. Patches are all you need?, 2022b. URL https://///ppeereeiien net/foorum2dd:7VHS5YY00
7
Guillermo Valle-Perez, Chico Q Camargo, and Ard A Louis. Deep learning generalizes because the parameter-funetion map is biased towards simple functions. International Conference on Learning
Representations, 2019. 1, 27
Gal Vardi. On the implicit bias in deep-learning algorithms, 2022. 28
Bhavya Vasudeva, Kameron Shahabi, and Vatsal Sharan. Mitigating simplicity bias in deep learning for improved ood generalization and robustness, 2023. 28
Bhavya Vasudeva, Puneesh Deora, and Christos Thrampoulidis, Implicit bias and fast convergence rates for self-attention, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gareet (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: http:////oeeennnseeeeeeeeecc neurips cw/paper.p.p..c/ httpe//ppeffleeee///eeeeeeeeeeee filles/paper/2017/EELL Paper. .df..
3f5ee243547deee
Johannes von Oswald, Eyvind Niklasson, E. Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in- in-context by gradient descent. In International Conference on Machine Learning, 2022. 27
Fali Wang, Zheng Lin, Zhengxiao Liu, Mingyu Zheng, Lei Wang, and Daren Zha. Macrobert: Maximizing certified region of bert to adversarial word substitutions. In Database Systems for Advanced Applications: 26th International Conference. DASFAA 2021, Taipei, Taiwan, April 11-14, 2021, Proceedings, Part 11 26, pp. 253-261. Springer, 2021. 28
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022. 27
Han Xiao, Kashif Rasul, and Roland Vollgraf. mashion-mmist: novel image dataset for benchmarking machine learning algorithms, 2017. 6, 29
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint 191.v01901 2019. 27, 28
Greg Yang. Tensor programs i: Wide feedforward or recurrent neural networks of any architecture are gaussian processes, 2021. 26
Greg Yang and Hadi Salman. fine-grained spectral perspective on neural networks, 2020. 2, 25, 26, 27
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018. URL https: 28
//openrevien
18

---

Published as a conference paper at ICLR 2025
APPENDIX
Additional Experiments
A.1 Synthetic Data and the MNIST Dataset..
19
A.2
Sensitivity with Random Noise instead of Token-wise Noise
19
A.3
Vision Tasks
20
A.4
Language Tasks
A.5 Sensitivity Progress Measure for Grokking
24
A.6 Sensitivity as a Progress Measure for Learning Sparse Parities
Proofs for Section
25
B.1 Proof of Proposition 2.1
26
B.2 Proof of Proposition 2.2
26
Related Work
27
Details of Experimental Settings
29
Limitations
30
A ADDITIONAL EXPERIMENTS
In this section, we include some additional results supplement the main experimental results for synthetic data well as the vision and language tasks.
A.1 SYNTHETIC DATA AND THE MNIST DATASET
In this section, we present some additional results for the low-sensitivity bias of a single-layer self-attention model (Eq. (3)) the synthetic dataset generated based on Definition 3.3, visualized in Fig. 2. Similar to the results in Section 3.1, we consider three data settings where using the sparse token leads function with lower sensitivity (Fig. 9, top row) and three settings where using the frequent token leads to lower sensitivity (Fig. 9, bottom row). The exact data settings and a comparison of the sensitivity values for each setting shown in Table 3. These results yield similar conclusions as in Section 3.1: in both cases, the model uses tokens which leads to a lower sensitivity function.
Continuing from the synthetic data, we now consider slightly more complicated dataset, namely MNIST (LeCun & Cortes, 2005). The MNIST dataset consists of 70k black-and-white images of handwritten digits of resolution 28 x 28. There are 60k images in the training set and 10k images in the test set. released under the CC BY-SA 3.0 license.
compare the sensitivity of ViT-small model with an MI.P on a binary digit classification task (<. 5 or 2 5). In our experiments, each image for the ViT-small model. For the MLP. the inputs are
divided into T 16 patches of size vectorized usual. With this setting, measure the sensitivity of the two models using patch token replacement as per Definition 1. As shown Figure 10, when achieving the same training accuracy, the VIT shows lower sensitivity compared the MLP.
A.2 SENSITIVITY WITH RANDOM NOISE INSTEAD OF TOKEN WISE NOISE
In this section, we consider changing the way we compute sensitivity, to see if the resulting metric also distinguishes transformers from other architectures. Instead of token-wise perturbations, add
19

---

Published as a conference paper at ICLR 2025
Table Comparison of sensitivity values for models that use only sparse or frequent tokens for the settings considered in Fig. 9.
Top row
Fig.
(n,
= 3, ng
Bottom row Fig.
3 [M2
Data Setting (ng, m)
(3.6)
15. 16)
(7.28)
(7, 10)
(17, 20)
(32. 36)
Using sparse tokens
U 0.2878
0
0.0339
0.0339
0,0899
Using frequent tokens
0.1815
0.4502
..n apernz
Fressent
arrelenant
Thain Accrney
501 let Ncurrte 931 Tett acurse
Llbratura (M)
Itaralites (U)
20ete122te 19
Lteratinn (U)
Itcrations 11
MULLEE
Trei Mccree X
Itaratuess
21ete 10
Eleration (U)
Itcratiens
Figure 9: Train and test dynamics for single-layer self-attention model (Eq. (3)) using the synthetic data visualized in Fig. 2: see Section 3.1 for details. The top row corresponds to the cases where the predictor that uses sparse tokens has lower sensitivity, while the bottom row corresponds to the cases where using the frequent tokens leads to lower sensitivity. The precise data settings for this figure, as well as comparison of sensitivity values, are shown in Table 3.
Train accuracy
Sensitivity
Train sccurcc
Senciivity
Adam
MLP VIT
0.2
12
30
Figure 11: Sensitivity using SGD and Adam. Com- parison of train accuracies and sensitivity values of the ViT-small model trained on the CIFAR-10 dataset using SGD and Adam optimizers.
Figure 10: Sensitivity on MNIST. ViT and MLP get similar accuracy, but the ViT has lower sensitivity.
Gaussian noise across the entire input with smaller variance that the transformer's sensitivity in this case is similar to the sensitivity with token-wise noise.
Table 4: Comparison of sensitivity values measured with random and patch-wis wise noise for various model-dataset settings. Token-wise perturbations lead to a larger gap between the sensitivity of transformer-based models compared to other architectures.
Model and dataset
Random noise
Token-wise noise
ResNet-18 on CIFAR-10
0.0172
0.0827
ViT-small CIFAR-10
0.0082
0.0050
LSTM on QQP
0.11
0.09
RoBERTa on QQP
0.05
0.03
In Table 4, we compare the sensitivity values at the end of training for ResNet- and ViT-small on the CIFAR-10 dataset (variance 0.025) and LSTM and RoBERTa on the QQP dataset (variance 0.5). We find that for random perturbations, the difference between sensitivity values is much smaller for CIFAR-10 10 and similar for the QQP dataset, compared to patch-wise perturbations. These results suggest that measuring sensitivity with patch-wise noise is indeed the metric that we should consider since it distinguishes transformers from other architectures with a larger gap.
A.3 VISION TASKS
Effect of Depth, Number of Heads and the Optimization Algorithm. In Fig. 11, we compare the sensitivity values of ViT-small model trained CIFAR-10 dataset with SGD and Adam
20

---

Published as a conference paper at ICLR 2025
optimization algorithms. Although the model trained with Adam has slightly higher sensitivity, the sensitivity values for both the models are quite similar. This indicates that the low-sensitivity bias is quite robust to the choice of the optimization algorithm.
In Fig. 12, we compare the sensitivity values of a ViT-small model with different depth and number of attention heads, when trained the CIFAR-10 10 dataset. Note that for main results, a model with depth 8 and 32 heads. We observe that the train accuracies and the sensitivity values remain the same across the different model settings. This indicates that the low-sensitivity bias is quite robust to the model setting.
Train
Senatirity
Tran socuncy
Sencitutt
04
Depth
0.4
0.5
01
Depth
Depth
02
32
07
(a) Sensitivity for varying depth
(b) Sensitivity for varying number of heads
Figure 12: Sensitivity for Various Model Settings. Comparison of train accuracies and sensitivity values on the CIFAR-10 dataset when varying the depth and number of heads of the ViT-small model. We observe that for the same train accuracy, the sensitivity values remain very similar for different model settings.
Train loss
Train accuracy
Test loss
Test accuracy
Sensttvvity
DenseNat-1 121 CNN
ResNet: 18 CNN
2
14
044
15
14
ConvMixer
VIT-small
3.
03
ViT-simple
10
20
43
30
10
20
Figure 13: Comparison of the sensitivity of two CNNs, two ViTs, and ConvMixer trained on the CIFAR-10 dataset, function of training epochs. For fair comparison, the figure also shows the train and test accuracies and loss values (cross-entropy loss). All models have similar accuracies but the ViTs have significantly lower sensitivity than the other models.
Effect of Variance. In Fig. 14, we compare the effect of the variance o² used while eval- uating sensitivity for different models trained on the CIFAR-10 dataset. We observe that the ViTs have significantly lower sensitivity than the other models and the difference becomes starker as the variance level increases.
6.8
Daroshie: 1a
0.6
CorsMoor
ViTsinple
02
Results on SVHN Dataset. Fig. 16 shows the training accuracy and sensitivity of a ResNet-18 and ViT-small model trained on SVHN dataset (Netzer et al., 2011). At the end of training, the sensitivity values are: 0.0516 for ResNet- 18 and 0.0147 for ViT-small. Similar to the observations for CIFAR-10, we see that the ViT has significantly lower sensitivity.
Figure 14: Sensitivity for Different Variances. Comparison of the sensitivity of two CNNs, two ViTs, and ConvMixer trained on the CIFAR-10 10 dataset,
function of different variance levels, at the end of training. The ViTs have significantly lower sensitivity at any variance and the difference grows as variance increases.
Additional Results on CIFAR-10-C. Fig. 17 and Fig. 18 show the test performance on various corruptions from the CIFAR- 10-C dataset with severity level 2 and 1, respectively. We observe that CNNs have lower test accuracies on corrupted images compared to ViT'.. Further, encouraging lower sensitivity in the VIT leads to better robustness.
A.4 LANGUAGE TASKS
Sensitivity Measured with Variance o² 4. Alternative to the main experiments with o² = 15, we also evaluate sensitivity with different comruption strength o² 4 on the QQP dataset, shown in
21

---

Published as a conference paper at ICLR 2025
Trsin accuraty
Sensilvity
Tianr eccurecy
Surodiedy
04
02
CNN
MIP Bet
MLP (igmard)
33
CNN
63
WT
03
25
10
Figure 15: Sensitivity on Fashion-MNIST. Compari- Figure 16: Sensitivity on SVHN. Compari- son of sensitivity of VIT with CNN, an MLP with son of sensitivity of a R8 CNN and LeakyReLU activation and an MLP with sigmoid acti- a ViT-small trained on SVHN dataset, as a vation, as function of training epochs. All the models function of training epochs. Both the models have similar accuracies but the ViT has significantly have similar accuracies but the ViT has signif- icantly lower sensitivity.
lower sensitivity.
Sunsltvitty
Clean Test
Gaussian
Blur
Mocion Blur
Zoom
Impulsn
Fog
Br fightness
Contrast
Buotic Transform
Pixelato
Rechieo- 18 CNN
DonseNic 121
WT-small
VIT simple
62
augmentation
VIT-mall regularinaiin
Figure 17: Comparison of the sensitivity, test accuracy on CIFAR-10 10 and test accuracies on various corruptions from the CIFAR-IO-C dataset (see Section 6 for details) of two CNNs and two ViTs trained on the CIFAR-10 dataset, as a function of the training epochs. We also compare with ViT- small trained with data augmenatiionirgglaazzzzii,, which encourage low sensitivity (see Section 6
for discussion).
Senstivity
Clean Test
Goussion
Glass Blur
Mosion Blur
Caussian Noie
Shot Noise
Impulso No
Snow
Frast
Fog
33
30
00
53 80
32
30
30
30
50
12
10
Brightnnas
Contract
Elartic Transform
Picelate
JPEC Compression
0.
04
VIT + data nugmensation
Figure 18: Comparison of the sensitivity, test accuracy on CIFAR- 10, and test accuracies on various corruptions from the CIFAR- 10-C dataset (see Section 6 for details) of ResNet- 18 CNN and a ViT-small model trained on the CIFAR-10 dataset, as a function of the training epochs. We also compare with ViT-small trained with data augmentation, which acts as a regularizer to encourage low sensitivity (see Section 6 for discussion). Here, we use severity level 1, while in Figure 17, we considered severity level 2.
Fig. 19. We observe the same trends in Figures 5 and 6: RoBERTa models have lower sensitivity than the LSTM and the LSTM is more sensitive to more recent tokens. These results indicate that the low-sensitivity bias is robust to the choice of corruption strength.
22

---

Published as a conference paper at ICLR 2025
Bobtitla Ciotimaa))
Ro
3.00
9.11
0.12
0,00
06403 Ieddr3aeld Seaas Sea32 46-00 Saute Seels
60
130
Fosition
Figure 19: Sensitivity on the QQP Dataset with Variance x² = 4. Results with alternative variance yield observations that are consistent with the setup in the main text. (Left) LSTM has higher sensitivity than the RoBERTa models. (Right) Softmax activation for RoBERTa induces higher sensitivity towards the CLS token.
Sensitivity Measured on the Validation Set. We also test sensitivity on the validation set for the two language datasets. As seen in Figure 20, the results on the validation set are consistent with those on the train set Fig. 5.
RoltERT (Sefteaa)
0.12
RARERTA (8al(mas)
LSTH
Rolttkt (Re1.)
0.10
(Ral())
0.00
Data Points Trained
Data Points Trained
(a) MRPC
(b) QQP
Figure 20: Sensitivity the Validation Sets. Similar the observation in Figure 5, the RoBERTa models have lower sensitivity than the LSTM for both the datasets. However, the difference between RoBERTa-ReLL and RoBERTa-softmax is less marginal on the validation set compared to the training set.
Sensitivity Measured with the GPT-2 Model. Here, we ablate the effect of the transformer architecture the sensitivity values. We compare GPT-based model with the BERT-based models used in our main experiments. The key difference lies in the construction of the attention masks: for GPT models, each token only observes the tokens that appear before whereas BERT models are bidirectional, therefore each token observes all the tokens in the sequence. In Fig. 21, we observe that the GPT-2 model has higher sensitivity compared to the RoBERTa models, but the sensitivity is significantly lower than the LSTM. The GPT-2 model is also relatively more sensitive to more recent tokens compared to the RoBERTa models, while also being sensitive to some CLS tokens.
LST RoBERTA (Sedmna))
BuBERTTA (Softmaz)
0.10
(Be1l))
0.30
RORERT (Ro11)
CP7-3
0.20
Dato Pointa Trained
Token Position
Figure 21: Sensitivity of GPT-2 on the QQP Dataset. (Left) We find that the RoBERTa models tend to have lower sensitivity than GPT-2, and all Transformer models have lower sensitivity than I.STM. (Right) The sensitivity per token of GPT-2 is more similar to LSMMs, which possibly due to their shared auto-regressive design.
23

---

Published as a conference paper at ICLR 2025
LS1M Lavvel
LStN 18 Loveel
Layero)
RaREAT Layord
0.100
RolERD (1) Layera)
Data Points Trained
Figure 22: Sensitivity for Different Model Depths. We vary the model depths of LSTM and RoBERTa on the QQP datasets and observe that LSTM models tend to have the same sensitivity throughout the entire training. RoBERTa model with 4 layers has slightly lower sensitivity with its 8-layer or 12-layer variants. RoBERTa, regardless of depths, have lower sensitivity than LSTMs.
A.5 SENSITIVITY AS A PROGRESS MEASURE FOR GROKKING
Grokking is phenomenon in which neural network suddenly and drastically improves its gener- alization ability after long period of training, even though it initially overfits to the training data (Nanda et al., 2023). During grokking, the model transitions from memorizing the training data to learning a more general solution, allowing it to perform well on unseen data. This often happens after many training steps, during which time the test accuracy remains low despite perfect training accuracy. The transition is abrupt, making grokking seem like emergent behavior where the model, after much training, "figures out" the correct approach to the task.
In more technical terms, this shift occurs the network amplifies structured mechanisms that enable generalization and removes components that only lead to memorization. Grokking has been observed in models trained with regularization techniques like weight decay algorithmic tasks, where the model learns an underlying structure that generalizes well beyond the training data. In (Nanda et al., 2023), the authors demonstrate that for the modular addition task, the model initially memorizes the training data, leading to low test accuracy. Later, the model discovers how to use trigonometric functions solve the task. However, the emergence of grokking is difficult measure practice, and it is not guaranteed that grokking will occur or that the model will find the correct approach to the task. Nevertheless, as shown in Figure 23, can clearly obscrve significant change sensitivity between epochs 3000 and 9800, during the saturation of the training loss. This suggests that the model discovering more robust solution to the task. We propose that sensitivity can serve as useful metric for assessing whether the model is learning to grok.
Similar Nanda et al. (2023)'s classification on phases: memorization, circuit formation, and cleanup, we claim that sensitivity also provides a progress measure, with extra phase of noise reduction:
From epoch 0 to 500, sensitivity drops significantly while training accuracy saturates to 100 and test accuracy increases but cannot saturate.
Memorization:
Circuit Formation: From epoch 500 to 3,000, the sensitivity goes up again but test accuracy remains low and flat. The dramatic fall in the weight norms suggests that circuit formation likely happens due to weight decay.
Clean up: From epoch 3,000 to 9,800, sensitivity starts to decrease, and the progress measure indicates that the model starts to learn to use Fourier features.
From epoch 9,800 onwards, sensitivity has an upward and then downward trend, and this is not caught by other mechanistic interpretability measures in Nanda al. (2023). This trend in sensitivity is due to further Fourier noise reduction, where initially, slightly perturbations to the number embeddings could drift the model performance where later on, model learns more robust Fourier basis.
Noise Reduction:
A.6 SENSITIVITY AS A PROGRESS MEASURE FOR LEARNING SPARSE PARITIES
In this section, we investigate if sensitivity also acts as a progress measure when training transformers on the sparse parity task, with input {=1}" and label y ILes J₁, where S C [d] with sparsity level p= |S| d.
24

---

Published as a conference paper at ICLR 2025
Train and Test Accuracy over Epochs
Sensitivity over Epochs
Weight ever Epochs
Seed
Seed
02-
Seed
Accuracy
0.8
trann accuracy
Accuracy sees
Accoracy Seed
A Saneaann Sane 3
Spoch 9000
Epoch 500
Spoch 2000
Weightt Norm
Epech $600
Ezech
500
Accurcy
Train accuracy
Accuracy Seed
033
Epach
2500 3050
2500 sotoo Lebou 13000 1rst0
18000 12500 17500
2502
1200 20000 12500 1/500
(a) Train & Test accuracy
(b) Sensitivity
(c) Weight norm
Figure 23: Sensitivity measures progress on modulo addition task + b mod 113 and indicates different stages of grokking.
We train two-layer transformer model on this task with 4 heads using Adam optimizer for 1000 epochs. We evaluate the sensitivity metric based on Eq. (I), using 10s samples to estimate the expectation over the Boolean cube. Fig. 24 shows the results six different settings in terms of (number train samples, batch size, embedding dimension, learning rate, seed):
Sea green: (50k, 250, 32, 0.001, 123)
Parrot green: (50k, 250, 32, 0.001, 42) Maroon: (10k, 100, 64, 0.001, 42)
Dark green: (50k, 250, 32, 0.001, 0) Coral: (5k, 25, 32, 0.0001, 42)
Pink: (5k, 50, 32, 0.0001, 42).
We observe that in all the settings, the train and test accuracy remain close 50% for the initial few epochs, while sensitivity increases, which indicates progress in learning the sparse parity task. When the model learns the sparse parity function, the sensitivity converges P = 0.1, which coincides with train and test accuracy going to 100%.
Train accuracy
Test accuracy
Sensitivity
0.2
0.9
0.9
0.1
0.5
0.6
0.5
Step
0.5
Step
Step
10
100 1k
10
100 1k
10
100
1k
Figure 24: Train accuracy, test accuracy, and sensitivity as a function of training epochs, when training two-layer transformer on the sparse parity task with dimension 40 and sparsity level Sensitivity increases while train and test accuracy are close to 50%, and converges to 0.1 when the model learns the sparse parity function, which coincides with train and test accuracy going to 100%.
B PROOFS FOR SECTION 2
We give brief overview of the CK and NTK here and refer the reader to (Lee et al., 2018; Yang & Salman, 2020) for more details.
Consider a model with L layers and widths {dHf= and an input æ. Let g' (æ) denote the output of the ( layer scaled by 1/2 Suppose we randomly initialize weights from the Gaussian distribution (N 1). can be shown that in the infinite width limit when min d' - oo, each element of g (æ)
[E[T]
is a Gaussian process (GP) with zero mean and kernel function K The kernel K corresponding to the last layer of the model is the CK. other words, it is the kernel induced by the embedding g f-1 (ar) when the model is initialized randomly. On the other hand, NTK corresponds to training the entire model instead of just the last layer. Intuitively, when the model parameters 0 stay close to initialization dg. the residual (x;; ) (x: 0u) behaves like a linear model with features given by the gradient at random initialization, V Vog (xx 0n), and the NTK is the kernel of this linear model. The spectra of these kernels provide insights about the implicit prior of randomly initialized
25

---

Published as a conference paper at ICLR 2025
model as well as the implicit bias of training using gradient descent (Yang & Salman, 2020). The closer these spectra are to the spectrum the target function, the better we can expect training using gradient descent generalize.
B. PROOF OF PROPOSITION 2.1
(Hron et al., 2020) show that the self-attention layer with linear attention and d' 1/2 scaling converges in distribution to (P(O, K) the infinite width limit, i.e. when the number of heads d" become large. For any layer E [L], let K' denote the kernel induced by the intermediate transformation when applying some nonlinearity & to the output of the previous layer I 1. Let f'y == (f., a E X, E [T]). where X' denotes the input space of æ. They show the following result for NNs with at least one linear attention layer, in the infinite width limit.
(Hron et al., 2020)). Let I [L], and be such that (6(x)] S c+ m|æ/ for some e, m. E TRtt Assume g' converges distribution g' 1 GP(0, K(-1), such that g'j" I-1 are independent for any * k. Then min(d.H d'3 co, g converges in distribution and 9.
Theorem B.1 (Theorem
g¹ (CP(O,K') with 9. and g'e independent for any l, and
K' œ²) = (2)g
Σ (Ri x Kas R2 x'). 1,==1
Similar results are also known for several non-linearities and other layers such as convolutional, dense, average pooling (Lee et al., 2018; 88; de G. Matthews et al., 2018; Garriga-Aloma Alonso al., 2019; Novak et al., 2019; Yang. 2021), as well as residual, positional encoding and layer normalization (Hron et al., 2020; Yang, 2021).
Consequently, any model composed of these layers, such as transformer with linear attention, also converges Gaussian process. This follows using an induction-basee based argument. It can easily be shown that the induced kernel takes the form
(x
3) B
| d
K K(x, y)
d
for some function V IR3 R. In addition, since E CD
they have the same norm, and v can be treated as a univariate function that only depends on c d (æ. y). i.e. V (c, 1, (c)..
Using this property and the following result, it follows that the kernel induced by transformer with linear attention is diagonalized by the Fourier basis {xu ]ug(4.
Theorem B.2 (Theorem 3.2 in (Yang & Salman, 2020)). On the d-dimensional boolean cube We) .fo every U S [d]. Xu is eigenfunction of K with eigenvalue
(d'
Mui == E [T"K K(x [1. a0-0
E
x
23 zo-
where 1 = (1...,,1) 1) col This definition of lnil does not depend on the choice S, only the cardinality of S. These are all of the eigenfunctions of K by dimensionality considerations.
Further, using the following result, it follows that transformers (with linear attention) exhibit weak spectral simplicity bias.
Theorem B.3 (Theorem 4.1 in (Yang & Salman, 2020)). Let be the CK or NTK of an MLP on a boolean cube Go Then the eigenvalues μk, & = O.... d, satisfy
Po P2 2 2 122k 2 .... μ1 #3 Z p22+11
B.2 PROOF OF PROPOSITION 2.2
First, we introduce the concept of noise stability (f) which measures the correlation between the outputs of function f for p-correlated pair (x, x as Q,( = E (('). Note that (f)
(x,z²),
is related to R,(f)== Pr [f(x) + f(x'(( as (f()) = 1 (f) (O'Donnell, 2014).
(=,,²))
26

---

Published as a conference paper at ICLR 2025
Using the following result, relate noise stability the Fourier weight of the function ff different degrees E [d].
Theorem B.4 (Theorem 2.49 in (O'Donnell, 2014)). For function f
0 {+1), the noise
stability for p-correlated pair (x) satisfies Qo(f)=
Σ USid
f1)) Σ [f], where
W'[s==
Σ Í(U).
UC -C
Clearly, (p(l) < 1 since the minimum degree of f is 0. Next, we use the following important result, which upper bounds the degree of f in terms of its maximum sensitivity, Smax (f) == max S(f,z), x).
Theorem B.5 (Theorem 1 1.4 in (Huang, 2019)). For function f :004 + {+1}. the degree D(f) of the multilinear polynomial which represents f satisfies D(f) (L)))((mmxx)))) (Smax(f))
Using this, and the fact that Qp(f) is minimized when the Fourier weight is concentrated on the highest degree term, we get the lower bound Qp(f) p (()m))()) .since p and Smax (f) E d].
Using the relation between Qp(f) and Rf( then finishes the proof.
C RELATED WORK
Understanding Transformers. The emergence of transformers as the go-to architecture for many tasks has inspired extensive work on understanding the internal mechanisms of transformers, including reverse-engineent engiinernnnn language models (Wang et al., 2022), the grokking phenomenon (Power et al., 2022; Nanda et al., 2023), manipulating attention maps (Hassid al., 2022; Kobayashi et al., 2024), automated circuit finding (Conmy et al., 2023), arithmetic computations (Hanna et al., 2023; Quirke & Barez, 2024), optimal token selection (Tarzanagh et al., 2023a;b; Vasudeva et al., 2024), and in-context learning (Brown et al., 2020; Garg et al., 2022; Akyürek et al., 2023; von Oswald et al., 2022: Fu et al., 2023; Bhattamishra al., 2023a; Guo al., 2024). Several works investigate why vision transformers (ViTs) outperform CNNs (Trockman & Kolter, 2022a; Raghu et al., 2021; Melas- Kyriazi, 2021), as well as other properties of ViTs, such robustness (adversarial) perturbations and distribution shifts (Bai et al., 2023; Shao et al., 2021; Mahmood et al., 2021; Bhojanapalli al., 2021; Naseer et al., 2021; Paul & Chen, 2022; Ghosal et al., 2022). Further, several works on mechanistic interpretability of transformers share a similar recipe of measuring sensitivity corruption with Gaussian noise (Meng et al., 2022; Conmy et al., 2023) but on hidden states rather than the input space.
Sensitivity and Spectral Bias. Sensitivity closely related to spectral bias (Yang & Salman, 2020), which is bias towards 'simple' functions in the Fourier space. Simple functions in the Fourier space generally correspond to low-frequency terms when the input space is continuous, and low-degree polynomials when the input space is discrete. Recent work has shown that deep networks prefer to use low-frequency Fourier functions on images (Xu et al., 2019), and low-degree Fourier terms on Boolean functions (Yang & Salman, 2020). We note that in contrast to some other notions of spectral bias, sensitivity also has the advantage that it can be efficiently estimated on data through sampling in contrast, estimating all the Fourier coefficients requires time exponential in the dimensionality of the data and hence can be computationally prohibitive (Xu et al., 2019).
Simplicity Bias in DL. Several works (Neyshabur al., 2014: Varle-Perez et al., 2019: Arpit et al., 2017; Geirhos et al., 2020) show that NNs prefer learning 'simple' functions over the data. Nakkiran et (2019) show that during the early stages of SGD training, the predictions of NNs can be approximated well by linear models. Morwani et al. (2023) show that 1-hidden-layed NNs exhibit simplicity bias to rely on low-dimensional projections of the data, while (Huh et al., 2021) empirically show that deep NNs find solutions with lower rank embeddings. (Shah et al., 2020) create synthetic datasets where features that can be separated by predictors with fewer piece-wise linear components considered simpler, and show that in the presence of simple and complex features with equal predictive power, NNs rely heavily on simple features. Geirhos et al. (2019) show that trained CNNs rely more image textures rather than image shapes to make predictions. Rahaman al. (2019a)
27

---

Published as a conference paper at ICLR 2025
use Fourier analysis tools and show that deep networks are biased towards learning low- uowfrrquency functions, and (Xu et al., 2019; Cao et al., 2021; Bietti & Mairal, 2019; Basri et al., 2019) provide further theoretical and empirical evidence for this.
Implicit Biases of Gradient Methods. Several works study the implicit bias of gradient- based methods for linear predictors and MLPs. Pioneering work by Soudry et al. (2018); Ji & Telgarsky (2018) revealed that linear models trained with gradient descent to minimize an exponentially-taie loss on linearly separable data converge (in direction) to the max-margin classifier. Following this, Nacson et al. (2019); Ji & Telgarsky (2021); Ji et al. (2021) derived fast convergence rates for tradient-based methods in this setting. Recent works show that MLPs trained with gradient flow/descent converge to KKT point of the corresponding max-margin problem in the parameter space, in both finite (Ji & Telgarsky, 2020; Lyu & Li, 2020) and infinite width (Chizat & Bach, 2020) regimes. Further, Phuong & Lampert (2021); 1); Frei et al. (2022); Kou et al. (2023) have also studied Rel.U//.eaky-Re.. networks trained with gradient descent on nearly orthogonal data. Li al. (2022) show that the training path in wver-parameterize models can be interpreted as mirror descent applied to an alternative objective. In regression problems, when minimizing the mean squared error, the bias manifests in the form of rank minimization (Arora et al., 2019; Li et al., 2021). Additionally, the implicit bias of other optimization algorithms, such as stochastic gradient descent and adaptive methods, has also been explored in various studies (Blanc al., 2020; HaoChen et al., 2021); ); see the recent survey ( (Vardi, 2022) for a detailed summary.
Robustness. Several research efforts have been made investigate the robustness of Transformers. Shao et al. (2021) showed that Transformers exhibit greater resistance to adversarial attacks compared other models. Additionally, Mahmood et al. (2021) highlighted the notably low transferability of adversarial examples between CNNs and ViTs. Subsequent research (Shen et al., 2023; Bhojanapalli et al., 2021; Paul & Chen, 2022) expanded this robustness examination to improve transformer- based language models. Shi et al. (2020) introduced the concept of robustness verification in Transformers. Various robust training methods have been suggested to enhance the robustness guarantees of models, often influenced by or stemming from their respective verification techniques. Shi et al. (2021) expedited the certified robust training process through the use of interval bound propagation. Wang et al. (2021) employed randomized smoothing to train BERT, aiming to maximize its certified robust space. Recent work of Bombari & Mondelli (2024) shows that mandoml--nitializz tend to have higher word-level sensitivity than fully connected layers. In contrast our work, they consider word sensitivity, which has been experimentally shown to be similar for transformers and LSTMs (Bhattamishra al., 2023b).
attention layers
Spurious Correlations. A common pitfall to the generalization of neural networks is the presence of spurious correlations (Sagawa et al., 2020). For example, Geirhos al. (2019) observed that trained CNNs are biased towards textures rather than shapes to make predictions for object recognition tasks. Such biases make NNs vulnerable to adversarial attacks. Gururangan et al. (2018) attribute the reliance of NNs on spurious features to confounding factors in data collection while Shah et al. (2020) attribute it to simplicity bias. Several works have studied the underlying causes of simplicity bias (Chiang, 2021; Nagarajan et al., 2021; Morwani et al., 2023; Huh et al., 2021: Lyu et al., 2021) and multiple methods have been developed to mitigate this bias and improve generalization (Pezeshki et al., 2020; Kirichenko et al., 2022; Vasudeva et al., 2023; Tiwari & Shenoy, 2023).
Data Augmentation. The essence of data augmentation is to impose some notion of regularization. The simplest design of data augmentation dates back to Robbins (1951) where image manipulation. e.g., flip, crop, and rotate, was introduced. Bishop (1995) proved that training with Gaussian noise is equivalent to Tikhonov regularization. We also note this observation is in parallel to our proposition in Section 6 that training with Gaussian noise promotes low sensitivity. Recently, mixup- based augmentation methods have been proposed improve model robustness by merging two images as well as their labels (Zhang et al., 2018). Several works also use a combination of existing augmentation techniques (Cubuk et al., 2019; Lim et al., 2019). A common belief is that data augmentation can improve model robustness (Rebuffi et al., 2021), and this work bridges the method (augmentation) and the outcome (robustness) with an explanation simplicity bias towards low sensitivity.
28

---

Published as a conference paper at ICLR 2025
D DETAILS OF EXPERIMENTAL SETTINGS
We use PyTorch (Paszke et al., 2019) as our code framework and as our implementation of LSTMs. PyTorch is licensed under the Modified BSD license.
Experimental Settings for Synthetic Data Experiments. We use standard SGD training with batch size 100. We consider Definition 3.3. Datasets, Model Architectures and Experimental Settings for Vision Tasks. We consider the following datasets:
and train with 1000 samples and test on 500 samples generated as per
Fashion-MNIST. Fashion-MNIST (Xiao et al., 2017) consists of 28 x 28 grayscale images of Zalando's articles. This is 10-class classification task with 60k training and 10k test images. released under the MIT license.
CIFAR-1O. The CIFAR-10 dataset (Krizhevsky, 2009) is a well-known object recognition dataset. It consists of 32 x 32 color images in 10 classes, with 64 images per class. There are 50k training and 10k test images. It is released under the MIT license.
SVHN. Street View House Numbers (SVHN) (Netzer et al., 2011) is real-world image dataset used as a digit classification benchmark. It contains 32 x 32 RGB images of printed digits (0 to 9) cropped from Google Street View images of house number plates. There are 60k images in the train set and 10k images in the test set. It is released under the CC BY 4.0 license.
ImageNet-lk. The ImageNet-Ik lk dataset (Russakovsky et al., 2015), also known as the ILSVRC (ImageNet Large Scale Visual Recognition Challenge) dataset, a widely used benchmark dataset in computer vision for tasks such as image classification, object detection, and image segmentation. There are 1000 different classes, and approximately 1.28 million training images of size 224 224. It is released under non-commercial research use license.
For all the datasets, we use the ViT-small architecture implementation available at https: //gp::://iire thub. wwwiiddainsvvit--ae
For the ResNet-18 model used in the experiments on CIFAR-10 and SVHN datasets, we use the implementation available at https:/ thub. cww/coo/aaaaaoe /////uuuallll pytorch- ci far. Additionally, for the DenseNet- 121 model, ConvMixer model and ViT-simple model used in the experiments httppppaaaaayooooc...AAAAt..raat and https: pettss:/gght //g thub. com released under the MIT license.
CIFAR-10, use the implementations available at https: wwwssaab/cooiiiiiii
//gi thub. wwwccomm/////eeeeeee
thub. respectively. All of these models
All the models are trained with SGD using batch size 50 for MNIST and 100 for the other datasets. We use patch size 7 for MNIST and 4 for the other datasets. We estimate the expectation over P in Definition 3.1 by replacing every patch with noisy patch 5 times, and sample about 30% of the training data to evaluate sensitivity.
For the MNIST experiments, we consider 1-hidden-layer MLP with 100 hidden units and LeakyReLU activation. We set depth as 2, number of heads as 1 and the hidden units in the MLP as 128 for the ViT. We train both models with a learning rate of 0.01.
For Reshion-MNISTY we set depth as 2, number of heads as 8 and the hidden units in the MLP as 256 for the ViT. We consider 2-hidden layer MLP with 512 and 128 hidden units, respectively. The CNN consists of two 2D convolutional layers with 32 output channels and kernel size 3 followed by a 2D MaxPool layer with both kernel size and stride as 2 and two fully connected layers with 128 hidden units. We use LeakyReLU activation for the CNN. We use learning rates of 0.1 for the MLP with Lea,yReLU, 0.5 for the MLP with sigmoid, 0.005 for the CNN and 0.1 for the VIT.
For CIFAR-10, we set depth as 8, number of heads 32 and the hidden units in the MLP as 256 for the ViT-small model, while these values are set as 6. 16, and 512 for the ViT-simple model. For the ConvMixer model, we use depth 6, embedding dimension 128 and kernel size 3. The learning rate is set as 0.1 for ViT-small, 0.2 for ViT-simple, 0.06 for Mixer, 0.001 for ResNet- -8 and 0.005 for DenseNet-13 121.
29

---

Published as a conference paper at ICLR 2025
For SVHN, most of the settings the same the CIFAR-10 experiments, except set the hidden units in the MLP as 512 for the ViT-small model and the learning rate is set as 0.0015 for ResNet- 18.
For ImageNet-IK: Ik, sample 20k samples from the training set and compare the sensitivity values of pre-trained ConvNext (ConvNextV2- Tiny) and ViT/L-16 with o² 15. Both achieved the same training and validation accuracy of 85%, and it ensures our sensitivity comparison is fair.
Datasets, Model Architectures and Experimental Settings for Language Tasks. We consider the following two binary classification datasets, which are relatively easy to learn without pretraining (Kovaleva et al., 2019).
MRPC. Microsoft Research Paraphrase Corpus (MRPC) (Dolan & Brockett, 2005) is a corpus that consists of 5801 sentence pairs. Each pair is labeled if is a paraphrase or not by human annotators. It has 4076 training examples and 1725 validation examples. It is released under the ODC-By or the Microsoft Research license.
QQP. Quora Question Pairs (QQP) (lyer et al., 2017) dataset is a corpus that consists of over 400k question pairs. Each question pair is annotated with a binary value indicating whether the two questions are paraphrases each other. has 364k training examples and 40k validation examples. It is released under the CC BY-SA 2.5 license.
For both RoBERTa models and LSTM models, we keep the same number of layers: layers. We set number of heads as 8 RoBERTa. We use the AdamW optimizer with a learning rate of 0.0001 and weight decay of 0.0001 for all the tasks. We also use dropout rate of 0.1. We use batch size of 32 for all the experiments. The used RoBERTa model is released on Huggingface http::/wwgggggaaa.ccccccbooooo/p http.////aaaacooo////ooottttote. base with MIT license.
Experimental Settings for Section 6. We set the learning rate as 0.16 and 0.2 when training the ViT-small with regularization and augmentation, respectively. We use a regularization strength of 0.25. The remaining settings are the same as for the other experiments. For computing the sharpness metrics, we approximate the expectation over the Gaussian noise by averaging over 5 repeats and set 0.005.
Experimental Settings for Appendix A.3. For the experiment with the Adam optimizer, we employ learning rate scheduler ensure that the accuracy on the train set is similar the model trained with SGD. The initial learning rate is 0.002 and after every 8 epochs, it is scaled by a factor of 0.5.
For the remaining experiments in this section, we consider the same settings as for the respective main experiment.
Compute Details. Experiments with synthetic data were run on Google Colab. Experiments on vision and language tasks were run on internal clusters using NVIDIA RTX A6000 GPUs with 48GB of VRAM. For the experiments on vision data, we use two GPUs and the runtime for each setting is about 17 hours. Experiments on language tasks use one GPU and the runtime for each experiment is about 24 hours.
E LIMITATIONS
In our theoretical results, we show that transformers exhibit weak spectral bias, similar to other NN architectures. An important direction for future work is to distinguish transformers from other architectures and show that they exhibit a stronger spectral bias.
Additionally, this work focuses on the inductive bias of the transformer architecture. However, other factors such as the data used while pre-training can also effect the biases these models exhibit on downstream tasks. It would be interesting to explore this effect in the future.
Similarly, the choice of the optimization algorithm used for training can also have an effect. In experiments the CIFAR-10 10 dataset (Fig. 11), However, conducting a more thorough comparison, e.g., by considering second-order optimization methods, can be important direction for future work.
that SGD and Adam very similar.
30


Reviews:
[{"review_id": "4RET9VLNgp", "reviewer": "Reviewer_pCqz", "summary": "The paper considers implicit biases for the transformer architecture. They describe sensitivity of a function as the change in the function value averaged over all possible element-wise changes to the function input, averaged or maxed over all inputs on a hypercube. Such functions can be described using polynomials with the sensitivity connected to the degree of the polynomial. The paper proves that a linear attention transformer is biased (in the eigenvalue of the NTK sense) toward low-sensitivity functions characterized by the degree. Then they go on to generalize the notion of sensitivity for neighborhoods of general (non-boolean) inputs.", "questions": "- What's the difference between sensitivity and adversarial/robustness that looks at neighborhoods?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Useful formalization of sensitivity \n- Interesting findings about low sensitivity, robustness, and sensitivity to different parts of the input (like last token in a sequence)\n- variety of tasks for better understanding of where architecture properties come from\n- connecting between grokking and sensitivity provides a new lens into understanding and improving DNN training.", "weaknesses": "- On line 141, \"where the eigenvalues are non-decreasing with the degree of the multi-linear monomials\" would be easier if it said \"eigenvalues do not decrease as the degree increases.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JYulT3zwV4", "reviewer": "Reviewer_A7Md", "summary": "This work builds on the theoretical notion of boolean sensitivity, extending it to an empirically measurable quantity and studying it for the case of transformers. It finds that transformers have lower input sensitivity on the training data, compared to other architectures, and that this is correlated with other phenomena such as test-time robustness, sharpness of minima, and grokking.", "questions": "Have you compared input sensitivity and perturbation robustness at test time? When if ever do they behave differently?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "The study in the paper is quite intriguing. A few things I liked:\n * Provides a new lens on what is different about transformers\n * Demonstrates phenomena consistently across many datasets\n * Provides a new lens on grokking not captured by the weight norm", "weaknesses": "There were a few key places where I felt the paper overclaimed or made dubious claims, which are enough for me to not favor acceptance. In particular:\n * Lower sensitivity leads to robustness: this is basically a restatement of the claim that Gaussian data augmentation improves robustness. This is a very well-known result, the authors do say that it is in line with other results in the literature, but I feel they are understating the extent to which this is well-trodden ground (for instance, Hendrycks, one of the authors of CIFAR-10-C, has published an entire line of work on data augmentation methods for improving robustness; Gaussian noise is the simplest of these and many others work far better).\n * Perhaps more importantly, this sentence does not seem merited: \"Together, these results indicate that the inductive bias of transformers to learn functions of lower sensitivity *explains* the improved robustness (to common corruptions) compared to CNNs.\" I am not sure what \"explains\" means, but there are many other interventions that improve robustness (such as the data augmentation methods mentioned above), and some of those might have better explanatory power.\n * It is not entirely clear whether input sensitivity is a *different* phenomena than test-time robustness to perturbations. The main difference is it is computed on the training set instead of the test set --- but are there cases where these come apart, or is test-time and train-time input sensitivity always highly correlated?\n    * I think the results could be interesting either way -- but if they are the same, then this is interesting mainly because it is a proxy for robustness that can be computed at training time; if they are different, then understanding the differences would be interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "971SiwmUYC", "reviewer": "Reviewer_kdeM", "summary": "The paper explores the inductive biases of transformers, particularly their tendency to learn low-sensitivity functions. It introduces sensitivity as a measure of how model predictions respond to token-wise perturbations in the input. By comparing transformers to other architectures like MLPs, CNNs, ConvMixers, and LSTMs across both vision and language tasks, the paper shows that transformers consistently exhibit lower sensitivity. This low-sensitivity bias is linked to improved robustness, flatter minima in the loss landscape, and hence better generalization. Additionally, the authors propose that sensitivity could act as a progress measure in training, and is linked to grokking.", "questions": "1. Related to weakness 1, how do you think the sensitivity and robustness relate to expressivity and performance in transformers?\n\n2. Lines 310-311 mention generalization capabilities of different models as a reason to investigate sensitivity during training. This made me curious, what do you think the connection between generalizability in representation learning or classification relates to generalizability in sensitivity (I think one direction of it is clear, but the other direction is not)?\n\n3. In line 394, you mention that use the same number of layers for LSTM and RoBERTa for fair comparison. How about the model size in terms of number of parameters? How many parameters in each model? And how do you think changing this could impact your results?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Key strength:** In addition to the general importance of developing rigorous understanding of how transformers work and why they show such remarkable properties, this paper proposes a novel perspective by looking into sensitivity. They rigorously define sensitivity and provide strong arguments on how it links to other important properties, such as robustness and generalization. They also show that it can track progress when grokking happens, which I think is an important finding and could potentially enable a series of future studies on grokking.\n\n**Other strengths:** Here are a list of other points that I commend the authors for:\n- The introduction is quite well written and motivates the main question quite well (thought it could be improved; see weaknesses). Similarly, the contributions are well explained at the end of the introduction.\n- The presentation of the paper is strong, and maintains a good balance between accessibility and rigor.\n- Propositions 2.1. And 2.2 are really interesting results on the spectral bias and sensitivity of transformers. \n- The authors explain the implications of their theory quite well.\n- The experimental design is thorough and well-tailored to validating the theory.\n- While I consider this a theoretical paper, the experiments are quite strong and cover various aspects of the paper’s main questions.", "weaknesses": "I do not see any major weakness. But there could be some improvements. See my suggestions for improvement, below.\n1. While the paper clearly explains that lower sensitivity is linked to higher robustness, trade-off/connection with expressivity and performance are not discussed. There is a well-established trade-off in various contexts (see, e.g., [1-2]), and it would further strengthen the paper to discuss this.\n\n2. Though I think the introduction is quite well-written, I think it under-emphasizes the findings of the paper on the role of sensitivity analysis. The authors conduct a rigorous analysis of the transformers sensitivity and use that to clarifies some of the important properties of transformers as I mentioned for the strengths, but while doing so, they also show, quite rigorously with strong theory and experiments, how sensitivity analysis could be used to understand generalization, grokking, etc. Near the end of the paper this realization caught my attention, and the authors actually do point this out more clearly in the Conclusion, but I think this can be better emphasized in the Introduction.\n\n3. I suggest the authors bring the Limitation section from the appendix to the main paper. The limitations are not discussed in the main paper, while it is always important to discuss them.\n\n4. This is a rather minor point and it might be a matter of taste: Do sections 5 and 6 really need to be separate sections? It seems like the findings are generally similar, and they could be merged in one section of empirical analysis of vision and language models.\n\n\n**References**\n\n[1] Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., & Jordan, M. (2019, May). Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning (pp. 7472-7482). PMLR.\n\n[2] Raghunathan, A., Xie, S. M., Yang, F., Duchi, J., & Liang, P. (2020). Understanding and mitigating the tradeoff between robustness and accuracy. arXiv preprint arXiv:2002.10716.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ChL5DkE378", "reviewer": "Reviewer_eYcM", "summary": "This work studies the sentitivity of functions defined by different deep learning architectures, comparing the specific case of Transformers with CNNs and Mixers. The work stems from previous work tha has studied sensitivity with Boolean inputs, and derives a formulation for token-based models. The authors make a connection between sensitivity and robustness, show how ViTs are less sensitive than other architectures and also show how sensitivity can be used for grokking analysis.\nExperiments on synthetic data are provided, as well as experiments using ViT on small datasets (CIFAR, SVHN, ImageNet) and LLMs on 2 datasets (MRPC and QQP).", "questions": "* Following up on my previous comment, the authors should clarify if the noising procedure is applied on patches (pixels) or token representations. Fig. 1 contradicts the text.\n  * Also, how is noising applied on CNNs?\n\n* How is $\\sigma$ important, and why were different $\\sigma$ chosen for the experiments in Section 4. I personally find it a drawback that one needs to find a right $\\sigma$, and that using different ones the conclusions might change. Also, bold claims are provided with a single (different) sigma per dataset, which raises some questions.\n\n* ViT/LLMs might produce different token scales, but $\\sigma$ is kept fixed. This can impact strongly some tokens and leave others almost noise-less. I find this also a negative point of this algorithm, since some \"topics\" might by-pass the noising.\n\n* How is a single attention representative of a large Transfomer in Section 3.1. I would ask the authors to elaborate on this.\n  * Additionally, why have a linear layer $U$ after another linear layer $W_v$, since the composition of both is already a linear layer?\n\n* In Fig. 4, only the training accuracy is provided. What about the test accuracies? It is known that models achieve perfect train accuracy, but the test accuracy might be very different though. Does the test accuracy correlate with the sensitivity (measured on train data as you already do)?\n\n* About the claim in L347 *_\"This shows that the observations on small-scale models studied in this section transfer to large-scale pretrained models.\"_*. By increasing scale, the sensitivity of a conv model has gone down to 0.0342, which is much lower than 0.0829 for ResNet-18. Also, ViT went up from 0.0014 to 0.0191. It would be fair to conclude that scaling up brings sensitivities closer, which would mean that small-scale does not transfer to large-scale. Also, one could go much larger in scale (larger ViT, larger datasets) and see if the trend is still maintained or sensitivities are even closer. \n\n* Claims in Section 5 are obtained with one Language model (Roberta) and one LSTM on 2 small datasets. I cannot agree with the claims being generic for *Language Tasks* with this setup. Moreover knowing that current LLMs have much different properties than LMs used in 2019 (Roberta).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Originality:**\n\nFocusing on sensitivity starting from the Boolean formulation is original. I also found the experiment on a synthetic vocabulary (3.1) original.\n\n**Clarity:**\n\nThe paper is well written, with clear language. The mathematical notation and formulation is also easy to read.\n\n**Significance:**\n\nThe study of sensitivity in current models is important for interpretability as well as to design better training strategies.", "weaknesses": "**Originality:**\n\nWhile sensitivity study has its originality, many previous works have studied sensitivity in many ways, for example by understanding the effects of image augmentations (see contrastive learning literature).\n\n**Quality:**\n\nThe experiments provided are either synthetic or use small models/datasets. This makes the claims in the paper weaker in my opinion. For example:\n* Results in Section 3 use synthetic data and a single attention layer. I would argue that, while still interesting, these experiments might not transfer to full models with several layers and multiple attention heads.\n  * Related to this experiment, other research has been carried out analyzing spurious correlations. For example, the work by Robert Geirhos (among others) has already shown that CNNs tend to learn from the easiest cues available. In the experiments in Section 3.1, these \"easy\" cues would be the sparse tokens. Once they become uninformative, the next available (but harder) cue are the frequent tokens. \n\n> Geirhos, Robert, et al. \"Shortcut learning in deep neural networks.\" Nature Machine Intelligence 2.11 (2020): 665-673.\n\n> Geirhos, Robert, et al. \"ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.\" arXiv preprint arXiv:1811.12231 (2018).\n\n* Results in Section 4 use small datasets (CIFAR, SVHN) and arguably a medium size dataset nowadays (ImageNet). The models used (Vit-simple/small) are far from real scenarios nowadays, and the compared architectures are also small (3-layer CNN in for example).\n\n* Results in Section 5 use a Roberta model (2019) which does not have the same properties as current LLMs. Also, this model is trained from scratch on small tasks, which also does not transfer to current abilities of LLMs.\n\nIn several cases, bold conclusions are extracted from a single model / single dataset experiment, with which I cannot agree. For example, the claim in L357 *_\"Thus, transformers learn lower sensitivity functions compared to MLPs, ConvMixers, and CNNs\"_* is validated with a 3-layer CNN on a small dataset like SVHN.\n\n**Clarity:**\n\n* It is not clear how the noising strategy is performed. The text mentions that _tokens_ are polluted with noise, however Fig 1 shows the noise applied to the pixel patch and says *_\" the original image and the corrupted image are fed into the same neural network\"_* (which implies that noise is applied at pixel level). The authors should clarify this important aspect.\n\n* It is also not clear how noising is applied to CNNs (which are not patch/token based).\n\n* Proposition 2.1 is harder to parse than the rest of the text, and it is hard to understand why it is important for the paper.\n\n**Significance:**\n\nWhile the objective of the paper is significant, the results provided and the size of the experiments laregly diminish the impact of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)