Elliptical Attention
Authors: Stefan Nielsen
Laziz Abdullaev
Rachel Teo
Tan Minh Nguyen
Venue: NeurIPS 2024 poster (2024)
Keywords: attention
non-parametric kernel regression
robustness
representation collapse
Primary area: deep_learning_architectures
TL;DR: We show that using a Mahalanobis metric within the attention mechanism reduces representation collapse, and improves robustness to contamination.

Abstract: Pairwise dot-product self-attention is key to the success of transformers that achieve state-of-the-art performance across a variety of applications in language and vision. This dot-product self-attention computes attention weights among the input tokens using Euclidean distance, which makes the model prone to representation collapse and vulnerable to contaminated samples. In this paper, we propose using a Mahalanobis distance metric for computing the attention weights to stretch the underlying feature space in directions of high contextual relevance. In particular, we define a hyper-ellipsoidal neighborhood around each query to increase the attention weights of the tokens lying in the contextually important directions. We term this novel class of attention Elliptical Attention. Our Elliptical Attention provides two benefits: 1) reducing representation collapse and 2) enhancing the model's robustness as the Elliptical Attention pays more attention to contextually relevant information, rather than focusing on some small subset of informative features. We empirically demonstrate the advantages of Elliptical Attention over the baseline dot-product attention and state-of-the-art attention methods on various practical tasks, including object classification, image
segmentation, and language modeling across different data modalities.

Elliptical Attention
Stefan K. Nielsen" FPT Software AI Center Ha Noi, Vietnam stefannvkppfptt.om
Laziz U. Abdullaev
Department of Mathematics National University of Singapore Singapore 119077, Singapore laziz abdullaev@u nus edu
Rachel S.Y. Teo Department of Mathematics National University of Singapore Singapore 190077, Singapore rachel. teo@u. nus edu
Tan M. Nguyen
Department of Mathematics National University of Singapore Singapore 119077, Singapore tanmn@nus. edu sg
Abstract
Pairwise dot- dot-product self-attention is key the success transformers that achieve state-of-tte tite-oftthe-rrt art performance across a variety of applications in language and vi- sion. This dot-product self-attention computes attention weights among the input tokens using Euclidean distance, which makes the model prone to representation collapse and vulnerable contaminated samples. In this paper, we propose using a Mahalanobis distance metric for computing the attention weights to stretch the underlying feature space in directions of high contextual relevance. In particular, define hyper-ellipsoidal neighborhood around each query increase the attention weights of the tokens lying in the contextually important directions. We term this novel class of attention Elliptical Attention. Our Elliptical Attention provides two benefits: 1) reducing representation collapse, and 2) enhancing the model's robustness as Elliptical Attention pays more attention to contextually relevant information, rather than focusing on some small subset of informative features. We empirically demonstrate the advantages of Elliptical Attention over the baseline dot-product attention and state-of-the-arr attention methods on vari- practical tasks, including object classification, image segmentation, and lan- guage modeling across different data modalities. The code is publicly available at http///lllllllllatttttttooooo.
https: github......
Introduction
Attention mechanisms and transformers [82] have achieved state of the art performance across wide variety of tasks in machine learning [27, 35, 75] and, in particular, within natural language processing [1, 2, 13, 65, 12], computer vision [16, 39, 78, 66, 62), and reinforcement learning [25, 5]. They have also demonstrated strong performance in knowledge transfer from pretraining tasks to various downstream tasks with weak or no supervision [63, 64, 15]. At the core of these models is the dot- product self-attention mechanism, which learns self-alignment between tokens in input sequence by estimating the relative importance of each token with respect all others. The mechanism then transforms each token into weighted average of the feature representations of the other tokens with weights proportional to the learned importance scores. The relative importance scores capture contextual information among tokens and are key to the success of the transformer architecture [83, 76, 8, 59, 36, 55].
"Equal contribution. Correspondence stefannvkp@fpt.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Imput
DeTT-Ellipical
DelT
DeiT-Elliptical
Figure 1: Comparison of Attention Heatmaps. Elliptical pays attention to more relevant information. DeiT Tocuses just subset of informative features while Elliptical considers a wider set of contextually relevant information, helping to produce more accurate and robust predictions. Attention scores min-max scaled for visualization purposes.
Recent work has begun exploring the connections between self-attention and non-parametric kernel regression [54, 23]. Under this interpretation, there an unknown, underlying function f mapping the tokens in the input sequence the output sequence. The self-attention mechanism estimates f by performing Nadaraya- Watson (NW) regression with isotropic Gaussian kernels. Our work leverages this perspective self-attention, where notice that Gaussian isotropic kernels spherically invariant. This has the drawback of assuming all dimensions of the feature space are equal terms of importance, meaning nearby tokens are assigned contextual relevance weights dependant only on their Euclidean distance from query, regardless of direction. From the non-parametric regression perspective, we show that spherical invariance in the kernel causes the estimator to suffer provably higher variance. This causes two connected disadvantages in the self-attention setting. First, high variance in the estimator impairs robustness as small contaminations in the ipput cause large, erroneous changes in the self-attention output. Second, the high variance of the estimator reduces the capacity of the self-attention mechanism as hidden representations passing through the model are increasingly composed of uninformative noise.
Contribution. In this work, we propose Elliptical Attention, new class of self-attention that constructs hyper-ellipsoidal, rather than hyper-spherical, neighborhoods around the attention queries. The key idea is stretch the neighborhoods around the queries to upweight keys in directions of high importance. We achieve this by computing Mahalanobis transformation that stretches the axes of the underlying feature space according a learned measure of coordinate-uee eeper-ellisooiall neighborhoods following this scheme allows the self-attention mechanism to learn nigher-quality contextual representations that prevent representation collapse while simultaneously exhibiting stronger robustness. We additionally propose an estimator of coordinate- werde wise relevance in the self-attention mechanism that can be computed highly efficiently and with no learnable parameters. We theoretically prove that our estimator accurately estimates the relative coordinate- wise relevance the feature space. Finally, our approach of constructing hyper-ellipsoidal neighborhoods linked to theoretical improvements in the mean squared error (MSE) of non-parametric estimators by reducing variance without introducing bias. We demonstrate that this provable reduction in variance is related to both representation collapse and robustness, proposing unifying framework for both phenomena. This framework is based on the geometry of the predictive neighborhood around queries in the attention mechanism. In summary, our contributions are three-fold:
relevance. Constructing
We develop the novel Elliptical Attention, which learns better contextual representations by constructing hyper-ellipsoidal neighborhoods around queries.
We propose an efficient estimator of the coordinate-wise relevance in the self-attention mech- anism, which requires no learnable parameters, and provide theoretical guarantees for this estimator.
3. We derive theoretical framework unifying representation collapse and robustness in trans- formers based only on the implicit geometry of the attention mechanism.
We empirically demonstrate that 1) Elliptical Attention outperforms baseline self-attention models in terms of accuracy and robustness on a variety of practical benchmarks, including WikiText- 103 language modelling, ImageNet- KK object classification, LRA long sequence modeling, and ADE20K image segmentation, 2) Elliptical Attention attains robust improvements with lower memory require- ments and faster computational speed than baseline robust transformers, and 3) Elliptical Attention can combined with state-of-the-ar robust transformers to further boost robust performance in ImageNet- 1K under adversarial attack.

---

query
relevant key
relevant
X1
X2
Figure 2: Left: The function does not vary in the æ2 axis so stretch the neighborhood in that direction. Right: The stretched ellipsoidal neighhorhood includes more keys.
Organization. We structure this paper follows: attention and non-parametric kernel regression. In Section 3, we illustrate the theoretical benefits of hyper-ellipsoidal neighborhoods, demonstrate how build the required transformation, and provide the full technical formulation of Elliptical Attention. We empirically validate the advantages of the Elliptical Attention in Section 4. Related work is discussed in Section 5 before presenting concluding remarks in Section 6. Proofs, technical details, and further experiments are provided in the Appendix.
In Section 2, present preliminaries self-
2 Background: Self-Attention and Non-Parametric Regression
We first provide preliminaries on the self-attention mechanism followed by background on its connection to the Nadaraya-Waite Watson (NW) estimator in non-parametric regression [48].
2.1
Self-Attention Mechanism
Given an input sequence X -....... END E R.Nxx of feature vectors, the self-attention mechanism transforms the input to H [hi,..., hN]
IR NxD, as follows:
q K3
h = Σ
softmax
Vg for i = 1,..., N.
(1)
SE[N]
The vectors q; kj, and vj are the query, key, and value vectors, respectively. They computed
[4...... .N] = Q = XW E RNDD [k,,..., kN]] K XW E RNND and
[uI...., UN = V = Xw E RND. where Wo, W E DDD,, W Wv R.xDD are the weight matrices. Eqn. can be expressed in matrix form as:
QK
H = sofumax
V,
(2)
where the softmax function applied row-wise to the matrix QK //D. QKT/DD. We refer to transformers built with Eqn. 2 standard transformers just transformers.
2.2 A Non-Parametrie Regression Perspective of Self-Attention
We now present the connection between self self-attention as described in Eqn. 1 and non-parametric regression. We first assume key and value vectors {ky, vj ]E(NN] are obtained from the following data generating process:
v = f(k) E, where e is random zero-mean noise [E{]] - 0, and ʃ the unknown function to be estimated. We consider the random design setting where the keys [kg))E[]) are i.i.d samples drawn from the marginal distribution p(k). We use p(v, k) to denote the joint distribution of pairs (v, k) obtained according to Eqn. 3. At any given new query q. we aim to estimate the unknown function f(q).
The NW estimator is a non-parametric estimator of the unknown f described by
[ R
v v.p(v, k)
f f(k) k[uik]
v e-p(olk)dv
dv,
(4)
RD
p(k)
3

---

where we apply zero-mean noise for the first equality and the definitions of conditional expectation and density for the second and final. Then, can be shown that by estimating the joint density p( v, k) and marginal density p p(k) using isotropic Gaussian kernels with bandwidth and evaluating the NW estimator a new query 4; we obtain
fa( = ΣJEIN) Us exp (-1lq, k2112/203)
(5)
ΣJE[NI exp (-1lq: k;,12/200
EjE(M) Us explq, 1//
Σ
softmax(q, k/ yo//
(6)
ΣJE(N exp(q, k2//22)
JE(M]
where choosing x² vD as the isotropic variance recovers the full attention mechanism. We present the full derivation in Appendix A.
Limitation of sell-attention. We see in Eqn. 5 that standard self-attention computes the relative im- portance scores between queries and keys via Euclidean distance. Euclidean distances are spherically invariant and therefore fail to consider coordinate proximity of k from qi influences its contextual relevance equally regardless of direction.
significance in the feature space, meaning the
Elliptical Attention: Leveraging Hyper-Ellipsoids to Pay More Attention Without Losing Focus
In this section, we first present how NW regression obtains a lower MSE by taking hyper-ellipsoidal neighborhoods around queries. We then construct the required hyper-ellipsoidal transformation via Mahalanobis metric. We present the framework relating robustness and representation collapse to the geometry of the query neighborhoods and show how our proposed scheme offers improvements in both areas. We then provide an efficient estimator of the coordinate-wist relevance before finally giving the full technical formulation of Elliptical Attention. Technical details on the implementation procedure are in Appendix E.
3.1 Improving NW Regression with Hyper-Ellipsoids
Distance-based estimators, such as the NW estimator, can obtain a lower MSE by taking hyper- ellipsoidal neighborhoods around queries [29, 30). The key idea is that we wish to stretch the axes of the underlying space in directions for which the true f in Eqn. 3 varies least.
Figure 2 shows a situation in which ff does not vary equally in all directions. This is actually a limiting case in which the function is sparse in the 22 direction. In the left sub- sub-figure, we show the result of stretching the Euclidean circular neigh-
Token Cosine Similarity
1.0
Dacaline Clppical
0.8
borhoods around each query in the x2 direction for which the
0.6
function does not vary. The right sub-figure then shows how
the resulting ellipse in the 22 direction can include additional data points without adding additional bias into the model. It is a well-established result that the variance of non-parametric estimates at a point is inversely proportional to the number of samples in that point's 'ss neighborhood, as the additional sam- plcs smooth out the effect of noise. As result, stretching the neighborhood, as shown in the right sub- sub-figure, decreases the variance. Crucially, including these additional samples does not cause the estimate to miss the true variation in the function, direction. By including points in this direction, we do not introduce bias into the estimate. Hence, we lower variance without the introduction of bias, obtaining lower MSE estimator. This intuition is formalized in Theorem 1 in Appendix C, which shows that the best achievable rate of convergence for estimators of non-sparse Lipschitz functions is of the order O(n feature space. However, when the function only depends on R C [dl coordinates, the rate improves to On 2/(2+1R)) In the case of approximate sparsity, when coordinate directions exhibit differing variability, the same intuition carries over as shown by the improvement in convergence rates in Theorem 2 in Appendix C.
0.4
0
2
4
8
11
10
12
14
Layer
Figure 3: Representation Collapse on
WikiText- 10Ff Elliptical Attention learns more diverse representations.
as there is variation in the 22
(mm-2/(2+4) -2/2+44) for a d dimensional
We leverage this analysis from non-parametric regression to motivate our Elliptical Attention. From the regression perspective, the self-attention mechanism, which performs NW regression, is able

---

to learn a lower MSE estimator of the true underlying f by reducing the variance of the estimator without (or with minimal) introduction of bias. From the attention perspective, this means queries pay higher attention to more relevant keys, producing more contextually meaningful attention scores and better, more robust learned representations.
3.2 Capturing Coordinate-wiss Variability and Building the Mahalanobis Transformation
We measure the variation in f in the 1th coordinate direction by the expectation the L1 norm of the tth directional derivative taken over all & Nx, where X₂ S R denotes the feature space. Roughly speaking, this quantity corresponds to the average absolute gradient of f in the zt direction throughout the space. Formally, this quantity is defined
Definition 1 (Coordinate-wiss Variability of f : - DD) The coordinate-wise variability of f: - R. with Jacobian matrix , E R RDxD in the ith direction is given by the quantity I == ( i
( E [D), where e, is an all-zero vector with a single I in the ith coordinate and y. the marginal distribution of over support Nx.
Remark This definition one of many possible. One could also take the supremum rather than the expectation or consider second derivatives. We select this definition averages over first derivatives are more easily estimated and the definition still captures the intuitive properties of variability.
Denoting estimates of the coordinate-wic -iise variability I by my we can then incorporate these quantities into distance function of the form
d(g, k) = k) ((q- - k),
where M = diag m1. m2, mp) ) is diagonal matrix whose diagonal elements are the estimates of [l 1 for E [D).
Remark 2 The metric described in Eqn. 7 is a form of Mahalanobis distance metric, which can be interpreted as first applying a transformation to the underlying space in which we stretch the coordinate axes by the diagonal elements of M. Therefore using this metric within the self aelf-attention computation produces the desired elper--elipsooial neighborhoods around queries.
In practice, we maxscale the estimates to obtain m, + -mm/ ma/Mmaa where momas 2 mi for all E [D]. This because we care about the relative magnitudes of the direction-wist wise variability as opposed to the absolute magnitudes. Under this interpretation, we identify the most variable dimension and stretch all others relative this direction.
Remark 3
3.3 Promoting Robustness and Avoiding Representation Collapse
Before providing the technical procedure for estimating M and the full technical formulation of Elliptical Attention in Section 3.5, we first theoretically analyze Propositions and 2 how the hyper-ellipsoidal transformation in Eqn.7 improves robustness and alleviates representation collapse.
Dimension-wist input sensitivity of Elliptical Attention and robustness. In Lemma 1, we show that when each input component is weighted according to the Mahalanobis transformation in Eqn. 7, the impact of perturbing the jh input coordinate on any coordinate of the output is proportional the corresponding weighting parameter with proportionality coefficient depending on the indices i and j.
Lemma Let Le R N denote the transformed Elliptical softmax operator for given of keys as M(x) = ESCIN explx Mkj) [exp(x: T Mk)), exp(x'
T Mkx)] for weight matrix M as in Eqn. 7. Then, the achievable rate of change of M(x) in ith input dimension proportional to my that is, sup=ex |J (x)))) x for all E [D] and j E [N] where J JM is the Jacobian matrix of M.
By virtue of Lemma 1, which is proven in Appendix I. we show in Proposition that choosing the weights properly scaled estimates of the underlying function variability, as in Elliptical Attention, the output vectors become less prone to large errors caused by noisy input while simultaneously respecting the dimension-wisg variability pattern of the true self-attention function.
5

---

Proposition 1 (Robustness of Elliptical Attention) Let f: RR RD be the true self- attentttnnion
function, fa be the Elliptical Attention estimator with metric d as described in Eqn. 7. Then for any index i E [N] and noise e R.D, the following error bound holds
llfala.) falq, + +ell م
Σ JE[N]
Vtr(K33))l;l ||e|l
(8)
where {K3 [Kz]jE[N] are constant diagonal matrices that depend only the key vectors.
Note that when the estimates are maxscaled so that m, < 1, the achievable output error of Elliptical Attention is lower than that of standard self-attention where m, = 1 for all i E [D]. Besides, when the true function exhibits approximate sparisity in some number of dimensions (i.e. m, - 0+ for majority of indices), the eπrоr bound in Eqn. 8 becomes significantly tighter for Elliptical Attention. The proof of Proposition 1 is provided in Appendix B.2.
Input smoothing and representation collapse. In each layer, the standard self-attention mechanism fits a noisy estimate of the true function which is then fed into subsequent layers and iteratively refit. The input to each attention layer is then partially composed of noise, which is cquivalently the common regularization method of random input smoothing. We show that by reducing the noise component in each layer, Elliptical Attention maintains expressive power and resists representation collapse. This is formalized in the following proposition:
Proposition 2 (Elliptical Attention maintains expressive power by reducing noise) Let h1y de- note the output of a transformer using Elliptical Attention with metric d as described in Eqn. 7 and h denote the output of transformer using standard self-attention layer l. Let D be the sampling distribution of the data and let RD. Then, for any h, ha and layer l. in expectation standard self- self-attention transformer attenuates towards c faster than Elliptical Attention. Formally, we have:
Ep hh - cl 2 Ep/h" ccl-
(9)
Proof is provided in Appendix B.3. Proposition 2 shows Elliptical Attention maintains better expressive power than standard self-attention. We find this empirically supported as shown in Fig 3.
3.4 An Efficient Estimator of the Coordinate-wise Variability
We propose simple difference-based estimator that effectively captures the coordinate-wiss vari- ability of the underlying function. Our estimator is easily and efficiently computed. It requires no additional learnable parameters and demands negligible additional memory. Let En denote em- pirical mean over n samples, v' (i) denote the 1t component of the vector at the gth layer, and x.,6+1 = {w )²) of = s(k') + + be the value feature space at neighboring layers & and e+1 where values are generated according to the process described in Eqn. 3. Then, our approach to estimating the 7th coordinate- wise variability is described in the following proposition.
Proposition 3 (Coordinate-wise Variability Estimator) Given a function f RD - RD with ath directional variation [I [1, E (D) and some ō > 0, the directional variation can be estimated by the quantity
10 ( o0(
mi
Em (w 2 1)
(10)
Remark 4 For the purposes of improving the performance of transformers by stretching the feature space according to the direction-vis wise variability of f. we note that consistent estimators of If for all i [D) are sufficient but not necessary. Instead, require only the weaker objective of accurately estimating the relative magnitudes of the direction-wiss variability. That is, if ll 2 11 1. we need only that m 2 my. This is because the theory requires only to identify coordinate directions of more or less variability and shrink or stretch the space accordingly.
The intuition behind our estimator in Eqn. 10 lies in prior lines of research studying transformers as an Euler discretization of continuous-time dynamic, usually as a system of first-order ordinary differential equations (ODEs) [40, 21, 53]. In fact, our estimator resembles the absolute value of a forward Euler discretization of the variability of the 7th component of a value vector over time Ov ()1 where the layers E and f+1 represent consecutive time points in an interval partition with the step size ö. We prove that our estimator in Eqn. 10 effectively estimates the relative magnitudes of the coordinate- wise variability of in Appendix B.5.

---

3.5 Full Technical Formulation of Elliptical Attention
We now present the full formulation of Elliptical Attention. Given the distance function d(s;:) in Eqn. 7, where M =
is a diagonal matrix with elements my as in Prop. 3, the M-norm can be defined as ||=|| M := Vx² Mæ, which produces hyper-ellipsoidal stretching in the feature space. Then, Elliptical Attention is defined as follows.
Definition (Elliptical Attention Computation) Let id RR + R. denote the Gaussian density kernel with variance o² I equipped with the -nnorm as defined above. Then the corresponding NW estimator at qi becomes
fa,D(g:): = Ejely Us exp (-1lg, k; 15lli/200
(11)
EjE(NI exp (- 11,, k; /202)
Then, the Elliptical Attention output for the ith query 9i given keys (ki ( and values {v: N corresponding to the NW estimator (11) with o² = =VD is given by
exp (g MKj///V LAks/VDD.
h4
Σ
softmax(g Mkj/V D)u,, (12)
Σ JEIN exp q, MK.//VDD
JE[N] where M- diag(m1........... mp) with m, defined as Eqn. 10 all E [D].
JEIM]
Eqn. 12 is equivalently expressed in matrix form as
QMK VD
H softmax
V.
(13)
Remark 5 We see from the form of Eqns. 12, 13 that standard self-attention is recoverable by setting M Ip- Under our framework, this implies that standard self-attention assumes the underlying regression function to have exactly equal variability in all coordinate directions.
Pseudocode for the Elliptical Attention computation is provided in Appendix F.12.
Experimental Results
In this section, we empirically justify the advantage of Elliptical Attention over baseline transformers
that take hyper-spheres around queries. We evaluate our method on robust Wikitext- 103 modeling under Word Swap contamination [45], ImageNet classification under a wide range of attacks [14, 67], the LRA benchmark [74], and ADE20K image segmentation [87]. We compare Elliptical Attention with state-of-the-art (SOTA) clean and robust models, including Performer [9], FourierFormer [54], Robust Vision Transformer [44], Fully Attentional Network (FAN) [89], Mixture of Gaussian Keys (MGK) [52]. Mixture-of Expert (MoE) based transformers, such as Switch transformer [18] and Generalist Langauge Model (GLaM) [17], and robust kernel density estimation (KDE) based transformers, such as Median of Means (MoM) and Scaled Projected KDE (SPKDE) [23]. We aim to show that i) Elliptical Attention offers substantive improvements over baseline models across tasks on both clean and contaminated data; ii) Elliptical Attention attains these improvements on contaminated data while reducing memory requirements and increasing computational speed compared to comparative robust models; iii) Elliptical Attention can be combined with SOTA robust transformers to further improve robustness with negligible increase in computational overhead. We compare Elliptical Attention with baselines of the same configuration. Results are averaged over runs with different seeds. Additional results and full details on experimental setup are in Appendix
4.1 Robust Language Modelling
Experimental setup. We adopt the experimental setup in [54, 23]. We pretrain and evaluate our models on the WikiText- 103 benchmark in comparison with the standard baseline Transformer [82], Performer [9], Transformer-MGG. [52], FourierFormer [54], and the robust kernel density estimation- based Transformers including Transforrme-SPKDDE and Transformer-MoM. [23]. All models use the 44M--prameter Transformer backbone. We pretrain all models on clean data for 125 epochs before attacking only the test set using a Word Swap Attack, which substitues random words with generic 'AAA' token a 2.5% swap rate. We report test perplexity (PPL) the performance metric.
7

---

Table I: Perplexity (PPL) on WikiText-103 under Word Swap contamination. Elliptical achieves top PPL in clean data and second best in contaminated. Best result bold and second best underlined.
Model
Clean Test PPL (4)
Contaminated Test PPL (1)
Transformer [82]
34.29
74,56
Performer [9]
33.49
73,48
Transformer-MGG [52]
33.21
71.03
FourierFormmer [54]
32.85
68.33
Transformer-SPK SPKEE [23]
32.18
54.97
Transformer- MoM [23]
34.68
52.14
Transformer-Fliipiien
32.00
52.59
Table 2: Top-1 Top-5 Test accuracy on ImageNet under adversarial attacks PGD, FGSM, and SPSA with perturbation budget 1/255. Best result shown in bold and second best shown underlined.
Clean Data Top
FGSM
PGD
SPSA
Method
Top 5
Top
Top
Top
Top
Top 1
Top
DeiT [78]
72.23
91.13
52.61
82.26
41.84
76.49
48.34
79.36
Distill [78]
74.32
93.72
53.24
84.07
41.72
76.43
49.56
80.14
FourierFormer [54]
73.25
91.66
53.08
83.95
41.34
76.19
48.79
79.57
RVT [44]
74.37
93.89
53.67
84.11
43.39
77.26
51.43
80.98
KDE [23]
72.58
91.34
52.25
81.52
41.38
76.41
48.61
79.68
DeiT-MoM [23]
71.94
91.08
55.76
85.23
43.78
78.85
49.38
80.02
DeiT-Elliptical
72.36
91.33
54.64
85.18
44.96
79.35
56.55
87.26
Table shows our Elliptical Transformer (Elliptical) achieves top test perplexity in clean data while also achieving second top test perplexity under data contamination by Word Swap [47], illustrating that the Elliptical Attention is highly robust and offers substantial advantages on clean data as well.
Results.
4.2 Image Classification under Adversarial Attack
Experimental setup. We adopt the experimental setup in [23]. We train and evaluate Elliptical on ImageNet-IK against standard vision transformers, including DeiT [78] and Distill [78], as well as the FourierFormer [54]. We also compare Elliptical with robust vision transformers, including DeiT-KDE [23]. DeiT-MoM [23], RVT [44], and FAN [89]. The DeiT backbone is the tiny configuration of 5.7M parameters. We train all models on clean ImageNet- K for 300 epochs before evaluating their top-1 1 and top-5 accuracy on the test dataset under fast gradient sign method (FGSM) [22], projected gradient descent (PGD) [42], and simultaneous perturbation stochastic approximation (SPSA) [81]. We also present results for performance against Auto Attack [11], which is an ensemble of auto PGD-Cross Entropy (APGD-CE), auto PGD-targeted (APGD-T), fast adaptive boundary-targeted (FAB-T), and Square. We display results for attacks individually and in default sequential mode.
Results. Table 2 shows Elliptical attains top robustness in PGD and SPSA and second top in FGSM while achieving highly competitive clean accuracy. DeiT DeiT-Elliptical is particularly impressive under black box attack SPSA, improving over the next best model, RVT, [44], by 10%. Table shows results on Auto Attack [11], where we see DeiT-Elliptee in each attack individually and sequentially. We again see strong performance against black box attack Square with 8.5% improvement. When combining with SOTA robust transformer, FAN (89), Elliptical Attention improves robustness to sequential Auto Attack and all individual attacks except FAB-T, for which it still remains highly competitive. This shows Elliptical Attention can further boost robustness when combined with SOTA robust models.
substantially outperforms standard DeiT
4.3 Long Sequence Modelling on the LRA Benchmark
Experimental setup. We adopt the setup in [7]. For each of the 5 tasks, equation calculation (ListOps) [50], review classification (Text) [41], document retrieval (Retrieval) [61], image classification (Image) [32], and image spatial dependencies (Pathfinder) [37], we compare Elliptical with standard Transformer [82], Linformer [26]. Reformer [28), Performer [9], and Longformer [3].
Results. Elliptical Attention achieves top or second top test accuracy in every task and top overall performance. This shows Elliptical Attention learns superior representations across a wide range of modalities in long-range contexts.
8

---

Table 3:
Test accuracy long range tasks: ListOps, Text, Retrieval, Image, and Pathfinder. Best result in bold and second best underlined.
Dataset (seq. length)
Trans. [82]
Lin. [26]
Re. [28]
Per: [9]
Long. [3]
Elliptical
ListOps (2K)
37.1
37.3
19.1
18.8
37.2
37.8
Text (4K)
65.0
55.9
64.9
63.8
64.6
65.6
Retrieval (4K)
79.4
79.4
78.6
78.6
81.0
80.3
Image (IK)
38.2
37.8
43,3
37.1
39.1
40.2
Pathfinder (IK)
74.2
67.6
69.4
69.9
73.0
73.2
Average Accuracy
58.5
55.6
55.1
53.6
59.0
59.4
Table 4: Top-1 and Top-5 Test accuracy on ImageNet under Auto Attack applied both individually and sequen- tially with perturbation budget 1/255. Best result is shown in bold.
DeiT [78]
DeiT-Elliptical
FAN [89]
FAN-Flliptical Top
Method
Top
Top
Top
Top 5
Top
Top
Top
Clean Data
72.23
91.13
72.36
91.33
76.31
93.42
76.38
93.53
APGD-CE
27.75
66.48
31.27
68.28
35.05
74.56
36.13
75.69
APGD-T
27.74
73.37
29.69
74.39
35.02
80.46
36.25
81.30
FAB-T
71.61
90.54
71.74
90.81
76.35
93.65
76.16
93.45
Square
43.55
80.96
47.25
81.65
56.75
88.05
58.38
88.20
Average
42.66
77.84
45.00
78.78
50.79
84.18
51.73
84.66
Sequential Attack
26.08
64.18
27.45
67.77
33.29
74.52
34.54
75.67
Table 5: Switch Transformer I Language Modeling
Table 6: GL.aM Language Modeling
Model
Test PPL (L)
Model
Test PPL (1)
Switch mradson [18]
35.33
GLAM-small [17]
58,27
Switch Elliptical medium
Ellipticalmmedium
34.67
SiMM-Elliticcal-mmal
56.69
Switch Transformer-lare [18]
31.18
aM---ddiimmm [17]
38.27
Switch Elliptical-larg
30.56
(LaM-Ellipticall-melli
36.34
4.4 Image Segmentation ADE20K
Experimental setup. We adopt the setup in [71]. The encoder is pretrained on ImageNet-1K following the same specification described in 4.2. In particular, the encoder is DeiT-tiny backbone of 5.7M parameters pretrained for 300 epochs. After pretraining, we then attach a decoder that contains 2-layer masked transformer and finetune the full encoder-decoder model for 64 epochs on ADE20K [88] image segmentation dataset.
Results. Table 7 reports pixel accuracy, mean accuracy, and mean intersection over union (IOU). Elliptical Attention boosts performance across all metries, with intersection over union, in particular, improving by substantive 4.7%.
4.5 Further Clean Data Language Modelling
Experimental setup. For experiments using Switch Transformer [18] and GLaM [17] backbones. we adopt the setup in [60]. In particular, we integrate Elliptical into small (70M parameters) and medium (220M parameters) GlaM backbones and train the models on WikiText-1033 for 80 and 120 epochs, respectively. We consider Switch backbones at medium (220M parameters) and large (388M parameters) configurations, both trained for 80 epochs. All models use top-2 expert routing. For the standard transformer experiments, continue with the setup of [54] and additionally present results for Elliptical in a medium configuration with 90M parameters trained for 100 epochs.
Results. We present in Tables 5 and 6 the performance of Elliptical in MoE backbones. We see moving from smaller to larger configurations, Elliptical maintains strong. consistent improvements in test PPL. We note particularly substantive improvements with scale in the GLaM backbone, where the small configuration Elliptical attains a 2.7% improvement, but the medium configuration this performance improvement almost doubles to 5.0%. Table 8 further shows that in the standard transformer backbone, Elliptical maintains substantive 6.8% improvement when scaling up to larger configuration. These results show that Elliptical Attention scales well with model size.

---

Table 7:
Image Segmentation Results
Table Wikitext- 103 Results
Model
Pixel Acc.
Avg Acc.
Avg IOU
Model
Test PPL (!)
DeiT [78]
77.93
46.30
35.44
Transformer-snen [82]
34.29
Elliptical
78.46
48.04
37.09
Elliptical-smn small
32.00
Transformer-mediumm [82]
29.60
Elliptical-mediug
27.60
5 Related Work
Theoretical Frameworks for Attention. Attention mechanisms have been studied from range of perspectives. [80] shows that self-attention can be derived from kernel similarity functions, and [77] points out that self-attention projects its query vectors onto the principal component axes of its key matrix in feature space. [56] formulates self-attention
the support vector expansion derived from a support vector regression problem, while [73] explains attention through nonlinear singular value decomposition of asymmetric kernels. Attention has also been explained through ordinary/partial differential equations, Gaussian mixture models, and graph-structured learning [40, 68, 72, 20, 52, 31, 86). [54, 23] show that self-attention performs Nadaraya- Vadaraya-Watson Watson regression with Gaussian isotropic kernels. This paper leverages this viewpoint and proposes modifying the Gaussian isotropic kernels include Mahalanobis metric which can be interpreted as stretching the hyper-spherical neighborhoods the kernel to hyper-ellipsoids.
Robust Transformers. In vision, [43] proposes an ensemble defense strategy to white-box attacks while [44] proposes position-aware attention scal- ing and patch-wise augmentation. Recently, [89] proposes a atlly-aatentional network to attain state-of-the- accu- racy corrupted image data. In language, [85] proposes
GPU Merrry
structurally aware table-text encoding, [38] proposes a ro-
bust end-to-end transformer for crisis detection, and [33]
proposes duration- hard attention. [6, 4] integrate Gaussian process into attention for autto--distribution detection, and [79] develops equivariant neural functional networks for transformers. These methodologies are moti- vated by their respective domain and tend to have limited generalizability to differing domains. Our approach, by contrast, proposes general framework that makes no assumption on the downstream task and requires addi- tional parameters and negligible computational overhead.
Avarage Computation Spead (iterationseeood))
Figure 4: ImageNet Efficiency: Comparison of throughput and max memory allocated for DeiT, Elliptical, RVT, RKDE, MoM on Tiny, Small. and Base sizes. Elliptical the most efficient robust model. Numerical analysis in Table 12 of Appendix F.
Mahalanobis Metrics. Mahalanobis metrics have been used predominantly in classical machine learning algorithms. In nearest-neighhor (NN) classification and regression, [84, 49] learn the metric through backpropagation. In NN KL divergence estimation, [58] learns Mahalanobis metric from density approximation. In kernel regression, [57] takes eigenvalues of the estimated Jacobian while [29, 30] estimate coordinate-wist variability of the true function. Our model similarly uses coordinate- wise variability of the unknown function to form the Mahalanobis transformation but instead uses a more efficient estimator that does not require materializing the prediction function and accommodates the self-attention setting. In general, our method is among the early work in incorporating Mahalanobis metries into the self-attention mechanism.
Conclusion and Future Work
In this paper, we present Elliptical Attention, a novel variant of attention that computes a Mahalanobis transformation to stretch the underlying feature space in directions of high contextual relevance. This transformation can be interpreted as modifying the hyper-spherical neighborhoods around queries to hyper-ellipsoids which upweight the attention paid to keys lying in important directions, enabling the transformer to learn better and more robust representations. This approach makes no assumptions on the downstream task, requires no learnable parameters, and can be applied to any transformer to boost clean and robust performance. A limitation of our work is that we use the values over layers to estimate the average direction- wise gradient of the true self-attention function, which makes the estimate prone to noise. For ongoing work, we are exploring more precise estimation methods with provable convergence guarantees that do not compromise efficiency.
10

---

Acknowledgments and Disclosure of Funding
This research project is supported by the National Research Foundation Singapore under the Al Singapore Programme (AISG Award No: ALSGG-TC-2223--12-S-00 supported by the Ministry of Education, Singapore, under the Academic Research Fund Tier (FY2023) (A-8002040-00-00, A- A-8002039-00-000 This research / project is also supported by the NUS Presidential Young Professorship Award (A-0009807-01-00)
This research / project is
Thanks to our anonymous reviewers, who provided valuable feedback which improved the paper substantially. Thanks also to Thai Ha for the many illuminating conversations.
References
[1] Rami AI-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI conference artificial intelligence, volume 33, pages 3159-3166, 2019.
[2] Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In International Conference on Learning Representations, 2019.
[3] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long- document transformer. arXiv preprint arXiv:2004.0511 055500 2020.
[4] Long Minh Bui, Tho Tran Huu, Duy Dinh, Tan Minh Nguyen, and Trong Nghia Hoang. Revis- iting kernel attention with correlated gaussian process representation. The 40th Conference on Uncertainty in Artificial Intelligence, 2024.
[5] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems,
15097,
2021.
[6] Wenlong Chen and Yingzhen Li. Calibrating transformers via sparse gaussian processes. In The Eleventh International Conference on Learning Representations, 2023.
[7] Yingyi Chen, Qinghua Tao, Francesco Tonin, and Johan Suykens. attention through asymmetric kernel svd in primal representation. Advances in Neural Informa- tion Processing Systems, 36, 2024.
Self-
[8] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 1724 1734, Doha, Qatar, October 2014. Association for Computational Linguistics.
[9] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021.
[10] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1310-1320. PMLR, 09-15 Jun 2019.
[11) Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning. pages
PMLR, 2020.
[12] Zihang Dai. Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-X XL: Attentive language models beyond a fixed-length context. Anna Korho- nen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics.

---

[13] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Uni- versal transformers. In International Conference on Learning Representations, 2019.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[15] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[17] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experre In International Conference on Machine Learning, pages 5547-5569. PMLR, 2022.
[18] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39 2022.
[19] Chris D Frost and Simon G Thompson. Correcting for regression dilution bias: comparison of methods for single predictor variable. Journal of the Royal Statistical Society: Series (Statistics in Society), 163, 2000.
[20] Prasad Gabbur, Manjot Bilkhu, and Javier Movellan. Probabilistic attention for interactive segmentation. Advances in Neural Information Processing Systems, 34:4448-4461 2021.
[21] Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clusters in self-attention dynamics. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[22] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
[23] Xing Han, Tongzheng Ren, Tan Nguyen, Khai Nguyen, Joydeep Ghosh, and Nhat Ho. Designing robust transformers using robust kernel density estimation. Advances in Neural Information Processing Systems, 36, 2024.
[24] Hang Xu Han Shi, JIAHUI GAO, Xiaodan Liang, Zhenguo Li, Lingpeng Kong, Stephen M. Lee, and James Kwok. Revisiting over- emersshoot in BERT from the perspective of graph. In International Conference on Learning Representations, 2022.
[25] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273- 1286, 2021.
[26] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnss: Fast autoregressive transformers with linear attention. In International conference on machine learming, pages 5156-5165. PMLR, 2020.
[27] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s): 2022.
[28] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference Learning Representations, 2020.
12

---

[29] Samory Kpotufe and Abdeslam Boularias. Gradient weights help nonparametric regressors, In Advances Neural Information Processing Systems, volume 25, 2012.
[30] Samory Kpotufe, Abdeslam Boularias, Thomas Schultz, and Kyoungok Kim. Gradients weights improve regression and classification. Journal of Machine Learning Research, 17(22): 1-34, 2016.
[31] Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Pro- cessing Systems, 14421618-21622 2021.
[32] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[33] Naihan Li, Yanqing Liu, Yu Wu, Shujie Liu, Sheng Zhao, and Ming Liu. Robutrans: A robust transformer-basee text-to-speech model. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8228-8223 2020.
[34] Hua Liang, Wolfgang Härdle, and Raymond J. Carroll. Estimation semiparametric partially linear errors-in variables model. Annals of Statistics, 27(5):1519-1535 Oct 1999.
irrri-li-variablls
[35] Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. A/ open, 3:1 11-13,, 2022.
[36] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In International Conference on Learning Representations, 2017.
[37] Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. Advances in neural information processing systems, 31, 2018.
[38] Junhua Liu, Trisha Singhal, Lucienne TM Blessing, Kristin L. Wood, and Kwan Hui Lim. Crisisbert: robust transformer for crisis classification and contextual crisis embedding. In Proceedings of the 32nd ACM conference on hypertext and social media, pages 133-141, 2021.
[39] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[40] Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2019.
[41] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Leariing word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150,
2011.
[42] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
[43] Kaleel Mahmood, Rigel Mahmood, and Marten Van Dijk. On the robustness vision trans- formers to adversarial examples. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7838-7847, 2021.
[44] Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 12042-12051, 2022.
[45] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
13

---

[46] Jeet Mohapatra, Ching Yun Ko, Tsui- Wei Weng, Pin- Yu Chen, Sijia Liu, and Luca Daniel. Higher-order certification for randomized smoothing. In Advances in Neural Information Processing Systems, 2020.
[47] John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjum Qi. TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP. In Qun Liu and David Schlangen, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 119-126, Online, October 2020. Association for Computational Linguistics.
[48] Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications, 11):144 1964.
[49] Youssef Nader, Leon Sixt, and Tim Landgraf. Donr: Differential nearest neighbors regression.
In International Conference on Machine Learning, pages 16296-16317. PMLR, 2022.
[50] Nikita Nangia and Samuel Bowman. ListOps: A diagnostic dataset for latent tree learning. In Silvio Ricardo Cordeiro, Shereen Oraby, Umashanthi Pavalanathan, and Kyeongmin Rim, edi- tors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 92-99, New Orleans, Louisiana, USA, June 2018. Association for Computational Linguistics.
[51] Tam Nguyen, Tan Nguyen, and Richard Baraniuk. Mitigating over-smoothing in transformers via regularized nonlocal functionals. Advances in Neural Information Processing Systems, 04880233-80255 2023.
[52] Tam Minh Nguyen, Tan Minh Nguyen, Dung DD DDLe, Duy Khuong Nguyen, Viet-Anh Tran, Richard Baraniuk, Nhat Ho, and Stanley Osher. Improving transformers with probabilistic attention keys. In International Conference on Machine Learning, pages 16595-16621. PMLR, 2022.
[53] Tam Minh Nguyen, Cesar A Uribe, Tan Minh Nguyen, and Richard Baraniuk. PIDformer: Transformer meets control theory. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 37776- 37797. PMLR, 21-27 Jul 2024.
[54] Tan Nguyen, Minh Pham, Tam Nguyen, Khai Nguyen, Stanley Osher, and Nhat Ho. Fourier- former: Transformer meets generalized fourier integral theorem. Advances in Neural Informa- tion Processing Systems, 35:29319 299395,,-99355, 2022.
[55] Tan Minh Nguyen, Tam Minh Nguyen, Hai Ngoc Do Khai Nguyen, Vishwanath Saragadam, Minh Pham, Nguyen Duy Khuong, Nhat Ho, and Stanley Osher. Improving transformer with an admixture of attention heads. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[56] Tan Minh Nguyen, Tam Minh Nguyen, Nhat Ho, Andrea L. Bertozzi, Richard Baraniuk, and Stanley Osher. A primal-dual framework for transformers and neural networks. In The Eleventh International Conference on Learning Representations, 2023.
[57] Jung-Kyun Noh, Masashi Sugiyama, Kee-Eung Kim, Frank Park, and Daniel D Lee. Generative local metric learning for kernel regression. Advances in neural information processing systems, 30, 2017.
[58] Yung-Kyun Noh, Masashi Sugiyama, Song Liu, Marthinus C Plessis, Frank Chongwoo Park, and Daniel D Lee. Bias reduction and metric learning for nearest-neighhor estimation of kullback-leibler divergence. In Artificial Intelligence and Statistics, pages 669-677. PMLR, 2014.
[59] Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. decomposable attention model for natural language inference. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages Austin, Texas, November 2016. Association for Computational Linguisties.
14

---

[60] Quang Pham. Giang Do, Huy Nguyen, TrungTin Nguyen, Chenghao Liu, Mina Sartipi, Binh T Nguyen, Savitha Ramasamy, Xiaoli Li, Steven Hoi, al. Competesmoe- effeccivve training of sparse mixture of experts via competition. arXiv preprint arXiv:2402.02552, 2024.
[61] Dragomir R Radev, Pradeep Muthukrishnan, Vahed Qazvinian, and Amjad Abu-Jbara. The acl anthology network corpus. Language Resources and Evaluation, 47:919 9944,9944449 2013.
[62] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 8763. PMLR, 2021.
[63] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[64] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 18))9, 2019.
[65] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 21(140):1-66 67, 2020.
[66] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference machine learning, pages 8821-8831, Pmlr, 2021.
[67] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 11::211-252, 2015.
[68] Michael E Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré Sinkformers: Transform- ers with doubly stochastic attention. In International Conference on Artificial Intelligence and Statistics, pages 3515-3530. PMLR, 2022.
[69] H. Sepanski, R. Knickerbocker, and R. Carroll. A semiparametric correction for attenuation. Journal of the American Statistical Association, 89(42)):1366 1860)::366333377 -176,, 1994.
[70] Charles Stone. Optimal global rates of convergence for nonparametric regression. The annals of statistics. pages 1040-1053, 1982.
[71] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7262-7272, 2021.
[72] Binh Tang and David Matteson. Probabilistic transformer for time series analysis. Advances in Neural Information Processing Systems, 34:23592-23608, 2021.
[73] Qinghua Tao, Francesco Tonin, Panagiotis Patrinos, and Johan A. K. Suykens. Nonlinear svd with asymmetric kernels: feature learning and asymmetric nyström method. CORR, abs/2306.07040, 2023.
[74] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena A benchmark for efficient transformers. In International Conference on Learning Representations, 2021.
[75] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1-28. 2022.
[76] Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korhonen, David Traum, and Lluis Marquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy, July 2019. Association for Computational Linguistics.
[77] Rachel Teo and Tan Nguyen. Unveiling the hidden structure of self-attention via kernel principal component analysis. Advances in Neural Information Processing Systems, 2024.
15

---

[78] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa. Alexandre Sablayrolles, and Hervé Jégou. Training bata-efficient image transformers & distillation through attention. In International conference machine learning, pages 10347-10357. PMLR, 2021.
[79] Viet-Hoang Tran, Thieu N Vo, An Nguyen The, Tho Tran Huu, Minh-Khoi Nguyen-Nhat, Thanh Tran, Duy-Tung Pham, and Tan Minh Nguyen. Equivariant neural functional networks for transformers. arXiv preprint arXiv:2410.000 04.009 2024.
[80] Yao- Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippp Morency, and Ruslan Salakhutdinov. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors, Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.P-JCNLPP), pages 4353, Hong Kong, China, November 2019. Association for Computational Linguistics.
[81] Jonathan Uesato, Brendan O' O'donoghue, Pushmeet Kohli, and Aaron Oord. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learning, pages 5025-5034. PMLR, 2018.
[82] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[83] Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in transformer language model. In Tal Linzen, Grzegorz Chrupata, Yonatan Belinkov, and Dieuwke Hupkes, editors, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63-76, Florence, Italy, August 2019. Association for Computational Linguistics.
[84] Kilian Q Weinberger and Lawrence Saul. Distance metric learning for large margin nearest neighbor classification. Journal of machine learning research, 10(2), 2009.
[85] Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. Tableformer: Robust transformer modeling for table-text encoding. arXiv preprint 20030:2203.00274, 2022.
[86] Shaolei Zhang and Yang Feng. Modeling concentrated cross-attention for neural machine translation with Gaussian mixture model. In Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 1401-1411, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
Moens, Xuanjing Huang, Lucia
[87] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
[88] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 127:302-321, 2019.
[89] Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International Conference on Machine Learning, pages 27378-27394 PMLR, 2022.
16

---

Supplement to "Elliptical Attention"
Table of Contents
Full Derivation Self-Attention as Non-Parametric Regression
17
Technical Proofs
18
B.1 Proof of Lemma
18
B.2 Proof of Proposition
19
B.3 Proof of Proposition 2
21
B.4
Eddge-preerrvation Perspective on Representation Collapse
22
B.5 Proof of Proposition 3.
23
B.6 Lipschitz smoothness in (X' d)
25
Additional Theorems
26
A Consistent Estimator
27
E Implementation Procedure and Computational Efficiency
28
Experimental Details and Additional Experiments
28
F.1
Out-of-Distrbbution Robustness and Data Corruption on ImageNet-A,R,C
28
F2
Representation Collapse
28
F3
Head Redundancy
29
F.4
Efficiency Results
29
F5
Elliptical Attention Mixture of Expert Architectures
29
F6
Additional Adversarial Attack Results on DeiT-Small Configuration
30
F.7
Wikitext- 103 Language Modelling and Word Swap Attack
30
F88
ImageNet Image Classification and Adversarial Attack
31
F9
LRA Long Sequence Classification.
31
F.10
ADE20K Image Segmentation
32
F11
Ablation Studies
33
F.12 Pseudocode
33
Broader Impacts
34
A Full Derivation of Self-Attention as Non-Parametric Regression
Recall NW estimator is non-parametric estimator of the unknown any given query described by
/ v p(vlk)du RD
p( J&P p(k)
/
f(k) = E[u]k]
dv,
where the first equality comes from the noise being zero mean, the second equality comes from the definition of conditional expectation and the final equality comes from the definition of conditional density. Eqn. 3 implies that if we can just obtain good estimates of the joint density p(n, k) and marginal density p(k) then we can estimate the required f(g). The Gaussian isotropic kernels with
17

---

bandwidth o are given by
Po (v,
Polu u,)po kz). (k) =
1 N
Σ JEIN]
Po (k kj),
(14)
N
JEIN]
where fo the multivariate Gaussian density function with diagonal covariance matrix o² PI.. Given the kernel density estimators in Eqn. 14, the unknown function can be estimated as
V U. du JED fk( {
EjEin) Pp Uz) k:) du
Do (k)
EJE(N] (0 kij)
EjEIN Pa ki,) Po - Bz)dw EjE[NT 1y40 kj)) ΣJE(N] Pa k;)
EjE[N Po k;))
Then, using the definition of the Gaussian isotropic kernel and evaluating the estimated function q: we have
Ug exp
(- 1q. k2
f(q.)
Σj exp
(-1lq, k,,l2
Σ N
j Dy exp [- |k /20 exp(q. happ(//////"" k,/o²)
Ej exp | (114,111 + |1,,11111// /33))11///))) |0 exp(a, ky)
Σj By exp(q, ko,//////²
Σ
softmax(g) ky /²)
Σn expla k//o²)
JE[N]
Remark 6 Note that relaxing the assumption of normalized keys, the standard unnormalized self- attention score can be written as
expla, k2/0 = exp (-1lg, [t]1(//20²) exp ((1la,11 + +11x,11)/220
exp (-11g; k,,1²/20²),
which shows that the dot- dot-product self-attention scores proportional to the NW kernel value with Euclidean distance. Hence the assumption of key normalization is sufficient to recover exactly the correspondence between self-atlention and NW kernel regression, but not necessary. Analogously, the unnormalized Elliptical Attention score takes the following form:
exp(a MM;//²) -exp (-d(q,, k,) /2////22) exp (CIqgll2u +1kg
x exp (-d(q,, k,)²/
where d(.,-) is the Mahalanobis distance used in Eqn. 7 and || laa is the norm in the transformed space with metric d. This observation justifies the use of the transformed dot product instead of the full Mahalanobis distance metric in Eqn. 12 as it preserves the proportionality relationship between the attention computation and the corresponding nonparametric regression estimator with chosen distance metric.
B Technical Proofs
In this section, we present the omitted theorem statements and technical proof's in the main body of the paper.
B.1 Proof of Lemmal
Let M RDD N be the transformed softmax operator as defined Lemma We wish find its Jacobian matrix given by
a.M. (g)
a.M. (g) Dq" aMalg) 0i"
aMi(g) 8q (Ma(g)
0gg
8MMa(a)
Jm(g)=
0q
8M
(Mx(g)
8Mx(g) 8y
0g
0g
18

---

to measure the sensitivity of each output dimension to a change in each input dimension. Let
M, IRD denote the j"h component of the output vector for j E [N], that is, for vector
q RD
exp(a Mkjj
Mj(g) ΣSENN] exp(q) Mk.)
(16)
Let q' and kj denote the 8th coordinates of vectors q and kj,, respectively. Then,
0
a dq
In(M((g))
q Mk, In Σ explq Mk.)
0q'
+E[M]
EVE(NI Dq exp(q' T Mk.)
-m.k)
ENE(N] exp(a Mk;s)
ki expla Mk.)
mękk
mi Σ
E
EVEIN exp(q MK.,)
SEIN]
kj Σ k.MM(aa)
my
NE[N]
Since the output of Eqn. 16 consists of only positive components, have
a Og
In(M,(q))- My(a)
Oq
mi K/- Σ ALM,.o0) Ma(a))
#EINI
Therefore, the triangle inequality gives
Mi(al
M2(9)
= mi kj Σ k.Ma(a))
0q:
#E[N]
Σ | (MM. (MMMMa)
S m;
M()))
(17)
=E(NIV))
We now bound each term individually. Consider the terms * first. Since 0 - M.(g) S 1, we can bound them as
| ( & [
(18)
Now recall that the inequality ab S (a + b)2 /4 holds for any real numbers and with equality holding at a = b. Therefore, for the first term, we obtain
11 MM < |KG] M2(g) + My(g))²
[g]]
(19)
4
4
Combining inequalities 17, 18 and 19, finally arrive
(m(g)) =
Dq
i Mg S
m,
Σ
kgl
= Ki
(20)
#E(N
for all E [D] and [N], where Kij denotes the coefficient in the bracket.
B.2 Proof of Proposition
Let us estimate the distance between two output vectors of Elliptical attention mechanism correspond- ing clean and contaminated query inputs, namely:
Σ
softmax (q" MM,//0)o, = Σ My
=
JE[N]
JE[N]
h Σ
softmax ((+ +6) Mk /00 Σ M;
JE[N]
JENM
19

---

where M is defined as in Lemma 1. We omit the keys and scaling parameter for convenience since they do not affect the analysis. Then,
h
hell
E - (
JEIN
- - +M.lg++)
jEIN]
Σ
(21)
JEIN]
E
(JM(G))
1
JEIN]
VE(D)
$
Σ
Σ VE(D)
k m ||
(22)
JEIN]
E
tr [ M²
| ||||,
JEIN]
where K, diag(K1j. K......; kD)) and Kij is defined as in Eqn. 20. Note that 21 follows from mean value theorem for some 3 E [0, and q q + BE while 22 follows from Lemma 1. O
It should be noted that Proposition addresses the impact of noise exclusively the query vectors.
However, the resulting bound can extended to account for noise the same technique utilized in the proof. For completeness, also provide the extension. Let M DR x R x x R - R.. be the Elliptical Softmax function defined as
tokens by employing
N
exp(q T Mk1)
Míq, kI,..,,,,N kN)
(23)
Eje(N exp(a Mk;)
exp(g MKN)]
Again, take the difference between output vectors calculated from clean and noisy tokens follows
h Σ M2(a
Eq. k Ek...., KN + Eu)(U; +
(24)
JEI
Σ
(25)
JEIN]
Let ||||| == max || [|e.1]]
denote the noise with the largest norm among query, key and
value noises. Then,
1h, h/l <
[Mç(a + Eg. kn + Ek. AN + Ex) - M, ki,...,,k)lllll kN)ll
JE[M]
Σ
Mila + Ege bs ek...... bN EE)ll
(26)
JEIN
Σ JE[N]
Σ ML,
|| + |||||-
(27)
SE(N] Following the same steps as Lemma 1, one can derive the bound
< m; . la'l (1- 35,z
M₁
oc m:
(28)
ak.
for s,j [N]. Therefore, we obtain
1+ Σ ++ V=(C ||ε||,
h/.
Alll
(29)
JE[N] sE(N+1]
where Cs.y are the diagonal matrices whose elements are the proportionality coefficients in the derived upper bounds.
20

---

B.3 Proof of Proposition 2
There avenues through which resistance to representation collapse. In this section, provide a proof based on noise propagation through layers, which decreases representation capacity representations in deeper layers increasingly composed of uninformative noise. We refer the reader to Appendix B.4 for an additional lens on representation collapse, where we show that Elliptical Attention is more sensitive the variation and local features of the underlying function.
Let the output at layer be denoted as ht, the standard self-attention estimator and Elliptical estimator fitted at layer &. be denoted f and få respectively, where d is the Mahalanobis metric described in Eqn. 7, and f be the true underlying function described in Eqn. 3. By assumption, f is higher variance estimator than fa for any layer. The output for either estimator at layer E can be decomposed into ground truth and noise as follows:
h' = ('(a)= f(q')
(30)
h - fa(a') [(a)))++)(())( +ny
(31)
where of ~ (0, Vy), (0, V.) are the noise components of the estimate at q' and f(q") the ground truth. By assumption of fa being lower variance, V. - is a positive We first require the following Assumption 1, which is described as:
matrix.
Assumption (Random Input Noise Causes Estimator Attenuation) Let f be any estimator of true function f and let the input x p drawn from marginal μ be randomly corrupted by random noise e ~ (0, V) of some unknown distribution and covariance matrix V. Let e be some constant. Then, random input noise atlenuates the estimator as follows:
fl + + - -cll < (=))(()) cll
(32)
Assumption 1 is well-studied phenomenon in parametric regression, often referred to as attenuation bias [69], regression dilution [19], or errors-in-vartes variables [34]. parametric regression, it can be
shown to have an exact form where the estimated gradients of the model are attenuated towards O proportional to the variance of the noise In non-parametrie regression, addition of input noise is often referred to random smoothing or random input smoothing [46, 10), and is well known to be used as regularization technique to introduce bias into the model. In non-parametric models, no exact closed forms exist to express the attenuation bias, but for our purposes we only note the attenuation exists and provide a general form in Assumption I.
The outputs of 30 and 31 then become the inputs to the following layer after being self-added, normal- ized, projected, and linearly transformed. For notational simplicity and because these operations do not change the analysis, we denote the input the next layer as the previous layer output qa g h². We therefore have the following process:
h ++ = fe+ (q f( =
(f(()((())))))))))(()))))))))))) +
(33)
where we see the output h² 8+1 is obtained by fitting fr to input z which is composed of ground truth f(q") and noise ef passed through from the previous layer.
The result then follows directly from the fact that in any given layer, the standard self-attention estimator produces noisier estimates, where that noise is then passed into the subsequent layer as input noise. This is
Bl 1+1
cl = fet1 ( (2) cl EJJ (
+ el + + cll
(34)
EI| ((l
(35)
Ellfa (f(g') + ccl
(36)
(((
c|l,
(37)
where line 35 follows from combining the fact that of is lower variance with Assumption and line 36 follows from the fact that X|| Rs E| Yll when X, Y have the same mean and roughly similar distribution.
21

---

Therefore we obtain at any layer & the following
h - cll S E 1
cl|
(38)
as required. 0
B.4 Edge-preservation Perspective on Representation Collapse
To further substantiate our findings on the mitigation of representation collapse in transformers, we now present an additional proposition that examines this phenomenon from a different perspective. In Proposition 4, we show that Elliptical attention reduces representation collapse by retaining the important local features (bumps etc.) better than the standard self-attention the case of sparse piece-wise constant functions.
Proposition 4 (Representation Collapse) Let A A CRD be a piece-wise constant function with (lA.(g) = f; R2 where A =Uier for some (possibly infinite) index I. Let qı and qn be the queries lying in any of the adjacent domain pieces with distant function values. Then, the Elliptical estimates at these queries retain the distance better than the standard self-attention estimates which is formulated as
El)fa(q2)) fa(q))l 2 f(a))l
(39)
Proof. Assume all output vectors normalized. Then, the Euclidean distance between two vectors determined by their dot product since
||a bl1² ||a||2 + |16| 2a bb
(40)
Without loss of generality, take A₁ and A2 be the two adjacent pieces so that f(a.) f. for i == 1,2. Denote f(q,) = h, and fa(g.) = has- Then, Eqn. 39 is equivalent to proving
ED hh th heal S (E h2),
(41)
where the expectation is taken over the whole sampling distribution D but the points qı E Ai and Q2 E A2 are fixed as described in the definition. We drop the subscript D as this will be the default
distribution for computing expectation unless specified otherwise. Let = cossim(s1, f2) fi f2 be the cosine similarity of the two piece-wise values. By definition of qı and 92 and since estimates work by averaging the output vectors with a small amount of noise, we have < min Ta |/ halj.
We now decompose hid and h2a in terms of components along and orthogonal f1 and f2. respectively:
hid (hlafi) + ft, h2a= =h fzz
(42)
where S. =0. Then, for their dot product, we have
h T h2d
((Tatt) si] ( + f
= ((Tas)) 2 + (hTafsS + (hzaf2))f fi + (f
(43)
The analogous decomposition of h fo2 can be obtained. By Theorem 2 we have that the Elliptical estimator lower variance and we have Ell,; h < Ellf. h This has the following implications:
1. 1 12 ETTTS BiA, f i.e. the component of h;s along f. is larger than that of h, and hence h h is closer to 1.
2. Due to the first implication above, the orthogonal component f becomes smaller in terms of magnitude so that f fi and (f are closer to for Elliptical compared to the standard self-attention.
These two arguments, combined with Eqn. 43, imply that in expectation highta is closer to 1 (fT fz) + = f. f2 = rs which, by definition, is the smallest dot product over S, and hence, rs I he2l < B h2] as desired. D
22

---

B.5 Proof of Proposition 3
The lemma below encapsulates the necessary calculations that will then be used in the following proofs.
Lemma 2 Given normally distributed zero mean random variable & ~ ~N(0, the expectation of random variable obtained by its absolute value is E|cl 2202/
Proof. Since & ~ N(0, g²), by definition of expectation, we have
x²a
E =
exp
dr
2TT02
2
x²
(- x²2
exp
dx
exp
20 (-20²)
dx
(44)
2
20
v V2T02
2
2
1' exp
dx
(45)
2 2
20
2
x²
-02 exp
TO²
20
10
20²
π
where we used the variable change x + (-z) in the first integral of 44 to obtain 45. 0
We derive the bounds for the impact of noise in 3, with respect to its variance, on our estimator 10 in Lemma 3. Henceforth, we omit the factor in Eqn. 10 since it does not affect the further analysis.
Lemma Given that the noise term in 3 follows a normal distribution with zero mean and variance 0², the following inequality
m, - - Se(kk) S
(46)
VT
holds for all E (D), where Sa denotes the 7th component of s(k²) (Ji (2)()) (n(k))
Proof. Since all value vectors are taken from the data generating process we have
m, (w of
E
f+ ua |0
= -] - (kkk²) S(
+ < 6+1 -
(47)
where ef and E, 2+1 denote the 8th components of the noise terms e' and E et+1, respectively. Note
that for real numbers a and b, we have by triangle inequality that |a + bl S |a| + [b] and la +
la - (-b)) 2 |||| - bill |al 1b). Applying these and the linearity of expectation to 47, we
obtain
( - f((ke) Ele - < m; El Sickk) + El cf
(48)
Recall that ef ~ (0, o²) and independent. Now we have that e, - of ~ (0, 20²) as the mean value does not change while variance accumulates when subtracting two zero-mean normal variables. Therefore, the Lemma 2 gives that
2 V
E
f+1
cl
Plugging this back into the inequalities 48, we get
2
2 V o.
(lf.((((()) f((k')
m, S -
which is equivalent to 46 as desired.
Remark 7 Note that in Lemma 3, we may also take into account the possible noise in the value
vectors. Let é.p ~ (N o.) be the noise in the values vectors as m Blo -af + G c,,0l-
s+l
Then, applying the triangle inequality, we obtain
E
2+1 - -,,ll S mi S E2 - -01+ Elett] c/,0l-
Now applying Lemma 2 and Lemma 3, we arrive
m - El (1 - f(k()]
S T
(o + ow).
23

---

Meen Perturbstions 500
Stpe M40 For SeRUE
C.00
0.00
0.7
Eooch
Lager
Figure 5: Left: Evolution of mean values of key perturbations over successive layers. Right: Mean key perturbattions at different layers after 300 epochs. The figures show that as the number of layers increases, mean key perturbations over layers stabilize around constant value.
Proof of Proposition We shall first make the following assumptions on the data generating process
3:
Assumption? The underlying coordinate system in the feature space Ng. is independent, implying that the function f IRD + +R in Eqn. 3 can be separated as f(k) (fi())).... fD(k))
Assumption 3 The noise term in Eqn. 3 has independent components with each component ef following normal distribution (O, f small o, for all j [D] and E N
Assumption The magnitude of each component of key perturbations across consecutive layers, defined as [k[+1 kél, follows distribution with small, layer-independen mean (8) and variance (V)
Remark The assumption of layer- eeyer-independence in Assumption 4, especially for deeper layers, is supported well empirically, as shown in Figure 5. Given the over-smoothing observed in transformers [24], where token representations stabilize after initial few layers. it also practical to assume that key perturbations across layers have relatively small mean and variance when modelled as random process.
Proof. Under the Assumptions 2, 3, 4, we show that I1f 2 11, implies my 2 my with high probability where m, is defined as in (10).
Directly from the Lemma 3, we have
2 m, -
V
Letting a -> O in this inequality, which feasible under the Assumption 3, one can get with a small error that
m, ~ -
(49)
which in turn implies that the impact of the noise in (10) negligible and the error of ignoring them can be controlled by the bounds given by (46). Now according to the theorem statement,
1 2 18f = E (k) 2 J
(k))
0 (k)
E
2 E
Ik:
REID
ME D
E (kk))
LE (k,
(50)
24

---

where we used the separabilityy of as given in Assumption 2 which simplifies the Jacobian matrix as
(f)(k)
05. (A:
0f (k)
0fi
(Kx)
(k,)
8kg
Blg
2k p
@fa(k)
((a (2)
ffak
@fa(ke)
aakka)
ffa (2a
(k(k)= =
0k
8kes
8k
ak
Økn
:
@fo(k)
85p (A)
(fD(k))
(fD(k))
(
8k1
Blg
Bk
Ok:
Bk2
Økp
(11
0
0
(2
0
(kD)
so that [J,(k)] (ka). Using the definition of derivative, the inequality 50 is equivalent to (1 + + f(kg)
E lim (' -T) ((()
E lim r-
(51)
1 (R. - for
Next, we note that for small 6, the limits in (51) can be approximated with 8 E {i,s}:
(K + 5) (((1)) ( + 5) f((f))
(52)
8
Let us choose 5 E|| Ml Then, by Chebyshev's 's inequality, we have for any that 22
82 S PP (11k5t 6 5
= P (8 - S. 1 K- < + =)
(53)
Given that the variance is sufficiently small as the Assumption 4, the inequality (53) implies that k(t1 se ke = 8 with high probability. Therefore, it follows from (52) with high probability that (K( -
(I - ((((
8
which, due to 49, equivalent to m, 2 my as desired.
B.6 Lipschitz smoothness in (X
Below we show how Lipschitz smoothness of f changes when moving from Euclidean to the Mahalanobis transformed space. We shall follow similar steps to [29] and [30] but for more general class of functions.
Proposition 5 (Change in Lipschitz smoothness for f) Suppose there exists a positive constant G3 such that ((k))l v for any E Xx and m, > 0 for all i (D). Then for any q, k. E Xx. the following inequality holds:
(1((a) - s(k)]
Σ (EID]
G m
d(q, k).
Proof. Let w = 9 g qll
denote the unit vector pointing from to q. The fundamental theorem of
calculus implies that
l1g- d f(g) f(k)
|14 k
dt [
tw) dt
w J tw) dt.
where , is the Jacobian matrix of f as usual. Starting with the distance between outputs f(q) and flk) we have
4 kll
11--kl -ll
(If(a) s(k)]l
TTJ ( (w) dt < 0
Σ
W: Vilk + (w) di
kE(D)
di < Σ Giwwxl / plla-bil dt
Will |V + tw)]
(E(D]
xE[D]
Σ Gilgs kgb
(54)
1E[D
25

---

where, as for all other vectors, denotes the 1th component of vector q. Now note that
mi (g) kg)²
qq k)
Míq k) d(q. k)
1g. ki - (q:
k;
++4
(55)
m
m4
mi
jzt
Combining 54 and 55, we finally attain
G. Vm
(I((g) - s(k)]
Σ iE(D)
dl k),
(56)
which completes the proof.
C Additional Theorems
The following Theorem is classic result from [70]. We refer the reader to their work for details.
Theorem (Minimax rate for functions of bounded variability [70]) Let Fx denote the class of distributions Px,y on X' x [0, 1] such that Vi E [d], the directional derivates of f(x) := == X = 2]
S X. Then for any f E Fx, estimator f. sample size n E N. there exists a č independent of n satisfying
satisfy [f;] lsup == sup x, filg))
inf sup R 11 f1| 2 222 (dx)24/244)) - 2/(2+d)
(57)
fn fet, X
Theorem (Improvement in MSE for approximately sparse functions [30]) Let the norm of the largest gradient be À suPre[d] fi) lluu and fa be an estimator in metric space (Ny, d) where d is defined as Eqn. Then,
E E SIB < inf sup BJJ 512
(58)
f Fx
Proof. We provide an abridged proof for completeness. We refer the reader to [30] for the full details.
First, the full bound is described as follows:
Ellfa - ffl1 v 202/2+T (CD)) n2 inf sup Elf - fll.
(59)
Fb
where d(x²) the d-diameter of X' defined as supa, d(x, x'). ) RC [D), S Cun S ('((nn)) RI. C' and C1 are universal constants and da 2 sup; fl /lspp /m,. Let
R D D DD R log(d())/eR)) log(1/c)
if 2 / d()(()) if < n d(x))
r(c) S
For bandwidth 6x r(ex) and let [R]| S r S D. Let > 0, ē be defined as the same ē in Theorem and E N, define the function Wnnd = re-rle) /nn////nnnnn and vs (e)(e) CieeD/n where G =((A/Cxd(xx))) Also define <(c) = D2
For any fixed n, let e n.d be solution to 3/ (,, = Φ(t). Solving for e, ndd obtains the following lower bound on the minmax rate of
20(e = 222 221222((22))))))) 2/(2++d)
(60)
For any n N there exists solution c,.a to the equation vn,a(c) = ((c) since r(c) is nondecreasing. Therefore it is possible to obtain the following:
Ex",y" Il fll2 < 20(En.d).
(61)
Since Φ is independent of and both an.d and sb of are strictly decreasing functions of n, we have that En and n, both tend to 0 + Therefore can define no such that, for all n 2 no. both En and m, are less than ex/ (x))
Thus, Vn 2 no. we have En.d a if, for all 0 < < ex/d(X)) wn.a(e) < (nn((),, which completes the proof D.
26

---

D A Consistent Estimator
In this section, present consistent centered lifference-basee quotient estimator of the coordinate- wise variability obtained by perturbing the estimated function in the jth direction and measuring the In norm of the difference. Similarly, this estimator requires no learnable parameters gradients. The estimator is described in the following proposition.
RDD with ith directional varia- tion | E (D), the directional variation can be estimated by the quantity
Proposition 6 (Consistent Estimator) Given function RDD
[ te:) - f(k le:)
mi
:= Er
(62)
2t
where t is a hyperparameter controlling the degree of locality of the estimator and E, denotes the empirical expectation for samples.
Despite m, in proposition 6's simple formulation, it is nonetheless a consistent estimator of the coordinate- wise variation in the underlying function. We utilize simplified version of a theorem from [30], adapted to suit our specific needs, as the original formulation is more detailed than necessary for our purposes.
Theorem (Consistency of Centered Difference-base -bsed Estimator for Scalar Function [30] Let ip IR + R. be smooth scalar function and || == Ex=yle, Vip be the coordinate-wis variability for that scalar function. Then, for any direction and any O < < 1/2, the following bound holds with probability of at least 1 26:
(x + le;) ((x le:))
11 S snn (2
(63)
2t
Note that the Theorem 3 is different from our setting by studying scalar function as opposed to a vector valued function. However, we show that the result can be generalized to the latter case in Corollary below via the estimator 62.
Corollary (Consistency of the Estimator (62) for Vector-valued Function) Let f D
Dr be a vector valued function and Ifill. be defined as in Definition 1. Then, for any direction i and any 8 1/2, the following bound holds with probability of at least 25:
mm
|5 S O000nnn2-2100 InDD/5))
(64)
Proof. We first derive the relation between the left hand side of 64 and its coordinate-wise differences as follows:
[I + te:)
-j(k- te:i)l1]
m
E,
Exong [kJs(k))slli)
2t
[k + (e:))
- [K,(k= te;))
E,,
Exaa
Σ le
V
(65)
2t
jEID]
jE(D)
(k + te,)
Fy - te;))
En
E le VS5l
(66)
28
EED
SE(D)
Σ SEID
m
-1171111))
(definition of m, and 11 i for components fy)
Σ
mi
(
(triangle inequality)
JE(D]
< On where line 65 follows from the definition of the & norm, line 66 follows from the linearity of expectation, the superscript j indicates that the case reduced to the scalar function case for each jth summand individually. Note that the probability of the last bound is at least (1 25 DDD since each
(Theorem 3)
component-ve wise bound holds with probability least - 25/D. However, since we can choose small enough such that 28 < 1, by Bernuull's 's incquality (1 - D) 2 1 - 2D8/D 1 - 26. D
27

---

Table 9: Evaluation of the performance of our model and DeiTT across multiple robustness benchmarks, using
appropriate evaluation metrics for each.
Dataset Metric
ImageNet-R Top-1
ImageNet-A Top-1
ImageNet-C mCE (1)
ImageNet-C C (Extra)
mCE (L)
DeiT Deit-Elliptiall
32.22
7.33
72.21
63.68
32.66
7.63
73.59
65.71
Despite the proven consistency of this estimator, we opt for the efficient estimator pre- sented in our main body described in Eqn 10. This because the consistent estimator requires materialising the prediction function - that is, computing a forward pass of the reif-attention mecha- - per dimension. This makes the consistent estimator unusable in most problem setlings. We present results for the consistent estimator in Appendix F.H..
Remark 9
E Implementation Procedure and Computational Efficiency
Training and Inference. Given Elliptical Attention requires keys and values from the previous layer in order to compute the required transformation, we can only implement Elliptical Attention from the second layer on. We incorporate our Elliptical Attention into both training and inference stages. This is because, firstly, Elliptical Attention is designed to offer improvements to both clean and contaminated data, and even in the presence of completely clean train and test data, it is advantageous to incorporate Elliptical Attention into both stages. Secondly, it is commonplace to encounter data contamination in test data and indeed also highly possible encounter in train data as well. Therefore, in the interest of robustness as well, we also incorporate Elliptical Attention into both stages.
Computational Efficiency. Computing the required transformation requires no learnable parame- ters and is obtained simply by averaging absolute differences in values over layers. These operations are therefore just of the order ObbhnD) (n) ) for batch size b, head number h, key/value length n, and dimension D. Hence upper-bound time complexity of the overall Transformer unaffected. We provide efficiency analysis in terms of computation speed and max GPU memory allocated (cal- culated by CUDA max_memoryy www.mmmmmmlllateetteeeaaee allocateedd in Figure 4, which shows that compared with baseline robust models, Elliptical is the fastest and most memory efficient. Elliptical exhibits no perceptible slowdown versus DeiT of the same configuration and only 0.99% increase in max memory allocated, which is why Elliptical and DeiT are shown as the same data point in the Figure 4.
F Experimental Details and Additional Experiments
F11 Out-of-Distribution Robustness and Data Corruption on ImageNet-A,R,C
ImageNet-A,R,C are benchmarks capturing a range of out-of-distribution and corrupted samples. ImageNet-A contains real world adversarially filtered images that fool current ImageNet classi- fiers. ImageNet-R contains various artistic renditions of object classes from the original ImageNet. ImageNet-C consists of 15 types of algorithmically generated corruptions with 5 levels of severity (e.g blurring, pixelation, speckle noise etc). Given that Elliptical Attention learns attention weights dependant on the transformation M, which itself dependant on the train data distribution, our proposed model is not designed for situations in which the test distribution is substantially different from the train distribution. This then includes OOD robustness and robustness to heavy corruption to the point where the underlying data distribution is fundamentally different. We nonetheless evaluate Elliptical Attention on ImageNet-A.RR, A.RR to assess these important forms of robustness as well. Table 9 shows that Elliptical Attention is still able to offer improvements over baseline DeïT' in terms of OOD robustness, while maintaining approximately the same performance as the baseline for ImtgeNet-C. Figure shows for Fog and Pixelate corruptions how Elliptical compares with DeiT over the 5 severity levels, where we see that at low severity levels Elliptical improves over DeiT, however the severity level gets too high Elliptical falls behind. This agrees with our expectation that as the severity level grows, the distribution is further shifted relative to the train distribution and so Elliptical Attention is unable improve performance.
F.2 Representation Collapse
We provide in Figure additional representation collapse results for ImageNet and ADE20K, showing that across modalities Elliptical Attention resists representation collapse.
28

---

ADE20K megnetttooon
Wikitext-
ImageNet Recognition
Cosine Similarity
Transformrr Baseline
1.0
Cliptical
0.7
0.6
0.5
0.7
10
2
10
12
30
Layer
Layer
Layer
Figure 6: Additional Representation Collapse Results ADE20K, WikiText-103 and ImageNet. Elliptical reduces token similarity over layers across a range of modalities
Table Additional Results on Imagenet Increasing Heads Maintaining Overall Embedding Dimension
Model
Num. Heads
Head Dim.
#Params.
Top-1 Accuracy
Top-5 Accuracy
DeiT
3
64
5M
72.23
91.13
Elliptical
3
64
5M
72.36
91.33
DeiT-6head
6
32
5M
72.34
91.22
Elliptical -hhaddcal-Ohead 6head
6
32
5M
73.00
91.77
F.3 Head Redundancy
We present Table 18 head redundancy results on the two large-scale tasks, WikiText-100 language modelling and ImageNet- K object classification. Mean La distance between vectorized attention heads, with the mean taken over a batch size 1000 and averaged layer-wise. see that Elliptical improves head redundancy on WikiText--O0 103 versus the baseline transformer while performing approxiamtely equally to the DeiT baseline on ImageNet.
Top-1 Accuracies at Different, Severity Levels (Fog)
Top-1 Accuracies at Different Severity Levels (Pixelate)
00
DelT Eliptical
DelT DelpEliptcll
35
Accurcy
SECHICCY
35
2.0
3.0
Severity
Severity
Figure Comparison Dei7 versus Deit Deit-Elliptical accuracies two types of compptoos, namely, Fog (left) and Pixclate (right). The figures show two out of many cases where DeiT- DeiT- Elliptica outperforms its counterpart while vanilla DeīT manages to exceed only at higher severity levels.
E.4 Efficiency Results
We present here the comparative efficiency results for DeiT and DeiT-Elliptica in side-by-side comparison at tiny, small, and base sizes, along with DeiT-Elliptical compared with other robust baselines.
F.5 Elliptical Attention in Mixture of Expert Architectures
We additionally evaluate Elliptical Attention within Mixture of Expert architectures. Specifically, we show in Tables 13, 14, and 15 the performance of Elliptical Attention within the Switch Transformer [18] and Generalized Language Model (GLaM) backbones [17].
29

---

Table 11: Side-by-side Efficiency comparison of DeiT and DeiT-Elliptica
Model
Compute Speed (ivs)
Max Memory (K) FLOPs / sample
#Params (M)
Timy
DeiT DeiT-Elliptical
0.347
2.08
1.77
5.7
0.342
2.12
1.79
5.7
% Change
-1.44%
1.92%
1.12%
Small
DeiT DeiT-Elliptical
0.297
4.89
6.91
22.1
0.289
4.96
6.99
22.1
% Change
-2.69%
1.43%
1.16%
Base
DeiT Deit-Elllipiicn
0.132
10.27
26.37
86.6
0.130
10.54
26.63
86.6
% Change -1.52%
2.63%
0.98%
Avg. % Change 1.88%
1.99%
1.09%
Table 12: Efficiency Comparison between Elliptical and baseline robust models
Model
Compute Speed (it/s)
Max Memory (K)
FLOPs/ / sample
#Params (M)
DeiT-MoM
0.331
2.24
1.74
5.7
DeiT-RKDE
0.271
3.12
1.77
5.7
DeiT-SPKDE
0.168
3.35
1.75
5.7
DeiT-RVT 0,292
3.91
1.89
7.1
DeiT-Elliptical
0.342
2.12
1.79
5.7
F.6 Additional Adversarial Attack Results on DeiT-Small Configuration
We present here additional results for DeiT and DeiT- at the Small configuration [78] (22.1M parameters) under adversarial attack. Table 16 shows the result of Elliptical against PGD, FGSM, and SPSA. Table 17 shows the results of Elliptical against Auto Attack. Given the larger model size, we attack with perturbation budget of 3/255.
F.7 Wikitext-100 103 Language Modelling and Word Swap Attack
Dataset. The WikiText- 1032 dataset contains around 268K words and its training set consists of about 28K articles with 103M tokens. This corresponds to text blocks of about 3600 words. The validation set and test sets consist of articles with 218K and 246K tokens respectively.
Corruption. Word Swap Text Attack² corrupts the data by substituting random words with generic token 'AAA'. We follow the setup of [23] and assess models by training them on clean data before attacking only the evaluation set using substitution rate of 2.5%.
Model, Optimizer & Train Specification. We adopt the training regime of [54]. To this end, the small backbone uses 16 layers, 8 heads of dimension 16, a feedforward layer of size 2048 and an embedding dimension of 128. We dropout rate of 0.1. 1. We trained with Adam using starting learning rate of 0.00025 and cosine scheduling under default PyTorch settings. We used a batch size of 96 and trained for 120 epochs and 2000 warmup steps. The train and evaluation target lengths were set to 256.
The medium backbone uses 16 layers, 8 heads of dimension 32, feedforward layer of size 2048 and embedding dimension of 256. We use a dropout rate 0.1. We trained with Adam using a starting
www..slesforre.com/. cttp:p/leeeeeeeeooiieeeeeeeeeeed
Implementation available gibhub.ccmm/QDaaa//exxtttttee
30

---

Elliptical layer all possible layers 2 through 12. We use Table 13: Elliptical Switch Transformers Pretrained WikiText-103 and Finetuned on Stanford Sentiment
Treebank (SST-2)
Model
Test PPL (J)
Finetune Test Acc. (1)
Switch Transformer- mediuum
35.33
76.27
Switch Elliptical- -eediu
34.67
77.32
Switch Transformer-large Switch Elliptical -aarec
31.18
76.79
30.56
78.08
Table 14; Elliptical Switch Transformers Pretrained on EnWik8 and Finetuned on Stanford Sentiment Treebank 2 (SST-2)
Model
Test BPC (4)
Finetune Test Acc. (1)
Switch Transformer
1.153
63.27
Switch Elliptical
1.142
67.75
learning rate 0.00025 and cosine scheduling under default PyTorch settings. We used a batch size of 56 and trained for 100 epochs and 2000 warmup steps. The train and evaluation target lengths were set to 384.
For Elliptical Attention, we use an Elliptical layer all possible layers 2 through 16. We use constant delta 1.
Compute Resources. GPUs.
All models are trained and evaluated two NVIDIA A100 SXM4 40GB
F88 ImageNet Image Classification and Adversarial Attack
Dataset. We use the full ImageNet dataset that contains 1.28M training images and 50K validation images. The model learns to predict the class of the input image among 1000 categories. We report the top-1 and top-5 accuracy on all experiments.
Corruption. We use attacks FGSM [22), PGD [42], and Auto Attack [11] with perturbation budget 1/255 while SPSA [81] uses perturbation budget 0.1. All attacks perturb under 1 norm. PGD attack uses 20 steps with step size of 0.15.
Model, Optimizer & Train Specification. The configuration follows the default DeiT tiny con- figuration [78]. In particular,
follow the experimental setup of [23, 54]. To this end, the DeiT backbone uses 12 layers, 3 heads of dimension 64, patch size 16, feedforward layer of size 768 and embedding dimension of 192. We train using Adam with a starting learning rate of 0.0005 using cosine scheduling under default PyTorch settings, momentum of 0.9, batch size of 256, 5 warmup epochs starting from 0.000001 and 10 cooldown epochs, for an overall train run of 300 epochs. The input size is 224 and we follow the default AutoAugment policy and color jitter 0.4.
For Elliptical Attention,
constant delta 1.
Compute Resources. We train and evaluate all models four NVIDIA A100 SXM4 40GB GPUs, with the exception of the robustness experiments on ImageNet-C which are conducted using four NVIDIA Tesla V100 SXM2 32GB GPUs.
F99
LRA Long Sequence Classification.
Dataset. The LRA benchmark consists tasks involving long range contexts of up to 4000 in sequence length. These tasks consist of equation calculation (ListOps) [50], review classification (Text) [41], document retrieval (Retrieval) [61], image classification (Image) [32] and image spatial dependencies (Pathfinder) [37].
Model, Optimizer & Train Specification. We adopt the same experimental setup as [7]. To that end, the Transformer backbone is set with 2 layers, hidden dimension of 128, attention heads
31

---

Table 15: Test Perplexity of Elliptical GLaM WikiTexxt1033 Modeling
Model
Test PPL
GLAM-sman
58.27
GLAM GLAM-Ellipticcll Elliptical-smaat
56.69
GLAM-medium
38.27
GLAM--Ellipiical-meddi
36.34
Table 16: DeiT and DeiT-Elliptical Accuracy ImageNet Under Adversarial Attacks PGD, FGSM, and SPSA with Small Backbone Configuration
Clean Data
PGD
FGSM
SPSA
Method
Top 1
Top 5
Top 1
Top 5
Top 1
Top 5
Top
Top 5
Deil'small 79.89
95.04 95.06
21.41 22.39
51.50
51.57
82.12
65.68
91.28 92.45
Elliptical-smal
79.92
54.02
51.86
82.87
72.02
Table 17: DeiT and DeiT-Elliptical Accuracy ImageNet under Auto Attack with Small Backbone Configura-
tion
Clean Data Top
APGD-CE
APGD-T
FAB-T
Square
Method
Top 5
Top
Top 5
Top
Top 5
Top 1
Top 5
Top 1
Top
DeiT-small
79.89 79.92
95.04 95.06
19.18 18.88
50.75
16.54 17.30
63.84 65.28
80.66 81.64
95.09
49.98 55.89
89.17
Elliptical-small
51.07
95.59
89.36
Table 18: Head Redundancy Results
Model
Num. Heads
Dim. Head
Distance
WikiText-103
Trunsformer-Smin Ellipticul-Smil -malll
8
16
5.40 1: 2.21
8
16
6.45 1: 2.38
ImageNet
DeiT Elliptical
3
5.11 + 1.67
3
4.98 + 1.54
of dimension 32, and embedding dimension of 64. We dropout rate of 0.1. Built on top of the standard transformer backbone, Reformer uses 2 hashes, Performer has 256 random feature dimensions and Linformer rate 0.0001 with linear decay. We use a batch size of 32 for ListOps, Retrieval, and Text and 256 for Image and Pathfinder32. We 1000, 175, 312, 800, and 1000 warmup steps for ListOps, Image, Pathfinder32, Retrieval, and Text respectively.
a projection dimension of 256. We train with Adam using learning
Elliptical places the Elliptical Attention layer on the final layer (as the only one possible) and uses delta equal to I.
Compute Resources. GPU.
All models trained and evaluated single NVIDIA A100 SXM4 40GB
F.10 ADE20K Image Segmentation
Dataset.
ADE20K [88] contains challenging scenes with fine-graine labels and one of the most challenging semantic segmentation datasets. The training set contains 20,210 images with 150 semantic classes. The validation and test contain 2,000 and 3,352 images respectively.
Model, Optimizer & Train Specification. We follow the experimental setup as in [71]. The encoder is pretrained on ImageNet following the specification described in F.8 using the setup in [78, 54]. That is, the encoder is DeiT backbone using 12 layers, 3 heads of dimension 64, patch
32

---

Table 19: Perplexity (PPL) of Elliptical and baselines WikiText-101 under Word Swap data contamination. Best results are bold. Our Elliptical method achieve substantially better robust PPL without compromising performance on clean data.
Model
Clean Test PPL (L)
Contaminated Test PPL (4)
Transformer
34.29
74,56
Performer
33.49
73,48
Transformer-MGK
33.21
71.03
FourierFormer
32.85
68.33
Transformer-SPKI -PEREE
32.18
54.97
Transformer-Mol
34.68
52.14
Elliptical
32.00
52.59
Random Ablation
37.84
46.82
Consistent Elliptcal-Consistentt
32.95
54.67
Elliptical-eanncatee Meanscale
31.94
52.78
size 16, feedforward layer of size 768 and embedding dimension of 192. We train using Adam with starting learning rate of 0.0005 using cosine scheduling under default PyTorch settings, momentum of 0.9, batch size of 256, 5 warmup epochs starting from 0.000001 and 10 cooldown epochs, for an overall train run of 300 epochs. The input size is 224 and we follow the default Auto-uggent policy and color jitter 0.4. After pretraining the encoder, we then attach as decoder a masked transformer consisting of 2 layers. Each layer contains heads of dimension 64, embedding dimension of 192 and feedforward dimension of 768. The decoder uses a dropout rate of 0.1. The full segmenter (encoder and decoder) is then finetuned using SGD with starting learning rate 0.001 and polynomial scheduling. The batch size is set to 8.
Compute Resources. All models are trained and evaluated on single NVIDIA A A100 SXM4 40GB GPU.
F.11
Ablation Studies
Ablation Models.
We consider the following models in our ablation studies:
Random Ablation. To validate the efficacy of our proposed estimator given in Eqn. 10, consider an alternate model in which M is populated by weights uniformly drawn from the [0, 1] interval followed by the same maxscaling as in Elliptical.
Alliptical-Meanscaale We ablate the effect of maxscaling by considering meanscaling of the estimates m,- That is, each m, + m,mn/ m scaled by the mean variability estimate m Ep [m.].
Elliptical-Consisten Consistent. We consider also the performance of Elliptical when using the consistent estimator of [[ described by Equation 62.
Language Modelling. Results are shown in Table 19. Amazingly, the random ablation model performs extremely well on contaminated data. In general, this most likely suggests that training model with randomness injected into the attention matrix can generate some robustness benefits, which intuitive. It does, less surprisingly, come at the cost of clean data performance, where Random Ablation performs almost 10% worse than baseline transformer.
F.12 Pseudocode
Algorithm 1 presents a pseudocode for implementing Elliptical Attention as given by Eqn. 13 on top of conventional self-attention.
ImageNet Classification and Attack. Table 21 shows the ablation model's performance on both clean ImageNet and under Auto Attack. The ablation model shows slight improvement the DeiT baseline in Top 1 accuracy, however Top 5 accuracy is substantially lower. Reasonable performance again Auto Attack is overall unsurprising given that the random Random Ablation model essentially employing random defence. Nonetheless, it still does not surpass the performance of Elliptical.
33

---

Algorithm Computation of Elliptical Attention
Require:
1: Tensor Q E BD
current layer queries >> current layer keys > current layer values previous layer values
2: Tensor K E RNxD
3: Tensor V NXD
4: Tensor VPay E RNXxD
5: float ő E R
step size head dimension
6: integer D N
7: function ELLIPTICALATTENTIONN K, V. Vpiee 8, D)
8: M +- ELLIPTICCALL WEIGHITS(V VPeee
D compute weight matrix M modify the dot- Not-product computation
9:
logits + M K' x VD
10:
attention (OOFMMXX(oggis)
11: output + attention x
12:
return output
13:
end function
14:
function ELLIPTICAL, WEIGHTS(V. Varey
15:
with torch.no gradggrrd() do
16:
N 4 V (0)
b sequence length
value_diff
(V-Veey) 3/5
18:
M + N x NORM(value Set_vvlllllfffflp p= 1, dim = 0) column- average of L1 norms
M <- DIAG (AG__EMBE((M)
> embed the vector into a diagonal matrix
20:
return M
end function
Table 20: Evaluation the performance our model and DeiT across multiple robustness benchmarks, using appropriate evaluation metrics for each.
Dataset
ImageNet-R Top-1
ImageNet-A Top-1
ImageNet-C mCE (1)
ImageNet-CC (Extra) mCE (L)
Metric
DeiT
25.38
3.65
72.21
63.68
Elliptical
31.37
6.76
73.59
65.71
Random Ablation
30.87
5.85
74.02
65.90
Elliptical-Consistent Consistent
31.46
6.71
82.92
71.74
Elliptical-Meancalee Meanscale
32.66
7.63
72.28
63.79
Table 21: Auto Attack Ablation Study: Top and Top 5 accuracies on clean ImageNet and under Auto Attack. The ablation model fails to fit the clean data well and is highly prone to adversarial attack.
Deif [78]
DeiT- Delipiallipiiaall
Random Ablation
Method
Top 1
Top
Top 1
Top
Top 1
Top 5
Clean Data
72.23
91.13
72.36
91.33
71.44
91.29
APGD-CE
27.75
66.48
31.27
68.28
27.85
61.74
APGD-T
27.74
73.37
29.69
74.39
28.60
68.72
FAB-T
71.61
90.54
71.74
90.81
68.54
89.43
Square
43.55
80.96
47.25
81.65
47.24
78.87
Average
42.66
77.84
45.00
78.78
43.06
74.69
Sequential Attack
26.08
64.18
27.45
67.77
26.33
60.85
G
Broader Impacts
Our research offers benefits to both clean data and robust performance. We in particular show improved results domains with wide social applicability. These include image segmentation, with benefits to self-driving cars, and language modeling, with benefits to AI chatbot assistants. We in particular show strong improvements against contamination by adversarial attack, which we hope protect vital Al systems from malicious actors, and competitive performance in contaminated
34

---

language modeling, which we hope can improve language models evaluated on imperfect data as is often the case the real world. There always possibility of misuse of AI systems. however our research shows substantive improvements in fundamental architectures and theory which we hope can spur further socially beneficial outcomes.
35

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our paper claims that using Mahalanobis metric inside of the self-attention mechanism to produce hyper-ellipsoidal neighborhoods around queries improves both robustness and representation collapse. We provide a theoretical framework for this with proofs and a large amount of empirical validation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We mention our future work (section 6) that our estimator is noisy. In Appendix D we provide and prove a consistent estimator but note it is too computationally expensive, hence we need to opt for our noisier but more efficient estimator. As a result we do not prove the consistency of our estimator, but just the weaker requirement which is that it accurately estimates the relative magnitudes of the direction-win variability, For this we assume approximate separability of the true function.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
36

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical results are numbered and have referenced section in the Appendix where all required lemmas and assumptions are stated and proven. All proofs make each step as clear as possible.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We include in Appendix F all hyperparameters, model configuration, training and inference specifications, and dataset details. For corruption, we additionally include all perturbation budgets, steps, step sizes, norm specification, and additional details. We provide the exact equation for our coordinate- wise variability estimator.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
37

---

reproducible, they (c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide all code used, packaged into folders for each set of experi- ments (e.g Wikitext- 103). Each folder then contains a bash to run the required result (c.g elliptical.sh
baseline.sh). Where possible, we also include bashes to download the data. Folders also contain readme files providing information on version and package dependencies.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
/tte///dddde//oeoubiiieeeee www.iiisssooiicyoicc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ wtipp.ccpppiiiiiiiiiiiieeiippppp public/guides: Soaes//ooobbbiiioonllll srriiisssiiocccyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: In Appendix we provide all dataset details, train and test splits, compute resources, and other experimental configuration information. We also include citations to other authors who we compare with for which we adopt the same experimental setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
38

---

7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In our head redundancy experiments we report error bars, in particular showing the difference in performance between DeïT and Elliptical is not statistically significant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide an efficiency analysis figure (Figure 4) containing computation time and memory resources. We also provide in Appenxix F the exact GPU resources used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htti:::////nuiipptt http//nnurrpp.eee hwtppuubllc/thhcccceeeeeeeeeeeee
Answer: [Yes]
Justification: Our research involves no human subjects or privacy concerns. Our research also has clear links discriminatory, unsafe, or harmful outcomes. Rather, it is in large part concerned with robustness, particularly to adversarial attack, we hope our research might be usable defend vital AI systems from ill-intentioned actors.
Guidelines:
39

---

The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss broader impacts in Appendix G. We see our improved accuracy and robustness offering societal benefits, for example with self-driving cars by improved image segmentation results or with our adversarially robust vision models to defend against ill- ol-iineentione actors.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper contains such models with high risk of misuse such pretrained
language models, image generators, or scraped datasets. We use widely accepted, stan- dard benchmark datasets and propose fundamental and general improvement to architecture.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
40

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In places where code has been borrowed from public repositories or baseline models have been implemented, we have duly credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com comddatasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We include details about training and implementation well limitations for our novel class of Elliptical Attention mechanisms.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not use any crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
41

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not use any crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
42


Reviews:
[{"review_id": "Et8F7y9VHj", "reviewer": "Reviewer_ySYF", "summary": "This paper propose a novel attention mechanism, named Elliptical Attention.\nElliptical Attention use a Mahalanobis distance metric to stretch the underlying feature space in directions of high contextual relevance.\nThe Elliptical Attention pays more attention to contextually relevant information, rather than focusing on some small subset of informative  features.\nThe Elliptical Attention can reduce the representation collapse and enhance the model’s robustness.\nThis method has theoretical support and extensive experimental results on object classification, image segmentation, and language modeling validate its superiority.\n\nThis is a fundamental research.", "questions": "The large language model is very popular now.\nThe reviewer is curious whether this Elliptical Attention mechanism complies with scaling laws.\nDoes the Elliptical Attention have the potential to replace self-attention in LLM?", "limitations": "No.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper exhibits strong theoretical originality.\nIt uses a Mahalanobis distance metric to calculate the attention weights instead of the traditional pairwise dot-product.\nThe Mahalanobis distance metric can stretch the underlying feature space in directions of high contextual relevance.\nThe proposed method is well-supported theoretically.\n\n2. The experiments are solid, which proves the effectiveness of the Elliptical Attention.\nIt includes object classification, image segmentation, and language modelling across different data modalities.\nIt analyzed the computational efficiency of this method and conducted a comprehensive research process.\n\n3. The motivation and core ideas of the paper are articulated clearly.", "weaknesses": "1. The experimental table does not provide the parameter count and FLOPS.\nThe reviewer thinks these comparative data will provide a clearer understanding of the differences in method performance.\n\n2. The state-of-the-art attention methods are not complete enough.\nThere are many improvements to classical attention mechanisms that were not compared in this paper, such as B-Attention in NeurIPS 2022. B-Attention utilizes the relationship between neighbours and improves the weight distribution of the attention mechanism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. [Provide FLOPs and parameter count]**\n\n**Answer:** We thank the reviewer for pointing out the important consideration of FLOPs and parameter count for a fuller understanding of the comparative efficiency analysis. We refer the reviewer to Tables A and B in the global response for the additional efficiency results comparing DeiT and DeiT-Elliptical side-by-side and Elliptical compared to the other robust baselines.\n\n**Q2. [Consider more SOTA methods, e.g B-Attention]**\n\n**Answer:** \nWith regard to the additional SOTA methods, we first refer the reviewer to tables C, D.1, and D.2 in the global response attachment where we incorporate Elliptical into mixture-of-expert architectures and obtain substantive improvements in both language modeling pretraining and downstream finetuning tasks.\n\nWith regard to B-Attention, we thank the reviewer for drawing our attention to this interesting, improved attention mechanism. This paper introduces a novel graph structure learning approach for images without inherent graph structures, employing single and multiple statistical tests (Sim-S and Sim-M) and refining the method with an efficient matrix form (B-Attention) for practical implementation. Furthermore, the high-level problem addressed in the paper – learning a robust similarity measure among high dimensional representations in the realistic setting of noisy features – is a significant and interesting perspective on our own problem setting in Elliptical Attention. The interaction between Elliptical and the B-Attention framework is indeed worth investigating, and we are currently working to integrate our mechanism into the B-Attention codebase and will present results at the earliest opportunity.\n\nAdditionally, we believe it is worth mentioning that our Elliptical Attention mechanism is well suited to the problem setting of noise in features, and we demonstrate both theoretically and empirically its ability to learn a meaningful similarity metric between tokens in this noise-corrupted setting. For theoretical propositions, we refer the reviewer to Appendix B.2 where we prove that the Elliptical mechanism attenuates the effect of noise in the inputs.  Furthermore, Remark 6 in B.5 links the robustness of the  Elliptical transformation itself to noise and shows that for reasonable bounds on the data corruption, the Elliptical transformation accurately captures the contextually important directions of the feature space. Empirically, we note that our strong performance on corrupted data as shown in tables 1,2 and 4 of our experimental results (section 4) offer strong evidence that Elliptical can indeed learn an accurate similarity metric between tokens in the presence of noise.\n\nWe therefore believe the theoretical background of Elliptical and B-Attention are likely to be harmonious and we will endeavour to provide additional results using this backbone as soon as possible.\n\n**Q3. [Does Elliptical comply with scaling laws and can Elliptical be integrated into LLMs?]**\n\n**Answer:**\nThe reviewer raises an interesting question and direction for further research. Below, we present results that offer evidence that Elliptical Attention indeed complies with scaling laws and behaves favorably in modern LLM architectures.\n\n**Elliptical Attention and Scaling Laws.** As proposed in [1], the scaling laws for transformer language models follow the form\n\n$L(N) = (\\frac{N_C}{N})^{-\\alpha_N}$, \n\nwhere $L$ is the test loss in nats, $N$ is the number of parameters excluding the vocabulary embedding layer, and $\\alpha_N$ is the degree of performance improvement expected as we scale up $N$, compute, or the amount of data used for training. $N_C$ is a constant depending on the vocabulary size and tokenization and does not have fundamental meaning. In our case, where models have limited numbers of parameters and are trained to convergence on sufficiently large datasets, the authors find $N_C \\sim 8.8 \\times 10^{13}$ and $\\alpha_N \\sim 0.076$.\n\nIn our language modeling experiments, our small model has $N = 9.43M$, and our medium one has $N = 20.97M$. By the given estimates of $N_C$ and $\\alpha_N$ and the scaling laws posited by the authors, this gives an anticipated relative decrease in test loss of the order\n$\\frac{20.97}{9.43}^{-0.076} = 0.941$. Empirically, we find our test loss reduces by a factor $\\frac{3.31782}{3.46574} = 0.957$. Hence, we find our empirical scaling results in our Elliptical language model to match almost exactly (with a margin of less than 2% difference) the existing scaling laws [1], justifying the promise of our Elliptical language model to attain the impressive and sustained improvements of LLMs.\n\n**Elliptical integration into modern LLM architectures.** New and cutting-edge LLMs are increasingly leveraging Mixture-of-expert (MoE) transformer architectures, for example Switch Transformer [2], Generalist Language Model (GLaM) [3], and Mixtral [4], along with proprietary models such as Grok and Gemini. Hence, to evaluate Elliptical Attention's compatibility with LLMs, beyond the topic of scale, we have assessed Elliptical Attention's performance within MoE transformer architectures.\n\nWe refer the reviewer to the global response, where we discuss the integration of Elliptical Attention into Switch Transformer and GLaM, as well as the empirical results and strong pretraining and finetuning improvements from the resultant models, in Tables C, D.1, and D.2.\n\n**References**\n\n[1] Kaplan, Jared, et al. \"Scaling laws for neural language models.\" (2020).\n\n[2] Fedus, William, Barret Zoph, and Noam Shazeer. \"Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.\" 2022 \n\n[3] Du, Nan, et al. \"Glam: Efficient scaling of language models with mixture-of-experts.\" 2022\n\n[4] Jiang, Albert Q., et al. \"Mixtral of experts.\" (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "whmC20AZzg", "reviewer": "Reviewer_EhBA", "summary": "The paper proposes a new class of self-attention mechanism for transformers. It uses Mahalanobis distance to form hyper-ellipsoidal attention regions around queries, aiming to improve model robustness and reduce representation collapse. This approach is demonstrated to be effective across various tasks like language modeling, image classification, and segmentation.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1) The shift to hyper-ellipsoidal attention regions is a novel approach that enhances attention mechanism's sensitivity to contextually relevant features and robustness against noise.\n\n(2) The paper provides a comprehensive set of experiments demonstrating improvements over traditional transformers in both clean and noisy conditions.\n\n(3) It offers a theoretical framework with in-depth discussion and analysis of the proposed mechanism.", "weaknesses": "(1) This paper fails to provide detailed setups of experiments, making it challenging to refer to other researchers and developers.\n\n(2) The proposed method is unclear for readers. I have not found the exact formulation of elliptical attention. Moreover, it would be helpful to include pseudo codes of core algorithms.\n\n(3) The Fig 3 and Fig 4 are in low resolution and would be blurred after zooming in.\n\n(4) This paper is not well written, with disconnected organization and many abbreviations. \n\n(5) Fig 2 is pretty confusing. What does x1 and x2 mean here? What is the insight of these two subfigures? The authors failed to provide enough explanation of this figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. [Provide further details on experimental setup]**\n\n**Answer:** We thank the reviewer for pointing out missing details on the experimental setup, and we have added to Appendix F additional details on hyperparameters, training procedure, and optimizer specification. Papers using the same experimental setup are cited as well to help direct comparison. \n\n**Q2. [The method is unclear and the exact formulation is difficult to find. Including pseudocode would be helpful]**\n\n**Answer:** \nThanks for your suggestion. Below we provide the requested pseudocode along with clarification on the method and formulation.\n\n**Proposed method and full technical formulation.** The proposed method replaces the Euclidean distance within self-attention with a Mahalanobis distance which stretches certain directions of the feature space in which attention is computed. These directions of the feature space correspond to contextually important directions and so by paying additional attention to tokens in those directions, the self-attention mechanism learns better representations. We discover these important directions by examining attention through the lens of non-parametric kernel regression, where we show that stretching the space in directions of *least* variability in the true underlying function obtains a lower mean squared error estimator. Adopting this approach of stretching the space from non-parametric regression into self-attention obtains our Elliptical Attention mechanism.\n\nTo improve the readability of the full formulation in Section 3.5, we have updated the manuscript by repeating the definitions of the novel components immediately after providing the full formulation. We hope this helps our method be more easily understood. For reference, this is $$\\boldsymbol{H} = \\mathrm{softmax} \\left( \\frac{\\boldsymbol Q\\boldsymbol M\\boldsymbol K^\\top}{\\sqrt{D}} \\right) \\boldsymbol V,$$\nwhere $\\boldsymbol Q$, $\\boldsymbol K$, $\\boldsymbol V$ are the queries, keys, and values, respectively, $D$ is the head dimension, and $\\boldsymbol M = \\mathrm{diag}(m_1, m_2 \\dots, m_D)$ is the diagonal Elliptical transformation with\n$$m_i := \\underset{ (\\boldsymbol v^\\ell, \\boldsymbol{v}^{\\ell + 1} ) \\in \\mathcal{X}{v}^{\\ell,\\ell+1} }{\\mathbb E_n} \\frac{| \\boldsymbol{v}^{\\ell+1}(i) - \\boldsymbol{v}^\\ell(i) |}{\\delta}$$\nwhere $\\boldsymbol v^\\ell(i)$ and $\\boldsymbol v^{\\ell+1}(i)$ are the $i^{th}$ dimension of values at adjacent layers $\\ell$ and $\\ell+1$, $\\mathbb E_n$ denotes the average over $n$ values, $\\mathcal{X}_{v}^{\\ell,\\ell+1}$ denotes the set of values across adjacent layers and $\\delta$ is a hyperparameter. \n\n**Algorithm in pseudocode.** We have included an algorithm for Elliptical Attention in Appendix F.9. For reference this is:\n\n**Algorithm: Computation of Elliptical Attention**\n\n**Input:**\n\n- Tensor $Q \\in \\mathbb{R}^{N \\times D}$ *# queries*\n- Tensor $K \\in \\mathbb{R}^{N \\times D}$ *# keys*\n- Tensor $V \\in \\mathbb{R}^{N \\times D}$ *# values*\n- Tensor $V^{\\text{prev}} \\in \\mathbb{R}^{N \\times D}$ *# previous layer values*\n- Float $\\delta \\in \\mathbb{R}_{+}$ *# step size*\n- Integer $D \\in \\mathbb{N}$ *# head dimension*\n\n**Function:** elliptical_attention($Q$, $K$, $V$, $V^{\\text{prev}}$, $\\delta$, $D$)\n\n1. $M \\leftarrow$ elliptical_weights($V$, $V^{\\text{prev}}$, $\\delta$) *# compute weight matrix $M$*\n2. $\\text{logits} \\leftarrow Q \\times M \\times K^\\top \\times \\frac{1}{\\sqrt{D}}$ *# modify the dot-product computation*\n3. $\\text{attention} \\leftarrow \\text{softmax}(\\text{logits})$\n4. $\\text{output} \\leftarrow \\text{attention} \\times V$\n5. **return** $\\text{output}$\n\n**Function:** elliptical_weights($V$, $V^{\\text{prev}}$, $\\delta$)\n\n1. **with** `torch.no_grad()` **do**\n    1. $N \\leftarrow V.\\text{size}(0)$ *# sequence length*\n    2. value_diff $\\leftarrow (V - V^{\\text{prev}}) / \\delta$\n    3. $M \\leftarrow \\frac{1}{N} \\times$ norm(value_diff, $p$=1, dim=0) *# column-wise average of $\\mathcal{L}_1$ norms*\n    4. $M \\leftarrow$ diag_embed($M$)\n2. **return** $M$\n\n**Q3. [Fig 3 and Fig 4 are low resolution]**\n\n**Answer:**\nWe thank the reviewer for catching this and have upgraded the figures' resolution.\n\n**Q4. [Organization is disconnected and there are many abbreviations]**\n\n**Answer:** \nWe endeavour to write with an intuitive, logical structure, and so we apologize if the organization was difficult to follow. If the reviewer has any specific suggestions on the organizational structure, we would be happy to implement them to improve readability.\n\nWe have also updated the manuscript, rectifying undefined abbreviations. \n\n\n**Q5. [Clarify Fig 2 and its insight]**\n\n**Answer:**\nFigure 2 aims to provide a graphical explanation of the theory in Section 3.1 that distance-based estimators, such as Nadaraya-Watson, can obtain a lower mean squared error by taking ellipsoidal neighborhoods around queries. The key idea is that stretching the neighborhood in directions of least variation in the underlying function allows the model to include more points in the estimate, reducing variance without increasing bias. \n\nSpecifically, Figure 2 shows a situation where $f$ (a function of two variables $x_1$ and $x_2$) does not vary equally in each direction $x_1$ and $x_2$. The left sub-figure shows the result of stretching the neighborhood around the query in the $x_2$ direction. The right sub-figure shows how this new ellipsoidal neighborhood includes 4 additional data points. This helps obtain a lower variance estimator, as more points included in the neighborhood smoothes out noise, while crucially not missing out on the true variation in $f$ because $f$ does not vary in the $x_2$ direction. Hence, we lower variance without introducing bias. This motivates our Elliptical Attention to pay more attention to keys lying in directions for which the true underlying function varies least.\n\nWe have updated the writing in section 3.1 accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M05aBVbHnS", "reviewer": "Reviewer_5EPe", "summary": "This manuscript introduces Elliptical Attention, a new approach employing the Mahalanobis distance metric to calculate attention weights. This method delineates a hyper-ellipsoidal neighborhood around each query, amplifying the attention weights of tokens situated in contextually pivotal directions. When compared with conventional self-attention mechanisms, Elliptical Attention exhibits a reduction in representation collapse and enhances the model's robustness.", "questions": "My question is to show more detailed experimental comparisons with the standard self-attention using different backbones.", "limitations": "The main limitation of this work is the need for a more extensive experimental comparison.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper provides a cogent blend of theoretical underpinnings and comprehensive experimental evidence supporting the efficacy of Elliptical Attention.\n2.\tThe empirical studies conducted across diverse research benchmarks reveal that Elliptical Attention is on par with or superior to existing attention mechanisms. Notably, it consistently surpasses the baseline standard self-attention when integrated with Transformer and DeiT architectures.", "weaknesses": "My primary reservations pertain to the experimental framework, which necessitates a more meticulous comparative analysis. Firstly, as Elliptical Attention is posited as an alternative to traditional attention computations, its juxtaposition with the standard Euclidean distance-based self-attention should be more exhaustive. For instance, within the ImageNet-1K classification task, it would be instructive to present results for both the vanilla Vision Transformer (ViT) and a modified ViT-Elliptical, wherein the standard self-attention is supplanted by Elliptical Attention. Further empirical comparisons between DeiT and DeiT-Elliptical across various model sizes and image resolutions on the ImageNet-1K classification task would substantiate the proposed method's advantages over the conventional self-attention.\n\nSecondly, despite assertions in the introduction regarding Elliptical Attention's reduced memory requirements and accelerated computational speed, Figure 4 shows negligible efficiency gains over the original DeiT. I would suggest the authors to furnish detailed results such as memory usage, number of model parameters, FLOPs, and throughputs for a lucid comparison. This can be done using ImageNet. They may also provide a side-by-side assessment of ViT against ViT-Elliptical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. [Provide results for DeiT and DeiT-Elliptical at additional model sizes]**\n\nFor convenience, we combine Tables E.1 and E.2 in the global response attachment into a single Table E below, which shows clean and robust performance of DeiT and DeiT-Elliptical at a larger 22.1M param model consisting of 12 layers, 384 embedding dimension over 6 heads and feed-forward size of 1536. This is the 'small' model size in [1]. We evaluate robustness using a comprehensive range of white box attacks PGD, FGSM, APGD-CE, APGD-T, and FAB-T, and black box attacks SPSA and Square. Due to the increased model size, we attack with a higher perturbation budget of 3/255.\n\n**Table E: Top-1 and Top-5 accuracy of DeiT and DeiT-Elliptical on ImageNet under Auto Attack at 22.1M Small Backbone Size**\n\n\n| Method           | Clean Data      | PGD            | FGSM            | SPSA | APGD-CE | APGD-T | FAB-T  | Square | Average |\n|------------------|-----------------|----------------|----------------|------------|---------|--------|--------|--------|---------|\n|                  | Top1/Top5   | Top1/Top5 | Top1/Top5 | Top1/Top5  | Top1/Top5  | Top1/Top5 | Top1/Top5 | Top1/Top5 | Top1/Top5 |\n| *DeiT-small*     | 79.89/95.04   | 21.41/51.50     | 51.57/82.12     | 65.68/91.28 | **19.18**/50.75 | 16.54/63.84      | 80.66/95.09      | 49.98/89.17      | 43.57/74.82     | \n| Elliptical-small | **79.92**/**95.06**   | **22.39**/**54.02**     | **51.86**/**82.87**     | **72.02/92.45** | 18.88/**51.07** | **17.30**/**65.28**  | **81.64**/**95.59**  | **55.89**/**89.36** | **45.71**/**75.80** |\n\nWe see in Table E that, as with the tiny architecture, Elliptical enhances robustness across a wide variety of attacks. In particular, we see on PGD an improvement of almost 5%, and once again, Elliptical excels against black box attacks with an improvement of almost 10% and 12% in SPSA and Square, respectively.\n\n**Q2. [Show experimental comparisons of Elliptical using additional backbones]**\n\n**Answer:** We agree with the reviewer that as Elliptical Attention is posited as an alternative to traditional attention, an exhaustive comparison across a variety of backbones is key to substantiating the empirical benefits. We refer the reviewer to Tables C, D.1, and D.2 in the global response attachment, where we show the performance gains of replacing self-attention in mixture-of-expert models with Elliptical Attention. We refer additionally to Table E above as added evidence of Elliptical Attention's strong robust performance at the larger DeiT configuration size.\n\nIn all, we endeavoured to provide as many tasks and backbones as possible with experimental results covering ImageNet in two different backbones and two different size configurations, language modeling and finetuning across two different model sizes covering both standard transformers and mixture-of-expert models, image segmentation, and long context analysis including CIFAR-10 object recognition, equation calculation, document retrieval, and image spatial dependency classification, all in comparison to the vanilla self-attention.  We hope that our results and these additional results offer satisfactory further evidence of our model's benefits.\n\n**Q3. [Clarify the efficiency proposition in comparison with DeiT and in comparison with the robust baselines. Provide further details on throughput, FLOPs, memory usage and parameters and compare DeiT and DeiT-Elliptical side-by-side]**\n\n**Answer:** We agree with the reviewer that there's some unintended ambiguity here that needs clarification and thank the reviewer for catching this. For the requested additional efficiency analysis, please see Table A in the global response attachment. For additional efficiency information regarding Elliptical in comparison to the other robust baselines of the same configuration, we also refer the reviewer to Table B in the global response attachment. We provide below the required clarification of the efficiency proposition made in the introduction.\n\n**Efficiency proposition clarification.** Our assertion in the introduction is that compared with comparative robust models, such as robust kernel density estimation-based approaches and Robust Vision Transformer (RVT), our model attains substantive robust performance gains at reduced memory requirements and accelerated throughput. Indeed, in Table B, we see Elliptical is both the fastest and least memory intensive. However, we do not intend to claim these efficiency benefits against the standard DeiT because, as you point out, the compute efficiency is basically the same. Essentially, our intention was to claim that the Elliptical Attention attains the efficiency gains when compared with other robust baseline models, not as in comparison to the backbone it is built atop of.\n\nIn other words, in reference to Table A, we can understand the robustness gains of Elliptical Attention as coming almost for free – whatever backbone we build Elliptical Attention on top of, we do not experience significant efficiency loss as measured by throughput, memory allocation, or FLOPs. This then means that compared to other robust models (which involve computationally intensive methods for robustifying) we are far more efficient.\n\nWe once again thank the reviewer for pointing out the source of confusion and we have adjusted the writing accordingly.\n\n**References**\n\n[1] Touvron, Hugo, et al. \"Training data-efficient image transformers & distillation through attention.\" International conference on machine learning. PMLR, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear AC and reviewers,

Thanks for your thoughtful reviews and valuable comments, which have helped us improve the paper significantly. We are encouraged by the endorsements that: 1) the proposed method of using hyper-ellipsoidal attention regions is a novel and theoretically well-supported approach to improving the performance of Transformers on clean and contaminated data (all reviewers); 2) the experimental analysis and validation is comprehensive, demonstrating improvements across a wide array of benchmarks (all reviewers); 3) the motivation and core ideas are cogent (reviewers 5EPe, ySYF) and the theoretical framework is discussed in depth (reviewer EhBA).

One of the main questions from reviewers was regarding additional efficiency analysis, in particular looking at Elliptical and DeiT side-by-side and Elliptical and the robust baselines in terms of throughput, FLOPs, memory allocation, and parameters. Another shared question was on the performance of Elliptical Attention within additional backbones. We address these questions here.

**Additional Efficiency Analysis.** We refer the reviewers to Tables A and B in the attachment where we present throughput, memory allocation, FLOPs, and parameters for DeiT and DeiT-Elliptical side-by-side and for DeiT-Elliptical against comparative robust baselines. We show that Elliptical is the fastest and most memory efficient model compared to the robust baselines. Against the DeiT backbone, we show at tiny, small, and base sizes that Elliptical Attention incurs almost no loss in efficiency.

**Elliptical Attention in Additional Backbones.** We refer the reviewers to Tables C, D.1 and D.2 where we incorporate Elliptical Attention into mixture-of-expert baseline architectures, Switch Transformer [1] and Generalist Language Model (GLaM) [2]. We show that Elliptical Attention is highly compatible with these additional backbones, achieving substantive improvements in language modeling pretraining and downstream finetuning tasks. We also include in Tables E.1 and E.2 additional adversarial robustness results at a larger configuration, where we see strong robust performance across a variety of attacks. 

We hope that our rebuttal has helped to clear concerns about our work. We are glad to answer any further questions you have on our submission and we would appreciate it if we could get your further feedback at your earliest convenience.

**References**

[1] Fedus, William, Barret Zoph, and Noam Shazeer. "Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity." 2022

[2] Du, Nan, et al. "Glam: Efficient scaling of language models with mixture-of-experts." 2022

Meta-review: This paper introduces the use of a Mahalanobis distance metric for computing attention weights, enabling the stretching of the underlying feature space in directions of high contextual relevance. The authors define a hyper-ellipsoidal neighborhood around each query to enhance the attention weights of tokens located in contextually significant directions.

While there are potential issues, such as the lack of detailed experimental setups, unclear model formulation, and areas needing improvement in writing and organization, this paper demonstrates strong theoretical originality. The shift to hyper-ellipsoidal attention regions is particularly novel. Additionally, the experiments are extensive, covering object classification, image segmentation, and language modeling across different data modalities. All reviewers have unanimously recommended this paper for acceptance.
Decision: Accept (poster)