Designing Robust Transformers using Robust Kernel Density Estimation
Authors: Xing Han
Tongzheng Ren
Tan Minh Nguyen
Khai Nguyen
Joydeep Ghosh
Nhat Ho
Venue: NeurIPS 2023 poster (2023)
Keywords: Transformers
Kernel Density Estimation
Robustness
TL;DR: We propose a suite of robust Transformers, capable of mitigating the impact of contaminated data across diverse data modalities.

Abstract: Transformer-based architectures have recently exhibited remarkable successes across different domains beyond just powering large language models. However, existing approaches typically focus on predictive accuracy and computational cost, largely ignoring certain other practical issues such as robustness to contaminated samples. In this paper, by re-interpreting the self-attention mechanism as a non-parametric kernel density estimator, we adapt classical robust kernel density estimation methods to develop novel classes of transformers that are resistant to adversarial attacks and data contamination. We first propose methods that down-weight outliers in RKHS when computing the self-attention operations. We empirically show that these methods produce improved performance over existing state-of-the-art methods, particularly on image data under adversarial attacks. Then we leverage the median-of-means principle to obtain another efficient approach that results in noticeably enhanced performance and robustness on language modeling and time series classification tasks. Our methods can be combined with existing transformers to augment their robust properties, thus promising to impact a wide variety of applications.

Designing Robust Transformers using Robust Kernel Density Estimation
Xing Han Department of ECE University of Texas Austin aaronhan223@ttexas. edu
Tongzheng Ren Department of Computer Science University of Texas at Austin
tongzhengdutexxs.cer
Tan Minh Nguyen Department of Mathematics University of California, Los Angeles
Khai Nguyen Department of Statistics and Data Sciences
University of Texas at Austin
tanmmguyen89uclla. edu
khainboutexas.. edu
Joydeep Ghosh Department of ECE University of Texas at Austin jghosh@utexas. edu
Nhat Ho
Department of Statistics and Data Sciences University of Texas at Austin minhnhat@utexas edu
Abstract
Transformer-base architectures have recently exhibited remarkable successes across different domains beyond just powering large language models. How- ever, existing approaches typically focus on predictive accuracy and computa- tional cost, largely ignoring certain other practical issues such as robustness to contaminated samples. In this paper, by re-interpreting the self-attention mecha- nism non- paraameeetric kernel density estimator, we adapt classical robust ker- nel density estimation methods to develop novel classes of transformers that are resistant to adversarial attacks and data contamination. We first propose meth- ods that down-weight outliers in RKHS when computing the self-attention oper- ations. We empirically show that these methods produce improved performance over existing state-of-the-ar methods, ial attacks. Then we leverage the particularly on image data under adversar- principle obtain another efficient approach that results in noticeably enhanced performance and robust- language modeling and time series classification tasks. Our methods be combined with existing transformers to augment their robust properties, thus promising to impact wide variety of applications.
1 Introduction
Attention mechanisms and transformers (Vaswani et al., 2017a) have drawn lots of attention in the machine learning community (Lin et al., 2021; Tay al., 2020; Khan et al., 2021). Now they among the best deep learning architectures for a variety of applications, including those in natural language processing (Devlin al., 2019; Al-Rfou et al., 2019; Dai et al., 2019; Child et al., 2019; Raffel et al., 2020; Baevski & Auli, 2019; Brown et al., 2020; Dehghani et al., 2019), computer vision (Dosovitskiy
al., 2021; Liu et al., 202lb; Touvron et al., 2021a: Ramesh al.. 2021; Radford et al., 2021; Fan et al., 2021; Liu et al., 2022), and reinforcement learning (Chen et al., 2021; Janner al., 2021). They are also known for their effectiveness in transferring knowledge from various pretraining tasks to different downstream applications with weak supervision or no supervision (Radford al., 2018, 2019; Devlin et al., 2019; Yang al., 2019; Liu et al., 2019).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

While there have been notable advancements, the robustness of the standard attention module re- mains an unresolved issue in the literature. In this paper, our goal is to reinforce the attention mech- anism and construct comprehensive framework for robust transformer models. To achieve this, we first revisit the interpretation of self-attention in transformers, viewing through the prism of the Nadaraya-Watse Watson (NW) estimator (Nadaraya, 1964) in non-parametric regression context. Within the transformer paradigm, the NW estimator is constructed based on the kernel density estimators (KDE) of the keys and queries. However, these KDEs are not immune to the issue of sample con- tamination (Kim & Scott, 2012). By conceptualizing the KDE as a solution to the kernel regression problem within Reproducing Kernel Hilbert Space (RKHS), art robust KDE techniques, such as those based on robust kernel regression and median of mean estimators. This facilitates the creation of substantially more robust self-attention mechanisms. The resulting suite of robust self-attention can be adapted to a variety of transformer architectures and tasks across different data modalities. We carry out exhaustive experiments covering vision, lan- guage modeling, and time-series classification. The results demonstrate that our approaches can uphold comparable accuracy clean data while exhibiting improved performance on contaminated data. Crucially, this is accomplished without introducing any extra parameters.
utilize range of state-of-the
Related Work on Robust Transformers: Vision Transformer (ViT) models (Dosovitskiy et al., 2020: Touvron et al., 2021b) have recently demonstrated impressive performance across various vi- compelling alternative to CNNs. A number of studies (e.g.,
sion tasks, positioning themselves Subramanya et al.. 2022; Paul & Chen, 2022; Bhojanapalli et al., 2021; Mahmood et al., 2021; Mao et al., 2022: Zhou et al., 2022) have proposed strategies to bolster the resilience of these models against common adversarial attacks on image data, thereby enhancing their generalizability across diverse datasets. For instance, Mahmood et al. (2021) provided empirical evidence of ViT's vulnera- bility to white-box adversarial attacks, while demonstrating that a straightforward ensemble defense could achieve remarkable robustness without compromising accuracy on clean data. Zhou et al. (2022) suggested fully attentional networks to enhance self-attention, achieving state-of-the-ar ac- curacy on corrupted images. Furthermore, Mao et al. (2022) conducted robustness analysis on various ViT building blocks, proposing position-an aware attention scaling and aatch-wise augmenta- tion to enhance the model's robustness and accuracy. However, these investigations are primarily geared toward vision-related tasks, which restricts their upplicability across different data modalí- ties. As an example, the position-based attention from Mao et al. (2022) induces a bi-directional information flow, which is limiting for position-sensitivv datasets such as text or sequences. These methods also introduce additional parameters. Beyond these ffcuued studies, robust trans- formers have also been explored in fields like text analysis and social media. Yang et al. (2022) delved into table understanding and suggested a robust, structurally aware table-text encoding archi- tecture to mitigate the effects of row and column order perturbations. Liu et al. (2021a) proposed a robust end-to-end transformer-basee model for crisis detection and recognition. Furthermore, Li et al. (2020) developed unique attention mechanism to create robust neural text-to- speech model capable of synthesizing both natural and stable audios. We have noted that these methods vary in their methodologies, largely due to differences in application domains, and therefore limiting their generalizability across diverse contexts.
Other Theoretical Frameworks for Attention Mechanisms: Attention mechanisms in transform- ers have been recently studied from different perspectives. Tsai al. (2019) show that attention can be derived from smoothing the inputs with appropriate kernels. Katharopoulos et al. (2020); Choro- manski et al. (2021); Wang et al. (2020) further linearize the softmax kernel in attention to attain family of efficient transformers with both linear computational and memory complexity. These linear attentions are proven in Cao (2021) to be equivalent to a Petrov-Galerkin projection (Reddy, 2004), thereby indicating that the softmax normalization in dot-product attention is sufficient but not necessary. Other frameworks for analyzing transformers that use ordinary/partial differential equa- tions include Lu et al. (2019); Sander et al. (2022). In addition, the Gaussian mixture model and graph-structured learning have been utilized to study attentions and transformers (Tang & Matteson, 2021; Gabbur et al., 2021: Zhang& & Feng, 2021; Wang et al., 2018: Kreuzer et al., 2021). Nguyen et al. (2022c) has linked the self-attention mechanism with a non- paraamettiic regression perspective, which offers enhanced interpretability of Transformers. Our approach draws upon this viewpoint, but focuses instead on how it can lead to robust solutions.
2

---

2 Self-Attention Mechanism from a Non-parametric Regression Perspective
Assume we have the key and value vectors {kj, V JE]N that is collected from the data generating process V f(k) + E, where some noise vectors with Eje] O, and the function that we want to estimate. We consider a random design setting where the key vectors [ej] are i.idd samples from the distribution p(k), and we use p(v, k) to denote the joint distribution of (v. k) defined by the data generating process. Our target is estimate f(q) for any new queries q. The NW estimator provides non parameeriet approach to estimating the function f. the main idea is that
V v.p(v,k) dv, p(k)
v p-v(vkk)dv f(k) = Elvik] = JD
(1)
where the first equation comes from the fact that B[e] O, the second equation comes from the definition of conditional expectation, and the last equation comes from the definition of conditional density. To provide estimation of f. just need to obtain estimations for both the joint density function p(v, k) and the marginal density function pkk). KDE commonly used for the density estimation problem (Rosenblatt, 1956; Parzen, 1962), which requires kernel ko with the bandwidth parameter a satisfies Jgo Ko (x
x' z')dx= = 1, Vx', and estimate the density as
(v, k))
1 N
Σ ka (lv, kj kyl) Po JEIN]
Σ JE[N]
ko (k - kk)
N
where [v, k] denotes the concatenation of v and k. Specifically, when ka is the isotropic Gaussian k,). Combine this with Eq. (1) and Eq.
kernel, we have Po (v, k) EzEiN (v (Vy--))kol (2), we can obtain the NW estimator of the function f f'as
(ks) = EjEN Vyko kj))
(3)
EjENNI lio k:))
Furthermore, it is not hard to show that if the keys (kihje[M] are normalized, the self- attention mechanism Fo (q.) in Eq. (3) is exactly the standard Softmax attention fo EjENN] softmax (q TqTk, ky)a²) V.. Such an assumption on the normalized key {k, ]SE[N] can mild, as in practice we always have a normalization step on the key to stabilizing the training of the trans- former (Schlag et al., 2021). we choose x² VD, where is the dimension of q and kj, then fo (q:) = h; As result, the self-attention mechanism in fact performs non- non- non-parametric regression with NW-estimator and isotropic Gaussian kernel when the keys are normalized.
KDE as a Regression Problem in RKHS We start with the formal definition of the RKHS. The space H₂ {f f : X - R} is the RKHS associated with kernel k where & X x X' + R. if it is Hilbert space with inner product (H and following properties:
k(x,-) Hes Vx X:
vf E Hu, f(x) (f,k(æ, He Aka the reproducing property.
Figure 1: The contour plots illustrate the den- sity estimation of the two-dimensional query vector embedding within transformer's atten- tion layer. The left plot employs the regular KDE method, as defined Eq. (4), whereas the right plot utilizes robustified version of the KDE method, which enhances KDE's robust- ness against outliers.
With slightly abuse of notation, we deñine ko (xx, æ²) (x =x)). By the definition of the RKHS and the KDE estimator, we know Do N EjENN| ko (a().)) Hi, and can be viewed as the optimal solution of the following least-square regression problem in RKHS:
P. = arg min P
Σ JE[N]
l (Ix..)) - 1ll
(4)
N
3

---

Image Patches
Attention Weight Factor
Later that year, she auctions breakfad Mayfair,
Glean Data
Eapod for
Ned which helps children's primary education
PGD Attack
Later that year, she auctioned breakfast Mayfain London, where she BAR around E4000 for the Pratham RAA which helps children's primany education.
Brightes
Figure 2: The application of Transformers with robust KDE attention on image and text is shown. (Left) The robust KDE self-attention generates varying weight factors for image patch embeddings under adversarial attacks or data corruption. The adversely impacted regions that would other- wise lead to incorrect predictions (Right) In the field of language modeling, the weight factors lend significance to essential keywords (highlighted in red). In the face of word swap attacks, the fortified self-attention mechanism, par- ticularly when utilizing the medians-of-meant principle, is proficient in disregarding or reducing the importance of less consequential words (marked in green). Consequently, this results in more resilient procedure during self-attention computations.
down-week weighted, ensuring enhanced accuracy and robustness.
Note that, in Eq. (4), the same weight factor 1/N is applied uniformly to each error term I|ki (3.).)) plli This approach functions effectively if there are no outliers in the set {kir
However, when outliers are present (for instance, when there is some such that ||A: (zz )l >> (I,, Vi E [N] # j). the error attributable to these outliers will overwhelmingly influence the total error, leading estimation. We illustrate the robustness issue of the KDE in Figure 1. The view that KDE is sus- ceptible outliers, coupled with the non- con-parametric understanding of the self-attention mechanism, implies potential lack of robustness in Transformers when handling outlier-rich data. We now of- fer fresh perspective on this robustness issue, introducing universal framework that is applicable across diverse data modalities.
significant deterioration in the overall density
3 Robust Transformers that Employ Robust Kernel Density Estimators
Drawing on the non- parameereeic regression formulation of attentttennoon we derive multiple robust variants of the NW-estimator and demonstrate their applicability in fortifying existing Transformers. We propose two distinet types of robust self-attention mechanisms and delve into the properties of each, potentially paving the way for Transformer variants that substantially more robust.
3.1 Down-weggh Outliers in RKHS
Inspired by robust regression (Fox& & Weisberg, 2002), a direct approach to achieving robust KDE involves down-weightte outliers in the RKHS. More specifically, substitute the least-square loss in Eq. (4) with a robust loss function resulting in the following formulation:
Probust arg min
Σ e (Ilko (x,, - pllme,)) ) = Σ JE[N]
(5)
PEH
JE[N]
Examples of the robust loss function p include the Huber loss (Huber, 1992). Hampel loss (Hampel et al., 1986), Welsch loss (Welsch & Becker, 1975) and Tukey loss (Fox & Weisberg, 2002). We empirically evaluate different loss functions in our experiments. The critical step here the set of weights w = (W1, ... ..) E AN, with each W, & vp (ZI,)((z,)))
estimate Probual |l |/. where (xx) = p'(x) Since Probust is defined via w, and w also depends on Probust- one can address this circular definition problem via an alternative updating algorithm proposed by Kim & Scott (2012).
The algorithm starts with randomly initialized (0) An, and performs alternative updates between Probust and w until the optimal Probus is reached at the fixed point (see details in Appendix A).
However, while this technique effectively diminishes the influence of outliers, it also comes with noticeable drawbacks. Firstly, it necessitates the appropriate selection of the robust loss function, which may entail additional effort to understand the patterns of outliers. Secondly, the iterative

---

Algorithm Procedure of Computing Attention Vector of Tranforor-KKDE/SSDDEAOO
1: Input: Q = [E(] K = [k3 JE(N), V [vi]e[N)) initial weights w
2: Normalize K = { İjE[N] along the head dimension.
3: Compute kereel function between each pair of sequence: Ko (). (Ko kg)R.jE(N).
(Optional) apply attention mask hi ()...
[MoM] Randomly sample subsets I,,...; IB size S. obtain the median block such that
Ejen, lio (q. k;) m 5 Ejer, (a - k.)),..... . Ejern ko - kk))3
(1 (k,,)) ((ke)-))-hhll Plain Ne
6: [RKDE] Update weights w for marginal/joint density by wfth ExC(a) of ( (hee Poodo (1)) Mike
7: [SPKDE] Obtain optimal weights w for marginal/joint density via solving equation (7).
h Ejein Yow, No (9,
8: [RKDE, SPKDEJ Obtain robust self-attention vector
Ejelnn
megian (a))
9: [MoM] Obtain attention vector h; Eser, "g kz) Zoen (0 (k,
updates might not successfully converge to the optimal solution. better alternative to assign higher weights to high-density regions and reduce the weights for atypical samples. The original KDE scaled and projected its nearest weighted KDE according to the L2 norm. Similar concepts have been studied by Scaled and Projected KDE (SPKDE) (Vandermeulen & Scott, 2014), which offer an improved set of weights that better defend against outliers the RKHS space. Specifically, given the scaling factor 3 > 1, and let C.V be the convex hull of (a (). :..... ( E Hag, Le., the space of weighted KDEs, the optimal density Probust is given by
B N
Probust arg min PECN
Σ JEIN]
Ro (Z
Hkr
which is guaranteed to have a unique minimizer since we are projecting in a Hilbert space and CN is closed and convex. Notice that, by definition, Probust can also be represented as Probust EJE[N (y)ka ) E AN, which is same as the formulation in Eq. (5). Then Eq. (6) can be written as quadratic programming (QP) problem over w.
min w Gw 2q Tw, subject to NREEE A
where G is the Gram matrix of [a)))E[N] with h and q G1 G Since h is a positive-definite kernel and each 2; is unique, the Gram matrix G is also positive-definite As a result, this QP problem is convex, and we can leverage commonly used solvers efficiently obtain the solution and the optimal density Probust.
Robust Self-Attention Mechanism We now introduce the robust self-attention mechanism that doinhweights atypical samples. We consider the density estimator of the joint distribution and the marginal distribution when using isotropic Gaussian kernel:
Probusa (v,
w Join her ([v3, ky), [v, k)), Probusa (k)
Σ
W
marginall ki (ke, k))
(8)
JEIN
JEINI
Following the non-parametric regression formulation of self-attention in Eq. (3), we obtain the robust self-attention mechanism as
h ESE(N) Vjue joino kor (g: k;)) marginal
(9)
EjE[N] W Ko (g. k,)
where jiin and marginal are obtained via either alternative updates or the QP solver. We term Trans- formers whose density from the non-parametric regression formulation of self-attention employs Eq. (5) and Eq. (6) as Transformer-RKDD and Transformer-SKDDD respectively. Figure 2 presents an example of the application of the attention weight factor during the learning process from image and text data. The derived weight factor can potentially emphasize elements relevant to the class while reducing the influence of detrimental ones. Note that, the computation of marginal J]E[M]
5

---

joint involves both keys and values vectors. During the empirical and (a) Je[N] are separate as w/ evaluation, we concatenate the keys and values along the head dimension to obtain the weights for the joint density Probust (v, k) and only use the key vectors for obtaining the set of weights for the marginal Pko In addition, munginal ب E Rixi for i.j = ,,...,,N .V are 2-dimensional ma- trices that include the pairwise weights between each position of the sequence and the rest of the positions. The weights are initialized uniformly across certain sequence length dimension. For experiments related to language modeling, we can leverage information from the attention mask to initialize the weights on the unmasked part of the sequence.
Limitations While constructing attention weight factors proves effective, they necessitate itera- tive algorithms to calculate the set of weights when computing self-attention at each layer. resulting in increased overall complexity. Moreover, the foundational contamination model for these meth- ods is the classical Huber contamination model (Huber, 2011), which requires assumptions about contamination distributions and parameters that may not be universally applicable, especially in dis- crete settings. To address these limitations, we propose a novel approach that circumvents these computational constraints while effectively fostering robust self-attention mechanisms.
3.2 Robust Sell-Attention via Median-of-Means Principle
The Median-of- Mf-ns (MoM) principle (Jerrum et al., 1986; Alon et al.. 1996) one other way to construct robust estimators. Rather than taking the average of all the observations, the sample split into several blocks over which the median is computed. The MoM principle also improved the robustness of KDE and demonstrated its statistical performance under a less restrictive outlier framework (Humbert et al., 2022). More importantly, it can be easily adapted to self-attention. Specifically, we randomly divide the keys {k, NN J--1
into B subsets II...., IB of equal size, namely, [hl |2l =... |T| = S. Then, the robust estimator of p(k) takes the following form:
(k) xx median (po.1, (k), ................................ Po (k)).
(10)
where we define Po. (k) = 5 Ejen, ko (k- (k)) for any E [B]. Similarly, the robust estimator of kkv,k) is as follows:
Probus (v, k) median (po,1, (v, k),.. Po (V, ky)
(11)
where Po.,, (v, = 3 Ejen, (v - Vy)ka k2) for any I E [B]. We now propose the self-
attention mechanism utilizing the median-of-means principle.
Median-of-Meant Self-Attention Mechanism we can consider the following robust estimation of the attention:
Given the robust estimators in Eq. (10) and (11),
Ejen, Uzko - k;))
h median (0o.1 k), ,o.....,,,......
(12)
I (q: k)}
where 1j is the block such that Po (q. k) achieves its median value in equation (11). In this context, the random subsets apply to input sequences rather than individual data points, distinguishing this approach from stochastic batches. It's worth noting that our proposed attention mechanism assumes that key and query vectors achieve their median on the same block. Consequently, we apply the me- dian block I obtained from the denominator into the numerator, rather than considering the median over all potential blocks, which leads to a process that is faster than computing median blocks on both sides. Moreover, the original MoM principle mandates that each subset be non-overlapping. i.e. It n lig = 0 for any < 4 # l2 < B. However, for structured, high-dimensional data, dividing into non-overlgpping blocks may result in the model only gaining partial perspective of the dataset, leading to sub-optimal performance. Therefore, we construct each subset by sampling with replacement from the original dataset, maintaining the sequential relationship thereafter. In particular, we have found this strategy to be effective in discrete contexts, such as identifying and filtering out aberrant words in sentence. As illustrated in the bottom right segment of Figure 2, under word swap attack, the MoM robust attention retains the subsequence that is most relevant to the content, while discarding unhelpful parts. The downside of MoM self attentotnn is also apparent: since the attention mechanism only accesses portion of the sequence, it is likely to result in subop- timal performance on clean datasets. The simplicity of MoM allows for easy integration with many existing models. We initially demonstrate that the MoM self-attention mechanism can enhance the recent state-of-the-ar Foureer (Nguyen et al., 2022c). The theoretical explanation for the ability to remove outliers using MoMMFourier attention can be found in Appendix C.

---

Perplexity (PPL) and negative likelihood loss (NLL) our methods (lower part) and basclines (upper part) on WikiText-106 103. The best results are highlighted in bold font and the second best are highlighted in underline. On clean data, Sereffomer-SPPDDD achieves better PPL and NLL than other baselines. Under random swap with outlier words., Transformers with MoM self-attention show much better performance.
Table 1:
Clean Data
Word Swap
Method (small version)
Valid PPL/Loss
Test PPL/Loss
Valid PPL/Loss
Test PPL/Loss
Transformer (Vaswami et al., 2017b)
33.15/3.51
34,29/3.54
72.28/4.45
74,56/4,53
Performer (Choromanski et al., 2021)
32.35/3.45 48
33.49/3.51 .1
71.64/4.42
73.48/4.49
sooomer-GGGK (Nguyen et al., 2022h)
32.28/3.44 .47
33.21/3.51
69.78/4.88
71.03/4.41
FourierFormer (Nguyen et al,, 2022c)
31.86/3.44 44
32.85/3.49
65.76/4.32
68.33/4.36
I Transtormer-KKl (Huber)
31.22/3.42
32.2913.47
52.1473.92
53.6873.99
TransformerRKDDD (Hampel)
31.24/3.48
32.35/3.48
55.61/3.98
57.92/4.03
Transformer-SKIDDE
31.05/3.41
32.18/3.46
51.36/3.89
54.97/3.96
Transformer-Moke
33.56/3.55
34.68/3.55
48.29/3.85
52.14/3.92
FourierFormer-Mor
32.26/3.47
33,14/3.50
47.66/3.81
50.96/3.85
3.3 Incorporating Robust Self-Attention Mechanisms into Transformers
Computational Efficiency The two types of robust attention mechanisms proposed above have their own distinct advantages. To expedite the computation for Transformer-RKDD and obtain the at- tention weight factor in a more efficient manner, we employ a single-step iteration on the alternative updates to approximate the optimal set of weights. Empirical results indicate that this one-step itera- tion can produce sufficiently precise results. For Transformer-SPKDDE, as the optimal set of weights is acquired via the QP solver, it demands more computation time but yields superior performance on both clean and contaminated data. As an alternative to weight-based methods, Transformer- MoM offers significantly greater efficiency while providing competitive performance, particularly with text data. The complete procedure for computing the attention vector for Transformer-RKe RKDE, Transformer-Sii SPPDDD and Transformer-Mon MoM is detailed in Algorithm
Training and Inference We incorporate our robust attention mechanisms into both training and inference stages of Transformers. Given the uncertainty about data cleanliness, there's s possibility of encountering contaminated samples at either stage. Therefore, it is worthwhile to defend against outliers throughout the entire process. In the training context, our methods modify the computation of attention vectors across each Transformer layer, making them less susceptible to contamina- tion from outliers. This entire process remains nonparametric, with no introduction of additional model parameters. However, the resulting attention vectors, whether shaped by re-weighting or the median-of- principle. diverge from those generated by the standard softmax attention. This distinction influences the model parameters learned during training. During inference, the test data undergoes similar procedure yield robust attention vectors. This ensures protection against po- tential outlier-induced disruptions within the test sequence. However, if we assume the availability of clean training set where contamination arises solely from distribution shifts, adversarial at- tacks, or data poisoning during inference - it is sufficient to restrict the application of the robust attention mechanism just the inference phase. This could considerably reduce the computational time required for robust attention vector calculation during training. In our empirical evaluation, we engaged the robust attention mechanism throughout both phases and recorded the associated compu- tational time during training. We found that on standard datasets like ImageNet- IK, WikiText- 103, and the UEA time-series classification, infusing the robust attention led drop in training loss. This suggests that training data itself may contain inherent noise or outliers.
4 Experimental Results
In this section, we provide empirical validation of the benefits of integrating our proposed robust KDE attention mechanisms compare these with the standard softmax Transformer across multiple datasets representing differ- ent modalities. These include language modeling on the WikiText- 103 dataset (Merity et al., 2016) (Section 4.1) and image classification on ImageNet (Russakovsky et al., 2015; Deng et al., 2009). Furthermore, we assess performance across multiple robustness benchmarks, namely ImageNet-C (Hendrycks & Dietterich, 2019), ImageNet-A (Hendrycks et al., 2021b), ImageNet-O (Hendrycks et al., 2021b), ImageNet-R (Hendrycks et al., 2021a), and ImageNet-SSetchh (Wang et al., 2019) (Section 4.2), as well as UEA time-series classification (Section 4.3). Our proposed robust trans-
into Transformer base models. We
7

---

Top-1 and top-5 accuracy (%) on ImageNet. The best results are highlighted in bold font and the second best are highlighted in underlines. RVT (Mao et al., 2022) and DeiT (Touvron et al., 2021b) achieve better results on clean data; meanwhile, Transformers incorporating robust self attentio hold stronger defense under different adversarial attacks while still achieving competitive performance on the original ImageNet.
Table 2:
Method
Clean
Data
FGSM
PGD
SPSA
Top 1
Top 5
Top
Top
Top
Top 5
Top 1
Top 5
ViT VIT (Dosovitskly et al., 2020)
12.23
91.13
52.61
82.26
41.84
76.49
48.34
79.36
DeiT (Touvron et al., 2021b)
74.32
93.72
53.24
84.07
41.72
76.43
49.56
80.14
RVT (Mao al., 2022)
74.37
93.89
53.67
84.11
43.39
77.26
51.43
80,98
FourierFormer (Nguyen et al.. 2022c)
73.25
91.66
53.08
83.95
41.34
76.19
48.79
79.57
VDT-RKDE DTEReDEE(Huberr (Huber)
72.83
91.44
55.83
85.89
44.15
79.06
52.42
82.03
ViT-RKDE (Hampel)
72.94
91.63
55.92
85.97
44.23
79.16
52.48
82.07
ViT-SPKDE
73.22
91.95
56.03
86.12
44.51
79.47
52.64
82.33
ViT-MoM
71.94
91.08
55.76
85.23
43.78
78.85
49.38
80.02
Fou
72.58
91.34
53.25
84.12
41.38
76.41
48.82
79.68
formers are compared with state-of-the-art models, including Performer (Choromanski et al., 2021),
MGK (Nguyen al., 2022a), RVT (Mao al., 2022), and FourierFormer (Nguyen et al., 2022c). All experiments were conducted on machines with 4 NVIDIA A-100 GPUs. For each experiment, Transforee-RKKD//SPPP rameter configurations.
were compared with other baselines under identical hyperpa-
4.1 Robust Language Modeling
WikiTeext- 103 is a language modeling dataset that contains collection of tokens extracted from good and featured articles from Wikipedia, which is suitable for models that can leverage long-term de- pendencies. We follow the standard configurations in Merity et al. (2016); Schlag et al. (2021) and splits the training data into L-word independent long segments. During evaluation, we process the text sequence using sliding window of size L and feed into the model with a batch size of 1. The last position of the sliding window is used for computing perplexity except in the first segment, where all positions evaluated as in Al-Rfou et al. (2019); Schlag et al. (2021).
In our experiments, we utilized the small and medium (shown in Appendix E) language models developed by Schlag et al. (2021). We configured the dimensions of key, value, and query to 128, and set the training and evaluation context length to 256. We contrasted our methods with Performer (Choromanski et al., 2021), Transformer-MGG (Nguyen et al., 2022a) and FourierFormer (Nguyen et al., 2022c), which have demonstrated competitive performance. For self-attention, we allocated 8 heads for our methods and Performer, and 4 for Transformer-MGE forward layer was set to 2048, with the number of layers established at 16. To prevent numerical instability, we used the log-sum exp trick in equation (3) when calculating the attention probability vector through the Gaussian kernel. We employed similar tactics when computing the attention weight factor of Transformer RRDEE, initially obtaining the weights in log space, followed by the log-sum sog-sum--r s-m-ssumppp------ee eep trick to compute robust self-attention as outlined in equation (9). For Transformer- MoM, the sampled subset sequences constituted 80% of the length of the original sequence.
The dimension of the feed-
Table presents the validation and test perplexity (PPL.) for several methods. Both Transformer- RKDE and SPKDE models exhibit an improvement over baseline PPL. and NLL on both validation and test sets. However, MoM-based models display slightly higher perplexity, a result of using only portion of the sequence. When the dataset is subjected stitutes selected keywords with a generic token " "AAA" during evaluation, our method, particularly MoM-based robust attention, yields significantly better results. This is particularly evident when filtering out infrequent words, where the median trick has proven its effectiveness. We also noticed robust attention compared to other baseline methods
word swap attack, which randomly sub-
superior robustness in EKDE/SPKDE-basse that were not protected from the attack. Our implementation of the word swap is based on the pub- licly available TextAttack code by Morris et al. (2020)'. We employed greedy search method with constraints on stop-word modifications provided by the TextAttack library.
4.2 Image Classification under Adversarial Attacks and Data Corruptions
For initial simplicity, we utilize the original ViT (tiny) as our base model. However, our methods are versatile and can be integrated with more advanced base models enhance their robustness. As our
Immleeenntatinn available at gitbub..com/DDaaaTTexttttte
8

---

Table 3: We evaluated the performance of proposed models across multiple robustmess benchmarks, using
appropriate evaluation metrics for each. the majority cases, our methods outperformed the baselines.
Dataset
ImageNet-C
ImageNet-A Top- Acct
ImageNet-O)
ImageNet-R
ImageNet-Sketch
Metric
mCEl
AUPRT
Top- Err Rate!
Top- Acc1
VIT"
71.14
0.18
18.31
96.89
37.13
DeÎT
70.26
0.73
19.56
94.23
41.68
RVT
68.57
9.45
22.14
63.12
50.07
FourierFormer
71.07
0.69
18.47
94.15
38.36
VIT-RKDE (Huber)
68.69
6.98
25.61
64.33
45.63
ViT- SIT-RKDE (Hampel)
68.55
6.34
26.14
62.16
45.76
ViT-SPKDE
68.34
8.29
28.42
61.39
50.13
ViT-MoM
69.11
2.29
28.08
76.44
36,92
FouM
70.62
2.42
28.86
74.15
39.44
FGSM
PGD
SPSA
50
Accoraa
VIT
30
Top A.cuuray
Dail
10
Foureer
RVT
30
20 SPIDE
10 VIT-HMM
20
Budget
Purturbation Budget
Purturbation Budget
(b)
(c)
Figure 3: top-1 classification accuracy V.S. perturbution budget curves on ImageNet against three untargeted attack methods under the loe norm. The proposed set of VIT with robust self-attention mechanisms shows stronger defense under all attack methods with different perturbation budgets.
approaches do not alter the model architecture, each model employs 5. 5.7M parameters. We have also implemented leading-edge methods, including DeiT with hard distillation (Touvron et al., 2021b), FourierFormer (Nguyen et al., 2022c), and Robust Vision Transformer (RVT) (Mao et al., 2022), as our baselines. It's important to note that, for a fair comparison with RVT, we only incorporated its position-aware attention scaling without further architectural modifications. Consequently, the sulting RVT model comprises approximately 7 7.2. parameters. To evaluate adversarial robustness, utilized adversarial examples generated by untargeted white-box attacks, which included the single-step attack method FGSM (Goodfellow et al., 2014), multi-step attack method PGD (Madry al., 2017), and score-bac score-based black-box attack method SPSA (Uesato et al., 20.8). These attacks were applied to the entire validation set of ImageNet. Each attack distorts the input image with perturbation budget
1/255 under loo norm, while the PGD attack uses 20 steps with step size of a = 0.15. In addition, we assessed our methods on multiple robustness benchmarks, which include images derived from the original ImageNet through algorithmic corruptions or outliers.
Table presents the results under ad- versarial attack. On clean ImageNet, our performance aligns closely with RVT and DeiT, the leading perform- ers. Notably, our methods outperform RVT under several adversarial attack types, particularly when employing the ViT-SPKDE method. Figure 3 il- lustrates the relationship between ac- curacy and perturbation budget across three attack methods. We observe that transformers equipped with robust self-attention mechanisms offer sig- nificantly enhanced defense capabili- ties across different perturbation bud- gets, with their advantages amplifying as the level of perturbation increases, as expected. We provide more ablation studies in Appendix E that explore dif-
Efficiency Memory Comparison
Nettook 3026-ottp
12300
Phacitmmm DIU Darnecyy
treull
x
X
4002
Avaroged Computation Spoed RRswe
Figure 4: Comparison of averaged computation speed (measured by iteration per second) and maximum GPU memory (measured by the CUDA max memory allocated function) on ImageNet classification task. The results are measured under the Trans- former base models with different capacities: Tiny (5.7M param- eters), Small (22M), and Base (86M).

---

ferent design choices for each proposed robust KDE attention. Table 3 displays the results across multiple robustmess benchmarks, employing appropriate evaluation metrics for each. In most in- stances, the best performance is achieved by our proposed methods, which clearly improve upon existing baselines. Additionally, Figure demonstrates the scalability of our methods as model size increases. We have excluded the SPKDE-based method from this analysis due to its heightened computational demands. The results indicate that both the computation speed and memory increase as model capacity expands for all methods, The MoM-based robust attention closely mirrors the vanilla Transformer, while the RKDE-based robust attention, with one-step approximation, also demonstrates scalability with larger models.
Table 4: A comparison of classification accuracy using the UEA Time Series Classification Archive. Trans-
formers that incorporate our proposed robust attention mechanisms outperform the existing baselines.
Model/Dataset
Ethanol
Heart
PEMS-SI
Spoken
UWave
Transformer
33.70
75.77
82.66
99.33
84,45
FourierFormer
36.12
76.42
86.70
99.00
86.66
Transformer- RKDE (Huber)
34.72
75.84
84.28
99.28
86,49
Transformer-SPKK
36.09
76.29
86.02
99.36
88.14
Transformer-
38.41
73.24
86.75
97.64
82.97
FourierFormer-Monn
39.89
74.11
87.63
98.12
85.43
4.3
UEA Time Series Classification
Lastly, we conducted experiments using five datasets from the UEA Time-Series Classification Archive (Bagnall al., 2018), and compared the outcomes across various methodologies (Table 4). The baseline implementation and datasets were adapted from Wu et al. (2022). Our findings indicate that our proposed approaches can notably enhance classification accuracy.
5 Conclusion and Future Work
In this work, we explored the link between the dot-product self-attention mechanism and non- parametric kernel regression. This led the development of family of fortified transformers, which leverage robust KDE as an alternative to dot-product attention, mitigating the impacts of con- taminated samples. We proposed two variants of robust self-attention mechanisms designed to either down-weight or filter out potential corrupted data, both of which can be seamlessly integrated into commonly used transformer models. As our ongoing effort, we are exploring more efficient tech- niques for estimating the weight set for robust KDE in extremely large models, and incorporating regularization strategies to mitigate the instability of kernel regression with outliers.
6
Acknowledgment
Xing Han and Joydeep Ghosh acknowledge support from Intuit Inc. Nhat Ho acknowledges support from the NSF IFML
2MMLL2019840 2019844 and the NSF Al Institute for Foundations of Machine Learning.
References
AI-Rfou, R., Choe, D., Constant, N., Guo, M., and Jones, L. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3159-3166, 2019.
Alon, N., Matias, Y., and Szegedy, M. The space complexity of approximating the frequency mo- ments. In Proceedings of the tienty-eighth annual ACM symposium Theory of computing, pp- 20-29, 1996.
Baevski, A. and Auli, M. Adaptive input representations for neural language modeling. In Inter- national Conference on Learning Representations, 2019, URL httpss//pperrrrii..e////uuumanna ByxZX20qFQ.
Bagnall, A., Dau, H. A., Lines, J., Flynn, M., Large, J., Bostrom, A., Southam, P., and Keogh, E. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv: 181 1.1.1.00 1.00055, 2018.
10

---

Bhojanapalli, S., Chakrabarti, A., Glasner, D., Li, D., Unterthiner, T., and Veit, A. Understanding robustness of transformers for image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10231-10241, 2021.
Brown, T., Mann, B.. Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Cao, S. Choose a transformer: Fourier or galerkin. Advances in Neural Information Processing
Systems, 34, 2021.
Chen. L.. Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1
2021.
Child, R., Gray, S.. Radford, A., and Sutskever, Generating long sequences with sparse transform- ers. arXiv preprint arXiv:/9044 10509, 2019.
Choromanski, K. M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J. Q., Mohiuddin, A., Kaiser, I.., Belanger, D. B., Colwell, J., and Weller, A. Rethinking attention with performers. International Conference on Learning Representations, 2021. URL,
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., and Salakhutdinov, R. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv: 2019.
Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, Universal transformers. termational Conference on Learning Representations, 2019. URL http:://oeemeevie.nne//oonn id=HyzdRiR9y7.
Deng. J., Dong, W., Socher, Li, Li-.., K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database, In 2009 IEEE conference on computer vision and pattern recognition, pp. 248- 255. leee, 2009.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 4171-4186. Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/1//N19-1422.. URL https:/accathhlooy.oog//119
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 1929, 2020.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. URL https://oeenrriew..t///ouutt YicbFiaadyiy..........
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, 7.,, Malik, J., and Feichtenhofer, C. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6824-6835, 2021.
Fox, J. and Weisberg, S. Robust regression. An R and S-Plus companion to applied regression, 91, 2002.
Gabbur, P., Bilkhu, M., and Movellan, Probabilistic attention for interactive segmentation. Ad- vances in Neural Information P Processing Systems, 34, 2021.
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 1412.6:412.6772 2014.

---

Hampel, F. R., Ronchetti, E. M., Rousseeuw, P., and Stahel, W. A. Robust statistics: the approach based on inftuence functions. Wiley-Intersciences New York, 1986.
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions 2019.
and perturbations. arXiv preprint
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of out-of-distriutiin gen- eralization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8349, 2021a.
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021b.
Huber, P. J. Robust estimation of location parameter. In Breakthroughs in statistics, pp. 492-518.
Springer, 1992.
Huber, P. Robust statistics. In International encyclopedia of statistical science, pp.
Springer, 2011.
Humbert, P., Le Bars, B., and Minvielle, L. Robust kernel density estimation with median -ff-mann---- means principle. In International Conference Machine Learning, pp. 9444-9465 PMLR, 2022.
Janner, M., Li, Q., and Levine, S. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273-1286, 2021.
Jerrum, M. R., Valiant, L. G., and Vazirani, V. V. Random generation of combinatorial structures from uniform distribution. Theoretical computer science, 43:169-188, 1986.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are mns: Fast autoregressive transformers with linear attention. In International Conference Machine Learning, pp. 5156- 5165. PMLR, 2020.
Khan, S., Naseer. M., Hayat, M., Zamir, S. W., Khan, F. S., and Shah, M. Transformers in vision: A survey. ACM Computing Surveys (CSUR), 2021.
Kim, J. and Scott, C. Robust kernel density estimation. The Journal of Machine Learning Research, 13:2529-2565, 2012.
Kreuzer, D., Beaini, D., Hamilton, W., Létourneau, V., and Tossou, P. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 34, 2021.
Li, N., Liu, Y., Wu, Y., Liu, S., Zhao, S., and Liu, M. Robutrans: A robust transformer-basee text- to-speech model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8228-8235, 2020.
Lin, T., Wang, Y., Liu, X., and Qiu, X. A survey of transformers. arXiv preprint arX1v:2106.04554, 2021.
Liu, J., Singhal, T., Blessing, L. T., Wood, K. L., and Lim, K. H. Crisisbert: a robust transformer for crisis classification and contextual crisis embedding. In Proceedings of the 32nd ACM Conference on Hypertext and Social Media, pp. 133,141, 2021a.
Liu, Y.. Ott. M.. Goyal, N., Du, J., Joshi, M.. Chen, D.. Levy, O., Lewis, M.. Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 190i 1692, 2019.
Liu, Z., Lin, Y. Cao. Y. Hu, H. Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierar- chical vision transformer using shifted windows. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 10012-10022, 2021b.
Liu, 7.,, Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., and Hu, H. Video swin transformer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
12

---

Lu, Y., Li. Z, He. D., Sun, Z., Dong, B., Qin. T., Wang, L., and Liu, T.Y.. Understanding and improving transformer from murti-particle dynamic system point of view. arXiv preprint arXiv: 190.00260 2019.
Madry, A., Makelov, A., Schmidt, L.,, Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: /706, 777600008. 2017.
Mahmood, K., Mahmood, R., and Van Dijk, M. On the robustness of vision transformers to adver- sarial examples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7838-7847, 2021.
Mao, X., Qi, G., Chen, Y., Li, X., Duan, R., Ye, S., He, Y., and Xue, H. Towards robust sion transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 12042-12051, 2022.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv: 07843, 2016.
Morris, X., Lifland, E., Yoo, J. Y., Grigsby, J., Jin, D., and Qi, Y. Textattack: A frame- work for adversarial attacks, data augmentation, and adversarial training in nlp. arXiv preprint arXiv:2005.05909, 2020.
Nadaraya, E. A. On estimating regression. Theory of Probability & Its Applications, 91):141-142, 1964.
Nguyen, T., Nguyen, T., Do, H., Nguyen, K., Saragadam, V., Pham, M., Nguyen, K., Ho, N., and Osher, Improving transformer with an admixture of attention heads. In Advances in Neural Information Processing Systems, 2022a.
Nguyen, T., Nguyen, Le, D., Nguyen, K., Tran, A., Baraniuk, R., Ho, N., and Osher, S. Improving transformers with probabilistic attention keys. In International Conference on Machine Learning, 2022b.
Nguyen, T., Pham, M., Nguyen, T., Nguyen, K., Osher, S.J., and Ho, N. Fourierformer: Transformer meets generalized Fourier integral theorem. Advances in Neural Information Processing Systems, 2022c.
Parzen, E. On estimation of probability density function and mode. The annals of mathematical statistics, 13(3)10055-177 1962.
Paul, S. and Chen, P.-Y. Vision transformers are robust learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 1001-2 2022.
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training OpenAl report, 2018.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervi- sion. In International Conference on Machine Learning, pp. 8748-8763 PMLR, 2021.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http://immloog/aaaccss//22220000
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero- shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Reddy, An introduction to the finite element method, volume 1221. McGraw-Hill New York, 2004.
13

---

Rosenblatt, M. Remarks on some nonparametric estimates of density function. The annals of mathematical statistics, pp. 832-837, 1956.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)22112255
journaa
2015.
Sander, M. E., Ablin, P., Blondel, M., and Peyré, G. Sinkformers: Transformers with doubly stochas- tic attention. In International Conference on Artificial Intelligence and Statistics, pp. 3515-3330. PMLR, 2022.
Schlag, I., Irie, K., and Schmidhuber, J. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, pp. 9355-93666 PMLR, 2021.
Subramanya, A., Saha, A., Koohpayegani, S. A., Tejankar, A., and Pirsiavash, H. Backdoor attacks on vision transformers. arXiv preprint 0rXiv:2206.08477, 2022.
Tang, B. and Matteson, D. S. Probabilistic transformer for time series analysis. In Beygelzimer,
A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances Neural Information Processing Systems, 2021. URL
Tay. Y., Dehghani, M., Bahri, D., and Metzler, D. Efficient transformers: A survey, arXiv preprint arXiv:2009.06732, 2020.
Touvron, H., Cord, M., Douze, M., Massa, F, Sablayrolles, A., and Jégou, H. Training data-cfficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021a.
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data cata-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021b.
Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., and Salakhutdinov, R. Transformer dis- section: An unified understanding for transformer's attention via the lens of kernel. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UCNLLP), pp. 4344-4353, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/119-111 URL http:///cccannooooooogg/////////
Uesato, J., O'donoghue, B., Kohli, P., and Oord, A. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learning, pp. 5025-5034. PMLR, 2018.
Vandermeulen, R. A. and Scott, C. Robust kernel density estimation by scaling and projection in hilbert space. Advances in Neural Information Processing Systems, 27, 2014.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L.,, and Polo- sukhin, I. Attention is all you need. In Advances in neural information processing systems, pp. 5998 6008, 2017a
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017b. URL https//poooeddinggnnuuiip.c..cpp 14a8550 Paper.... Paper.pdf.
Wang, H., Ge, S., Lipton, Z., and Xing, E. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Wang, S., Li, B., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.00 047688 2020.
Wang, X., Girshick, R., Gupta, A., and He, K. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
14

---

Welsch, R. E. and Becker, R. A. Robust non-linear regression using the dogleg algorithm. Technical report, National Bureau of Economic Research, 1975.
Wu, H., Wu, J., Xu, J., Wang, J., and Long, M. Flowformer: Linearizing transformers with conser- vation flows. In International Conference on Machine Learning, 2022.
Yang, J., Gupta, A., Upadhyay, S., He, L.., Goel, R., and Paul, S. Tableformer: Robust transformer modeling for table-text encoding. arXiv preprint arXiv:2203.00274 2022.
Yang. Z.,, Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv: 08237, 2019.
Zhang. S. and Feng, Y. Modeling concentrated cross-attention for neural machine translation with Gaussian mixture model. In Findings of the Association for Computational Linguis- fies: EMNLP 2021, pp. 1401-1411, Punta Cana, Dominican Republic, November 2021, As- sociation for Computational Linguistics. doi: 10.18653/v1/ findings-emmlpp112). 121. URL http://aclantoolgy.or///// www.22200.....---eeeee 1entinggeemml..11...
Zhou, D.. Yu, Z., Xie, E., Xiao, C., Anandkumar, A., Feng, J., and Alvarez, J. M. Understanding the robustness in vision transformers. In International Conference Machine Learning, pp- 77374-27394. PMLR, 2022.
15

---

Supplementary Material of "Designing Robust Transformers using Robust Kernel Density Estimation"
A
The Non-parametrie Regression Perspective of Self-Attention
Given an input sequence X ........ ØN T E NRND>= of N feature vectors, the self-attention mechanism transforms into another sequence [h1,--. .xn]' IR as follows:
q: kg VD Uj for i 1, I....,N.
h Σ
softmax
(13)
JE[N]
The vectors q. kú and v are the query, key and value vectors, respectively. They are computed as follows:
9 42 .........
qNl Q= XW E ERND
k1, k2,
kN = K = XW ER
(14)
[1,, 12, UN) = V = XW, ERNXD.
where Wo, WK E RDDD. Wy E RDxDD= are the weight matrices. Equation (13) can be written in the following equivalent matrix form:
QK
softmax
V
(15)
where the softmax function is applied to each row of the matrix (QKT)/VD. Equation (15) is also called the "softmax attention". Assume we have the key and value vectors (kz, Vy)se[N] that is collected from the data generating process
v v= = f(k) tE,
(16)
where é is some noise vectors with B[e] = O, and f is the function that we want to estimate. If (kgyEEN] i.i.d. samples from the distribution p(k),, and p(v, k) the joint distribution of (v, k) defined by equation (16), we have
/ 2
v kll(,k) dv, p(k)
f(k) [[[[k]]
p(vik)dv
(17)
We need to obtain estimations for both the joint density function p(v, k) and the marginal density function pkk) to obtain function f, popular approach is the kernel density estimation:
(v k)
Σ JEIN]
ko (lv, A:] [v, k,))
(18)
(k(
Σ
Ko (k - (k;;)
(19)
N
JE(N]
where [v, k] denotes the concatenation of and k. ki could be isotropic Gaussian kernel: ki x') exp (-1|x x²|1? (1//(222)) )), we have
Pa (v. k)
ka - (e(v-vy)kall (k k;). Σ
(20)
N
JE[N]
Combining equations (19), (20), and (17), we obtain the NW estimator of the function f as
v vo k)
(k)
dv
(21)
JRD
(k)
EJE(N) ((v--v)koo (k- -k;)
Jgp
dv EjE[N] ko (k k;)
EiE(N) to (k k2) I V -ooov- ((vv-)))vv vj)dv
ΣjEN kio k:)
ΣjEMM Vykio (k kj) ΣjE(N Ro (k,)
(22)
16

---

Now we show how the self-attention mechanism is related to the NW estimator. If the keys [ENN are normalized
fo(9) EjEIN) V exp (-1gq k k,]12/2002)
EJE(NI exp (-llq k,,1l//207
EJEIN) V, exp [- (Iall2 + ||> /200] exp (q ky/o²)
=
EJEIN exp | (1lg]12 2 [k,l(3) /20 exp (q k 02)
exp (q) kg//²2)
LjE(N] explaTks/0" V
JEIN]
Σ
softmax (q)k// kg/o²) Vj.
(23)
JE[N]
Then estimating the softmax attention is equivalent to estimating fo (q).
B Details on Leveraging Robust KDE on Transformers
For simplicity, the Huber loss function the demonstrating example, which is defined as follows:
x² /2 0 x aa 2r
p(x)
xxx)= = oxr a²
(24)
where a is a constant. The solution to this robust regression problem has the following form:
Proposition 1. Assume the robust loss function p is non-decreasing in [0, p(0) and
lim p(x) Define ((x)= == ((()) and assume (0) lim p'(x) exists and finite. Then
the optimal Probust can be written as
Protuss
Σ
wyko (zz.
JE(N]
where w = (wi,-.. ,WN) E AN. with each Wy oc vb (Il (xz,.) - Probuse [ Here An denotes the ne-imennsooal probability simplex.
Proof. The proof of Proposition 1 mainly adapted from the proof in Kim & Scott (2012). Here, provide proof of completeness. For any He. denote
(
Σ p (1kkaaa) JE(M]
- pllpHn..)
N
Then have the following lemma regarding the Gateaux differential of J and necessary condition for Protust to be optimal solution of the robust loss objective function in equation (5).
Lemma 1. Given the assumptions on the robust loss function in Proposition 1, the Gateaux dif- ferential of p E Hr, with incremental h E Hk,, defined as 5J (p: h), is
J(p + Th) J(p) - H),
6J(p; = lim T-40
where the function V Hr. - Hy, is defined as:
V(p)
Σ JE[N]
v (1ko - pllme,)) (Ko (2x))))) - p).
N
A necessary condition for Produus is V (Prosnnes) = 0.
17

---

The proof of Lemma can be found in Lemma 1 of Kim & Scott (2012). Based on the necessary condition for Protust Lemma 1, i... V (Protusa) O. we have
Σ JE[N]
Probuse (atnesrlltte (2t
Probust) O.
Direct algebra indicates that Probust EJENN) ( where (w), ,WN) AN. and Wj & vé (T.j.) - -Pobuaall ( As consequence, we obtain the conclusion of the proposition.
For the Huber loss function, we have that
1, 0 OsIsa
((x)
a/2,
Hence, when the error (at -), Probust |7 over the threshold a, the final estimator will down-weight the importance of Ko (x.. This is in sharp contrast with the standard KDE method, which will assign uniform weights all of the k (zr. :) As mentioned in the main paper, the estimator provided Proposition is circularly defined, as Probust defined via w, and w depends on Probust- Such issue can be addressed by estimating with iterative algorithm termed as kemnel- ized iteratively re- re-weighted least-squares (KIRWLS). The algorithm starts with randomly initialized (0)) E An and perform the following iterative updates between two steps:
wb (1 - Probuss
(k) Probust
Σ SE[N]
w/ (1-1)) Ko (2)
(o
(R) wi
(25)
ΣjEN v (11oo(8).)) ploc
Her
Note that, the optimal Probust is the fixed point of this iterative update, and the KIRWLS algorithm converges under standard regularity conditions. Furthermore, one can directly compute the term ko (zz..) - - via the reproducing property:
Me.
Kin (Zx::)) :)- Protuss
= -2
Σ me(N]
(k (k-1)) k) (Im: xz,) + Ko (z)j, xz))
Alite
Σ
Wm
(k-1) (k-1) kin (mm Dnn)
(26)
ME(N].EE[[
Therefore, the weights can be updated without mapping the data to the Hilbert space.
C Fourier Attention with Median of Means
We introduce the Fourier Attention coupled with the Median of Means (MoM) principle and show how this is robust to outliers. For any given function ó IR + R and radius R, we randomly divide
the keys (kr heeN into B subsets II...., IB of equal size where |41 [1] sint
| S.
Define PR. (qi) Exem as
I o(
ssin Riay -N,)), then the MoM Fourier attention is defined Rigis kis)
1 EiEL V. 1- oi
IP
sin (n(R(gg k1))))
h 3
Rias KGp)
(27)
modian (pn,1, (q)) PR.ID (p))
where Im is the block such that pnn k)) achieves its median value. To shed light into the robustness
of Transformers that use Eq. (27) the attention mechanism. we demonstrate that the estimator
pa(()) median . {q)} is a robust estimator of the density function p(q) of the
PRRI
keys. We first introduce a few notations that are useful for stating this result. Denote C = {1 <
i < N h; is clean) and O = [1 < i < N k; is outlier). Then, we have C C 0 and
CUO = (1,2.... N). The following result establishes a high probability upper bound on the sup-norm between (R(g) and p(g).
18

---

Theorem 1. Assume that the function & satisfies 6(sin(2)/2))d = 0 for all S S m. and I
dz < oo for some m E N. Furthermore, the density function p(q) satisfies supy (p(g) < oc. The number of blocks B and the number of outliers |O] are such that B (2 + O) where 6 is the failure probability. Then, with A sufficiently large and ó sufficiently small, with probability at least expl-2A2 B) we find that
[0]
B for the radius R
1 |pR - poloo S C( Rmm+I +V
BRD log R (ogg)22)) (2/0)
N
where C. some universal constans.
Remark The result of Theorem indicates by choosing O Am+1)=+0 ) the rate of PR to p under the supremum norm is O 3tm Xm11)T)). With that choice of R, when N approaches infinity, the MoM estimator PR is a consistent estimator of the clean distribution of the keys. This confirms the validity of using Pn to robustify p and similarly the usage of MoM Fourier attention Eq. (27) as a robust attention for Transformers.
Proof. From the formulation of the MoM estimator (n(4), we obtain the following inequality
B (1) p(2)22)) 2 2
(sup (p(()) - p(q)) 2 c) C {sup Σ
This bound indicates that to bound P(láx(()) p(a) 2 e), it is sufficient to bound P((sup, Enn 1 llpn, p(a)lee) B).. Indeed, for each 1 b S B, find that
Therefore, we have
(sup, (a)-))))2e)))
which leads to supg E (p0,1, (p00)22 shows that
1=1 (sup, (10n This inequality
B P(ssup Σ
B 2
(PR.. p()))())
2
})
P
(oup, (0) p(g))2e))
To ease the presentation, denote W. ( Ig n = 0}. Then, the following inequalities hold
(1xn.1, (0) and B 5 b B
B
Σ W. Σ Wo
sup, ( [I,.I1 (p))2())
beb
D
W. 101
be
Σ (Wb [[(,)) B P(sup Ir,,, (g) p(g)] <) + +10,, beB
where we assume without loss of generality that E B, which is possible due to the assumption that B > (2 ()) We now prove the following uniform concentration bound:
Lemma 2. Assume that (b2) S C for all |z| < 1 for some universal constant C. We have
RD log R log (2/() )) < 8.
P(sup pR, (q) - p(q) 2 C(
Rm+I +
41
q
19

---

Proof of Lemma 2. By the triangle inequality, we have
sup IRRI ( (q) p(q)) S sup PR, (q) - ElpR, (9)l + sup ElRR.. (q)) p(g)).
q
4
To bound s PR. (g) E (g)l,,
Bemstein's inequality along with the brack-
eting entropy under L1 norm in the space of queries q- In particular, we denote Y;
A I1 6 sin, Ríg, Rsa22) where A -SR ((= d2 for all i 41 Since sin(R(g) Mgg keg)
< D. we obtain that [Y]] S CD CDRP/AD where C' is the constant such that ((2) C' when S 1. Furthermore, E||y]]
kg)))/(((g) k)))) < 1 for all 1 <
By choose
C n
)) then find that R2 log (1/g(2
11
6 P(sup (q) p(g)) >e) 5 2(2+
Collecting the above inequalities leads to
B P((sup (10m:11000-p00022 2 z S exp(-2BA
where A
As consequence, we obtain the conclusion of the theorem.
2+5
D Dataset Information
WikiTexx- The dataset contains around 268 words and its training set consists about 28 articles with 103M tokens, this corresponds text blocks of about 3600 words. The validation set and test sets consist of 60 articles with 218K and 246K tokens respectively.
ImageNet We use the full ImageNet dataset that contains 1.28M training images and 50K vali- dation images. The model learns to predict the class of the input image among 1000 categories. We report the top-1 and top-5 accuracy on all experiments. The following ImageNet variants are test sets that are used to evaluate model performance.
ImageNet-C For robustness on common image corruptions, we use ImageNet-C (Hendrycks & Dietterich, 2019) which consists of 15 types of algorithmically generated corruptions with five levels of severity. ImageNet-C uses the mean corruption error (mCE) as a metric: the smaller mCE means the more robust the model under corruption.
This dataset contains real-world adversarially filtered images that fool current Ima- geNet classifiers. A 200-class subset of the original ImageNet-1K3 1000 classes is selected that errors among these 200 classes would be considered egregious, which cover most broad categories spanned by ImageNet-IK.
ImageNet-A
ImageNet-OO This dataset contains adversarially filtered examples for ImageNet out-of- distribution detectors. The dataset contains samples from ImageNet-22K but not from ImageNet- IK, where samples that are wrongly classified as an ImageNet- KK class with high confidence by ResNet-50 are selected. We use AUPR (area under precision-recall as the evaluation metric.
This dataset contains various artistic renditions of object classes from the original ImageNet dataset, which discouraged by the original ImageNet. ImageNet-R contains 30,000 image renditions for 200 ImageNet classes, where subset of the ImageNet-1K K classes is chosen.
ImageNet-R
This dataset contains 50,000 images, 50 images for each of the 1000 ImageNet classes. The dataset is constructed with Google Image queries "sketch of xxx", where is the standard class name. The search is only performed within the "black and white" color scheme.
ImageNet-Sketch
modeling-
20

---

E Ablation Studies
In this section, provide additional results and ablation studies that focus on different design choices for the proposed robust KDE attention mechanisms. The detailed experimental settings can be found in the caption of each table.
Table 5: Perplexity (PPL) and negative likelihood loss (NLL) of our methods (lower part) and baselines (upper part) WikiText-105 103 using medium version of Transformer. The best results are highlighted in bold font and the second best are highlighted in underline. On clean data, Transforer-SPKDDE achieves better PPL and NLL than other baselines. Under random swap with outlier words, Transformers with MoM self-attention show much better performance.
Clean Data
Word Swap
Method (median version)
Valid PPL/Loss
Test PoS/Loss
Valid POSS
Test PPL/Loss
Transtormer (Vaswam et al., 20176)
27.90/3.2
29.60/3.37
63.36/4.31
68. 68.12/4.36
Performer (Choromanski et al., 2021)
27.34/3.31
29.51/3.36
64.72/4.30
67.43/4.34
Transformer-Me MGGK (Nguyen et al., 2022b)
27.28/3.31
29.24/3.36
64.46/4.30
67.31/4.33
FoumeerFormer (Nguyen et al.. 2022c)
26.51/3.29
28.01/3.33
63.74/4.28
65.27/4.31
Transformer- RKDE (Huber)
26.12/3.28
27.8913.32
49.37/3.85
51.22/3.89
Transformer-KIl RKDE (Hampel)
25.87/3.27
27. 21.44/3.31
48.62/3.83
51.03/3.88
Transformer-SKKDE
25.76/3.27
27.35/3.31
46.91/3.79
49.14/3.84
Transformer-Mok.
28.26/3.34
29.98/3.38
45.35/3.75
47.92/3.81
FourierFormer-MMok,
27.13/3.31
29.02/3.36
43,23/3.71
44.97/3.74
Table 6: Test PPL/NLL loss versus the parameter a of Huber loss function defined in Eq. (24) (upper) and Hampel loss function (Kim & Scott, 2012) (lower; we x and 3 as parameters b and c) on original and word-swapped Wiki-103 dataset. The best results are highlighted bold font and the second best are highlighted in underline. We choose a 0.4 in rest of the experiments.
Robust Loss Parameter
0.1
0,2
0.4
0.6
0.8
Clean Data
32.92/3.48
32.87/3.48
32.29/3.47
32.38/3.48
32.46/3.48
32.48/3.48
Word Swap
55.82/3.99
55.97/3.99
55.68/3.99
56.89/4.01
57.26/4.01
57.37/4.01
Clean Data
32.67/3.48
32.32/3.48
32.35/3.48
32.47/3.48
32.53/3.48
32.58/3.48
Word Swap
58.02/4.03
57.86/4.03
57.92/4.03
58.24/4.04
58.37/4.04
58.43/4.04
Table 7: Top-1 classification accuracy on ImageNet versus the parameter of Huber loss function defined in Eq. (24) under different settings. The best results are highlighted in bold font and the second best are highlighted in underline. We choose 0.2 in rest of the experiments.
Huber Loss Parameter
0.1
0.2
0.4
0.6
0.8
Clean Data
71.45
72.83
71.62
71.07
70.65
70.34
FGSM
56.72
55.83
55.34
54.87
54.02
52.98
PGD
46.37
44.15
43.87
43.25
42.69
41.96
SPSA
52.38
52.42
51.69
51.34
50.97
48.22
Imagenet-C
45.37
45.58
45.63
45.26
44.63
43.76
21

---

Table 8: Top-1 classification accuracy on ImageNet versus the parameter a of Hampel loss function defined in Kim & Scott (2012) under different settings. We use 2 and 3 parameters b and The best results are highlighted in bold font and the second best are highlighted in underline. We choose a 0.2 in rest of the experiments.
Hampel Loss Parameter
0.1
0.2
0.4
0.6
0.8
Clean Data
71.63
72.94
71.84
71.23
70.87
70.41
FGSM
56.42
55.92
55.83
55.66
54.97
53.68
PGD
45.18
44.23
43.89
43.62
43.01
42.34
SPSA
52.96
52.48
52.13
51.46
50.92
50.23
Imagenet-C
44.76
45.61
46.04
46.13
45.82
45.31
the parameter of SPKDE defined in Eq. (6) under different settings. / = = > 1, where & is the percentage of anomalous samples. A larger 3 indicates more robust model. The best results are highlighted bold font and the second best highlighted in underline. We choose 1.4 in of the experiments.
Table 9: Top-1 classification accuracy on ImageNet
8
1.05
1.2
1.4
1.6
1.8
2
Clean Data
74.25
73.56
73.22
73.01
72.86
72.64
FGSM
53.69
55.08
56.03
55.37
54.21
53.86
PGD
42.31
43.68
44.51
44.32
44.17
43.71
SPSA
51.29
52.02
52.64
52.84
52.16
51.39
Imagenet-C
44.68
45.49
44.76
44.21
43.96
43.33
Table 10: Top-1 classification accuracy ImageNet versus the number of iterations of the KIRWLS algorithm in Eq. (25) employed in Transformer-RKDE. Since the increased number of iterations does not lead to significant improvements of performance while the computational cost is much higher, we use the single-step iteration of the KIRWLS algorithm in Transformer-RKDE.
Huber Loss
Hampel Loss
Iteration #
1
2
3
5
5
Clean Data
72.83
72.91
72.95
72.98
72.94
72.99
73.01
73.02
FGSM
55,83
55.89
55.92
55.94
55.92
55.96
55.97
55.99
PGD
44.15
44.17
44.17
44.18
44.23
44.26
44.28
44.31
SPSA
52.42
52.44
52.45
52.45
52.48
52.53
52.55
52.56
Imagenet-C
45.58
45.61
45.62
45.62
45.61
45.66
45.68
45.71
Table 11: Computation time (measured by seconds per iteration) of baseline methods, Transformer- SPKDE, Transformer-Molt and SoinsformerRRKD with different number of KIRWLS iterations. Seaeffomer-SPKKDD requires longer time since it directly obtains the optimal set of weights via the QP solver.
Iterations of KIRWLS
DeiT
RVT
SPKDE
MoM-KDE
2
5
Time (s/it)
0.43
0.51
0.68
0.84
0.35
0.41
1.45
0.37
22

---

(a)
(b)
(c)
(d)
Figure 5: Contour plots of density estimation of the 2-dimensional query vector embedding in an attention layer of the transformer when using (b) KDE (Eq. (4)) and (c) RKDE after one iteration of Eq. (25) with Huber loss (Eq. (24)), (d) KDE with median- of-msans principle (Eq. (10)), where (a) is the true density function. We draw 1000 samples (gray circles) from a multivariate normal density and 100 outliers (red cross) from a gamma distribution as the contaminating density. RKDE and KDE with the median-of- means principle can be less affected by contaminated samples when computing self-attention as nonparametric regression.
23


Reviews:
[{"review_id": "rygXHBgjF4", "reviewer": "Reviewer_SN7k", "summary": "The paper describes and leverages the connection between the self-attention mechanism and KDE estimation to motivate replacing dot-product attention with a robust KDE approach to enable more robust transformer models. The experiments illustrate that the proposed approach can maintain performance on clean data while improving performance on contaminated data.", "questions": "N/A", "limitations": "The authors accurately describe limitations and potential solutions throughout the paper and in the Conclusion and Future Work section.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Given the wide use of transformer models, the paper tackles an important topic, as robustness to training data contamination can improve their performance.\n* The proposed approach is well motivated by the connection to non-parametric regression.\n* The experiments show very promising results on both clean data and contaminated data, across \n* The proposed approach does not introduce additional parameters.", "weaknesses": "Practicality of the approach for large datasets: the proposed approach incurs additional computational complexity even with the proposed approximations. The experiments are performed on small versions of the models. It's unclear how well this can be applied to large scale training of transformer models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer `SN7k`,\n\n\nWe would like to thank you for your valuable comments and positive feedback. We now address your major concern below.\n\n**Q: The proposed approach incurs additional computational complexity even with the proposed approximations. The experiments are performed on small versions of the models. It's unclear how well this can be applied to large-scale training of transformer models.**\n\n**A**: With respect to computational complexity, it's crucial to highlight that if we operate under the assumption of having a clean training set—where contaminated samples due to factors like distribution shifts, adversarial attacks, or data poisoning only manifest during the inference stage—then the robust attention mechanism only needs application during inference. This can lead to a significant reduction in the time spent calculating the robust attention vector during training. However, when we don't have prior knowledge about the quality of the data, contamination can potentially occur during both training and inference. In such cases, it becomes prudent to invest the extra computational time to shield against outliers throughout the entire pipeline.\n\nIn terms of models in different scales, for the robust language modeling task outlined in Section 4.1, we've carried out assessments using both the small and medium versions of Transformers. You'll find these detailed in Table 5 in Appendix E, where the pertinent text is highlighted in red. Additionally, we've also contrasted the time complexity and memory usage of Vision Transformers employing tiny, small, and base Transformer backbones respectively - see Figure 4 and the accompanying discussion for this.\n\nWe wish to emphasize that we've done our utmost to ensure that our experimental results are as thorough as possible. However, we were constrained by both time and the available computational resources at the time of submission. Before the rebuttal process begin, we were able to perform extra experiments using Transformers with greater capacities. The results of these investigations can be found in the general announcement.\n\nPlease do not hesitate to let us know if you have additional questions.\n\nSincerely,\n\nAuthors"}, {"review_id": "l3a2xTGfPD", "reviewer": "Reviewer_c4Ug", "summary": "This paper introduces fortified transformers that employ robust KDE as an alternative to dot-product attention, self-attention, mitigating the impacts of attacked images. Especially, this paper proposed two variants of robust self-attention mechanisms based on kernel density estimators (KDE). Extensive experimental results demonstrate the effectiveness of the proposed approach on diverse tasks.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-written.\n- This paper analyzes the characteristics of the proposed methods, especially their limitation.\n- Extensive experiments are conducted on diverse tasks with multiple modalities.", "weaknesses": "- The paper offers a comprehensive study of several methods, but a dominant strategy does not emerge as the results seem to depend largely on the task at hand, as evidenced in the data tables provided.\n- Although the stated limitations are acknowledged, a more in-depth analysis of these shortcomings would enhance the persuasive power. \n   - It is acknowledged that RKDE and SPKDE possess computational inefficiencies. Then, how computationally expensive they are than the original ones which are not robust? It would be better to provide the experimental results.\n   - Suboptimality is a possible limitation of MoM, then how much the extent of MoM's accessibility of MoM is restricted compared to the originals? How those restricted accessibility affects the performances?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer `c4Ug`,\n\nWe would like to thank you for your constructive feedback and insightful suggestions! Below we summarize your major questions and address your concerns.\n\n**Q: The paper offers a comprehensive study of several methods, but a dominant strategy does not emerge as the results seem to depend largely on the task at hand, as evidenced in the data tables provided.**\n\n**A**: Thank you for your constructive feedback - it's greatly appreciated and will certainly help us refine and improve our manuscript. Strengthening model robustness has been a major focus in this field for a long time, yet there isn't a one-size-fits-all solution that ensures robustness under all circumstances. This is because the appropriate approach is heavily dependent on several factors, including the nature of the outlier distribution, the data modality, model architectures, and so on.\n\nOur initial aim was to devise a robust attention mechanism that could serve as a standard component for Transformer models, regardless of the task at hand. This was based on the perspectives of nonparametric regression for the self-attention mechanism. However, our experimental findings across various tasks have shown the unique advantages of individual robust attention mechanisms.\n\nIt's important to note that our conclusion doesn't revolve around a specific method, but rather the general methodology for designing robust Transformers from the nonparametric regression perspective of the self-attention mechanism.\n\n**Q: It is acknowledged that RKDE and SPKDE possess computational inefficiencies. Then, how computationally expensive they are than the original ones which are not robust? It would be better to provide the experimental results.**\n\n**A**: We've made sure to include a comprehensive comparison of computation times across all the methods featured in our experiments, utilizing the widely-recognized ImageNet-1K dataset. You'll find this in Table 11 in the Appendix. We've also conducted an extra evaluation using the WikiText-103 dataset for the language modeling task; this can be found in the recently added PDF file. In addition, all results of our new investigations can be found in the general announcement.\n\n**Q: Suboptimality is a possible limitation of MoM, then how much the extent of MoM's accessibility of MoM is restricted compared to the originals? How those restricted accessibility affects the performances?**\n\n**A**: Firstly, we'd like to highlight that the suboptimality we observed with the MoM-based robust self-attention is primarily on clean datasets. As we mentioned in the main paper (lines 221 - 223), MoM's attention mechanism uses only a portion of the sequences, which is what leads to this suboptimal performance. However, this approach tends to be more effective in discrete contexts, for instance, when identifying and filtering out aberrant words in a sentence.\n\nIn response to your question, the reach of MoM largely hinges on how we define the subsets of the input sequence $I_l, l \\in [B]$, such as the size of the subset $\\mathcal{S}$, whether the subset overlaps with others, and whether the sequential relationship is retained within the subset. These factors are pivotal to the extent Transformers can recover the full view of the input sequence.\n\nIn our experiments, each subset's size, $\\mathcal{S}$, is 80% of the original sequence length. We construct each subset by sampling with replacement from the original sequence while maintaining the original sequential relationship. Our empirical results indicate that the performance on clean datasets closely mirrors that of the standard softmax attention, with little impact on performance.\n\nFurthermore, as we've discussed in Section 4.1, the robust attention that uses the Median-of-Means principle can be useful in defending against word swap attacks.\n\nPlease do not hesitate to let us know if you have any additional questions.\n\nSincerely,\n\nAuthors"}, {"review_id": "FGRXzbymhq", "reviewer": "Reviewer_KTaQ", "summary": "The paper addresses robustness of transformer architecture in both natural language and vision contexts.\nMotivated by classical nonparametric estimation problem, the authors propose a procedure to robustly estimate the self-attention values. While not loosing too much in performance on the original data, the method obtains improvements on corrupted data, with several benchmarks considered. The paper proposes a number of methods to address an important problem of robustness to adversarial contaminations and outliers.\n\nPros:\n- sound experimental results. the proposed method beats sota on a number of benchmarks.\n- ablation study is included in the appendix\n- comparison of time\n\nCons:\n- I did not understand from reading the introduction whether this method is a post-hoc modification or requires in-processing training\n- the theoretical contribution is questionable, I have some questions for Theorem 1 in the appendix, see below\n- lack of originality, the strongest part of the paper is experimental results.\n\nminor comments:\n- line 84: it is best to refer to Nguyen et al to motivate using this form of regression\n- line 84: usually conditional mean is equal to zero E[eps | k] = 0\n- line 86: I suggest to make a precise statement that the derivations only hold for Gaussian kernel earlier.\n- line 256 \"We follow...\" correct the grammar in this sentence\n- theorem 1 in appendix: the sentence \"by adapting Lemma 1 to uniform case\". How exactly do you adapt this to uniform case without any loss in the upper bound? I don't think it's that easy to turn a pointwise bound into a uniform one, so I reckon you do not have a proof of the claimed theorem.\n\n\nQuestions:\n\n- do the methods require fine-tunning the parameters?\n\n- how do you choose hyper-parameters, such as number of bins in MoM, cut-off parameter in Huber loss, etc?\n\n- it is completely unclear from the paper whether robust modification only concerns with inference time, or it changes the training as well?\n\n- robust estimators such as MoM are typically robust towards a corruptions of finite amount of data points. Does it mean that you can be robust to one pixel modifications? Or even one 16x16 patch modifications?\n\n- in the experiments, does the hyperparameter choice procedure require the access to contaminated data?", "questions": "-", "limitations": "-", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "-", "weaknesses": "-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer `KTaQ`,\n\nWe're really grateful for the thorough and constructive feedback you've provided. Your writing suggestions are highly valuable, and we plan to incorporate all of them in our revised manuscript. We now summarize and address your major questions below.\n\n**Cons 1 & Q3**\n\n**A**: Our intention is to integrate our robust attention mechanism into both the training and inference stages. Considering that we lack prior knowledge of the data, contaminated samples could potentially emerge during both stages. Consequently, it is worthwhile to safeguard against outliers throughout the process.\n\nIn the context of model training, our methods adapt the computation of attention vectors across each Transformer layer, rendering them more resistant to outlier contamination. It's crucial to emphasize that this entire procedure is nonparametric and does not introduce any new model parameters. Nevertheless, the robust attention mechanism, whether it's based on re-weighting or the median-of-means principle, results in attention vectors that are distinct from those generated by the standard softmax attention. This, in turn, impacts the model parameters learned during the training procedure.\n\nDuring the inference stage, the test data undergoes the same procedure to generate robust attention vectors, helping to shield against potential contamination from outliers in the test sequence.\n\nHowever, under the assumption that a clean training set is available, and contamination from distribution shifts, adversarial attacks, or data poisoning only occurs during the inference stage, the application of the robust attention mechanism may be limited to inference. This approach could considerably reduce the computational time required for robust attention vector calculation during training.\n\nFor the empirical evaluation, we applied our robust attention mechanism in both stages and documented the related computation time during training. We found that on standard datasets such as ImageNet-1K, WikiText-103, and the UEA time-series classification, the incorporation of robust attention could also lead to reduced training loss. This suggests that training data itself may contain inherent noise or outliers in many cases.\n\n\n**Cons 3**\n\n**A**: While we respectfully contest the statement in question, our study marks a pioneering effort to interpret the robustness issue of Transformers through a nonparametric lens. In this context, we have undertaken significant work to identify effective methodologies capable of tackling this issue.\n\nInitially, our exploration involved linking self-attention to kernel regression problems within the realm of RKHS. This link helped us communicate the primary takeaway of our research, namely, that the robustness concern associated with Transformers can be traced back to the vulnerability of kernel regression problems when confronted with data outliers. However, given the computational constraints and presumptions related to contamination distributions, we were driven to seek alternate methods that could circumvent these issues.\n\nSubsequently, we uncovered a straightforward yet effective approach centered on the Median-of-Means principle, which has shown empirical values in discrete contexts. This approach acted as an adjunct methodology for down-weighting outliers in RKHS.\n\nBesides the empirical evidence drawn from a broad spectrum of applications, we argue that each individual component of our study—including the problem formulation, the adaptation of nonparametric methodologies to Transformer settings, and the resolution of practical implementation issues encountered throughout the process—represents a significant, stand-alone contribution to the community.\n\n\n**Cons 2**\n\n**A**: Thank you for your comment, and we apologize for any confusion our initial argument may have caused. Indeed, the uniform concentration bound can be adapted from the argument used in Theorem 2 in [1], in conjunction with Lemma 1 in [2]. This is possible when we extend the Fourier Integral Theorem in its generalized form. We will revise the paper to reflect this change.\n\nThe core idea behind the uniform concentration bound is the utilization of Bernstein's inequality, along with the bracketing entropy under the $\\mathbb{L}_{1}$ norm of the functions in the space of queries, denoted as $q$. The details of this argument can be found within Theorem 2 in [1]. \n\n[1] N. Ho and S. Walker. Multivariate Smoothing via the Fourier Integral Theorem and Fourier Kernel. Arxiv preprint, 2020.\n\n[2] T. Nguyen, M. Pham, T. Nguyen, K. Nguyen, S. Osher, N. Ho. Fourierformer:Transformer meets generalized Fourier integral theorem. NeurIPS, 2022.\n\n**Q1 & Q5**\n\n**A**: The methods require fine-tuning and the full set of parameters can be found in the code we've provided. There are several key hyperparameters including \n\n(1) the cut-off parameters in the Huber and Hampel loss function; \n\n(2) the iteration count in the Kernelized Iterative Reweighted Least Squares algorithm; \n\n(3) the scaling factor in the scaled and projected KDE; \n\n(4) the number of partitions or blocks in the Median-of-Means procedure.\n\nOur initial approach involves setting these parameters within a commonly used range, after which we execute a grid search within this predefined space. There is no need to require access to contaminated data during this procedure. The outcomes of this hyperparameter tuning process are detailed in Tables 6 through 10 in the appendix.\n\n\n**Q4**\n\n**A**: As discussed in Section 3.2 of our paper, the robust attention mechanism employing Median-of-Means tends to excel in discrete contexts. When applied to modifications at the pixel or patch level, this approach doesn't demonstrate a clear advantage compared to other methods. This observation is substantiated by the experimental results obtained from Vision Transformers, as presented in Section 4.2 of our paper.\n\nPlease let us know if you have additional questions.\n\nSincerely,\n\nAuthors"}, {"review_id": "KsCQ2v4MYz", "reviewer": "Reviewer_ATq2", "summary": "This paper re-interpret the self-attention mechanism in transformer architectures as a non-parametric kernel density estimator (KDE). Leveraging existing robustness results on classical KDE, the authors develop new variants of transformers; empirical results on a variety of datasets demonstrate how these variants are resistant to adversarial attacks and data contamination.", "questions": "In my view, the paper makes a strong contribution. Could the authors please address my questions below.\n\n- In line 93 when the kernel $k_{\\sigma}$ is introduced, it should be specified what set does this kernel act on and the fact that the considered class of kernels is translation invariant $\\tilde k_\\sigma(x,y) = k_\\sigma(x-y)$. For example, equation (2) is ambiguous because the kernels used in the joint density and marginal density estimations are different (indexed on different spaces).\n- In line 95-96, it would be good to add an expression to define the mentioned isotropic Gaussian kernel.\n- In lines 186-187 it is mentioned that the contruction of attention weight factors necessitate iterative algorithms to calculate the set of weights when computing self-attention at each layer; however, if (sometimes) the weight estimation boils down to a finite dimensional quadratic optimisation problem with linear constraints such as equation (7), can't the weights be determined by simply solving a linear system of equations?\n- Kernel regression often involves regularisation; this setup is not considered in the paper. Can the authors comment on the potential role of regularisation?", "limitations": "The authors have discussed limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper provides a simple yet powerful re-interpretation of the self-attention mechanism through the lens of KDE. This re-interpretation allows to leverage results from robust KDE to design better transformer architectures. Kernel methods are notably amenable to theoretical analysis; therefore, links between transformers and kernels as the one proposed in this paper can help sheding lights on justify the empirical successess of transformer architectures. The paper is in general well-written and its structure is easy to follow. ", "weaknesses": "Links between Transformers and kernel methods have been proposed in prior literature, see for example [1]. Although an extensive literature review is proposed by the authors, some important references such as [1] are omitted. \n\n**References**\n[1] Tsai, Yao-Hung Hubert, et al. \"Transformer dissection: a unified understanding of transformer's attention via the lens of kernel.\" arXiv preprint arXiv:1908.11775 (2019).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer `ATq2`,\n\nWe greatly value your constructive feedback and insights. We will address your suggestions regarding the introduction of kernel functions (questions 1 and 2) and will incorporate any missing citations you've noted in the upcoming version of our manuscript. Thank you for your significant contribution to improving our work. We address your remaining questions below.\n\n**Q: If (sometimes) the weight estimation boils down to a finite-dimensional quadratic optimisation problem with linear constraints such as equation (7), can't the weights be determined by simply solving a linear system of equations?**\n\n**A**: Thank you for raising this point. To the best of our understanding, in a finite-dimensional quadratic optimization problem, one can combine all quadratic terms to transform the problem into a linear equation. However, due to the linear constraint in our formulation, the constraints that result from this transformation remain nonlinear in relation to the quadratic terms. As such, we cannot derive a system of linear equations using this method. We apologize if our interpretation of your question is not accurate and are open to exploring this further with more specific guidance.\n\n\n**Q: Kernel regression often involves regularisation; this setup is not considered in the paper. Can the authors comment on the potential role of regularisation?**\n\n**A**: Thank you for this astute observation. Regularization in kernel regression indeed presents another viable strategy for managing the instability of kernel regression when dealing with data contaminated by errors. Consequently, this perspective may pave the way for developing an alternative robust Transformer model.\n\nHowever, the reason we didn't address regularized kernel regression in our paper is due to the already comprehensive analysis of three different robust Transformers presented, which rendered the paper quite dense. We therefore thought it prudent to earmark the development of a robust Transformer using regularized kernel regression approaches for future research.\n\nPlease do not hesitate to let us know if you have any additional questions.\n\nSincerely,\n\nAuthors"}, {"review_id": "pGcxme1wUK", "reviewer": "Reviewer_3B8c", "summary": "In this work the authors present novel adaptation of a nonstandard self attention module, like those used in transformers. Previous works have demonstrated that the weighting matrix in self attention can be evaluated using a positive kernel rather than the inner product, which ends up looking like a kernel regressor, which itself can be understood through kernel density estimation. In this work the authors propose using robust versions KDEs in place of KDEs for this method. The authors introduce and describe 3 existing versions of robust KDEs for this and apply their method to several experimental settings on a few data types. \n\nOn image datasets the authors find that, while their method does not improve the standard performance on clean datasets, the methods improve performance when robustness is advantageous, eg. adversarial samples, dataset shift, etc.. On time series and NLP datasets the authors observe some improvement on the clean data as well as unclean data.", "questions": "The omega values in the robust KDEs depend on the values k_i (they are functions of $k_i$). During training do the gradients take into account how these omega values will be modified as the functions for k_i are modified? Its a bit opaque to me how the robustness plays into the optimization.\n\n Could one just train using the standard KDE and then at test time use a robust KDE?", "limitations": "None", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The core contribution of this paper is providing a new deep learning methodology that yields substantial performance improvements. The method seems to work on several datatypes and robustness scenarios indicating a fundamental improvement to the general transformer methodology, rather than something that is very application/task/architecture specific. The results also give some insight as to origins of non-robustness of transformer architecture. Its not obvious that key attention values being \"heavy-tailed\" may make a transformer sensitive to word swapping.\n\nAn additional strength of this paper is its novelty. This work is brings methods far outside of what one usually sees in deep learning research, which is also a plus. \n\nOverall the paper is fairly clearly written with a nice overview of a few robust KDEs and introduction to using KDEs for transformers.", "weaknesses": "The primary weakness, in my opinion, is that the proposed method is simply swapping out a component of a transformer with a existing robust version of that component, and is somewhat trivial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer `3B8c`,\n\nWe sincerely appreciate your constructive comments and positive feedback. Such insights play a crucial role in refining our manuscript. We now address the primary concerns you've raised.\n\n**Q: The proposed method is simply swapping out a component of a transformer with an existing robust version of that component.**\n\n\n**A**: While we respectfully contest the statement in question, our study marks an important effort to interpret the robustness issue of Transformers through a nonparametric lens. In this context, we have undertaken significant work to identify effective methodologies capable of tackling this issue.\n\nInitially, our exploration involved linking self-attention to kernel regression problems within the realm of RKHS. This link helped us communicate the primary takeaway of our research, namely, that the robustness concern associated with Transformers can be traced back to the vulnerability of kernel regression problems when confronted with data outliers. However, given the computational constraints and presumptions related to contamination distributions, we were driven to seek alternate methods that could circumvent these issues.\n\nSubsequently, we uncovered a straightforward yet effective approach centered on the principles of the Median-of-Means, which has shown empirical values in discrete contexts. This approach acted as an adjunct methodology for down-weighting outliers in RKHS.\n\nBesides the empirical evidence drawn from a broad spectrum of applications, we argue that each individual component of our study—including the problem formulation, the adaptation of nonparametric methodologies to Transformer settings, and the resolution of practical implementation issues encountered throughout the process—represents a significant, stand-alone contribution to the community.\n\n\n**Q: During training do the gradients take into account how these omega values will be modified as the functions for k_i are modified?**\n\n**A**: From your inquiry, it appears you are primarily referencing the robust self-attention mechanism achieved by down-weighting outliers in RKHS. In our actual implementation, the weight set $\\omega$ is kept **separate** from the gradient updates of the Transformer models. Specifically, the `requires_grad` attribute is set to `False` for the weight tensor. The determination of this set of weights is achieved via the iterative methods elaborated upon in our paper. This design choice is motivated by two principal reasons:\n\n(1) Allowing gradient updates for the weight tensors considerably decelerates the training process and, in our observations, doesn't produce any discernible improvements in the end results.\n\n(2) Permitting gradient updates for these tensors could potentially disrupt the gradient flow, making it unstable.\n\nHowever, this weight set does affect the learned parameters of Transformers. Should there be outliers in the training dataset, this auxiliary weight set introduced would generate attention vectors that differ from those produced by the standard softmax attention mechanism. This, in turn, will influence the model parameters learned during the training phase.\n\n\n**Q: Could one just train using the standard KDE and then at test time use a robust KDE?**\n\n**A**: Yes. Our initial intention is to integrate our robust attention mechanism into both the training and inference stages. Considering that we lack prior knowledge of the data, contaminated samples could potentially emerge during both stages. Consequently, it is worthwhile to safeguard against outliers throughout the process.\n\nIn the context of model training, our methods adapt the computation of attention vectors across each Transformer layer, rendering them more resistant to outlier contamination. It's crucial to emphasize that this entire procedure is nonparametric and does not introduce any new model parameters. Nevertheless, the robust attention mechanism, whether it's based on re-weighting or the median-of-means principle, results in attention vectors that are distinct from those generated by the standard softmax attention. \n\nDuring the inference stage, the test data undergoes the same procedure to generate robust attention vectors, helping to shield against potential contamination from outliers in the test sequence.\n\nHowever, under the assumption that a clean training set is available, and contaminated samples from distribution shifts, adversarial attacks, or data poisoning only occur during the inference stage, the robust attention mechanism only needs application during inference. This can lead to a significant reduction in the time spent calculating the robust attention vector during training.\n\nPlease do not hesitate to let us know if you have any additional questions.\n\nSincerely,\n\nAuthors"}]

Author rebuttal:
Dear Reviewers,

We greatly appreciate the time and effort you've dedicated to offering valuable feedback for the enhancement of our manuscript. We'd like to address a frequently asked question and provide details on our further investigations.

1. **Is this method purely post-hoc, or does it necessitate modifications during the training process? Put differently, is it feasible to train using the standard KDE and then employ a robust KDE only during testing?**

We aim to incorporate our robust attention mechanism into both the training and inference stages. Given the uncertainty about data cleanliness, there's a possibility of encountering contaminated samples at either stage. Thus, it is worthwhile to defend against outliers throughout the entire process.

In the training context, our methods modify the computation of attention vectors across each Transformer layer, making them less susceptible to contamination from outliers. Importantly, this entire process remains nonparametric, with no introduction of additional model parameters. However, the resulting attention vectors, whether shaped by re-weighting or the median-of-means principle, diverge from those generated by the standard softmax attention. This distinction influences the model parameters learned during training.

During inference, the test data undergoes a similar procedure to yield robust attention vectors. This ensures protection against potential outlier-induced disruptions within the test sequence.

However, if we assume the availability of a clean training set — where contamination arises solely from distribution shifts, adversarial attacks, or data poisoning during inference — it is sufficient to restrict the application of the robust attention mechanism to just the inference phase. This could considerably reduce the computational time required for robust attention vector calculation during training.

In our empirical evaluation, we engaged the robust attention mechanism throughout both phases and recorded the associated computational time during training. We found that on standard datasets like ImageNet-1K, WikiText-103, and the UEA time-series classification, infusing the robust attention led to a drop in training loss. This suggests that training data itself may contain inherent noise or outliers.

2. **Large-scale experiments and computational time.**

We wish to emphasize that we've done our utmost to ensure that our experimental results are as thorough as possible. However, we were constrained by both time and the available computational resources at the time of submission. Before the rebuttal process begin, we were able to perform extra experiments using Transformers with greater capacities. 

We have also monitored the computational time for each task in the paper. As mentioned in the former question, the computational time can be greatly reduced if the robust attention mechanism is only applied in the inference stage. 

Please find the additional results in the attached PDF file.

Sincerely,

Authors

Meta-review: The paper proposes to use robust kernel density estimation (KDE) in place of the standard dot-product self-attention in Transformers to improve the robustness of models on corrupted data in both vision and language tasks. All five reviewers rated this paper as above acceptance threshold or more positive. Reviewers mentioned that the re-interpretation of self-attention as KDE is a novel perspective, the motivation of using robustness KDE is strong and the empirical evidence of improved robustness is extensive. The weaknesses brought by the reviewers are partly addressed in rebuttal and outweighed by the merits. The AC is pleased to recommend acceptance of this paper.
Decision: Accept (poster)