Robustify Transformers with Robust Kernel Density Estimation
Authors: Xing Han
Tongzheng Ren
Tan Minh Nguyen
Khai Nguyen
Joydeep Ghosh
Nhat Ho
Venue: Submitted to ICLR 2023 (2023)
Keywords: Transformers
Kernel Density Estimation
Robustness

Abstract: Recent advances in Transformer architecture have empowered its empirical success in various tasks across different domains. However, existing works mainly focus on improving the standard accuracy and computational cost, without considering the robustness of contaminated samples. Existing work (Nguyen et al, 2022, FourierFormer) has shown that the self-attention mechanism, which is the center of the Transformer architecture, can be viewed as a non-parametric estimator based on the well-known kernel density estimation (KDE). This motivates us to leverage the robust kernel density estimation (RKDE) in the self-attention mechanism, to alleviate the issue of the contamination of data by down-weighting the weight of bad samples in the estimation process. The modified self-attention mechanism can be incorporated into different Transformer variants. Empirical results on language modeling and image classification tasks demonstrate the effectiveness of this approach.

Under review as a conference paper at ICLR 2023
ROBUSTIFY TRANSFORMERS WITH ROBUST KERNEL DENSITY ESTIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in Transformer architecture have empowered its empirical success in various tasks across different domains. However, existing works mainly focus on improving the standard accuracy and computational cost, without considering the robustness of contaminated samples. Existing work (Nguyen et al., 2022) has shown that the self-attentic architecture, can be viewed as non non-parametric estimator based on the well-known kernel density estimation (KDE). This motivates us to leverage a set of robust kernel density estimation methods in the self-attention mechanism, to alleviate the issue the contamination of data by wowg-weighting the weight of bad samples the estimation process. The modified self-attention mechanism can be incorporated into different Transformer variants. Empirical results on language modeling and image classification tasks demonstrate the effectiveness of this approach.
mechanism, which the center of the Transformer
INTRODUCTION
Attention mechanisms and transformers (Vaswani et al., 2017) have been widely used in machine learning community (Lin et al., 2021: Tay al., 2020; Khan et al., 2021). Transformer-based models are now among the best deep learning architectures on a variety of applications, including those in natural language processing (Devlin et al., 2019; Al-Rfou al., 2019; Dai al., 2019; Child et al., 2019; Raffel et al., 2020; Baevski & Auli, 2019; Brown et al., 2020; Dehghani et al., 2019), computer vision (Dosovitskiy et al., 2021; Liu et al., 2021; Touvron et al., 2021a; Ramesh al., 2021; Radford et al., 2021; Fan et al., 2021: Liu et al., 2022), and reinforcement learning (Chen al., 2021; Janner et al., 2021). Transformers have also been well-known for their effectiveness in transferring knowledge from pretraining tasks to downstream applications with weak supervision or no supervision (Radford et al., 2018; 2019; Devlin al., 2019; Yang et al., 2019; Liu et al., 2019).
Contribution Despite having an appealing performance, the robustness of the conventional attention module still remains an open question in the literature. In this paper, to robustify the attention mechanism and transformer models, we first revisit the interpretation of the self-attention in the transformer as the Nadaraya- Watson (NW) estimator (Nadaraya, 1964) in a -onpaaamettric regression problem in the recent work of Nguyen al. (2022). Putting in the context of transformer, the NW estimator is constructed mainly based on the kernel density estimators (KDE) of the keys and queries. However, the KDE is not robust to the outliers (Kim & Scott, 2012), which leads to the robustness issue of the NW estimator and the self-attention in transformer when there are outliers in the data. To improve the robustness of the KDE, first show that the KDE can be viewed as an optimal solution of the kernel regression problem in the reproducing kernel Hilbert space (RKHS). Then, to robustify the KDE, we can either robustify the loss function of the kernel regression problem via some robust loss functions, such as the well-known Huber loss function (Huber, 1992). or reweight the contaminated densities via scaling and projecting the original densities. The family of robust KDE can be used to construct set of novel robust attentions in transformer, which also improves the robustness issue of the transformer. In summary, our contribution two-fold:
By connecting the dot-produce self-attention mechanism in transformer with the nonpara- metric kernel regression problem in reproducing kernel Hilbert space (RKHS), propose a novel robust transformer framework, based on replacing the dot-product attention by an attention arising from of robust kernel density estimators associated with the robust

---

Under review as a conference paper at ICLR 2023
kernel regression problem. Comparing to the standard soft-max transformer, the family of robustified transformers only requires computing an extra set of weights.
Extensive experiments on both vision and language modeling tasks demonstrate that our proposed framework has favorable performance under various attacks. Furthermore, the proposed robust transformer framework flexible and can incorporated into different Transformer variants.
Organization The paper organized as follows. In Section 2, we provide background on self- Matarayy-Waaton attention mechanism in Transformer and its connection to the Nadaraya- Watson (NW) estimator in the nonparametric regression problem, which can be constructed via KDE. In Section 3, we first connect the KDE to kernel regression problem in the reproducing kernel Hilbert space (RKHS) and demonstrate that it is not robust to the outliers. Then, we construct the robust self-attention of robust KDE methods. We empirically validate
mechanism for the Transformer by leveraging the advantage of the proposed robust self-attension mechanism, over the standard softmax transformer along with other baselines over both language modeling and image classification tasks in Section Finally, we discuss the related works in Section 5 while conclude the paper in Section 6.
2 BACKGROUND: SELF-ATTENTION MECHANISM FROM A NON-PARAMETRIC REGRESSION PERSPECTIVE
In this section, first provide background the self-attention mechanism in transformer in Section 2. We then revisit the connection between the self-attention and the Nadaraya-Ware Watson estimator in nonparametric regression problem in Section 2.2.
2.1 SELF-ATTENTION MECHANISM
Given an input sequence X [ ØNI E NND, of N feature vectors, the self-attention transforms it into another sequence H [h,,... _NN]' NxD. as follows:
q ki Vy for i -I...,,N
h JE[N]
softmax
where the scalar softmax((q)) /D) can be understood the attention h; pays to the input feature zy. The vectors qi ky, and vy are the query, key, and value vectors, respectively, and are computed follows:
[q 42
qND
= Q = XW ERD
ki, ka
kN
K XW ERD
(01, 02
uNd
= XWI ERND..
where Wo, WK DxDs Wv l8 DoXDs are the weight matrices. Equation can be written as:
QK
H softmax
V.
where the softmax function is applied each row of the matrix (QK) 3/ DVD equation 3 is also called the "softmax attention".. For each query vector q for i 1,... N, an equivalent form of equation 3 to compute the output vector h, is given by
h softmax
qi
K V3
Σ a. JE[N]
(4)
JE[N]
In this paper, we call transformer built with softmax attention standard transformer transformer.
2.2 A NON-PARAMETRE REGRESSION PERSPECTIVE OF SELF-ATTENTION
We now review the connection between the sele-attention mechanism in equation and the non- parametric regression, which has been discussed in the recent work (Nguyen et al., 2022). Assume
2

---

Under review as a conference paper at ICLR 2023
we have the key and value vectors {kj, VyljeiN] that is collected from the following data generating process:
V = f(k)+ +
(5)
where is some noise vectors with B|c] = 0, and f is the unknown function that we want to estimate. We consider random design setting where the key vectors jey]) are i..d. samples from the distribution pkk), and we use p(v, k) to denote the joint distribution of (v, k) defined by equation 5. Our target is to estimate f(q) for any new queries q.
Nadaraya (1964) provides non-parametric approach to estimate the function f. which is known as the the Nadaraya- Watson (NW) estimator, the kernel regression estimator or the local constant estimator. The main idea of the NW estimator is that
v (V dv, V v(v(vkk)dv | p(k)
[(k) Elv kc
(6)
J22
where the first equation comes from the fact that E[E] = O, the second equation comes from the definition of conditional expectation and the last inequality comes from the definition of the conditional density. With equation 6, we know, to provide an estimation of f. we just need to obtain estimations for both the joint density function p(v, k) and the marginal density function p(k). One of the most popular approaches for the density estimation problem is the kernel density estimation (KDE) (Rosenblatt, 1956; Parzen, 1962), which requires kernel K with the bandwidth parameter satisfies Jgp (x(x - x')dæ = 1, Va',,, and estimate the density as
kv,
1
Σ No ([v, (vy, (k) JE[N]
Σ
Ko (k- kz), -kj);
N
N
JE[N]
where [v, k| denotes the concatenation of v and &.. Specifically, when Ho is the isotropic Gaussian kernel kn =x') exp
(-11a: x =x'11=////// x211²/(20²)). we have
1
(v. k)
Σ
lv -v))ko kz).
N
JE[N]
Given the kernel density estimators equation and equation 8, as well as the formulation in equation 6, we obtain the NW estimator of the function f:
(k)) [
v (v, dv { v. Eje(N ko (v - v,)ko (K kg) dv (k)
J8D
EjE( ko (k k,)
ΣJENN lk - -kj). l'v (v -e-vvy -vvyv
ΣJEN lio l ---h k:))
ΣJEM Vikio (k- kj)
(9)
EJE(N) ho kk))
Now we show how the self self-attention mechanism is related the NW estimator. Note that
(g( = EjEiM v, exp (-11q k k//02/200)
ΣJE(N exp (- 11q - 12
E EjE[M V, exp [- (11/ / + |k, [12) /202 exp ku,k//0²))
(10)
EjEN exp (1 + +1,1111)) 2 exp (q k /2)
If the keys (ksbeE(N) are normalized, we can further simplify fo( in equation 9 to
f.)))) ΣjENN V, exp (qk} /02)
Σ
softmax TaammaxxgTk///ae (oTk//o))
(11)
EjE[N] exp (gk) /02)
JEIN]
Such an assumption on the normalized key (k: (EI)) can be mild, as in practice we always have an normalization step the key to stabilize the training of the transformer (Schlag et al., 2021). If choose 2² VD, where D is the dimension of q and k,. then fo (q:) h; As result, the self-attention mechanism in fact performs a non- parametric regression with NW-estimator and isotropic Gaussian kernel when the keys are normalized.
3

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
Figure 1: Contour plots of density estimation of the 2-dimensional query vector embedding in an attention layer of the transformer when using (b) KDE (equation 12) and (c) RKDE (equation 13) with Huber loss (equation 14), where (a) is the true density function. We draw 1000 samples (gray circles) from a multivariate normal density and 100 outliers (red cross) from a gamma distribution as the contaminating density. RKDE can be less affected by outliers when computing self-attention as nonparametric regression.
3 ROBUSTIFY TRANSFORMER WITH ROBUST KERNEL DENSITY ESTIMATION
As we have seen in Section 2, the self-attention mechanism can be interpreted as NW estimator for the unknown function where the density is estimated with KDE using the isotropic Gaussian kernel. In this section, first re-interpret KDE as regression in the Reproducing Kernel Hilbert Space (RKHS), which shows that the vanilla KDE is sensitive to the data corruption. Instead, we observe that, variants of the kernel density estimation such robust KDE (Kim & Scott, 2012) and scaled projection KDE (Vandermeulen & Scott, 2014), can down-weight the importance of the potential corrupted data and obtain a robust density estimator. Based on the variants, derive the corresponding robust version of the NW-estimator, and show how to use this to replace the self-attention mechanism, and eventually lead more robust Transformer variants.
3.1 KDE AS REGRESSION PROBLEM IN RKHS
We start from the formal definition of the RKHS. The space H4 = [f RJ is called an RKHS associated with the kernel k where k X X R, if it Hilbert space with the following two properties: (1) k(x,) :) E He, Vx E X: (2) the reproducing property: Vf E H. f()) k/=, -)),., where (:: He denotes the RKHS inner product. With slightly abuse of notation, we define ki (2t x') (x x²). By the definition of the RKHS and the KDE estimator, know Po 3 ΣjE(N] Ko (2r,.)) Hi. In fact, Po is the optimal solution of the following least-square regression problem in RKHS:
Po arg min
Σ (1,.) pll
(12)
H JE[N]
Note that, in equation 12, we have the same weight 1/N on each of the error Iika (0z,.) -0ppppl pll Ne. This works well if there are no outliers in (ao (z,, JEN] However, when we have outliers (e-g.. when there exists some j. such that (> -) Ma, >> [ (x,, ml Vi [N], / j), the error on the outliers will dominate the whole error and lead to substantially worse estimation on the entire density. We illustrate the robustness issue of the KDE in Figure 1.
Combining the viewpoint that KDE is not robust to outliers with the interpretation of section 2.2 implies that the transformer is also not robust when there are outliers in the data. The robustness issue of transformer has mostly been studied in the vision domain, such as (Mahmood et al., 2021; ; Mao et al., 2022: Zhou et al., 2022). These works modify the original architectures of vision transformer and introduces extra parameters. A representative one is Mao et al. (2022), which proposed position- based attention by adding on another fully connected layer. However, this approach will cause bi- bi-directional information flow for positional-sensitte therefore limited to image data. We take a different view of the robustness problem the RKHS domain and provide unified framework for different data modalities.
dataset such text sequences and is
4

---

Under review as a conference paper at ICLR 2023
3.2 ROBUST KDE
Motivated by the robust regression (Fox & Weisberg, 2002), Kim & Scott (2012) proposed a robust version of KDE, by replacing the least-square loss in equation 12 with robust loss function p:
Probust
arg min
E P (1I,,((z,)) pll JE(N)
(13)
PE
Examples of the robust loss functions p include the Huber loss (Huber, 1992), Hampel loss (Hampel al., 1986), Welsch loss (Welsch Becker, 1975) and Tukey loss (Fox & Weisherg, 2002). We empirically evaluate different loss functions in experiments. For simplicity, use the Huber loss function as the demonstrating example, which defined as follows:
x² /2, a²/2,
0 S x x
p(x) = ax
(14)
Ø,
where a a constant, Kim & Scott (2012) shows the solution of this robust regression problem has the following form:
Proposition 1. Assume the robust loss function p is non-decreasing in [0, 20), p(0) and
l pla) = O. Define vb(x)= == g((z) and assume (00) l gl(x) exists and finite. Then the
optimal Probust can be wrilten as
Protuss
Σ
wyyka (z)z
SE[N]
where w = (wi,... ,NN E AN, and wy x (1lia( n-dimensional simplex.
Probucr (l Here Dn denotes the
The proof of this proposition can be found in Appendix A. For Huber loss function, we have that
1, a/x,
0 S Z<zsa < x.
vxx) =
Hence, when the error [ )).- Probuse [Ha,, is the threshold a, the final estimator will down-weight the importance of k. (z) This is in sharp contrast with the standard KDE method, which will assign uniform weights to all the Ko (x),)). One additional issue is that, the estimator provided in Proposition is circularly defined, Probust defined via w, and w depends Probus- To address this issue, Kim & Scott (2012) proposed to estimate with an iterative algorithm termed kernelized iteratively re-weighted least-squares (KIRWLS) algorithm. The algorithm starts with some randomly initialized (0) An, and perform the following iterative updates:
(((()))))))
((k) -Probusst
Probust
Σ JE[N]
w (k- Ko (Z)
wi
Hike
(15)
ΣjEIn vp (11 Probust
HNO
Note that, the optimal Probus is the fixed point of this iterative updates, and Kim & Scott (2012) shows that the proposed algorithm converges under standard regularity conditions. Furthermore, one can directly compute the term
hi, ( Probust
.(k)
via the reproducing property:
Huo
Kn (2ng))
(arg) hi H) 2 (io (10 -) P.
Prubuutt
Has
Hio
Hoo
-ko (2 (t);) 2
Σ
with 1) Kio (Xm, xti)
mE[N]
Σ
(k-))) (k))) ko (2 Rnn)
mE M
Therefore, the weights be updated without mapping the data the Hilbert space.
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Procedure of Computing Attention Vector of Transfomerr-KDD//PKKDD
1: Input: Q= qq [N) K = [ky ( V he(j) initial weights (0)
2: Normalize K = (k3 JyeiN] along the head dimension.
3: Compute kernel function between each pair of sequence: (o (Q (ko ky)RsSENN]
4: (Optional) apply attention mask on k (Q, ().
(1saatess) Potle Nen
5: [RKDE] Update weights w (0) for marginal/joint density by wi
EvE(a v ( seeda Mao
6: [SPKDE] Obtain optimal weights for marginal/joint density via solving equation 17.
7: Obtain attention vector via robust self-attention h. = EyE[M] yyul (a (k,))
ΣSE[N] ")
magina ko (ks))kk))
Scaled Projection KDE (SPKDE) Vandermeulen & Scott (2014) is one other option of robust KDE in the RKHS space. It essentially scale the original KDE and project it to its nearest weighted KDE the L2 norm. The resulting weighted KDE can allocate more weight high density regions and truncate the weights for anomalous samples. Specifically, given the scaling factor 3 > 1, and let C,* be the convex hull of Ko (#1, .)...,))). Ko (ZN, E Hk, ic., the space of weighted KDEs, the optimal density Probust is given by
Protust = arg min
||
Σ (o -PlleE, JEIN]
(16)
PECN
which is guaranteed to have a unique minimizer since we are projecting Hilbert space and C.N is closed and convex. Note that, Probust can also be represented as Probust ΣjENNj Wyk (xy, :), w E A N, which is similar to robust KDE by Kim & Scott (2012). Then equation 16 can be written as a quadratic programming (QP) problem over w. Let G be the Gram matrix of k2 and q = G1 A
N then
the QP can be written follows
min w Gw 2g Twu w, subject to E A
(17)
Since the Gram matrix G is defined to sestiie-semidefiiite,, this QP is convex. practice, one can leverage commonly used solvers efficiently obtain the solution and the optimal density Probust-
3.3 ROBUST
MECHANISM
Now we describe the robust self-attention mechanism we use. We consider the density estimator of the joint distribution and the marginal distribution from the robust KDE:
Probust v, k)
Σ
J joint
ho ([v,, k,y. [v, &)), Probust =
Σ
w
marginal
kig (k:,, k)
JE[M]
JE[N]
With the similar computation, the robust self-attention mechanism use is defined
h; ESE(N) Vjul
joint ko (q: k:))
(18)
EjE[N] W/
marginal k (g: kg))
where whint and marginal are obtained via either the KIRWLS algorithm or results from the QP solver. We term the transformer models that employ robust KDE and SPKDE Transformer-RKe RKDE and Transformer- SPKDE, respectively. We will show in our experiments on language modeling and image classification that SPKDE performs better empirically as it finds the optimal set of weights.
Remark 1. Note that, the computation of {w marginall ]jE[M] and ( ]J[[N] are separate as wolin involves both keys and values vectors. During the empirical evaluation, we concatenate the keys and values along the head dimension to obtain the weights for the joint density Probust (v. k) and only use the key vectors for obtaining the set of weights for the marginal Protuer (k). In addition, marginal joint E Rixi for i 1, 1,...,N are 2-dimensional matrices that includes the pairwise weights between each position of the sequence and the rest of the positions. The weights are initialized uniformly across certain sequence length dimension. For experiments related to language modeling, we can leverage information from attention mask to initialize the weights on the unmasked part of sequence. To speed up the computation for Transformer-RKDD) we use single-step iteration

---

Under review as a conference paper at ICLR 2023
Table Perplexity (PPL) and negative likelihood loss (NLL) of our methods and baselines on WikiText- 103 dataset. The best results are highlighted bold font and the second best results are highlighted in underline. Transformer-RKIl and Sansformer-SPKDE achieve competitive performance to the baseline methods while shows much hetter PPI. and NLL under random swap with outlier words.
Clean Data
Word Swap
Method
Valid PPL/Loss
Test PPL/Loss
Valid PPL/Loss
Test PPL/Loss
Standard Softmax
33.52/3.51
34.59/3.54
72.28/4.45
74.56/4.53
Transformer-KDD KDE
33.34/3.51
34.37/3.54
71.94/4.43
73.75/4.49
Transformer-RKDE (Huber)
33.22/3.50
34.29/3.54
52.14/3.92
55.68/3.99
Transformer-R RKDE (Hampel) 33.24/3.50
34.35/3.54
55.61/3.98
57.92/4.03
Transformer:SPKDD
33.05/3.49
34.18/3.53
51.36/3.89
54.97/3.96
equation 15 to approximate the optimal set of weights. Empirical results have shown that this one-step iteration can achieve sufficiently accurate results. For Transformer-SPKII
we find the optimal set of weights via the QP solver. This strategy is shown to be effective during the empirical evaluation on both image and text data. The procedure of computing the attention vector for Transformer-RKD RKDE and Transformer- SPKDE can be found at Algorithm I.
EXPERIMENTAL RESULTS
In this section, we empirically validate the advantage our proposed transformer integrated with robust KDE attention (Tansfomer--KKDE/PPDD)) over the standard softmax transformer and its nonparametric regression variant (Transformer- KeDe in equation 9) on two large-scale datasets: lan- guage modeling WikiText- 103 dataset (Merity al., 2016) (Section 4.1) and image classification on Imagenet (Russakovsky et al., 2015; Deng et al., 2009) and Imagenet-C: (Hendrycks & Dietterich, 2019) (Section 4.2). Our experiments have shown that: (1) Transformer with robust KDE attention can reach competitive performance with baseline methods on a variety of tasks with different data modalities, this can be achieved without modifying the model architecture or introducing extra parameters; (2) the advantage of Transformer with robust KDE attention is more prominent when there is contamination of samples either text image data. All of our experiments are performed on the NVIDIA A-100 GPUs. For each experiment, we compare Transformer-RDDESSPP RKDE/SPKDE with other baselines under the same hyper-parameter configurations. The implementation to reproduce our results can be found at www.gynww..ppo..ceeeeemootttttta
4.1 ROBUST LANGUAGE MODELING
Dataset: WikiText- 103 is language modeling dataset that contains collection of tokens extracted from good and featured articles from Wikipedia, which is suitable for models that can leverage long-term dependencies. The dataset contains around 268K words and its training about 28K articles with 103M tokens, this corresponds to text blocks of about 3600 words. The validation and sets consist of 60 articles with 218K and 246K tokens respectively. We follow the standard configurations in Merity et al. (2016); Schlag et al. (2021) and splits the training data into L-word independent long segments. During evaluation, we process the text sequence using sliding window of size L and feed into the model with batch size of 1. The last position of the sliding window is used for computing perplexity except in the first segment, where all positions are evaluated as in AI-Rfou et al. (2019); Schlag et al. (2021).
consists of
Implementation Details: We used the language models developed by Schlag et (2021) in our experiments. The dimensions of key, value, and query are set to 128, and the training and evaluation context length are set to 256. As for self-attention, we set the number heads as 8, the dimension of feed-forward layer as 2048, and the number of layers as 16. To avoid numerical instability. we apply the Log-sum-ext trick in equation 9 when computing the attention probability vector through the Gaussian kernel. We apply similar tricks when computing the weights of KIRWLS algorithm, where we first obtain the weights in log space, followed by the log- song-sum-aas sum-exp trick to compute robust self-attention in equation 18.
7

---

Under review as a conference paper at ICLR 2023
Table 2: Top-1. top-5 accuracy (%) and mean corruption error (mCE) of DeiT with different attentions. The best results are highlighted in bold font and the second best are highlighted in underline. RVT (Mao et al., 2022) achieves better results clean data and corrupted imagenet; DieT with robust KDE attention achieve hetter results under different adversarial attacks while still achieve competitive performance corrupted imagenet.
Clean Data
FGSM
PGD
SPSA
Imagenet-C
Method
Top
Top 5
Top
Top 5
Top
Top 5
Top
Top 5
Top
mCE
Baseline DeiT
72.23
91.13
52.61
82.26
41.84
76.49
48.34
79.36
42.38
71.14
RVT
74.37
93.89
53.67
84.11
43.39
77.26
51.43
80.98
45.64
68.57
DeÎT-KDE
72.58
91.34
52.25
81.52
41.38
76.41
48.61
79.68
42.63
70.78
DeiT-RKDE (Huber)
72.83
91.44
55.83
85.89
44.15
79.06
52.42
82.03
45,58
68.69
DeiT-RKDE (Hampel)
72.94
91.63
55.92
85.97
44.23
79.16
52.48
82.07
45.61
68.67
DeiT-SPKDE
73.22
91.95
56.03
86.12
44.51
79.47
52.64
82.33
44.76
69.34
Results: In Table 1, we report the validation and test PPL of Transformer-RKI (with Huber and Hampel loss functions), Transformer-RKII versus the softmax transformer and its nonparametric regression variant. Based on the derivation in equation 11, we would expect Transformer- -eeefforrrr--EEE to have similar performance with softmax transformer. Meanwhile, Transformer-RKe and SPKDE is able to improve baselines PPL and NLL in both validation and test sets.
We can observe more obvious improvement when the dataset is under a word swap attack, which randomly replace selected keywords of input data by a generic token "AA "AAA" during evaluation. Our method, particularly SPKDE- robust attention, achieves much better results for down-weighting rare words, and therefore more robust such kind of attack. Our implementation on word swap is based on the public code TextAttack by Morris et al. (2020)', while we use the greedy search method with the constraints on stop-words words modification from the TextAttack library.
4.2 IMAGE CLASSIFICATION UNDER ADVERSARIAL ATTACK
We use the full ImageNet dataset that contains 1.28M training images and 50K validation images. The model learns to predict the class of the input image among 1000 categories. We report the top-1 and top-5 accuracy on all experiments. For robustness on common image corruptions, we use ImageNet-C (Hendrycks & Dietterich, 2019) which consists of 15 types of algorithmically generated corruptions with five levels of severity. ImageNet-C uses the mean corruption error (mCE) as metric, while the smaller mCE means the more robust of the model under corruptions.
Dataset:
Our method uses the same training configurations DeiT-Tiny (Touvron et al., 202 Given that all approaches do not modify the model architecture, each employed model has 5.7 M parameters. We also implemented a state-of-the-art robust vision transformer (RVT) model (Mao et al., 2022) as a baseline. For a fair comparison, we only implemented its position-aware attention scaling without further modifications on model architecture. The resulting model has around 7.2M parameters. To evaluate adversarial robustness, we apply adversarial examples generated by untargeted white-box attacks including single-step attack method FGSM (Goodfellow al., 2014), multi-step attack method PGD (Madry et al., 2017) and score-based black-box attack method SPSA (Uesato et al., 2018). The attacks are applied on 100% of the validation set of ImageNet. Both these attacks perturb the input image with perturbation budget = 1/255 under loo norm; while PGD attack uses 20 steps with step size a 0.15.
Implementation Details:
Results: We summarize the results in Table 2. RVT achieves better performance on clean and corrupted imagenet. The set of DeiT with robust KDE attention can also obtain very close results with RVT under these settings while leading to much better results under different adversarial attacks. Figure 2 shows the relationship between accuracy versus perturbation budget using three attack methods. Our proposed methods can improve the accuracy under different perturbation budget and exhibits greater advantage with higher perturbation strength. We provide more ablation studies in Appendix B regarding to different design choices of the proposed robust KDE attention.
Immleemnntation availahle at giihuh..cm//DDaaa/Teettttaa
8

---

Under review as a conference paper at ICLR 2023
FGSM
PGD
SPSA
50
55
DeiT
DeiT
DeiT
Huber
Huber
50
Huber
Accuracy
45
SPKDE
Hampel
Accuracy
30
Hampel
SPKDE
Accuracy
Hampel
SPKDE
RVTT
RVT
80
RVT
35
20
25
30
15
20
Purturbation Budget
Purturbation Budget
Purturbation Budget
(a)
(b)
(c)
Figure 2: The top-1 I classification accuracy perturbation budget 255 curves ImageNet against three untargeted attack methods under the to norm. Among all the competing methods, DeiT with robust KDE attention models show better robustness under all attack methods with different perturbation budgets.
RELATED WORKS
Vision Transformer (ViT) models (Dosovitskiy et al., 2020; Touvron et al., 2021b) recently achieved exemplary performance on variety of vision tasks that can be used as a strong alternative to CNNs. To ensure its generalization ability on different datasets, many works (e.g., Subramanya et al., 2022; Paul & Chen, 2022; Bhojanapalli et al., 2021) have studied the robustness of VIT under different types of attacks. Mahmood al. (2021) empirically shows that ViT is vuinerable to white-box adversarial attack but simple ensemble defense can achieve unprecedented robustness without sacrificing clean accuracy. Mao et al. (2022) performs robustness analysis on different building blocks of ViT and proposed position-awan attention scaling and patch- wise augmentation that improved robustness and accuracy of ViT models. More recently, Zhou et al. (2022) proposed fully attentional networks to improve the self-attention and achieved state-of-the-an accuracy on corrupted images. However, these works focus on improving the architectural design of VIT targeted for some specific tasks, which lacks general framework improving the robustness of transformers. In addition, most of the recent works studying robustness of transformer concentrate on vision related tasks and cannot generalize across different data modalities.
Robustness of Transformer:
Theoretical Frameworks of Attention Mechanisms: Attention mechanisms in transformers have been recently studied from different perspectives. Tsai et al. (2019) shows that attention can be derived from smoothing the inputs with appropriate kernels. Katharopoulos et al. (2020): Choromanski et al. (2021); Wang et al. (2020) further linearize the softmax kernel in attention to attain a family of efficient transformers with both linear computational and memory complexity. These linear attentions are proven in Cao (2021) to be equivalent to a -altov-Galerkin projection (Reddy, 2004), thereby indicating that the softmax normalization in dot-product attention is sufficient but not necessary. Other frameworks for analyzing transformers that use ordinary/partial differential equations include Lu et al. (2019); Sander et al. (2022). In addition, the Gaussian mixture model and graph-structured learning have been utilized to study attentions and transformers (Tang & Matteson, 2021: Gabbur al., 2021; Zhang& & Feng, 2021; Wang al., 2018; Shaw et al., 2018; Kreuzer al., 2021).
CONCLUSION AND FUTURE WORKS
In this paper, via the connection between the dot-product self-attention mechanism in transformer with nonparametric kernel regression problem, we developed a family of robustified transformers by leveraging robust kernel density estimation as a replacement of dot-product attention alleviate the effect from outliers. We show that the optimal estimation of potentially contaminated density functions via robust KDE requires computing set of weights, which can be flexibly integrated when computing attentions in commonly used transformer models. Empirical evaluations have shown that Transformer-RKKe RKDE can improve performance on clean data while demonstrate robust results under various attacks on both vision and language modeling tasks. The robust KDE attention we developed has the merit of generalizing to the whole family of transformer models, which we intended to demonstrate as a future work. Meanwhile, we will also investigate better and more efficient approach to estimate the set of weights for robust kernel density estimations.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rami Al-Rfou, Dokook Choc, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3159-31 3166, 2019.
Alexci Baevski and Michael Auli. Adaptive input representations for neural language modeling. In International Conference on Learning Representations, 2019. URL https//opeeneviic.ee///ouuuu id-ByxZX20qFQ.
Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit. Understanding robustness of transformers for image classification. P Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10231-10241, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Shuhao Cao. Choose a transformer: Fourier or galerkin. Advances in Neural Information Processing Systems, 34, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neurul information processing systems, 34:1 5084-15097, 2021.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprini arXiv: 1904. 1904.00999 2019.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021. URL https:
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv:1901.02860 2019.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019. URL https:
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19- 1423. URL https: Maclanthology.orrg/N 1423.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 11929, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https///openrvveew.nel
10

---

Under review as a conference paper at ICLR 2023
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6824-6835, 2021.
John Fox and Sanford Weisberg, Robust regression. An R and S-Plus companion to applied regression. 91, 2002.
Prasad Gabbur, Manjot Bilkhu, and Javier Movellan. Probabilistic attention for interactive segmenta- tion. Advances in Neural Information Processing Systems, 34, 2021.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 141v:5412.65722 2014.
Frank R Hampel, Elvezio M Ronchetti, Peter Rousseeuw, and Werner A Stahel. Robust statistics: the approach based on inftuence fimctions. Wiley-Intersciennee; New York, 1986.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019.
Peter Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pp. 492-518. Springer, 1992.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 14:1273-1286, 2021.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Flcuret. Transformers are mns:: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pp. 5156-5165 PMLR, 2020.
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM Computing Surveys (CSUR), 2021.
JooSeuk Kim and Clayton Scott. Robust kernel density estimation. The Journal of Machine Learning Research, 3:2529-25665 2012.
Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 34, 2021.
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. arXiv preprint arXiv:2106. 106.52006.04554, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012- 10022, 2021.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yar Liu. Understanding and improving transformer from multi-particle dynamic system point of view. arXiv preprint arXiv:1906.02762. 2019.
Aleksander Madry. Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1000 06083, 2017.
Kaleel Mahmood, Rigel Mahmood, and Marten Van Dijk. On the robustness of vision transformers to adversarial examples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7838-7847 2021.
11

---

Under review as a conference paper at ICLR 2023
Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li. Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12042-1205/) 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: (TXXiv:6099078433. 2016.
John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. arXiv preprint arXiv:2005.05900 2020.
Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications, 9(1): 141-142, 1964.
Tan Nguyen, Minh Pham, Tam Nguyen, Khai Nguyen, Stanley J Osher, and Nhat Ho. Fourier- former: Transformer meets generalized Fourier integral theorem. Advances Neural Information Processing Systems, 2022.
Emanuel Parzen. On estimation of a probability density function and mode. The annals of mathemat- ical statistics, 1365)110 1962.
Sayak Paul and Pin- Yu Chen. Vision transformers are robust learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2071-2088 1-281,, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. OpenAl report, 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
JN Reddy. An introduction to the finite element method, volume 1221. McGraw-Hill New York, 2004.
Murray Rosenblatt. Remarks some nonparametric estimates of density function. The annals of mathematical statistics, pp. 832-837, 1956.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211225) 2015.
Michael Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Sinkformers:
Transformers with doubly stochastic attention. In International Conference Artificial Intelligence and Statistics, pp. 3515-3530. PMLR, 2022.
Imanol Schlag, Kazuki Irie, and Jürgen Schnidhuber. Linear transformers are secretly fast weight programmers. In International Conference Machine Learning, pp. 9355-9366. PMLR, 2021.
12

---

Under review as a conference paper at ICLR 2023
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 2 (Short Papers), pp. 464-468, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/ N18-2074. URL htwpp////aalantoooogoog.g/ 2074.
Akshayvarun Subramanya, Aniruddha Saha, Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Backdoor attacks on vision transformers. arXiv preprint arXiv:2206.088: 084777 2022.
Binh Tang and David S. Matteson. Probabilistic transformer for time series analysis. In A. Beygelz- imer, Y. Dauphin, P. Liang. and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL http:///wwwwooooooo///////dd///a
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. arXiv preprint XXXi::2009006732.. 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé
Jégou. Training data-efficient image transformers distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021a.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé
Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021b.
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, I Louis-Philippe Morency, and Ruslan Salakhut- dinov. Transformer dissection: An unified understanding for transformer's s attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4344 4353, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 1/t/5//////191111 1443. 1443. URL https//accaathooeg..o////
Jonathan Uesato, Brendan O' O'donoghue, Pushmeet Kohli, and Aaron Oord. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learning, pp. 5025-5034. PMLR, 2018
Robert A Vandermeulen and Clayton Scott. Robust kernel density estimation by scaling and projection in hilbert space. Advances in Neural Information Processing Systems, 27, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0477 2020.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
Roy E Welsch and Richard A Becker. Robust non-linear regression using the dogleg algorithm. Technical report, National Bureau of Economic Research, 1975.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint 1xXiv:9966.0233777 2019.
Shaolei Zhang and Yang Feng. Modeling concentrated cross-attention for neural machine translation with Gaussian mixture model. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1401-1411, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2011 http://22.2..nnn--oonooo findings-emmili 121. URL https://aclanthology org/2021 findings---nnn.nn www.iinns-emmll.p.....
Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International Conference Machine Learning, pp. -23994. PMLR, 2022.
13

---

Under review as a conference paper at ICLR 2023
Supplementary Material of "Robustify Transformers with Robust Kernel Density Estimation"
A PROOF OF PROPOSITION
Proposition 2. Assume the robust loss function p is non-decreasing in [0, p(0) and
limg=0 pla) 0. Define (xx) p'(z) and assume (0)) lim (((t) exists and finite. Then the
optimal Probust can be written as
Probuss
Σ jE(N]
wyko (2)g.));
where w (wi,.... WN) E AN. and w x 4 (11,o(23)))) Probuarllaa..) Here Dn denotes the n-dimensional simplex.
Proof. The proof of Proposition 2 mainly adapted from the proof in Kim Scott (2012). Here, we provide proof of completeness. For any p Hk,, we denote
(p)
Σ ( -Pl JEIN]
N
Then we have the following lemma regarding the Gateaux differential of J and a necessary condition for Probust to be optimal solution of the robust loss objective function in equation 13.
Lemma Given the assumptions on the robust loss function p Proposition the Gateaux differential of J at p Hx, with incremental h Hk.- defined as 8. ( h), is
J(p + Th) J(p) - (VV h)) h)H He.
8J(p; h) = -iim T-+0
where the function V Hr. Hks is defined as:
V()
Σ JE[N]
& (1hio - pllHee) (ai;))); - p).
N
A necessary condition for Probust is V (Probust) = 0.
The proof of Lemma can be found in Lemma of Kim & Scott (2012). Based on the necessary condition for Probust in Lemma 1, i.c., V (Probust) O, we have
Σ d (xrg.)) Probusa MaanssllHHas (H..)) (305)))
prrbuss) 0.
JE[N]
Direct algebra indicates that Probust EJEIN] (2),)) where (wi,... ,WN) AN. and Wj x vb (1lio - -PobusullHa ). As a consequence, we obtain the conclusion the proposition.
B ABLATION STUDIES
14

---

Under review as a conference paper at ICLR 2023
Table 3: Text PPL/NLL loss versus the parameter a of Huber loss function defined in equation 14 (upper) and Hampel loss function (Kim & Scott, 2012) (lower; we use 2 x a and 3 x as parameters b and on original and word-swapped Wiki-103 dataset. The best results are highlighted in bold font and the second best are highlighted in underline. We choose 0.4 in rest of the experiments.
Robust Loss Parameter
0.1
0.2
0.4
0.6
0.8
Clean Data
34.92/3.57
34.87/3.56
34.29/3.54
34.38/3.54
34.46/3.54
34.48/3.54
Word Swap
56,82/4.01
55.97/3.99
55.68/3.99
57.89/4.03
58.26/4.04
58.37/4.04
Clean Data
34.67/3.55
34.32/3.54
34.35/3.54
34.47/3.54
34.53/3.54
34.58/3.54
Word Swap
58.02/4.03
57.86/4.03
57.92/4.03
58.24/4.04
58,37/4.04
58.43/4.04
Table Top-1 classification accuracy on ImageNet versus the parameter Huber loss function defined in equation 14 under different settings. The best results are highlighted in bold font and the second best are highlighted underline. We choose 0.2 rest of the experiments.
Huber Loss Parameter
0.1
0.2
0.4
0.6
0.8
Clean Data
71.45
72.83
71.62
71.07
70.65
70.34
FGSM
56.72
55.83
55.34
54.87
54.02
52.98
PGD
46.37
44.15
43.87
43.25
42.69
41.96
SPSA
52.38
52.42
51.69
51.34
50.97
48.22
Imagenet-C
45.37
45.58
45.63
45.26
44.63
43.76
Table Top-1 classification accuracy on ImageNet versus the parameter a of Hampel loss function defined in Kim & Scott (2012) under different settings. We use 2 x and 3 x parameters b and The best results are highlighted in bold font and the second best are highlighted in underline. choose 0.2 in rest of the experiments.
Hampel Loss Parameter
0.1
0.2
0.4
0.6
0.8
Clean Data
71.63
72.94
71.84
71.23
70.87
70.41
FGSM
56.42
55.92
55.83
55.66
54.97
53.68
PGD
45.18
44.23
43.89
43.62
43.01
42.34
SPSA
52.96
52.48
52.13
51.46
50.92
50.23
Imagenet-C
44.76
45.61
46.04
46.13
45.82
45.31
Table 6: Top-1 classification accuracy ImageNet versus the parameter of SPKDE defined in equation 16 under different settings. 5 --g 1, where E the percentage of anomalous samples. larger indicates a more robust model. The best results are highlighted bold font and the second best are highlighted in underline. We choose , 1.4 in of the experiments.
1
3
1.05
1.2
1.4
1.6
1.8
Clean Data
74.25
73.56
73.22
73.01
72.86
72.64
FGSM
53.69
55.08
56.03
55.37
54.21
53.86
PGD
42.31
43.68
44.51
44.32
44.17
43.71
SPSA
51.29
52.02
52.64
52.84
52.
51.39
Imagenet-C
44.68
45.49
44.76
44.21
43.96
43.33
15

---

Under review as a conference paper at ICLR 2023
Table 7: Top-1 classification accuracy ImageNet versus the number of iterations of the KIRWLS algorithm in equation employed Transformer- RKDE. Since the increased number of iterations does not lead to significant improvements of performance while the computational cost is much higher, we use the single-step iteration of the KIRWLS algorithm Transformer-RKDDi
Huber Loss
Hampel Loss
Iteration #
1
2
3
Clean Data
72.83
72.91
72.95
72.98
72.94
72.99
73.01
73.02
FGSM
55.83
55.89
55.92
55.94
55.92
55.96
55.97
55.99
PGD
44.15
44.17
44.17
44.18
44.23
44.26
44.28
44.31
SPSA
52.42
52.44
52.45
52.45
52.48
52.53
52.55
52.56
Imagenet-C
45.58
45.61
45.62
45.62
45.61
45.66
45.68
45.71
Table Computation time (measured by seconds per iteration) of baseline methods, Transformer- SPKDE and Transformer-RKDE with different number of KIRWLS iterations. Transformer-SPKDD requires longer time since directly obtains the optimal set of weights via the QP solver.
Iterations of KIRWLS
DeiT
RVT
SPKDE
2
5
Time (s/it)
0.43
0.51
0.68
0.84
0.35
0.41
1.45
16


Reviews:
[{"review_id": "e3VUbMykV-x", "reviewer": "Reviewer_NtCQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a robust variant of the standard Transformer named Transformer-RKDE to improve the robustness of data with contaminated samples. The idea is based on the interpretation that the self-attention in transformer can be viewed as a non-parametric estimator based on the kernel density estimation (KDE). The authors re-interpret KDE as a regression in the Reproducing Kernel Hilbert Space (RKHS) and indicate that the vanilla KDE is sensitive to data corruption. Therefore, the authors introduce a robust version of the KDE and use the robust KDEs to construct the robust self-attention mechanism for the Transformer.", "summary_of_the_review": "Overall, the innovation is somewhat interesting to me. However, I have some concerns about its efficiency and practical application value.", "strength_and_weaknesses": "Strengths:\n+ The idea seems interesting. Starting from the inherent problem of self-attention (the KDE is not robust to the outliers), the authors introduce RKDE, the robust version of the KDE, to construct robust attention in transformer.\n+ The experiments verify the effectiveness of the proposed model in both language modeling and image classification tasks.\n \nWeaknesses:\n+ Lack of quantitative analysis on the efficiency of the proposed model. Since additional weight calculation requires iteration, I wonder whether it will affect the efficiency of the algorithm.\n+ Because the experimental verification is mainly designed for different adversarial attacks, I wonder how the final effect of the proposed method is compared with other models designed for adversarial attacks. This can better show the significance of the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YyWWMgY10XW", "reviewer": "Reviewer_cLTJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper advocates to use robust kernel density estimation in the self-attention mechanism to mitigate the issue of data contamination. In particular, the idea is to down-weight the weight of bad samples in the estimation process. The authors conduct empirical experiments to demonstrate the effectiveness of the proposal.", "summary_of_the_review": "I fail to identify the technical contribution of this manuscript over existing literature.", "strength_and_weaknesses": "Strength:\n1. The problem and issues to address are well motivated.\n\nWeakness:\n1. The paper appears to be largely overlapped with the existing literature of Kim and Scott (2012), which is intensively cited in the manuscript.\n2. The robust self-attention mechanism does not have strong theoretical support ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "CVZAO-xREhK", "reviewer": "Reviewer_tAEV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a robustified transformer model motivated by the robust kernel density estimation. The proposed model alleviates the influence of bad data (e.g., outliers). The empirical studies on NLP and CV tasks demonstrate the effectiveness of the proposed methods.", "summary_of_the_review": "This paper proposes a variant of Transformer motivated by robust kernel estimation. In particular, a decayed weight $\\omega$ is computed to minimize the influence of the outliers, and in high noise settings, the proposed method reaches promising numerical performance.\n\nMy major concern is the writing of the paper which could be significantly improved. I also suggest authors to further highlight the computation cost of the iterative approach or show the single path can yield an accurate enough estimation of $\\hat{p}_{\\mathrm{robust}}^{(k)}$.\n\n\nMinor Issue: I suggest authors use \\eqref instead of \\ref when referring to the equation number.\n\n\nAt the current stage, I tend to reject this paper but I'm willing to change my evaluation after rebuttal.\n\n\n", "strength_and_weaknesses": "Strength:\nThe proposed method enjoys promising numerical performance when the noise level is high.\n\nWeakness:\n\n1.  The writing could be significantly improved. For example, equations (5-11) in subsection 2.2 on page 3 are almost analogous to equations (3-8) in Nguyen et al. (2022). It may not necessary to use a whole page to restate the existing result in the literature. \nMoreover, in Aglrotihm 1 on page 6, a) the $\\hat{p}_{\\mathrm{robust}}^{(k)}(k)$ is used without declaration, in the input\nline the initial weights of $\\omega\\_{\\mathrm{marginal}}^{(0)}$ and $\\omega\\_{\\mathrm{joint}}^{(0)}$  are not used, and I don't find the definition of $\\omega_j^{\\mathrm{joint}}$.\n\n2. The bond between theoretical motivation and real implementation is kind of weak. For example, the theoretical analysis requires the solve for $\\hat{p}_{\\mathrm{robust}}^{(k)}$ via an iterative method (e.g., KIRWLS). However, when checking the code, the RBFMultiHeadAttn class in Line 391 of src/mem_transformer.py, I don't find the for loop in the forward function and it seems the attention \nbeing computed via only a single path...  If it is the case, the $\\hat{p}\\_{\\mathrm{robust}}^{(k)}$ can be far away from its underlying *optimal* value. If it is not the case, I suggest authors report the computing cost of the proposed method since the multi-round of the iterative method can increase the computation time by a significant amount. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "By05fSexVi", "reviewer": "Reviewer_3QyS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper develops transformer-RKDE by leveraging robust kernel density estimation as a replacement of dot-product attention. The authors show that the optimal estimation of density functions via robust KDE requires computing a set of weights by solving an iterative re-weighted least-square problem. Multiple experiments including NLP and vision demonstrate the proposed method on performance and efficiency improvement. ", "summary_of_the_review": "The paper simply employs the robust KDE (20212 JMRL) for transformer models. The idea is great and the connection is natural and straightforward but the major contribution seems marginal. \n\nI also expect to see the performance benefits compared with other robust transformer models not only KDE-based vanilla models. That's important to demonstrate the necessity of the proposed robust KDE. ", "strength_and_weaknesses": "Strength \n1. The topic is interesting and important \n2. The paper is clear to understand and easy to follow \n3. Related works and backgrounds are well discussed and explained.\n4. Strong experiments are shown the benefits of the proposed method\n\nWeakness\n1. The novelty is weak and lacks of a major contribution\n2. The comparison is not strong and lacks of other robust transformer models instead of KDE-based models ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-QnnTmWvFt6", "reviewer": "Reviewer_N1gC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authoor leverage the robust kernel density estimation (RKDE) in the self-attention mechanism, to alleviate the issue of the contamination of data by down-weighting the weight of bad samples in the estimation process.  Empirical results on language modeling and image classification tasks have demonstrated the effectiveness of this approach. The motivation is clear, but still have few concerns.", "summary_of_the_review": "Motivation of this paper is clear, but the reviwe of the related works and baselines are insufficient.", "strength_and_weaknesses": "strengths:\n1. propose a novel robust transformer framework which replace the dotproduct attention by an attention arising from the robust kernel density estimators (RKDE) associated with the robust kernel regression problem.\n2. the proposed method only requires computing an extra set of weights.\n\nweaknesses:\n1. My most critical concern is regarding the review of the related works. Several robust transformer methods have been proposed, I suggest that the author should review those works in the related works, please see an incomplete list here, Mao X, Qi G, Chen Y, et al. Towards robust vision transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 12042-12051. Li N, Liu Y, Wu Y, et al. Robutrans: A robust transformer-based text-to-speech model[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(05): 8228-8235. Liu J, Singhal T, Blessing L T M, et al. Crisisbert: a robust transformer for crisis classification and contextual crisis embedding[C]//Proceedings of the 32nd ACM Conference on Hypertext and Social Media. 2021: 133-141. Yang J, Gupta A, Upadhyay S, et al. TableFormer: Robust Transformer Modeling for Table-Text Encoding[J]. arXiv preprint arXiv:2203.00274, 2022.\n2. The second concern is the baselines. The current baseline methods seem insufficient. Why does the author choose these methods as the baseline? Why does comparison with these methods can prove the effectiveness of the proposed method? I think the authors should select some similar methods (see weaknesses 1) as a baseline to prove the effectiveness of the proposed method.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject