Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis
Authors: Rachel Teo
Tan Minh Nguyen
Venue: NeurIPS 2024 poster (2024)
Keywords: Transformers
Attention
Kernel Principal Component Analysis
Primary area: deep_learning_architectures
TL;DR: We show that self-attention projects its query vectors onto the principal component axes of its key matrix in a feature space and propose Attention with Robust Principal Components, a novel robust attention that is resilient to data contamination.

Abstract: The remarkable success of transformers in sequence modeling tasks, spanning various applications in natural language processing and computer vision, is attributed to the critical role of self-attention. Similar to the development of most deep learning models, the construction of these attention mechanisms relies on heuristics and experience. In our work, we derive self-attention from kernel principal component analysis (kernel PCA) and show that self-attention projects its query vectors onto the principal component axes of its key matrix in a feature space. We then formulate the exact formula for the value matrix in self-attention, theoretically and empirically demonstrating that this value matrix captures the eigenvectors of the Gram matrix of the key vectors in self-attention. Leveraging our kernel PCA framework, we propose Attention with Robust Principal Components (RPC-Attention), a novel class of robust attention that is resilient to data contamination. We empirically demonstrate the advantages of RPC-Attention over softmax attention on the ImageNet-1K object classification, WikiText-103 language modeling, and ADE20K image segmentation task.

Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis
Rachel S.Y. Teo Department of Mathematics National University Singapore rachel.. tsy@u. mus edu
Tan M. Nguyen Department of Mathematics
National University of Singapore
tanmn@nus. Gannn@nus.edu.sg.
Abstract
The remarkable success of transformers in sequence modeling tasks, spanning various applications in natural language processing and computer vision, is at- tributed the critical role of self-attention. Similar to the development of most deep learning models, the construction of these attention mechanisms relies on heuristics and experience. In our work. derive self-attention from kernel prin- cipal component analysis (kernel PCA) and show that self-attention projects its query vectors onto the principal component axes of key matrix a feature space. We then formulate the exact formula for the value matrix in self-attention, theoretically and empirically demonstrating that this value matrix captures the eigenvectors of the Gram matrix of the key vectors in self-attention. Leveraging our kernel PCA framework, we propose Attention with Robust Principal Compo- nents (RPC-Attention). novel class of robust attention that is resilient to data contamination. We empirically demonstrate softmax attention on the ImageNet- I K object classification, WikiTText- 103 language modeling, and ADE20K image segmentation task. The code is publicly available at htttps://gite http:///gggiuub.nnnaacateeeeeeee com.raacttsy/KKAACCCC....
advantages RPC-Attention over
Introduction
Transformers [83] have emerged the preeminent model for tackling a myriad of challenging
problems in natural language processing [4, 20, 1, 66, 9, 39, 17], computer vision [22. 78, 43, 63, 38, 24), reinforcement learning [13, 35, 94, 40], and other applications [93, 68, 36, 88). The effectiveness of transformers is rooted in their ability to learn from unlabeled data and take advantage of pre-trained models for downstream tasks that involve diverse data modalities with limited supervision [64, 65, 90, 97, 67]. At the core of the transformer's success lies the self-attention mechanism, which serves as the fundamental building block of a transformer model. This mechanism enables each token in sequence to aggregate information from other tokens by computing weighted average based on similarity scores between their feature representations. Facilitating dynamic interactions among tokens, this attention mechanism allows tokens to selectively attend to others, thereby attaining contextual representation [5, 56, 60, 42]. The flexibility in capturing diverse syntactic and semantic relationships is an important factor contributing the success of transformers [77, 85, 33, 84, 53).
Self-Attention. For a given input sequence := [m...... N T E NDND= of feature vectors, self-attention transforms X into the output sequence H in the following two steps:
Step 1: The input sequence is projected into the query matrix Q, the key matrix K, and the value matrix V via three linear transformations
Q XW. = XW V = XW,,
(1)
Please correspond to: tanmnθnus.c edd..ss
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

where WQ. WK E RDDD and Wy E R R.xD, are the weight matrices. We denote Q == IN K = k,,,.. kv]", and V == -o,,... uNl", where the vectors q ki V2 for 91 1,... N are the query, key, and value vectors, respectively.
Step 2: The output sequence H == [hi,'.. ,Ny]' is then computed as follows
H softmax (KK/VDD AV, AV,
(2)
where the softmax function is applied to each row of the matrix QKT/VD. The matrix A softiaax QK E NRN and its component dij for i, = I, NN are called the attention matrix VD
and attention scores, respectively. For each query vector q, for i = -,- .N, an equivalent form of Eqn. (2) to compute the output vector h; is given by
h Σ softmax (g.K///D)o)
The self-attention computed by Eqn. (2) and (3) is called the scaled dot-product or softmax attention. In our paper, we call transformer that uses this attention the softmax transformer. The structure that the attention matrix A learns from training determines the ability of the self- attention to capture contextual representations for each token.
Despite their impressive achievements, the development of most attention layers rely on intuitions and heuristic approaches. The quest for a systematic and principled framework for studying and synthesizing attention layers has remained challenging.
Contribution. We study and analyze self-attention in transformers from the perspective of kernel principal component analysis (kernel PCA). In particular, we develop a novel connection between self-attention and kernel PCA, showing that self-attention projects its query vectors Q,, i = 1, N, onto principal component axes of the key matrix K in a feature space. We then inspect the structure of the value matrix V of self-attention suggested by our kernel PCA framework, validating V captures the eigenvectors of the Gram matrix of the key vectors kz. = .,....N.. .... Using our framework, we then propose new class of robust attention, namely the Attention with Robust Principal Components (RPC-Attention). Our contribution is three-fold
1. We derive self-attention from kernel PCA, showing that the attention outputs are projections of the query vectors onto the principal components axes of the key matrix K in a feature space.
2. We discover and validate that the value matrix V in self-attention captures the eigenvectors of the Gram matrix of the key vectors kj.j N..
3. We develop the Attention with Robust Principal Components (RPC-Attention), new attention mechanism that is resilient to data contamination, using our kernel PCA framework.
We empirically demonstrate the benefits of RPC-Attention on the ImageNet- IK object classification, ADE20K image segmentation, and large scale WikiTeext-10 103 language modeling tasks. We further illustrate the robustness of RPC-Attention through our evaluations on popular, standard robustness benchmarks, as well as various white and black box adversarial attacks on ImageNet-1K K images, 15 different types of corruptions on the ADE20K dataset, and word swap attack on WikiText-103.
2 Principal Component Analysis of Attention
In this section, we will derive attention from kernel PCA. Suppose we are given a dataset M (ki...., ..) C. RD, Here, ki...., KN are attention keys in self-attention. As in kernel PCA, we first project these data points into feature space using a feature map ((xx) == ((x)/g(x), where (b-) is nonlinear transformation from IK D RD', and g(-) is a vector-scalar function that computes scaling factor for (xx). We center the projected data as follows:
pkkk)) =p(kk
Σ pkky)
y==

---

Letting C' be the covariance matrix of the centered data in feature space, its eigenvector expansion is
1 N N
C= (kk))
Σ
Cuu = daule d I....,,.. DV.
(5)
==1
Plugging C' into (5), we obtain
Σ ((k))) ua} = dauas
(6)
Thus, provided that Xa 0, the eigenvector un is given by a linear combination of the (kk,) and with d NX, (kk,) ud, can be written as
Ud
adjsp(k)).
(7)
/=1
2.1 Deriving Attention from Kernel PCA
In order to derive self-attention from kernel PCA, we consider the query vector qi in self-attention as new data point. The projection h; of a new test point qi onto the principal components un in Eqn. (7), for d I..... Du, is given by
N
1 N
h,(d) p(q.)) u Σ ==1
(
Σ
aajsp(a.) ((pp(a))(<ppkk))))
Σ (kk 5'=1
N
N k(q;s kj) adj
N
k(q.,,y)
Σ
N
adj
g(q:)
g(k;)
N
g(q.)
g
(k(z,
==
j'=1
1=1
N k(q;:kg)
g(q:) (g(k))
adj
N j'=1
Σ
N
g(k:))
any
=
N
klq., k,)
g(q:)
Cdj.
(8)
j=1
where the kernel k(x, y) = ((x)) T (by) and vaj = o(k, 1 E Li'==1 [(,,) We further let
the self-attention's value vectors vj [Uis: ... vD E R D. j 1,...; N. and rewrite the projection h; as h4 = Enn k(q,, f v. Selecting g(x) = En k(x,, kj) and k(x, y) =
exp(x //v v/VD), we obtain a formula of attention:
k(g;, kj)
h
Uy Σ softmax (g. kz/ kt/VVD/y
(9)
2 k(qs, kz))
Recovering Self-Attention: Eqn. (9) matches the exact formula of self-attention as in Eqn. (3). Thus, can view outputs h; of self-attention projections of the query vectors g., 1,..., N, onto Dr principal components in a feature space:
[4((q)) (0()).....(((Ng)))))) (u...... up.)
(10)
Computing the Value Vectors: As derived above, the self-attention' 's value vectors vy are given ass
by: Uj = .........".. "'p.,, E Dox1, j 1,...,N, N, where Udj = a(k,) N E s(k)) d =
aay
1.... D.. Since (kk) be calculated g(k,) - k(k,, ky ), in order to compute Uj need to determine the coefficients dijj...,, .up. for 1 N.
We define (xc 3) ((x) ( and the Gram matrix K2 NxN with elements K (i,
kip (k,, k,). Substituting the linear expansion Eqn. (7) into (6), we attain
Σ p(k,) (kk,)
any (kk)) Xa
Σ j=1
auy (k;
3

---

We multiply both sides of the above by (((k)) to obtain Kzas= N with the column vector au (ad1,"' [dN]" Nx1, We compute dd by solving
a = NaNaa.
(11)
The calculation of K is provided Remark 1. We summarize our results in the following theorem.
Theorem (Softmax Attention as Principal Component Projections) Given a set M of vec- tors, M == kyy ID,, a kernel k(x, y) := exp(x T "y/DD), and a vector-scalar functionn g(x) == Σソ= k(z, k2), self- telf-atttntion performs kernel PCA and projects a query vector qı E R onto principal component axes of M in an infinite dimensional feature space up as follows
h; softmax (1, (g. (..k//D)or
The feature space vp is induced by the kernel kip (z, y) = k(w,y) and the value vectors Oj = g'
[Uij...., E F2 =....... N, where Vdj == g(k,)
g(k,)) - Dy. The column vectors ag = [al:...,,ad." E RNNxI can be determined by solving the eigenvalue problem defined in Eqn. (11). This constraint by can be relaxed by letting the self-attention learn v; from data via a linear projection of the input z. Le. Vj Wyzz where Wv is a learnable matrix.
Bay
Remark (Calculating the Gram Matrix Kp) In the eigenvalue problem defined in Eqn. (11), the centered Gram matrix Kep can be computed from the uncentered Gram matrix K₂p with elements K (i ki (k,, K,) = sp (ks)) <<(kk) In particular, K2 = K₂ - 11y K K IN + 1y KpIN. where y denotes the Nx N matrix which every element takes the value 1/ 1/N [8] (See Appendix B). k(a.y)
Here, the kernel kip (a, y) = g(m)g(y)
Remark 2 (Determining D.,) The feature space is infinite dimensional, so we can find infinitely many principal components. However, the number nonzero eigenvalues of C' in Eqn. (5) cannot exceed N, the number of data points, since C' has rank at most equal to N. Notice that only principal components corresponding to nonzero eigenvalues are used for projections in kernel PCA. Thus, D.. the number of principal components used for projections as in Eqn. (10), must be less than or equal N, i.e., D. S N.
Remark 3 (Parameterization of the Value Matrix V) Different parameterizations of the value matrix V can result in different self-attention architectures. For instance, the projection h, of a query vector qi onto the principal components Udd d = 1,.... D., in Eqn. (8) can be rewritten as N Σ ===l o(k;) g(kg)
(.(d)- = Σ k(g,, kz))
N
Odj
g g(ky) ady
j=1 g(qi)
g(kg)
N
Letting Vaj = od and 833' o(k,)
g(k,)
obtain
g(k,))
k(g,, k.)
h;(d) Σ
Udj
Σ y'11
Siz Uay
g(q:)
Following the same derivation as above, we can write the projection h; as an attention
softmax (gT (1.
(TTTs//VD)
1
Σ
N
1=1
8'=1
The matrix form of this new attention form is as follows:
H = softmax QQK /VD)(I-S)) SVV,
(12)
where is an identity matrix, and S is the matrix whose elements S(j.j') = Spy. We name the self-attention defined by Eqn. (12) the Scaled Attention. Even though the softmax attention in (2) and the Scaled Attention in (12) are mathematically equivalent according to our kernel PCA framework, the training procedure might cause the self-attention models that are derived from different parameterizations to have different performances.

---

2.2 Analysis on the Convergence of Self-Attention Layers to Kernel PCA
In this section, we discuss empirical justifications that after training, the value vectors vj parameter- ized as a 1-layer linear network, i.e., v; Wyzj. I...., ... in self-attention converge to the values predicted by our kernel PCA framework in Theorem 1. In other words, we provide evidence that the self-attention layers in transformers try to learn their value vectors vj to perform the kemnll PCA.
2.2.1 Projection Error Minimization
PCA can be formulated based on projection error minimization as well. In particular, PCA minimizes the average projection cost defined the mean squared distance between the original data points and their projections [62]. Given our kernel PCA framework in Theorem 1, this implies that self-attention minimizes the following projection error
J proj
Σ ()x(()))
Σ d=1
haruall
-
(13)
N
N
In the derivation above, we leverage the or- thonormality of (ua)d= Pr and has = (p(x)) ud- Here, notice that we can compute |lp((:))1 from qi and [k5), exp(g, q. //VV)/(E) exp(q, T kt/VD))2
Train_loss
Test_loss
Reconstruction Loss
60
[1<(()))l
55
In Fig. I, we empirically show that a transformer model minimizes the projection loss Jpoj during training. Here, we train vision transformer [22]. ViT-tiny model in particular, classification task and compute the average of Junj across attention heads and layers. This result suggests that during training, the transformer learns to perform PCA at each self-attention layer by implicitly minimizing the projection loss Jproj- Thus, the value vector Vj. j I,..., N. in self-attention layers converge to the values specified in Theorem 1. We provide more details the computation of prooj in Appendix C.
50
45
the ImageNet-1K
40
35
100
Epochs
Figure 1: Projection loss vs. training epochs of ViT-tiny model. The reconstruction loss is aver- aged over the batch, heads, and layers. The down- ward trend suggests that the model is implicitly minimizing this projection loss.
2.2.2 Learning Eigenvectors of K in Eqn. (11)
In this section, we study empirical results that confirm Eqn. (11). In particular, we aim to verify that after training, the value matrix V "......,,""" captures the eigenvectors a,, d 1...., D,,. of the Gram matrix K.p.
We first compute a, in terms of V. Recall from Eqn. (11) that a, = [adi, ad2, .. AdN] We denote the diagonal matrix G == diag(1 (k))))))))... (..g/))...,1////// 1/9(kx)), )), the matrix A = aa,..... a ap. and
rewrite the value vectors Uj. j 1, ,,..., N. as follows:
an.Li g(k;)
an. bi'l
(kg
N
g(k,)
N
==1 g (k))
y'=
The value matrix V in self-attention is then given by
V= GA Gl A (I-IINN)
Thus, given the value matrix V X XWJ that the self-attention learns after training, the estimation āg of ag can be computed as
ā (I 1N) G ([
We empirically verify that K_d Na.
= = (n...., 7N] where 71 = IN const, which
confirms that di is eigenvector of K.p In particular, in Fig. 2(Left), we plot the average pairwise
5

---

Average attention heads and layers
Layer
Layer
6.06646
0.010
0.500422
0.500000
0.005
-0.850423
0000473
0.000845
25
50
80
50
60
7.0097945
10
0.000
Layer
Layer 11
-0.005
-0.010
10
20
30
40
50
60
Index the principal companent axis
Index of principal component
Figure 2: Mean and standard deviation of the absolute differences of elements the constant = I...., De. The mean should 0 with small standard deviations when daj are close to values predicted in Theorem 1. For comparison, we observe that the max, min, mean, and median of the absolute values of all the eigenvalues, averaged over all attention heads and layers, are 648 46, 4.65, 40.07, and 17.73, respectively, which much greater than the values of %/3
1
.N, for each principal component axis of K The results are averaged over all attention heads and all layers in the 12-layer ViT-tiny model trained on ImageNet- IK. As can be seen in our figure, the absolute difference between any pair of % and 7j is almost with very small standard deviation. Similar results are observed each layer when averaging over all attention heads in that layer. In Fig. 2(Right), we show the results for Layers 2, observe that the max, min, mean, and median of the absolute values of these D, cigenvalues, averaged over all attention heads and layers, are 648.46, 4.65, 40.07, and 17.73, respectively, which are much greater than the values of h%| T/l. These
absolute differences of 7/s and 7/ i
5, 8, and 11 in the model. For comparison,
results empirically justify that K.
const and the value matrix captures the eigenvectors of Nå. K after the transformer model trained, as suggested in Eqn. (11).
In order to prove that after training transformer with the value vectors vj parameterized as Wy au
Bay j 1,..., N, using stochastic gradient descent, daj converges g(k,) N E okk, stated Theorem
is sufficient to prove that after the training, the outputs h. self-attention become projections of the query vectors g., i = 1,..., N, onto DT principal component axes in the feature space yp, i.e., the eigenvectors u1, UDD of the covariance matrix To theoretically prove this result for a multi-layer multi-head softmax transformer trained to explicitly minimize particular loss, c.g., cross-entropy or L2 loss, using stochastic gradient descent is indeed challenging due to the highly nonconvex nature of the optimization problem and the nonlinear structure of the model. Our experimental results in Section 2.2.1 and 2.2.2 empirically justify this result and serve as guidance for the theoretical proof, which we leave for future work.
3 Robust Softmax Attention
In this section, we employ our kernel PCA framework Section 2 to derive a new class of robust attention, namely, Attention with Robust Principal Components (RPC-Attention).. It is well-known problem that both PCA and kernel PCA are sensitive to grossly corrupted data routinely encountered in modern applications [11, 49, 34, 19]. Since self-attention performs kernel PCA by projecting the query vectors q:, i 1,..., , N, onto principal components in a feature space as derived in Section 2.1, it is also vulnerable data corruption and perturbation. Our RPC-Attention robustifies self-attention by solving convex program known as Principal Component Pursuit (PCP) [11].
3.1 Principal Component Pursuit
Given corrupted measurement matrix M E NxD both PCA and PCP aim to recover low-rank matrix L E N xD from M. However, while PCA models the corruption by a small Gaussian noise term, PCP models the corruption by matrix S E Nx that have arbitrarily large magnitude with sparse supports. In particular, PCP solves the following convex optimization problem:
minimizer.s
[|]|| + A||S||
subject to L + S = M,

---

Algorithm 1 Principal Attention Pursuit (PAP)
initialize: S. Yo 0: H > O.
while not converged do
compute Sk41 1
SM/U(K Le + H YY):
compute L Softmax K - SL41 1 ar
Yk, K Skt1 -M);
compute YE+1 Ys + u(K L6+1 Sk+1); end while output: L.
where ||| is the nuclear norm of L, ie.. the sum of the singular values of L, and || = > id S is the --oormm of S. From [11], under minimal assumptions
the rank and sparsity of L and S the PCP solution exactly recovers the low-rank component L and the sparse component S. Since PCP can recover the principal components of a data matrix even when positive fraction of the measurements are arbitvarily corrupted, it is more robust than PCA.
3.2 Attention with Robust Principal Components
In self-attention, following our kernel PCA framework in Section 2, the dataset M is given as M- = {k...... kN } Rº and .i...., YN are key vectors. Thus, the key matrix K := [*,,-.., kx] RNN in self-attention can be set as the measurement matrix M in PCP. Then, the PCP for self- attention can be formulated as
minimizer,5
LII+ + S||S||1 subject to S+ K.
(14)
Following [11], we utilize the Alternating Direction Method of Multipliers (ADMM) algorithm introduced in [41, 91] to solve the convex program (14). The augmented Lagrangian of (14) is C(L, Y) = | + A|| + KY,K-I,
S) + //2 K - L L-
An ADMM solves the convex program (14) by iterating the following steps until convergence: [1] setting S = arg ming L(L,, S, Y)), [2] setting Lx+1 = arg min, L(L, Sk++ Yx). and [3] updating the Lagrange multiplier matrix Ys+1 Y2 + , Lk+1 Sk+1). We define
(x)x) = sgn(x) max [|r] -T,0) 0) as an element-wise shrinkage operator and D, US (
as singular value thresholding operator with the singular value decomposition of X UEV* As proven in [11], we can rewrite steps [1] and [2] as
arg min S, Y) Sx/u (K
1 YY; arg min ((L, Dr - S p YY)
5
Dr finds a low-rank approximation of K S 1 YY. Thus, we obtain an approximation of the above equation by replacing D,, by low-rank approximation operator. Such approximation takes a step towards the minimum of C L, S, Y) when fixing S and Y. It has been empirically observed and theoretically proven that the output matrix H of self-attention is low-rank, phenomenon known as over-smoothing or rank collapse [71, 86, 21]. Therefore, we can replace 'D by self-attention operator. The ADMM method applied to self-attention, which we name Principal Attention Pursuit (PAP), is given by Algorithm 1. We define our RPC-Attention as
Definition (Attention with Robust Principal Components) An RPC-Attention performs the PAP in Algorithm for n iterations with X as a hyperparameter: For the key matrix K E NND RPC-Attention sets μ. DD/ ||| K |l as suggested in [11], where |||]| = Eid Kidl. The output
matrix H of RPC-Attention is set be the low-rank output matrix L from PAP.
4 Experimental Results
We aim to numerically show that: (i) RPC-Attention achieves competitive or even better accuracy than the baseline softmax attention clean data, and (ii) the advantages of RPC-Attention are more prominent when there is a contamination of samples across different types of data and a variety of tasks. We also validate the performance of the Scaled Attention proposed in Remark 3.
Throughout our experiments, we compare the performance of our proposed models with the baseline softmax attention of the same configuration. All of our results are averaged over 5 runs with different seeds and run on 4 100 GPU. Details on the models and training settings are provided in Appendix and additional experimental results are provided in Appendix F. Primarily, we focus on a ViT-tiny model backbone [22], but included in the appendix are experiments on a larger model backbone, ViT-base, and state of the art (SOTA) robust model, Fully Attentional Networks (FAN) [96].
7

---

Table I: Top-1, Top-5 accuracy (%). mean corruption enror (mCE), and area under the precision-recall curve (AUPR) of RPC-SymViT and SymViT clean ImageNet-1K data and popular standard robustmess benchmarks for image classification. RPC-SymViT (niter/layerl) applies n PAP iterations only at the first layer. RPC- SymViT (riter/all-layer) applies PAP iterations at all layers.
IN-IK
IN-R
IN-A Top-1 f
IN-C
IN-O
Model
Top-1 f
Top-5 f
Top-1 f
Top-1 f
mCB
AUPR
Vim (baseline)
70.44
90.17
28.98
6.51
41.45
74.75
17.43
RPC-Sym (4iter/layerl)
70.94
90.47
29.99
6.96
42.35
73.58
19.32
RPC-Sym ViT (Siter/layerl)
71.31
90.59
30.28
7.27
42.43
73,43
20.35
PPC-SymVV1 (Giter/layerl)
71.49
90.68
30.03
7.33
42.76
73.03
20.29
RPC-SymViT (2iyer
70.59
90.15
29.23
7.55
41.64
74.52
19.18
Table 2: Top-1/5 1ip-1/5 accuracy (%) on attacked ImageNet-1K data. RPC-SymViT (viter/layerl) applies n PAP iterations at the first layer. RPC-SymViL (witer/alll-ayer) applies v. PAP iterations at all layers.
SymViT (baseline)
RPC-SymViT (4iter/layerl)
RPC-SymViT RPC-SymViT (biter/layerl)
RPC-Sym (2iter/all-laye)
Attack
Metric/Model
(Siter/layerl)
Top-17 Top-5 f
4.98
5.15
5.20 11.13
5.11
6.12
PGD
10.41
11.20
11.34
13.24
Top-1 1 f
23.38
26.62
26.75
27.22
29.20
FGSM
Top-5 f
53.82
56.87
57.19
57.55
59.63
Top-1 f
47.94
48.13
49.29
48.75
51.01
SPSA
Top-5 f
82.63
82.87
83.52
83.27
83.66
Top-1 1 f
67.91
68.48
68.60
68.99
67.60
SL.D
Top-5 1
89.68
90.16
90.18
90.38
89.79
Top-1 f 5
48.44
50.00
50.08
50.36
48.77
CW
72.68
74.24
74.14
74.46
72.91
Top-1 f Top-5
67.81
68.37
68.72
68.81
68.09
Noise
88.51
89.00
89.13
89.27
88.64
Top-1 1 Top-5 63.48
23.09
24.56
24.68
24.74
23.51
AutoAttack
65.11
65.09
65.13
64.06
4.1
Vision Tasks: ImageNet-1K Object Classification
We implement PAP in Algorithm in the symmetric softmax attention layers of ViT-tiny model and compare it to the standard symmetric model as our baseline. We refer to our model RPC-Sym VIT and the bascline model as Sym VIT. For RPC-SymViL we denote by RPC-SymVi VIT (miter/layer1), )), n = 4, 5, 6, to maintain the computational efficiency of the model, apply PAP iterations only at the first layer to recover cleaner data matrix that is then sent to the subsequent layers in the model. In the second setting, which we denote by RPC-Sym VIT Initer/all-layer),
study two settings. In the first setting, which
2, apply PAP iterations all layers. We note that iterative scheme has the potential to have an increased computational load, hence, we provide a comparison on the number of flops per sample, time per sample, memory and number of parameters between RPC-Sym ViT and the baseline in Appendix E.9, showing a comparable efficiency.
Robustness against Data Corruptions. To benchmark robustness data corruptions, we use the standard datasets, ImageNet-C [3N-C)]]]], ImageNet-A (IN-A), ImageNet-O (IN-O) [32], and ImageNet-R (IN-R) [30]. We provide details each dataset and the metrics for evaluation in Appendix A. The direction of increasing or decreasing values of these metrics signifying greater robustness are indicated in the Table I with arrow, along with the results each dataset.
Across all evaluations. RPC- Sym outperforms the Sym baseline, thereby justifying the advantages of our RPC-Attention. Particularly, RPC-Sym ViT with 6 iterations in the 1st layer achieves an improvement
over 1% in terms of accuracy on clean ImageNet-1K validation set and almost 3 AUPR on ImageNet-O compared to the SymViT. This result is consistent with the intuition that higher number of iterations executed on a consistent data matrix leads to cleaner data. The full details of all corruption types are presented in Appendix E.3.

---

2/ter/allllayer
4lter/layer1
SymVIT
Scaled Attention $
Baseline VIT
Biter/layer1
5iter/laye11
Scaled Attention Ct X
Asym
Top-1 accuracy log scale)
PGD
FGSM
Validation Top-1 Accuracy
Validation Loss
Top-1 accuracy (%)
2,0
1.8
Log(loss)
24
Perturbation budget
Perturbation budget
Epoch
Epoch
Figure 3: Left: Top-1 I accuracy of RPC-SyyVVII vs. baseline Sym evaluated on PGD/FGSM attacked
ImageNet- IK validation set across increasing perturbation budgets. Right: Validation top-1 accuracy (% and loss of Scaled Attention the baseline asymmetric softmax attention ViT for the first 50 training epochs.
Table 3: Validation/test perplexity (PPL) on clean WikiText- 103 and word swap attacked dataset. RPC-Symlm (miter/layerns-rre) applies iterations of PAP only in layers nı r2 of the model.
Wikitext-103
Attacked Wikitext-103
Model
Valid-ppl L Test-ppl
Valid-ppl
Test-ppl $
Symlm
38.37
36.36
42,90
44.32
RPC-Symlm
37.38
35,26
41.83
43.16
(4iter/layer1-4)
Robustness under Adversarial Attacks. We report the top-1 and top-5 accuracy of the models
on ImageNet- IK distorted by white box attacks, including PGD [45], FGSM [28], SLD [79], and CW [12] attacks. We further examine the models on the black-box attack SPSA [82], AutoAttack [18], and noise-adding attack. In Table 2, we present the results of all attacks, where PGD and FGSM are reported for the maximum perturbation budget.
On all attacks, RPC-SymViT outperforms Sym by a substantial margin, demonstrating the effectiveness of RPC-Altention. Notably, as the square attack in AutoAttack also score-based black box attack, in order to further verify our method, we include the results of APC-SymViTT (6iter/layerl) and the baseline SiT this attack in the Appendix E.4 in which our model also performs better. This result together with our model's considerable improvement over the baseline on the black-box SPSA attack justifies that RPC-Attention's 's robustmess against adversarial attacks not due to any form of gradient masking. In addition, we illustrate RPC-Sym increasing perturbations for PGD and FGSM in Fig. 3(left). Details on the evaluation of the models under all attacks are provided in Appendix A.2.
S---Syyiii ViT's ' robustness across
ADE20K Image Segmentation. We continue evaluating the benefits of our method by implementing RPC-Attention in a Segmenter model [72] and providing results on the ADE20K image segmen- tation task [95]. Table 6 in Appendix A.3 shows that Segmenter with the RPC-Sym ViT backbone outperforms the Segmenter with the baseline Sym ViT backbone on both clean and corrupted data.
4.2 Language Tasks: WikiText- 103 Language Modeling
We assess our model on the large-scale WikiText-101 103 language modeling task [47]. Using a standard transformer language model [83], with symmetric attention layer (Symlm), replace it with an RPC-Attention layer (RPC-Symlm). As with RPC-SymViT, we only implement RPC-Attention in the first four layers of Symlm and run PAP for 4 iterations to results of the validation and test perplexity (PPL.) summarized in Table 3 validate the advantage of method, surpassing the baseline by least 1 PPL. for all datasets.
computational overhead. The
4.3 Validating the Benefits of Scaled Attention
In this section, we provide an empirical comparison between the Scaled Attention in Remark 3 and the softmax attention. We train an asymmetric ViT-tiny model with two different versions of Scaled Attention. While the exact value of S can be mathematically formulated as in Remark 3, it

---

might lead to numerical errors that are difficult to handle. Therefore, in the first version of Scaled Attention, we let S be learned parameter matrix in each layer (Scaled Attention S in Fig. 3(right)). In the second version of Scaled Attention (Scaled Attention a x Asym in Fig. 3(right)) given that "زز' = g(ky)) /g()), we rewrite S as the product of a symmetric softmax attention matrix and the reciprocal of its transpose. The model learns this reciprocal by a learnable scalar a and we let S == a.mmm.mmm where Asmm is the symmetric softmax attention. More details are in Appendix E. I.
Fig. 3(right) shows the top-1 validation accuracy and loss over 50 epochs when training ViT models equipped with Scaled Attention and softmax attention on the ImageNet- 1K object classification task. The full training curve can also be found in Appendix E.1. The results suggest that both versions of Scaled Attention outperform the softmax attention. This provides further evidence that aelf-attention learns to approximate a kernel PCA since the Scaled Attention with more explicit structure of the value matrix V suggested in Theorem obtains better performance than softmax attention.
Related Works
Theoretical Perspectives of Attention Mechanisms. The study of the attention mechanism in transformers through different theoretical frameworks has been expanding. [81] shows that attention can be analyzed as kernel smoother over the inputs using appropriate kernel score that is the similarities between them. [16. 87, 37, 55] reduce the quadratic complexity of transformers by linearizing the softmax kernel to improve the computational and memory efficiency. In addition, there are works interpreting transformers using the frameworks of ordinary/partial differential equations [50, 51, 44, 69, 27, 26] and from probabilistic viewpoints with Gaussian Mixture Models [52, 23, 75, 54, 92]. [15] provides a new perspective by emphasizing the asymmetry of the softmax kernel and recovers the self-attention mechanism from an asymmetric Kernel Singular Value Decomposition (KSVD) using the duality of the optimization problem. Another related work views transformers from the perspective of Support Vector Machines [57. 76]. We discuss [15] and [76]. as well as the approaches that use matrix decomposition and iterative algorithms in deep models, in Appendix F, Separate from these works, our kernel PCA perspective derives softmax attention as a projection of the query vectors in feature space. Using our framework, we are able to predict the exact explicit form of the value matrix in self-attention, demonstrating that this matrix captures the eigenvectors of the Gram matrix of the key vectors in a feature space. Our work is the first to show this insight.
Robustness of Transformers. There have been many works studying the robustness of Vision Transformers (ViT) [22] against different types of attacks [7, 61, 73, 96). Recent work that serves to address this include [46], whereby new training strategies and architectural adjustments are proposed to improve the robustness of VIT. In addition, [58] suggests employing a Mahalanobis distance metric calculate attention weights, expanding the feature space along directions of high contextual relevance, thereby enhancing the model's robustness. Also, [29] adapts traditional robust kernel density estimation techniques to create new classes of transformers that are resilient to adversarial attacks and data contamination.
[14, 10] integrate a Gaussian process into attention for out-of- distribution detection, and [80] develops equivariant neural functional networks for transformers. Our RPC-Attention is orthogonal to these methods.
6 Concluding Remarks
In this paper, we derive self-attention from kernel principal component analysis (kernel PCA) as projection of the query vectors onto the principal component axes of the key matrix in feature space. Using our kernel PCA framework, we derive a new class of robust attention, namely the Attention with Robust Principal Components (RPC-Attention), that is resilient to data contamination. A limitation of RPC-Attention is its derivation from an iterative algorithm that leads to its unrolling architecture, increasing the computational cost of the model. In our paper, we mitigate this by only replacing softmax attention with RPC-AAttnntion in the first layer of the model and demonstrate that doing so is sufficiently effective for robustness. In addition, we provide comparison of the efficiency of PtC-AAtention with softmax attention in Appendix E.9 and find that we are comparable across all metrics at test time while only slightly less efficient during training. It is also interesting extend our kernel PCA framework to explain -ulti-layer transformers. We leave these exciting directions as future work.
10

---

Acknowledgments and Disclosure of Funding
This research project is supported by the National Research Foundation Singapore under the Al Singapore Programme (AISG Award No: (CGG2-TC-2023-012-SGGII This research / project is supported by the Ministry of Education, Singapore, under the Academic Research Fund Tier (FY2023) (A-8002040-00-00, A-8002033-00-000, This research / NUS Presidential Young Professorship Award (A-0009807-01-00).
project is also supported by the
References
[1] Rami AI-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 3159-3166, 2019.
[2] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. International conference on machine learning, pages 136-145. PMLR, 2017.
[3] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: qquery-efficiiee black-box adversarial attack via random search. In European conference on computer vision, pages 484-501. Springer, 2020.
[4] Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In International Conference on Learning Representations, 2019.
[5] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2015.
[6] Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in neural information processing systems, 32, 2019.
[7] Srinadh Bhojanapalli,, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit. Understanding robustness of transformers for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10231-10241, 2021.
[8] Christopher M Bishop and Nasser M. Nasrabadi. Pattern recognition and machine learning. volume 4. Springer, 2006.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[10] Long Minh Bui, Tho Tran Huu, Duy Dinh, Tan Minh Nguyen, and Trong Nghia Hoang, Revis- iting kernel attention with correlated gaussian process representation. In The 40th Conference on Uncertainty in Artificial Intelligence, 2024.
[11] Emmanuel J Candès, Xiaodong Li. Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-37, 2011.
[12] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks, In 2017 ieee symposium on security and privacy (sp). pages 39-57. leee, 2017.
[13] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 1504-
2021.
[14) Wenlong Chen and Yingzhen Li. Calibrating transformers via sparse gaussian processes. In The Eleventh International Conference on Learning Representations. 2023.
11

---

[15] Yingyi Chen, Qinghua Tao, Francesco Tonin, and Johan Suykens. attention through asymmetric kernel svd in primal representation. Advances in Neural Informa- lion Processing Systems, 36, 2024.
Self-
[16] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021.
[17] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Kathy
[18] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206-2216. PMLR, 2020.
[19] Fernando De la Torre and Michael J Black. Robust principal component analysis for computer vision. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 1, pages 362-369. IEEE, 2001.
[20] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-41888 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[21] Yihe Dong, Jean- nan-BBptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference Machine Learning, pages 2793-2803. PMLR, 2021.
[22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
[23] Prasad Gabbur, Manjot Bilkhu, and Javier Movellan. Probabilistic attention for interactive segmentation. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems. volume 34, pages 4448 4460. Curran Associates, Inc., 2021.
[24] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023.
[25] Zhengyang Geng, Meng-Hao Guo, Hongxu Chen, Xia Li, Ke Wei, and Zhouchen Lin. Is attention better than matrix decomposition? In International Conference Learning Represen- tations, 2021.
[26] Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. mathematical perspective transformers. arXiv preprint arXiv:2312. 2023.
12

---

[27] Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clusters in self-attention dynamics. Advances in Neural Information Processing Systems, 36, 2024.
[28] lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
[29] Xing Han, Tongzheng Ren, Tan Nguyen, Khai Nguyen, Joydeep Ghosh, and Nhat Ho. Designing robust transformers using robust kernel density estimation. Advances in Neural Information Processing Systems, 36, 2024.
[30] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of distriibitt generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8340-8349, 2021.
[31] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[32] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15262-15271, 2021.
[33] John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UCNLP), pages 2733-2743, Hong Kong, China, November 2019, Association for Computational Linguistics.
[34] Mia Hubert, Peter J Rousseeuw, and Karlien Vanden Branden. Robpca: a new approach to robust principal component analysis. Technometrics, 47(1):64-79, 2005.
[35] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273- 1286, 2021.
[36] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583 --858,98888 2021.
[37] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns:
Fast autoregressive transformers with linear attention. In International conference on machine learning. pages 5156-5165. PMLR, 2020.
[38] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s): 2022.
[39] Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199-22215 Curran Associates, Inc., 2022.
[40] Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. Advances in Neural Information Processing Systems, 35:27921-27936. 2022.
[41] Zhouchen Lin, Minming Chen, and Yi Ma. The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. arXiv preprint arXiv:1009.500 2010.
[42] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. International Conference on Learning Representations, 2017.
13

---

[43] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[44] Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie- Yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint arXiv: I 099960007622 2019.
[45] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018.
[46] Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. In Proceedings of the IEEE/CVF conference on Computer Vision and Puttern Recognition, pages 12042-12051, 2022.
[47] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
[48] John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: framework for adversarial attacks, data augmentation, and adversarial training nip. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 119-126, 2020.
[49] Minh Nguyen and Fernando Torre Robust kernel principal component analysis. Advances in Neural Information Processing Systems, 21, 2008
[50] Tam Nguyen, Tan Nguyen, and Richard Baraniuk. Mitigating over-smoothing in transformers via regularized nonlocal functionals. Advances in Neural Information Processing Systems, 36:80233 80256, 2023.
[51] Tam Nguyen, César Uribe, Tan M Nguyen, and Richard G Baraniuk. Pidformer: Transformer meets control theory. In International Conference on Machine Learning. PMLR, 2024.
[52] Tam Minh Nguyen, Tan Minh Nguyen, Dung DD Le, Duy Khuong Nguyen, Viet-Anh Tran, Richard Baraniuk, Nhat Ho, and Stanley Osher. Improving transformers with probabilistic attention keys. In International Conference on Machine Learning, pages 16621, PMLR, 2022.
[53] Tan Nguyen, Tam Nguyen, Hai Do Khai Nguyen, Vishwanath Saragadam, Minh Pham, Khuong Duy Nguyen, Nhat Ho, and Stanley Osher. Improving transformer with an admixture of attention heads. Advances in neural information processing systems, 35:27937-27952, 2022.
[54] Tan Nguyen, Minh Pham, Tam Nguyen, Khai Nguyen, Stanley Osher, and Nhat Ho. Fourier- former: Transformer meets generalized fourier integral theorem. Advances in Neural Informa- lion Processing Systems, 35:29319-29333. 2022.
[55] Tan Nguyen, Vai Suliafu, Stanley Osher, Long Chen, and Bao Wang. Fmmformer: Efficient and flexible transformer via decomposed near-field and far-field attention. Advances in neural information processing systems, 01-29449-29466 2021.
[56) Tan M Nguyen, Tam Nguyen, Long Bui, Hai Do, Duy Khuong Nguyen, Dung D Le. Hung Tran-The, Nhat Ho, Stan J Osher, and Richard G Baraniuk. A probabilistic framework for pruning transformers via finite admixture of keys. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2023.
[57] Tan Minh Nguyen, Tam Minh Nguyen, Nhat Ho, Andrea L. Bertozzi, Richard Baraniuk, and Stanley Osher. A primal-dual framework for transformers and neural networks. In The Fleventh International Conference Learning Representations, 2023.
[58] Stefan K Nielsen, Laziz Abdullaev, Rachel Teo, and Tan Nguyen. Elliptical attention. Advances in Neural Information Processing Systems, 2024.
14

---

[59] Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Kurakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy, Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang, Yi-Lin Juang, Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gierke, Yinpeng Dong, David Berthelot, Paul Hendricks, Jonas Rauber, and Rujun Long. Technical report on the cleverhans v2.1. 1..1.0 adversarial examples library. arXiv preprint arXiv:1610.00760 2018.
[60] Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2255-2255, Austin, Texas, November 2016. Association for Computational Linguistics.
[61] Sayak Paul and Pin-Yo Chen. Vision transformers are robust learners. In Proceedings of the AAAI conference on Artificial Intelligence, volume 36, pages 2071-2002 2022.
[62] Karl Pearson. Lüi. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11):559 572, 1901.
[63] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8763. PMLR, 2021.
[64] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. OpenAl report, 2018
[65] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[66] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 2020.
[67] Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sereu, and Alexander Rives. Msa transformer. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8844-8856. PMLR, 18-24 Jul 2021.
[68] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 1 118(15), 2021.
[69] Michael E Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Sinkformers: Transform- ers with doubly stochastic attention. In International Conference on Artificial Intelligence and Statistics, pages 3515-3530 PMLR, 2022.
[70] Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference Machine Learning, pages 9355-9366. PMLR, 2021.
[71] Han Shi, Jiahui Gao, Hang Xu, Xiaodan Liang, Zhenguo Li, Lingpeng Kong, Stephen M. S. Lee, and James Kwok. Revisiting over- werrssoothing in BERT from the perspective of graph. In International Conference Learning Representations, 2022.
[72] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEF/CVF international conference on computer vision, pages 7262-7272, 2021.
[73] Akshayvarun Subramanya, Aniruddha Saha, Soroush Abbasi Koohpayegani, Ajinkya Te- jankar, and Hamed Pirsiavash. Backdoor attacks on vision transformers. arXiv preprint arXiv:2206.0847 2022.
15

---

[74] J.A.K. Suykens, T. Van Gestel, J. Vandewalle, and B. De Moor. A support vector machine formulation to pea analysis and its kemel version. IEEE Transactions on Neural Networks, 14(2):447-450, 2003.
[75] Binh Tang and David S. Matteson. Probabilistic transformer for time series analysis. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[76] Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, and Samet Oymak. Transform- ers support vector machines. arXiv preprint arXiv:2308. 2023.
[77] Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy, July 2019. Association for Computational Linguistics.
[78] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[79] Florian Tramer and Dan Boneh. Adversarial training and robustness for multiple perturbations. Advances in neural information processing systems, 32, 2019.
[80] Viet-Hoang Tran, Thieu N Vo, An Nguyen The, Tho Tran Huu, Minh-Khoi Nguyen- Thanh Tran, Duy-Tung Pham, and Tan Minh Nguyen. Equivariant neural functional networks for transformers. arXiv preprint arXiv:2410.04209 2024.
[81] Yao- Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-PPilipee Morency, and Ruslan Salakhutdinov. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors, Proceed- ings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLL pages 4344 4353, Hong Kong, China, November 2019. Association for Computational Linguisties.
[82] Jonathan Uesato, Brendan O' O'donoghue, Pushmeet Kohli, and Aaron Oord. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learning, pages
PMLR, 2018.
[83] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 1998-6008, 2017.
[84] Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpret- ing Neural Networks for NLP, pages 63-76, Florence, Italy, August 2019, Association for Computational Linguistics.
[85] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, Florence, Italy, July 2019, Association for Computational Linguistics.
[86] Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice. In International Conference on Learning Representations, 2022.
[87] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0 04068, 2020.
[88] Zifeng Wang and Jimeng Sun. Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35:2902-2915, 2022.
[89] Yongyi Yang, Tang Liu, Yangkun Wang, Jinjing Zhou, Quan Gan, Zhewei Wei, Zheng Zhang, Zengfeng Huang, and David Wipf. Graph neural networks inspired by classical iterative algorithms. In International Conference on Machine Learning, pages 11773-11783, PMLR, 2021.
16

---

[90] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bie Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[91] Xiaoming Yuan and Junfeng Yang. Sparse and low-rank matrix decomposition via alternating direction method. Pacific Journal of Optimization, 9:167, 2013.
[92] Shaolei Zhang and Yang Feng. Modeling concentrated coos-aateention for neural machine translation with Gaussian mixture model. In Marie-Francine Moens. Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 1401-1411, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[93] Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. Deep learning based recommender system: survey and new perspectives. ACM Computing Surveys (CSUR), 52(1): 2019.
[94] Qinging Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, edi- tors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 27042-2701 -22059. PMLR, 17-23 Jul 2022.
[95] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In 2017 IEEE Conference Computer Vision and Puttern Recognition (CVPR), pages 5122-5130, 2017.
[96] Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International Conference on Machine Learning, pages 27378-27394. PMLR, 2022.
[97] Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. A robustly optimized BERT pre-training approach with post-training. In Sheng Li, Maosong Sun, Yang Liu, Hua Wu, Kang Liu, Wanxiang Che, Shizhu He, and Gaoqi Rao, editors, Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218-1227, Huhhot, China, August 2021. Chinese Information Processing Society of China.
17

---

Supplement to "Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis"
Table of Contents
Experiment Details
18
A.1 Robustness against Data Corruptions
19
A.2 Robustness under Adversarial Attacks
19
A.3 ADE20K Image Segmentation
19
A.4 WikřText-103 Language Modeling
20
B Calculating the Gram Matrix K
20
Plotting Jproj in Section 2.2.1
20
D Principal Component Pursuit
21
Additional Experimental Results
22
E.1
Reparameterization of Self-Attention
22
E.2
Pairwise Absolute Differences of and 7
23
E.3
ImageNet-C Results by Corruption Type
................................
23
E,4
Square Attack
................................
23
E.5 Results ADE20K Image Segmentation
24
E.6
Results RPC-SymViT-base
24
E.7 Results on RPC-Attention in FAN
25
E,8 Results on downstream Natural Language Understanding tasks
25
E.9 Computational Efficiency
27
E.10
Results Robust Asymmetric Attention
27
Further Discussion on Related Works
27
Broader Impacts
28
A Experiment Details
Implementation Details of RPC-SymViT: Our RPC-Sym models have 5.2M parameters, the same as the SymViT baseline. We use standard tiny configuration with 12 transformer layers, attention heads per layer, and a model dimension of 192 and simply replace softmax attention with RPC-Attention. We follow the training settings as in [78] and their implementation available at In Segmenter model,
https://github.co https://gthub.com/ffcccoookk cwt.//accccoooresccchheeoo..
the same RPC-Sym VIT setting to replace the baseline SymViT backbone. We follow the training details in [72] and their code is publicly available well,
There are hyperparameters: μæ this parameter controls the singular value thresholding operator in the PAP algorithm. We set f to the recommended value given in Definition 1; 2) X: this is regularization parameter that controls the sparsity of the corruption matrix S. We finetune for training and observe that RPC-Sym with À = yields the best performance for models with iterations per layer and = yields the best performance for models with iterations only the first layer; 3) the number of iterations of the PAP algorithm in RPC-Attention layer.
18

---

Implementation Details of RPC-Symlm: For our language model, we use standard transformer language model [83], with a symmetric attention layer. The model has a dimension of 128 for the keys, queries and values, while the training and evaluation context length is set at 256. There are 16 layers altogether and heads per layer. Similarly, we replace the softmax attention with RPC-Attention only in the first four layers to save on computational overhead. There are the same 3 hyperparameters as in RPC-SymVii ViT and we use the same value for μ. = 4 and n = 4. We also follow the standard training settings as in [70, 47] and the code base developed by [70], available here httpgggihh.cooSSS///Iai
A.1 Robustness against Data Corruptions
We use the ImageNet- IK dataset that contains 1.28M training images and 50K validation images. There are 1000 classes of images and the model learns an image classification task. For robustness to common corruptions, we use ImageNet-C (IN-C) [31] which consists of 15 different types of corruptions applied to the ImageNet- IK validation set with 5 levels of severity. To test robustness to both input data distribution shifts as well as label distribution shifts, we use ImageNet- (IN-A) and ImageNet-O (IN-O) [32] respectively. Both of these datasets contain a 200 class subset of ImageNet-1K 1K classes with adversarially filtered images. Finally, we test our model on ImageNet-R (IN-R) [30] which contains various artistic renditions of images. This evaluates the model's generalization ability to abstract visual renditions.
Datasets:
Metrics: On ImageNet- IK, ImageNet-A and ImageNet-R, we report the top-1 accuracies for all experiments. We include top-5 accuracies on ImageNet-IK. On ImageNet-C, the standard metric for evaluation the mean Corruption Error (mCE). To calculate this, we average the top-1 error rate for each corruption type across the 5 levels of severity and divide them by AlexNet's 's average errors, then take the final average across all corruption types. We report the area under the precision-recal curve (AUPR) for ImageNet-O which requires anomaly scores. The score is obtained by taking the negative of the highest softmax probability output by the model. The direction of increasing or decreasing values of these metrics signifying greater robustness will be indicated in the table with an arrow.
A.2 Robustness under Adversarial Attacks
We evaluate the robustness of our method against adversarial attacks using CleverHans [59] and AutoAttack [18]. All attacks are executed on ImageNet-ING IK's validation set, and each model evaluated on the whole set. In particular, we use untargeted, white box attacks such as PGD, FGSM, SLD and CW. In addition, we provide results on gradient-fre free black box attack, SPSA, diverse AttoAttack, in the standard setting and a simple noise-adding attack. AutoAAtack consists of untargeted APGD-CE, targeted APGD-DLR, DLR, targeted Fast Adaptive Boundary (FAB) and Square Attack. For further justification of the benefit of our method, we evaluate a variant of our model, RPC-Sym VIT (6iter/layerl) solely on the black-box Square Attack and show that we are robust against black box attacks as well. This is provided Appendix F.4. We use perturbation budget of = 1/255 with the L.0 norm to manipulate the images and evaluate each model with an incremental increase in perturbation for PGD and FGSM. In SPSA, we run 40 steps per attack with a perturbation budget of 0.1. PGD attack uses step size of a 0.15 for 20 steps, where at each step the image is adjusted slightly to maximize the model's loss. Similarly, FGSM does this for a single step. For each attack, we report the top-1 and t accuracy of the model on the distorted dataset with the maximum perturbation budget in Table and across all different perturbations in Figure 3.
Attacks:
A.3 ADE20K Image Segmentation
Dataset: The ADE20K dataset includes complex scenes featuring highly detailed labels, making one of the most challenging segmentation tasks. The training set contains 20,210 images spread across 150 distinct semantic categories. The validation set includes 2,000 images, while the test set comprises 3,352 images. The metrics report for this task are the Mean Accuracy (%) and Mean Intersection- Cerpeettioo--ve-nnnieeee (IOU).
19

---

A.4 WikiText-103 Language Modeling
The WikiText- 103 dataset [47] is derived from Wikipedia articles and is designed to capture long-range contextual dependencies. The training set contains about 28,000 articles, with a total of 103 million words. Each article is divided into text blocks with approximately 3,600 words. The validation and test sets have 218,000 and 246,000 words, respectively, with both sets comprising 60 articles and totaling about 268,000 words. For evaluation, we use a batch size of and apply sliding window of length L to process the text sequences. When calculating perplexity (PPL), we focus only on the final position, except in the first segment, where we evaluate all positions. We corrupt the both validation and test datasets to demonstrate the robustness of RPC-Attention using TextAttack's 's word swap attack [48] create the attacked WikiText-103 dataset. This adversarial attack randomly replaces words in the dataset with a generic "AAA" for evaluation making it difficult for the model to predict the next word in the sequence correctly.
Dataset:
B Calculating the Gram Matrix K
In this section, we will show that the centered Gram matrix K yp can be computed from the uncentered Gram matrix K with elements (t.j) (ki, kj) <p(k;) (kk)), In particular, K K Iy K K l + x KylN, where v denotes the N x N matrix in which every element takes the value 1 1/N. Our centered feature vector has the form
<p(k;) (o((k)
Σ (ky))
y'=1
Then the elements of the centered Gram matrix are given as follows
K (1, = kap k;))
-k;))) T [{((k;))
=k((k) ( (kg))
Σ
(k:))
(kkk)) (ek))T((k))
1
Σ
((kky) T (p(())
1
( (p(k))
N
N
N2
N
-kilkis k:j)
1 N
Σ (ki,ki
1
Σ
(ky k:j) +
1
kipp (ki kz)
N
N2
1=1
which expressed in matrix form, gives the result.
C Plotting Jproj in Section 2.2.1
In Section 1. we plotted the reconstruction loss at each epoch of training. Here, we provide the details of the calculation of this loss. From Eqn. (13),
N
Du Σ
L l(p(g)) N
2
d=1
N
(11 111.113).
(1
In the above, b2 is simply our attention output and [1 (p(ga)T<((e)
2a 4./VDD
EN ed to /VD2a for i L,..., N, can be calculated using the kernel trick follows. Let A be the softmax attention
20

---

matrix, a1 its first column and af denote the element-wise product.
2n 4//VB
25 ei R//VDyy
log(a = log(
(E)=) ex
2 /v/DD
E e A//D//DD
= =log(
log(
22q
(2 ARkks/DDy2
Σ E9 oai 1//
20 /v/DD
= log(
log(
-ogg(a))
(E exxxs/VDD3
2
24 KI/VD
(og(a})
2/Tkn/VD
2q kn/VD
EEjn
N edi 22/D0/2
-log(a
//
Then,
2q T ki k//VD
[|<(()))117
-log(a?)
/e 2/
[|(((a))]1
eTNaN/VD
D Principal Component Pursuit
Let M, S,S NxD be the matrix of our corrupted measurements, the low-rank matrix we seek to recover and sparse corruption matrix respectively. The optimization problem we would like to solve is
minimize
|| + A|ST| L+S = M
(15)
subject to
Under minimal assumptions that the low-rank component L not sparse (i.e., L satisfies the incoherence condition defined in [11]), and the sparse component is not low-rank (i.e., the sparsity pattern of is selected uniformly at random), we will follow the author's choice of algorithm to solve the PCP which is the Alternating Direction Method of Multipliers (ADMM).
The ADMM algorithm uses the augmented Lagrangian,
I(L,S,Y) || + S|| + (Y,M - L S) + H/2 L S- and solves a sequence
optimization problems. We iterate through setting S arg ming I(Lx, S, Y2) and L+-1 arg min ((L, Sk+1, Yt) before updating the Langrange multi- plier matrix Ys+1 = Y2 + μ(M L Sx-1). The advantage the algorithm is that it turns a complicated optimisation problem into sub problems that have straightforward and efficient solutions. Without much difficulty we can show that:
arg min (LL S.Y) =Sx/ (M L L+ -+
arg min -D - S - L
21

---

Scaled Attention S
Scaled Attention OXX Asym
Baseline
Validation Top-1 Accuracy
Validation Loss
2.00
1.75
60
1.50
Top-1 accuracy
Log(loss)
1.25
40
1.00
0.75
20
0.50
0.25
50
100
200 250 Epoch
100 150 200 250 300
Epoch
Figure 4: Plot of the validation top-1 accuracy (%) and loss on a log scale of the bascline asymmetric attention ViT and two variants with the parameterization of Remark. 3. The curves plotted for the full training time and show S trained as matrix parameter well scalar parameter scaling symmetric attention matrix.
where S (a) = sgn(x) max (|x) - T.0) 0) is the shrinkage operator, extended to matrices by applying
element- wise and D2 (X) (E) V is the singular value thresholding operator with the singular value decomposition of X = UVV*
The algorithm is summarised as below
Algorithm 2 ADMM for Principal Components Pursuit
initialize: So Yo 0; μr,
0.
while not converged
compute S Sx/y (M- LA + YY);
compute LE+1 = (M Sk+1 -Y)):
compute Yk+1 YL + a(M LK+1 - Sk+1)i end while
output: L, S.
E Additional Experimental Results
E.1 Reparameterization of Self-Attention
Let Agym [a:y] be the softmax attention matrix. Then, from Remark 3, S has the following form and we multiply the numerator and denominator by 1/ K (kj, ky)) to obtain much more convenient expression.
g(k,))
85'
(
g ((k))
1/ (Ke, ky)
(kagr)
1/ (k3, kiz))
g(kg)
+ K K(ke, ke)
K K(kj, ((k,,,y)))
g(k,)
K (Kg ky)
K(kj, kkz))
g(kz))
aj's
ajj'
1
S N Asym 1/AT sym
In Fig. 4, we plot the full training curve of the two versions of Scaled Attention, Scaled Attention S and Scaled Attention a Asym. We observe that the parameterized models with Scaled Attention do converge more quickly and even obtain a slightly higher validation top-1 accuracy of 73.02% for the scalar variant as compared the standard asymmetric ViT 72. 18%. The final validation loss also lower at 1.18 and 1.23 respectively.
22

---

Layer
Layer
Layer
0.0703
000846
0.0351
0. 0.004423
0.0000952
0,000000
-0.0703
-0 000846
0.0001904
Layer 4
Layer
Layer 6
0,0003596
0.0000946
0.04066
0.0001738
0.02033
0.0000000
0.0000000
0.00000
0001796
0.0000473
-0.0003566
10
20
30
40
50
60
20
30
50
10
20
30
40
50
Layer 7
Layer
Layer
0.001177
0.02977
0.000589
01.88
0.0000
0.000589
0.01488
0.0696
0.001177
-0.02977
-0.1392
Layer 10
Layer 11
Layer 12
0.0571
0.000576
0.0286
0.000788
0.0000
000000
0.000000
0.0286
-0 001043
0.000288
0.0571
10
20
30
40
50
60
002087
10
20
40
50
60
0.000576
10
20
50
60
Index of the principal component axis
Figure 5: Plot of the and standard deviation of the differences in coordinate values of constant vector 1 for 1,..., Dy, for all 12 layers of ViT-tiny model. The should be with small standard deviations
when Maj R3
auj E
a(R))
E.2 Pairwise Absolute Differences of and 7
In Figure 5, we provide the plot of the absolute differences of the coordinates of Kgdd in all 12 Nā. layers of ViT-tiny model as elaborated in Section 2.2.2. This would be constant vector within our framework and the plots provide empirical evidence to justify our claim. In all of the layers, the means are noticeably close to 0 and the standard deviations are also small suggesting that we indeed recovered constant vector.
E.3 ImageNet-CC Results by Corruption Type
In Table 4, we provide the full results various RPC-Sym VIT models against the standard Sym VIT on ImageNet-C for all corruption types averaged over the 5 severity levels. In all types, except for Zoom Blur, the RPC-Sym SymVi ViT model with 6 iterations the 1st layer outperforms Sym
E.4 Square Attack
Square attack [3] is score-based black box attack that does not use local gradient information, hence it is not affected by gradient masking. It operates based on random search scheme that modifies square-shaped regions such that at each iteration, the perturbation lies approximately at the boundary of the feasible set. We evaluate our VPC-SSm ImageNet-1K K validation set and compare it to the baseline. Our result of the top-1 and top-5 accuracy Table illustrates that the effectiveness RPC-Attention against adversarial attacks is not solely due to a form gradient masking as still significantly outperform the baseline square.
variant on square attacked
23

---

Table 4: Top-1 accuracy (%) and mean corruption error (mCE) of all RPC-Sym variants and SymViT V on each corruption type in ImageNet-C. RPC-SymVVT (miter/layerl) applies PAP iterations only the first layer. RPC-SymVIT vuiter/all-layen applies n PAP iterations at all layers.
Model/
SymViT
RPC- VyM
RPC-SymViT
RPC-SymViT RPC-SymViT
Corruption Type
Metric
Maler/alllayeen
(4iter/layerl)
(Siter/layerI)
(6iter/layer
Top-I f
63.21
62.97
63.74
64.19
64.31
Brightness
mCE |
65.16
65.69
64.22
63.43
63.22
Top-1 f
50.25
49.59
49,30
49.84
50.18
Contrast
mCE &
58.53
59.08
59.42
58.79
58.39
Top-I
35.47
35.61
35.54
35.26
36.38
Defocus Blur
mCE +
78.71
78.53
78.62
78.96
77.59
Top-1 t
44.26
44.52
44.68
44.84
44.72
Elastic Transform
mCE +
86.28
85.87
85.63
85.38
85.56
Top-I f
42.72
45.49
46.03
46.59
46.40
Fog
mCE! 4
69.91
66.53
65.87
65.19
65.42
Top-1 f
44.55
45.06
45.93
46.24
46.27
Frost
mCE +
67.08
66.46
65.41
65.04
65.01
Top-1 f
40.97
41.64
42.22
41.67
42.51
Gaussian Noise
mCE +
66.59
65.84
65.19
65,80
64.85
Top-1 f
27.79
28.12
28.48
27.87
28.05
Glass Blur
mCE L
87.39
87.00
86.56
87.30
87.07
Top-1 f
38.56
38.98
40.24
39.46
40.33
Impulse Noise
mCE 4
66.59
66.14
64.77
65.61
64.67
Top-I
46.28
46.23
47.92
48.41
48.63
JPEG Compression
mCE 4
88.58
88.65
85.87
85.06
84.70
Top-1 1
40.47
40.00
41.72
41.81
42.22
Motion Blur
mCE
75.75
76.43
74.16
74.04
73.51
Top-I 39.36
39.81
41.13
41.86
42.25
Pixelate
mCE
84.47
83.84
82.01
81.00
80.45
Top-1 38.83
39.20
39.52
39.03
39.71
Shot Noise
mCE|
68.38
67.98
67.61
68.16
67.40
Top-1
38.27
37.87
38.75
39.12
38.90
Snow
mCE
71.22
71.67
70.66
70.23
70.48
Top-1 f
30.69
29.49
29.97
30.20
30.48
7oom Blur
mCE +
86.82
88.32
87.72
87.43
87.08
Table 5: Top-1 and Top-5 accuracy (%) of RPC-SyyVTT(((icc/lyerr))) and baseline Sym VTT on square attacked ImageNet- 1K validation data. RPC-SymVIT (niter/layerl) applies n PAP iterations only at the first layer.
Square attack
Model
Top-1 1
Top-5
SymViT (baseline)
41.50
79.79
RPC-SymViT (6iter/laycrl)
43.64
80.81
E.5 Results on ADE20K Image Segmentation
We further evaluate the benefits of our method by implementing RPC-Attention in a Segmenter model [72] and providing results the ADE20K image segmentation task [95] in Table 6. We report the Mean Accuracy and Mean Cetrrsectioo-vve---iie
(IOU) for a Segmenter with a RPC-Sym ViT backbone and compare it against baseline Sym ViT backbone. To assess the robustness of each model, we corrupt the ADE20K dataset with the same 15 corruption types in ImageNet-C and report the same metrics averaged over all corruption types in Table 6 as well. Details of our implementation can be found in Appendix A.3, On both datasets, APC-SymViT outperforms the baseline substantially.
E.6 Results PPC-SymViT-bbase
To show that RPC-Attention is not limited to small-scale models, we conduct experiments larger VymViT-base
model, Sym ViT-base with transformer layers, heads per layer, and hidden dimension 768.
We train ViT-bbe with RPC-Attention in all layers using iterations on ImageNet- IK for

---

Table 6: Mean accuracy (%) and mean Intersection-Over--niie (IOU) of RPC-SymViT and SymViT on clean
ADE20K and coruuteed ADE20K dataset. RPC-SymViT (niter/layerl) applies PAP iterations only at the first layer. RPC-Sym (fitte/all-layer) applies n PAP iterations at all layers.
ADE20K
Corrupted ADE20K
Model
Mean Acc.
Mean IOU
Mean Ace. f Mean IOU
SymViT
44.27
34.00
14.85
10.47
RPC-SymViT (4iter/layerl)
45.61
34,69
16.06
11.47
RPC-SymVil (Siter/layerl)
45.51
34.63
16.15
11.35
RPC-SymViT (6iter/layerl)
45.27
34.24
16.44
11.60
RPC-SymViT (Ziter/all-aaeer)
43.61
33.5
15.04
10.64
Table 7: Top-1, top-5 accuracy (%), mean corruption error (mCE), and area under the precision-reae recall curve
(AUPR) of RPC-SymmVTT-baee ViT-hase and Sym Vim-iT- on clean ImageNet-1K K data and popular standard robustness benchmarks for image classification. RPC-SymmVTT-e ViT-hase (xiter/all-layer) applies n PAP iterations at all layers.
IN-IK
IN-A
IN-C
IN-O
Model
Top-1 f
Top-5 f
Top-1* f
Top-1 f
mCE $
AUPR
Sym ViT-hase (baseline)
80.62
94.78
24.03
58.88
52.57
23.96
RPC-Sym VyVV--baeee (2iter/all-layer)
80.72
94.82
24.97
59.29
52.00
26.01
Table 8: Top-1 top-5 accuracy (%) of RPC-Sym base and SymViT-base on PGD and FGSM attacked
ImageNet- IK validation data with the highest perturbation budget. RPC-Sym VyC--Symi Vise (niter/all -lyer) applies n PAP iterations at all layers.
PGD
FGSM
Model
Top-1 f
Top-5 f
Top-1 1
Top-5 1
Sym ViT-hase (baseline)
12.11
25.55
53.61
78.05
RPC-SymViT-base (2iter/all-layer)
13.41
26.68
54.68
78.06
300 epochs. We refer to this model as RPC-Sym base (2iter/allllayer). We compare our RPC- VymViT-base with the baseline the same standard robustness benchmarks before, ImageNet-C. ImageNet-A, and ImageNet-O, as well as on white box attacks PGD and FGSM with the highest perturbation budgets. These results are in Table and respectively and show that RPC-Attention is also effective Sym
E.7
Results on RPC-Attention in FAN
To validate that RPC-Attention is also complementary with SOTA transformer models and can be combined with those methods to improve the SOTA results, we have conducted additional experiments in which we incorporate transformer models that obtain SOTA results on accuracy and robustness. FAN model augments the channel attention (CA) block. This CA
RPC-Attention with FAN [96]. FAN is one of the top
MLP layer that follows the standard self-attention with computes an attention matrix along the channel dimension, taking advantage of the feature covariance and allowing the model to filter out irrelevant information.
We use the FAN- VAN-ViT- (FAN-tiny) variant with symmetric attention for training. In our RPC- Attention FAN (RPC-FAN-tiny), we replace the attention blocks in the first layer of FAN with our RPC-Attention that runs 4 PAP iterations with hyperparameter values of = and μ = DD/4|| K ||. Both RPC-FAN-ting and the baseline FAN-tiny are trained for 300 epochs on the ImageNet- IK object classification task. We summarize our results in Tables and 10. RPC-FAN-tiny outperforms the baseline FAN-tiny all evaluated benchmarks, including ImageNet-Ik, 1k, ImageNet- R, and ImageNet-A. Additionally, on PGD and FGSM attacked data, RPC-FAN-tiny significantly outperforms FAN-tiny by over 3%.
E.8 Results downstream Natural Language Understanding tasks
We conduct additional experiments on several downstream Natural I Language Understanding tasks to illustrate the effectiveness of RPC-Attention during fine-tuning well. The Stanford Sentiment
25

---

Table 9: Top-1, top-5 accuracy (%) of RPC-FAN-tiny and FAN-tiny clean ImageNet- IK data and popular standard robustness benchmarks for image classification. RPC-FAN-tiny (niter/layerI) applies PAP iterations only at the first layer.
IN-IK
IN-R
IN-A
Model
Top-1
Top-5
Top-1 f
Top-1 f
FAN-tiny (baseline)
77.89
94.20
41.79
13.40
RPC-FAN-tiny (4iter/layerl)
77.98
94.27
42.02
13.55
Table 10: Top-1 and lop-5 accuracy (%) of RPC-FAN-iny and FAN-tiny on PGD and FGSM attacked ImageNet- IK validation data with the highest perturbation budget. RPC-FAN-ViT (miter/layerl) applies n PAP iterations only at the first layer.
PGD
FGSM
Model
Top-1
Top-5
Top-1 f Top-5 f
FAN-tiny (baseline)
2.92
4.86
32.01
61.72
RPC-FAN-ting (4iter/layer1)
6.25
10.01
35.12
63.40
Table 11: Train and validation accuracy (%) of RPC-SymSMoE, SymSMoE, RPC-SMoE (asymmetric) and SMoE (asymmetric) on SST-2 and IMDB downstrcam tasks when pre-trained on VikiText-100 RPC only implemented during fine-tuning, not pre-training and runs for 2 iterations for all layers in each model.
SST-2
IMDB
Model
Train Ace
Valid Acc t
Train Ace f
Valid Acc t
SymSMoE (baseline)
63.12
69.20
RPC-SymSMoE
74.48
76.27
SMoE (baseline)
53.20
69.38
64.12 74.36
65.74
RPC-SMoE
58.87
70.63
73.96
Table 12: Validation and accuracy of RPC- implemented a pre-trained transformer language model (RPC-LM) during fine-tuning versus the baseline transformer language model (Baseline-L.M) We fine- tune both models on the 5-class Stanford Sentiment Treebank (SST-5) task with 2 iterations for all layers in the RPC model.
Model
Valid Acc f
Test Acc 1
Baseline-LM
46.51
49.23
RPC-LM
48.68
50.36
Treebank v2 (SST-2) and IMDB Sentiment Analysis (IMDB) tasks are binary classifications where the goal is to determine if sentences have positive or negative sentiments. We use baseline Sparse Mixture of Experts (SMoE) models, one with symmetric attention (SymSMoE) and one with asymmetric attention (SMoE), pre-trained on WikiText-103 without RPC, then fine-tuned with PPPC-SymSMME/RPC...
and without RPC-Attention for comparison. Their results can be found in Table 11. We observe that RPC-models outperform the baseline models significantly on these tasks.
On the 5-class sentiment classification task, Stanford Sentiment Treebank (SST-5), we use a pre- trained transformer language model from the NAACL 2019 tutorial on "Transfer Learning in Natural Language Processing" for fine-tuning. Their publicly available code can be found at https://github con/huggingfacc/nnace haaece///aal_ttaasfeerree learning tutorial. The objective of the task is to determine if the sentences have negative, somewhat negative, neutral, somewhat positive, or positive sentiments. We implement RPC-Attention during fine-tumin tueint only (RPC-LM) and compare the results with the baseline model (Baseline-L.M) on SST-5 in Table 12. As can seen from the table, our RPC- Attention is applicable to pre-trained language models and performs significantly better than the baseline. Hence, RPC-Attention highly effective in downstream natural language understanding tasks and versatile in its application.
26

---

Table 13: Flop per sample, run time per sample, memory and number of parameters of each RPC-SymViT variant compared to the SymViT baseline. RPC-SymViT (niter/layerl) applies PAP iterations only at the first. layer. RPC-SymViT (riter/all-layer applies n PAP iterations at all layers.
Sec/Sample (Training)
Sec/Sample (Test)
Memory (Training)
Memory (Test)
Model
Flop/Sample
Parameters
VymViT (baseline)
1.17M
0.079
0.010
1435MB
1181MB
5.2M
RPC-SymViT (4iter/layerl)
1.22M
0,082
0,010
1441MB
1181MB
5.2M
RPC-SymViT (Siter/layerl)
1.23M
0.084
0,010
1443MB
1181MB
5.2M
RPC-SymViT (6iter/layerl)
1.25M
0.085
0.011
1443MB
1181MB
5.2M
RPC-SymVit VTT (2iter/layer)
1.35M
0.092
0.017
1461MB
1181MB
5.2M
Table 14:
Top-1, top-3 accuracy (%) and AUPR of an implementation RPC-Atiention on asymmetric attention evaluated on ImagetNet-1K validation set, ImageNet-R, ImageNet-A and ImageNet-O. These results are compared to the standard asymmetric VIT.
IN-IK
IN-R
IN-A
IN-O
Model
Top-1 f
Top-5
Top-1
Top-1 1
AUPR
VIT
72.11
90.97
32.41
7.65
17.36
RPC-AsymViT (4iter/layerl)
72.34
91.12
32.23
7.75
17.54
RPC-AsymViT (Sym/Asym)
72.34
91.38
32.79
7.23
17.58
E.9 Computational Efficiency
possible limitation of introducing an iterative scheme into deep model is a significant increase in computational overhead. We aim to alleviate that concern by providing the number of flops per sample, run time per sample, memory and number of parameters of each RPC-Sym ViT variant and the Sym baseline during both training and test time in Table 13. We observe that RPC-Attentin is comparable to the baseline softmax attention across all metrics at test time while only slightly less efficient than the baseline in terms of the number of flops, run time per sample, and memory during training.
E.10 Results on Robust Asymmetric Attention
In this section, we report the results of extending the RPC-Sym model to the asymmetric attention. However, the PAP Algorithm. 1 is not designed for multiple data matrices, it is not as effective in the asymmetric case. We implement two variations of the algorithm in an asymmetric attention ViT-tiny with 12 layers. In the 4iter/layerl version, similar to the symmetric attention case, we run 4 iterations of the algorithm only in the 1st layer of the model by replacing Softmax K S μ "Y,, K Sk+1 μ Yx) with Softmax (Q K Sk+1 - μ Yy) For the second version, labeled Sym/Asym, we run the algorithm for 4 iterations in a symmetric attention mechanism in the 1st layer, followed by the standard asymmetric attention in layers 2 to 12. We compare these RPC-Asym ViT models to the asymmetric softmax attention VIT.
As we can see from Table 14, both variants only improve slightly over the benchmark on most of the corrupted datasets. Such result confirms our intuition that the ADMM algorithm does not extend easily to multiple corrupted matrices as it only solves an objective function involving single low-rank data matrix and its sparse corruption matrix.
F Further Discussion on Related Works
[15] provides a new perspective by emphasizing the asymmetry of the softmax kernel and recovers the self-attention mechanism from an asymmetric Kernel Singular Value Decomposition (KSVD) using the duality of the optimization problem. Separately, our kernel PCA perspective derives softmax attention and addresses the asymmetry of attention through a projection of the query vectors in feature space. While there are similarities between KSVD and kernel PCA, in the sense of finding low rank approximations and low dimensional representations of the data, the primal objective functions are different and we do not need to consider the dual form. Another related work views transformers from the perspective of Support Vector Machines [76]. Though, kernel PCA can be formulated in similar fashion a least squares SVM problem explained in [74], our work focuses the forward
27

---

pass of attention and show that it can recover a kernel PCA solution. In contrast, [74] examines the backward pass and optimization geometry of an attention layer towards a hard margin SVM solution that separates optimal tokens from non-optimal ones, under certain assumptions of the loss function, initial conditions, and certain SVM constraints. Furthermore, using our framework, we are able to predict the exact explicit form of the value matrix in self-attention, demonstrating that this matrix captures the eigenvectors of the Gram matrix of the key vectors in a feature space, To the best of our knowledge, our work is the first to show this insight.
Hamburger in [25] models the global context discovery as the low-rank recovery of the input tensor and solves it via matrix decomposition. Both Hamburger and our Attention with Robust Principal Components (RPC-Attention) try recover clean signal subspaces via computing low- rank approximation of given matrix. The key differences between our RPC-Attention and Hamburger are: (1) Our RPC-Attention finds a low-rank approximation of the key matrix while Hamburger finds a low-rank approximation of the input matrix, and (2) Our RPC-Attention models the corruption by sparse matrix while Hamburger does not enforce this condition. The entries of this sparse corruption can have an arbitrarily large magnitude and help model grossly corrupted observations in which only a portion the observation vector is contaminated by gross error. Numerous critical applications exist where the data being examined can be naturally represented as combination of a low-rank matrix and a sparse contribution, such as video surveillance, face recognition, and collaborative filtering [11].
[89] derives each component in a Graph Neural Network (GNN) from the unfolded iterations of robust descent algorithms applied to minimizing principled graph regularized energy function. In particular, propagation layers and nonlinear activations implement proximal gradient updates, and graph attention results from iterative reweighted least squares (IRLS). While this is an interesting approach, it has not been extended to explaining the architecture of transformers, including self- attention, yet. In contrast, our kernel principal component analysis (kernel PCA) allows us to derive self-attention in transformers, showing that the attention outputs are projections of the query vectors onto the principal components axes of the key matrix in feature space.
[2] and [6] implement each layer as an optimization and fixed- fixed-point solver, respectively. In particular, an OptNet layer in [2] solves a quadratic program, and a Deep Equilibrium layer in [6] computes the fixed point of a nonlinear transformation. Different from these layers, our RPC-Attention solves Principal Component Pursuit convex program. Also, both OptNet layer in [2] and Deep Equilibrium layer in [6] do not shed light on the derivation and formulation of self-attention, which our kernel PCA framework does.
G Broader Impacts
Our research improves both clean data processing and robust performance, especially in areas with significant social relevance. Specifically, demonstrate enhanced results in image segmentation, benefiting self-driving cars, and language modeling, enhancing AI chatbot assistants. We show notable improvements in resisting adversarial attacks, aiming to protect crucial AI systems from malicious actors. Additionally, we achieve competitive performance in language modeling with contaminated data, which is often encountered in real-world situations. Despite the potential for Al misuse, our research presents substantial advancements in fundamental architectures and theory, which we hope will inspire further socially beneficial developments.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction are clearly stated in the Contribution in the Introduction. These claims accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the Conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
29

---

Answer: [Yes]
Justification: All theoretical results in the paper are given together with the full set of assumptions and complete/correct proofs (See Section 2.1 in our manuscript).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the experiment details in the Experiment Details Section in the Appendix of our manuscript. We also provide the source code so that the results in the paper can be easily reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
30

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide the source code so that the results in the paper can be easily repro- duced. We verify our proposed methods using public benchmarks (See the Experimental Results Section in our manuscript)
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (tttps//niip.cc/public
Quides/CoodeSbbmissinooooiic for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (htips: hetppuulllgguiieeSoeeeuuuuieieee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the training and test details necessary understand the results in the Experiment Details Section in the Appendix of our manuscript.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report error bars suitably and correctly defined of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
31

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources for all experi- ments in our Experimental Results Section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:://euuupscccccuuuucettteeee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in Appendix G.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
32

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We cite the githubs we and the baselines compare with in manuscript.
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
33

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswitheode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
34

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "ThB7cSuZHe", "reviewer": "Reviewer_9vVz", "summary": "This work introduces a method for understanding attention mechanisms using kernel principal component analysis (kernel PCA). From this perspective, self-attention is viewed as projecting the query vectors onto the principal components of the key matrix. Building on this framework, this work develops a new robust attention mechanism, termed Attention with Robust Principal Components (RPC-Attention).\n\nRPC-Attention formulates the attention layer as an optimization problem of low-rank matrix recovery under sparse noise conditions, through Robust Principal Component Analysis. It utilizes the Alternating Direction Method of Multipliers (ADMM) algorithm to perform the forward pass of RPC-Attention. The proposed RPC-Attention layer converges within 4 to 6 iterations during the forward pass and shows robustness against data corruption and adversarial attacks.", "questions": "1. Gradient Flow: Iterative optimization layers often face issues with unstable gradient flow. How does RPC-Attention handle gradient propagation through the layer? Does it utilize implicit differentiation [3,4], inexact gradient methods [1], or Backpropagation Through Time (BPTT)? Specifically, are there any instabilities when differentiating through the softmax operator?\n\n2. Convergence: How does the RPC-Attention layer converge in the forward pass when using 4 or 6 iters? What if training the layer using 4 iterations and performing inference with 6 iterations? Would improved convergence at test time enhance the results?", "limitations": "The authors have discussed the limitations in the conclusion section.\n\nGiven the connection between attention mechanisms and Graph Neural Networks (GNNs), it would be interesting to see further exploration of RPC-Attention applied to graph data.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Soundness: The analysis of the attention mechanism presented in the manuscript is generally sound. The development of RPC-Attention is well-motivated by the formulation of standard self-attention.\n\n2. Writing: The paper is well-motivated and written with clarity.\n\n3. Evaluations: The evaluations encompass a wide range of common tasks in computer vision and language modeling. These evaluations support the claim of the proposed RPC-Attention's robustness against data corruption and adversarial attacks.", "weaknesses": "1. Related works. The manuscript omits several relevant works. References [1,2] are particularly pertinent given the connection between attention mechanisms and decomposition methods for recovering clean signal subspaces discussed in this work. Additionally, RPC-Attention, as an iterative optimization layer, also links to OptNet [3] and Deep Equilibrium Models (DEQs) [4].\n\n    [1] Is Attention Better Than Matrix Decomposition? \n\n    [2] Graph Neural Networks Inspired by Classical Iterative Algorithms\n\n    [3] OptNet: Differentiable Optimization as a Layer in Neural Networks\n\n   [4] Deep Equilibrium Models\n\n2. Speed. A major deficiency of RPC-Attention is that it adds cost to the inference speed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Related works. The manuscript omits several relevant works such as those below**\n\n**[1] Is Attention Better Than Matrix Decomposition?**\n\n**[2] Graph Neural Networks Inspired by Classical Iterative Algorithms**\n\n**[3] OptNet: Differentiable Optimization as a Layer in Neural Networks**\n\n**[4] Deep Equilibrium Models**\n\n**Answer:** Thanks for your suggestion. We agree with the reviewer that [1], [2], [3], and [4] should be discussed in the Related Work section. \n\nHamburger in [1] models the global context discovery as the low-rank recovery of the input tensor and solves it via matrix decomposition.  Both Hamburger and our Attention with Robust Principal Components (RPC-Attention) try to recover clean signal subspaces via computing a low-rank approximation of a given matrix. The key differences between our RPC-Attention and Hamburger are: (1) Our RPC-Attention finds a low-rank approximation of the key matrix $K$ while Hamburger finds a low-rank approximation of the input matrix $X$, and (2) Our RPC-Attention models the corruption by a sparse matrix while Hamburger does not enforce this condition. The entries of this sparse corruption can have an arbitrarily large magnitude and help model grossly corrupted observations in which only a portion of the observation vector is contaminated by gross error. Numerous critical applications exist where the data being examined can be naturally represented as a combination of a low-rank matrix and a sparse contribution, such as video surveillance, face recognition, and collaborative filtering [Candes, 2009]. \n\n[2] derives each component in a Graph Neural Network (GNN) from the unfolded iterations of robust descent algorithms applied to minimizing a principled graph regularized energy function. In particular, propagation layers and nonlinear activations implement proximal gradient updates, and graph attention results from iterative reweighted least squares (IRLS). While this is an interesting approach, it has not been extended to explaining the architecture of transformers, including self-attention, yet. Even though graph attention in GNNs and self-attention in transformers share many similarities, they are not the same. For example, query, key, and value matrices are introduced in the transformer's self-attention but not GNN's graph attention. In contrast, our kernel principal component analysis (kernel PCA) allows us to derive self-attention in transformers, showing that the attention outputs are projections of the query vectors onto the principal components axes of the key matrix $K$ in a feature space.\n\n[3] and [4] implement each layer as an optimization and fixed-point solver, respectively. In particular, an OptNet layer in [3] solves a quadratic program, and a Deep Equilibrium layer in [4] computes the fixed point of a nonlinear transformation. Different from these layers, our RPC attention solves a Principal Component Pursuit - a convex program. Also, both OptNet layer in [3] and Deep Equilibrium layer in [4] do not shed light on the derivation and formulation of self-attention, which our kernel PCA framework does. \n\nAlso, we would like to point out that our RPC-attention aims at improving the robustness of self-attention mechanism to data contamination (See Tables 1, 2, and 3 in our main text and Tables 4, 5, 6, 7, 8, 9, 10, and 12 in our Appendix). None of the methods proposed in [1], [2], [3], and [4] addresses the model's robustness, neither theoretically nor empirically.\n\nFollowing the reviewer's suggestion, we will include the discussion above in the Related Work section of our revision. \n\n**Q2. Speed. RPC-Attention adds cost to the inference speed.**\n\n**Answer:** Please see our answer to Q1 in the Global Rebuttal.\n\n**Q3. Gradient Flow: How does RPC-Attention handle gradient propagation through the layer? Does it utilize implicit differentiation [3,4], inexact gradient methods [1], or Backpropagation Through Time (BPTT)? Are there any instabilities when differentiating through the softmax operator?**\n\n**Answer:** As shown in Algorithm 1, the Principal Attention Pursuit (PAP) in our RPC-Attention is quite simple. Most operators in PAP are linear except for the softmax operator and the shrinkage operator $S_{\\lambda/\\mu}$. Like the softmax operator in self-attention, the softmax operator in RPC-Attention is easy to differentiate through. Similarly, like the ReLU operator in feedforward layers, the shrinkage operator $S_{\\lambda/\\mu}$ is also easy to differentiate through. We do not encounter any instabilities with RPC-Attention, and the standard backpropagation algorithm is used to handle gradient propagation in RPC-Attention. Fig. 1 (Right) in the attached PDF in Global Rebuttal shows that training/validation loss curve of the transformer with RPC attention is quite stable.\n\n**Q4. Convergence: RPC-Attention layer's convergence in the forward pass when using 4 or 6 iters? What if training the layer using 4 iters and performing inference with 6 iters? Would improved convergence at test time enhance the results?**\n\n**Answer:** In Figure 1 (Left) in the attached PDF, we plot the objective loss given in Eqn. (14) in the main text of our manuscript versus the number of PAP iterations in RPC-Attention (see Algorithm 1 in our manuscript) for models that use 4 and 6 PAP iterations. In Table 3 in the attached PDF, we report the results of using 6 iterations at inference on a model trained with 4 iterations.  \n\n**Q5. Further exploration of RPC-Attention applied to graph data.**\n\n**Answer:** Thanks for your suggestion. We agree with the reviewer that given the connection between the attention mechanism and GNNs, as pointed out in [Joshi, 2020], our RPC-Attention can be extended to apply to GNNs. For example, RPC-Attention can be incorporated into the graph attention of Graph Attention Networks [Veličković, 2018]. \n\n**References**\n\nCandès, E. J., et al. Robust principal component analysis? 2011.\n\nVeličković, P, et al. Graph Attention Networks. 2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K6AbVmlS0A", "reviewer": "Reviewer_8tie", "summary": "The paper derives self-attention from kernel principal component analysis (kernel PCA), showing that the attention outputs are projections of  query vectors onto the principal component axes of the key matrix in a feature space. Using this kernel PCA framework, the authors propose Attention with Robust Principal Components (RPC-Attention), a robust attention that is resilient to data contamination, and demonstrate its effectiveness on ImageNet-1K object classification, WikiText-103 language14modeling, and ADE20K image segmentation task.", "questions": "For language tasks, ppl may not accurately reflect the final performance. Have the authors conducted experiments on other downstream natural language understanding tasks to evaluate the effectiveness? Will RPA be applicable to pre-trained language models?", "limitations": "Please refer to questions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper offers a novel perspective on attention mechanisms, proposing that attention outputs are projections of query vectors onto the principal component axes of the key matrix in a feature space. \n2. The authors provide a detailed mathematical proof to support this claim.\n3. The paper's experimental evaluation is comprehensive, demonstrating the applicability of the proposed RPC-attention mechanism across both vision and language tasks. \n4. The writing is clear and well-organized, making the paper a pleasure to read.", "weaknesses": "1. The results in Table 1 show that the best performance on different tasks is achieved with varying settings. For example,  RPC achieves the best performance with 6 iterations/layer 1 on IN-1K, while RPC achieves the best performance with 2 iterations/all layers on IN-A. Moreover, for IN-1K top-5, 2 iterations/all layers even performs worse than the baseline attention. This suggests that more RPC attention is not always better for different tasks, and sometimes even performs worse than the baseline. Can authors provide an explanation for this phenomenon?\n\n2. As shown in Appendix Table 11, the computational cost of RPC attention increases substantially with the number of iterations/layers. This raises concerns about its practical applicability when extended to multi-layer settings. Specifically, it is unclear whether the performance gains will be sufficient to justify the increased computational cost. This directly affects the method's usability in applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The results in Table 1 show that the best performance on different tasks is achieved with varying settings. For example, RPC achieves the best performance with 6 iterations/layer 1 on IN-1K, while RPC achieves the best performance with 2 iterations/all layers on IN-A. Moreover, for IN-1K top-5, 2 iterations/all layers even performs worse than the baseline attention. This suggests that more RPC attention is not always better for different tasks, and sometimes even performs worse than the baseline. Can authors provide an explanation for this phenomenon?**\n\n**Answer:** Our RPC-Attention needs more iterations to converge. While applying RPC to all layers leverages more advantages of RPC, 2 iterations/layer is too little for convergence. Also, for 4-5-6 iter/layer1, more iterations tend to give better results since the algorithm converges better. However, sometimes 6 iters perform worse than 5 iters. This might be due to overshooting. \n\n**Q2. As shown in Appendix Table 11, the computational cost of RPC attention increases substantially with the number of iterations/layers. This raises concerns about its practical applicability when extended to multi-layer settings. Specifically, it is unclear whether the performance gains will be sufficient to justify the increased computational cost. This directly affects the method's usability in applications.** \n\n**Answer:** Since we only apply RPC-Attention to the first layer of the transformer model, it adds minimal cost to the inference speed. In particular, in Table 11 in our Appendix, we show the inference runtime cost (in second/sample) of our RPC-Attention. As can be seen in that table, during inference (\"Test\" column in Table 11), our RPC-Attentions with 4iter/layer1 (4 Principal Attention Pursuit (PAP) iterations at layer 1) and 5iter/layer1 (5 PAP iterations at layer 1) have the same inference runtime as the baseline SymViT, 0.01 second/sample. Our RPC-Attention with 6 iter/layer1 (6 PAP iterations at layer 1) only requires slighly more inference time than the baseline, 0.011 second/sample vs. 0.01 second/sample, which is not a \"major deficiency\" as the reviewer suggested. Due to the effectiveness of our RPC-Attention, applying it at the first layer of a transformer model is already good enough to improve the model's robustness significantly (see Tables 1, 2, and 3 in our main text and Tables 4, 5, 6, 7, 8, 9, 10, and 12 in our Appendix). Note that results reported in Table 11 also show that our RPC-Attention is as memory, parameter, FLOP, and training efficient as the baseline SymViT.\n\n**Q3. For language tasks, ppl may not accurately reflect the final performance. Have the authors conducted experiments on other downstream natural language understanding tasks to evaluate the effectiveness? Will RPA be applicable to pre-trained language models?** \n\n**Answer:** Thanks for your suggestion. We have conducted additional experiments on downstream natural language understanding tasks and their results can be found in Table 4 of the attached PDF. RPC-models outperform the baseline models significantly on these tasks. Particularly, we use 2 baseline Sparse Mixture of Experts (SMoE) models, one with symmetric attention (SymSMoE) and one with asymmetric attention (SMoE), pretrained on WikiText-103 without RPC. We finetune these models on the Stanford Sentiment Treebank v2 (SST2) task and IMDB Sentiment Analysis (IMDB) task, without RPC as our baseline and with RPC for comparison. Due to the tight schedule, we did not report SymSMoE's results on IMDB and will do so during the discussion period. We further aim to test our method on more downstream tasks and will continue updating the results in the mean time. From the current results presented, we observe strong advantages of RPC on downstream tasks with large increases in training and validation accuracies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m7DxAFG1BX", "reviewer": "Reviewer_hhSg", "summary": "The paper proposes a new perspective for understanding the underlying operation learned by scaled dot-product self-attention from a kernel PCA perspective. The paper recovers the self-attention formulation starting from PCA over feature projections of data points. In particular, the authors show that different parameterizations for the value matrix, while mathematically equivalent, can lead to different optimization problems and therefore different outcomes. Based on the new perspective provided in the paper, the authors propose RPC-Attention, a method to address one of the known vulnerabilities of PCA with corrupted data. The authors provide several experiments to demonstrate the effectiveness of RPC-Attention.", "questions": "1.  Given the new perspective provided in the paper, do the authors have any insights into how to reduce the complexity of self-attention while retaining the same quality?", "limitations": "Yes", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper provides a very interesting and important perspective for understanding self-attention from the lens of Kernel PCA. Given the ubiquity and effectiveness of the transformer architecture, such insights are extremely valuable.\n2.  The paper is very well written, and the derivations are clear and well-explained.  \n3.  In addition to clearly explaining the new perspective for understanding self-attention, the authors went ahead and addressed one of the weaknesses self-attention might suffer from given this new understanding. This is very valuable and gives a good example of how new perspectives can pave the way to key improvements in the underlying mechanisms.\n3. The experimental results reflect the effectiveness of the proposed RPC-Attention method in improving the robustness of the models against adversarial attaches and corrupted data.", "weaknesses": "1. [minor] The experiments use relatively small-scale models and short training runs (e.g. 50 epochs on Imagenet). While not a core issue given the objective of the paper, it would be nice to see results in more standardized settings (e.g. ViT-Base with 300 epochs)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. [minor] The experiments use relatively small-scale models and short training runs (e.g. 50 epochs on Imagenet). While not a core issue given the objective of the paper, it would be nice to see results in more standardized settings (e.g. ViT-Base with 300 epochs)**\n\n**Answer:** Thanks for your suggestion. All of our models are trained on 300 epochs, we provide the plot of 50 epochs in the main text for emphasis on the faster initial convergence during training. A plot of the full training curve with 300 epochs is provided in Appendix E.1, Fig. 4. Following the reviewer's suggestion, we have conducted an additional experiment on ViT-Base for the ImageNet object classification task with 300 epochs and reported our results in Table 1 and 2 in the attached PDF. Our RPC-ViT-Base outperforms the baseline in terms of clean accuracy and is also more robust to data perturbation and adversarial attacks. \n\n**Q2. Given the new perspective provided in the paper, do the authors have any insights into how to reduce the complexity of self-attention while retaining the same quality?** \n\n**Answer:** Our kernel principal component analysis (kernel PCA) framework allows the development of efficient attentions. In the following, we will show how to derive two popular efficient attentions: the linear attention in [Katharopoulos, 2020] and the sparse attention in [Child, 2019]. Combining linear and sparse attentions helps reduce the complexity of self-attention while maintaining its quality [Nguyen, 2021; Zhu, 2021; Chen, 2021].\n\n\n*Deriving the linear attention:* The self-attention output in Eqn. (8) in our main text can be re-written as follows:\n\n$$\n h_{i} = \\sum_{j=1}^{N}\\frac{\\phi(q_i)^{\\top}\\phi(k_j)}{g(q_i)}v_j = \\frac{\\sum_{j=1}^{N}\\phi(q_i)^{\\top}\\phi(k_j)v_j}{\\sum_{j'=1}^{N}\\phi(q_i)^{\\top}\\phi(k_{j'})}.\n$$\n\nHere, we again choose $g(x) := \\sum_{j=1}^{N}\\phi(x)^{\\top}\\phi(k_j)$. Following the derivation in [Katharopoulos, 2020], we use the associative property of matrix multiplication to obtain\n\n$$\n h_{i} = \\frac{\\phi(q_i)^{\\top}\\sum_{j=1}^{N}\\phi(k_j)v_{j}^{\\top}}{\\phi(q_i)^{\\top}\\sum_{j'=1}^{N}\\phi(k_{j'})}.\n$$\n\nWe can then choose $\\phi(x) = \\text{elu}(x) + 1$ to achieve the linear attention in [Katharopoulos, 2020], which is one of the most popular efficient attentions.\n\n*Deriving the sparse attention*: For each query $q_i$, we consider a subset $\\mathcal{M_{i}}=\\\\{k_{\\ell_{i}(1)}, k_{\\ell_{i}(2)},\\dots, k_{\\ell_{i}(L)}\\\\}$ of the  dataset $\\mathcal{M}=\\\\{k_1, k_2, \\dots, k_N\\\\}$, where $\\mathcal{L_{i}} = \\\\{\\ell_{i}(1), \\ell_{i}(2), \\dots, \\ell_{i}(L)\\\\} \\subset \\\\{1, 2, \\dots, N\\\\}$. Following Eqn. (8) and (9) in our main text, the projection $h_i$ of the query $q_i$ onto $D_v$ principal components of   $\\mathcal{M_i}$ is given by:\n\n$$\nh_{i} = \\sum_{j=1}^{N}1_{\\mathcal{L_{i}}}(j)\\frac{k(q_i, k_j)}{g(q_i)}v_j,\n$$\n\nwhere $1_{\\mathcal{L_{i}}}(j) = 1 \\text{ if } j\\in \\mathcal{L_{i}}$. Note that the subsets $\\mathcal{M_i}$ are different for different $q_i$, $i=1,\\dots,N$. Again, as in Section 2.1 in our manuscript, selecting $g(x) := \\sum_{j=1}^{N} k(x,k_j)$ and $k(x, y) = \\exp(x^{\\top}y/\\sqrt{D})$, we obtain a formula of the sparse attention in [Child, 2019] where the binary matrix $(1_{\\mathcal{L_{i}}}(j))_{i,j=1}^{N}$ becomes the sparse masking matrix.\n\n**References**\n\nKatharopoulos, A, et al. Transformers are rnns: Fast autoregressive transformers with linear attention. 2020.\n\nChild, R., et al. Generating long sequences with sparse transformers. 2019.\n\nNguyen, T., et al. Fmmformer: Efficient and flexible transformer via decomposed near-field and far-field attention. 2021.\n\nZhu, C., et al. Long-short transformer: Efficient transformers for language and vision. 2021.\n\nChen, B., et al. Scatterbrain: Unifying sparse and low-rank attention. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LdnDYRxAAe", "reviewer": "Reviewer_85Cd", "summary": "This paper studies the self-attention mechanism. The authors recover self-attention from kernel principle component analysis (kernel PCA). The authors empirically verify that the projection error is being minimized during training, suggesting that the transformer model is actually training to perform PCA during training. Also, the authors empirically confirm that the value matrix capture the eigenvectors of the Gram matrix. Lastly, using the analysis framework, the authors propose a robust attention based on principal component pursuit that is robust to input corruption and attacks.", "questions": "see weakness section.", "limitations": "The authors discussed a limitation about the potential efficiency issue of the proposed algorithm.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Viewing self-attention from kernel PCA perspective is very interesting. \n2. The kernel PCA analysis of self-attention is supported with empirical evidence. \n3. The analysis not only provides the theoretical understanding of self-attention, but also helps guiding the design of a new class of robust self-attention that is robust to input corruption and attacks.", "weaknesses": "1. The RPC-Attention uses an iterative algorithm PAP to solve the convex formulation of principal component pursuit. How is back-propagation via PAP algorithm calculated? Does it lead to any instability? \n2. The appendix E8 shows the runtime and memory of RPC-Attention is comparable to baseline despite using an iterative algorithm. I am curious about why it is not much slower. Is there any analysis? \n3. I noticed that the baseline SymViT that uses symmetric attention has significantly worse accuracy compared to vanilla ViT on clean ImageNet (for tiny model, SymViT accuracy is 70.44, while vanilla ViT is around 75). I am wondering what would be the reason. If this is caused by the use of symmetric attention, is there a way to adopt RPC-Attention for vanilla ViT? \n4. Given that RPC-Attention is more robust and as efficient as baseline, why the majority of experiments only apply it on the first few layers or the first layer, but not apply RPC-Attention on all layers (for example, like 6iter/all-layer).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The RPC-Attention uses an iterative algorithm PAP to solve the convex formulation of principal component pursuit. How is back-propagation via PAP algorithm calculated? Does it lead to any instability?** \n\n**Answer:** As shown in Algorithm 1 in the main text, the Principal Attention Pursuit (PAP) in our RPC-Attention is quite simple. Most operators in PAP are linear except for the softmax operator and the shrinkage operator $S_{\\lambda/\\mu}$. Like the softmax operator in self-attention, the softmax operator in RPC-Attention is easy to differentiate through. Similarly, like the ReLU operator in feedforward layers, the shrinkage operator $S_{\\lambda/\\mu}$ is also easy to differentiate through. We do not encounter any instabilities with RPC-Attention, and the standard backpropagation algorithm is used to handle gradient propagation in RPC-Attention. As can be seen in Figure 1 (Right) in the attached PDF, the training and validation loss curve of the transformer with RPC attention is quite stable.\n\n**Q2. The appendix E8 shows the runtime and memory of RPC-Attention is comparable to baseline despite using an iterative algorithm. I am curious about why it is not much slower. Is there any analysis?** \n\n**Answer:** Since we only apply RPC-Attention to the first layer of the transformer model, it adds minimal cost to the runtime and memory. In particular, in Table 11 in our Appendix, we show the inference runtime cost (in second/sample) of our RPC-Attention. As can be seen in that table, during inference (\"Test\" column in Table 11), our RPC-Attentions with 4iter/layer1 (4 Principal Attention Pursuit (PAP) iterations at layer 1) and 5iter/layer1 (5 PAP iterations at layer 1) have the same inference runtime and use the memory during inference as the baseline SymViT, 0.01 second/sample and 1181MB, respectively. Our RPC-Attention with 6 iter/layer1 (6 PAP iterations at layer 1) only requires slighly more inference time than the baseline, 0.011 second/sample vs. 0.01 second/sample, and uses the same memory during inference as the baseline, 1181MB. Due to the effectiveness of our RPC-Attention, applying it at the first layer of a transformer model is already good enough to improve the model's robustness significantly (see Tables 1, 2, and 3 in our main text and Tables 4, 5, 6, 7, 8, 9, and 10 in our Appendix). Note that results reported in Table 11 also show that our RPC-Attention is as parameter, FLOP, and training efficient as the baseline SymViT.\n\n**Q3. I noticed that the baseline SymViT that uses symmetric attention has significantly worse accuracy compared to vanilla ViT on clean ImageNet (for the tiny model, SymViT accuracy is 70.44, while vanilla ViT is around 75). I am wondering what would be the reason. If this is caused by the use of symmetric attention, is there a way to adopt RPC-Attention for vanilla ViT?**\n\n**Answer:** Thanks for your comments. For the ImageNet object classification task, as the reviewer pointed out and as reported in our paper (see Tables 1 and 12), SymViT has worse performance than the asymmetric vanilla ViT. However, for other tasks, such as neural machine translation (NMT) [Vaswani, 2017] and sequence prediction (SP) [Dai et al., 2019], symmetric attention has comparable or even better performance than asymmetric attention, as reported in [Tsai, 2019]. These evidences show the promise of applying our RPC-Attention to practical models for enhancing their accuracy and robustness. \n\nIn Appendix E.9, we discuss the extension of RPC-Attention to asymmetric attention. We report the results of the RPC-Asymmetric ViT (RPC-AsymViT) vs. baseline asymmetric ViT in Table 12. Our RPC-AsymVit improves over the baseline on most of the corrupted datasets.  However, as explained in Appendix E.9, as the PAP in Algorithm 1 in our manuscript is not designed for multiple data matrices, it is not as effective in the asymmetric case as in the symmetric case.\n\nAdditionally, in Table 4 of the attached PDF, we provide positive results of finetuning an asymmetric Sparse Mixture of Experts (SMoE) language model using RPC on the downstream natural language understanding tasks, Stanford Sentiment Treebank v2 (SST2) and IMDB Sentiment Analysis (IMDB). The significant improvements over the baseline SMoE validate the effectiveness of both of our methods, symmetric RPC-Attention and asymmetric RPC-Attention.\n\n**Q4. Given that RPC-Attention is more robust and as efficient as baseline, why the majority of experiments only apply it on the first few layers or the first layer, but not apply RPC-Attention on all layers (for example, like 6iter/all-layer).** \n\n**Answer:** As mentioned in our answer to your Q2 above, due to the effectiveness of our RPC-Attention, applying it at the first layer of a transformer model is already good enough to improve the model's robustness significantly (see Tables 1, 2, and 3 in our main text and Tables 4, 5, 6, 7, 8, 9, and 10 in our Appendix). This strategy also adds minimal cost to the runtime and memory.\n\nWhile applying RPC at all layers, as in RPC-SymViT (2iter/all-layer) in Table 1, 2, 4, and 6 leverages more advantages of RPC, it is not really necessary in most settings due to its small improvement over applying RPC at the first layer and its higher computational overhead (see Table 11).\n\n\n**References**\n\nVaswani, A., et al. Attention is all you need. 2017.\n\nDai, Z., et al. Transformer-xl: Attentive language models beyond a fixed-length context. 2019. \n\nTsai, Y. H. H., et al. Transformer dissection: a unified understanding of transformer's attention via the lens of kernel. 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "svjsQYoqv9", "reviewer": "Reviewer_rAS8", "summary": "This paper introduces a new method, Kernel PCA Attention (RPC-Attention), designed to enhance the self-attention mechanisms in transformers widely used in sequence modeling for both natural language processing and computer vision. The method applies principal component analysis to derive self-attention kernels, which helps in projecting key vectors into a feature space that prioritizes essential components, thereby optimizing task performance. The effectiveness of RPC-Attention is empirically validated through improved results in object classification, language modeling on WikiText-103, and image segmentation on the ADE20K dataset.", "questions": "1. Did the authors try larger models for RPC-Attention?", "limitations": "The authors have discussed their limitations; Scalability might be a vital limitation to the models proposed in this paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper reveals the intrinsic connection between the self-attention mechanism and PCA/kernel PCA, providing a thorough mathematical explanation。\n\n2. The derivation part is convincing and interesting.", "weaknesses": "1. My main concern is about the practicality of the novel RPC-Attention mechanism. In your experiments, RPC-SymViT can only scale to a small size, and it does not show a significant performance advantage compared to ViT-Small.\n\n2. Additionally, RPC-SymViT does not have an advantage in computational speed; for example, in Table 11, the inference time for RPC-SymViT (2iter/layer) is 70% higher than that of ViT. As it stands, the authors have not convinced me to abandon the traditional ViT in favor of RPC-SymViT.\n\n3. I am also concerned about the scalability of this model, which can outperform ViT by ~1% on ImageNet at the tiny size, but the performance becomes very close at the small size. What happens if the model is scaled up even more? Do the assumptions in the paper still hold when the feature dimension is higher?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. My main concern is about the practicality of the novel RPC-Attention mechanism. In your experiments, RPC-SymViT can only scale to a small size, and it does not show a significant performance advantage compared to ViT-Small.**\n\n**I am also concerned about the scalability of this model, which can outperform ViT by ~1% on ImageNet at the tiny size, but the performance becomes very close at the small size. What happens if the model is scaled up even more? Do the assumptions in the paper still hold when the feature dimension is higher?** \n\n**Did the authors try larger models for RPC-Attention?** \n\n**Answer:** We have conducted an additional experiment on RPC-SymViT-Base for the ImageNet object classification task to address the reviewer's concern about the scalability and practicality of our RPC-Attention and reported our results in Table 1 and 2 in the attached PDF. Our RPC-SymViT-Base outperforms the baseline in terms of clean accuracy and is also more robust to data perturbation and adversarial attacks than the baseline. Specifically, we improve on ImageNet-O by over 2 AUPR and on PGD and FGSM by more than 1% in top-1 accuracy. These results further demonstrate the effectiveness of RPC-Attention even in larger models. \n\nOur assumptions in Remark 2, do not hold when the feature dimension is higher than the sequence length as the number of principal components cannot exceed the rank of the covariance matrix. But in practical tasks, the feature dimension is very often smaller than the sequence length since, in those tasks, transformers are used to capture long-range dependency in the input sequence. \n\n**Q2. Additionally, RPC-SymViT does not have an advantage in computational speed; for example, in Table 11, the inference time for RPC-SymViT (2iter/layer) is 70% higher than that of ViT. As it stands, the authors have not convinced me to abandon the traditional ViT in favor of RPC-SymViT.**\n\n**Answer:** In Table 11, in our first three settings, RPC-SymViT (4iter/layer1, 5iter/layer1, 6iter/layer1), we only apply RPC-Attention to the first layer of the transformer model. This adds minimal cost to the inference speed. In particular, in Table 11, we show the inference runtime cost (in second/sample) of our RPC-Attention. As can be seen in that table, during inference (\"Test\" column in Table 11), our RPC-Attention with 4iter/layer1 (4 Principal Attention Pursuit (PAP) iterations at layer 1) and 5iter/layer1 (5 PAP iterations at layer 1) have the same inference runtime as the baseline SymViT, 0.01 second/sample. Our RPC-Attention with 6 iter/layer1 (6 PAP iterations at layer 1) only requires slighly more inference time than the baseline, 0.011 second/sample vs. 0.01 second/sample. Due to the effectiveness of our RPC-Attention, applying it at the first layer of a transformer model is already good enough to improve the model's robustness significantly (see Tables 1, 2, and 3 in our main text and Tables 4, 5, 6, 7, 8, 9, 10, and 12 in our Appendix). Note that results reported in Table 11 also show that our RPC-Attention is as memory, parameter, FLOP, and training efficient as the baseline SymViT.\n\nWhile applying RPC at all layers, as in RPC-SymViT (2iter/layer), leverages more advantages of RPC, it is not really necessary in most settings due to its small improvement over applying RPC at the first layer and its higher computational overhead, as the reviewer mentioned.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Global Rebuttal

Dear AC and reviewers,

Thanks for your thoughtful reviews and valuable comments, which have helped us improve the paper significantly. We are encouraged by the endorsements that: 1) The perspective for understanding self-attention from the lens of Kernel PCA provided by the paper is very interesting (Reviewer hhSG, 85Cd), important (Reviewer hhSG), novel (Reviewer 8tie), and intrinsic (Reviewer rAS8); 2) The kernel PCA analysis of self-attention is sound (Reviewer 9vVz) with detailed mathematical derivation and proof to support the claim (Reviewer 8tie, hhSg, rAS8) and is supported with empirical evidence (Reviewer 85Cd); 3) The experimental evaluation is comprehensive, demonstrating the applicability of the proposed RPC-attention mechanism across both vision and language tasks (Reviewer 8tie, 9vVz) and supporting the claim of the proposed RPC-Attention's robustness against data corruption and adversarial attacks (Reviewer 9vVz, hhSg). We have included the additional experimental results requested by the reviewers in the 1-page attached PDF.

One of the main concerns from the reviewers is that our RPC-Attention might add cost to the inference speed and require more memory. Another concern is that we need to verify the performance of our RPC-Attention on a larger model, like ViT-Base with 300 epochs. Additionally, the reviewers ask about the stability of back-propagation through PAP iterations. We address these questions here.

**Q1: Efficiency of RPC-Attention**

**Answer:** Since we only apply RPC-Attention to the first layer of the transformer model, it adds minimal cost to the inference speed. In particular, in Table 11 in our Appendix, we show the inference runtime cost (in second/sample) of our RPC-Attention. As can be seen in that table, during inference ("Test" column in Table 11), our RPC-Attentions with 4iter/layer1 (4 Principal Attention Pursuit (PAP) iterations at layer 1) and 5iter/layer1 (5 PAP iterations at layer 1) have the same inference runtime as the baseline SymViT, 0.01 second/sample. Our RPC-Attention with 6 iter/layer1 (6 PAP iterations at layer 1) only requires slighly more inference time than the baseline, 0.011 second/sample vs. 0.01 second/sample, which is not a "major deficiency". Due to the effectiveness of our RPC-Attention, applying it at the first layer of a transformer model is already good enough to improve the model's robustness significantly (see Tables 1, 2, and 3 in our main text and Tables 4, 5, 6, 7, 8, 9, 10, and 12 in our Appendix). Note that results reported in Table 11 also show that our RPC-Attention is as memory, parameter, FLOP, and training efficient as the baseline SymViT.

While applying RPC at all layers, as in RPC-SymViT (2iter/layer), leverages more advantages of RPC, it is not really necessary in most settings due to its small improvement over applying RPC at the first layer and its higher computational overhead, as the reviewer mentioned.

**Q2: Experiments on a larger model, e.g., ViT-Base with 300 epochs**

**Answer:** We have conducted an additional experiment on ViT-Base for the ImageNet object classification task with 300 epochs and reported our results in Tables 1 and 2 in the attached PDF. Our RPC-SymViT-Base outperforms the baseline in terms of clean accuracy and is also more robust to data perturbation (Table 1) and adversarial attacks (Table 2) than the baseline.

**Q3 (Gradient Flow): Stability of back-propagation through PAP iterations**

**Answer:** As shown in Algorithm 1 in the main text, the Principal Attention Pursuit (PAP) in our RPC-Attention is quite simple. Most operators in PAP are linear except for the softmax operator and the shrinkage operator $S_{\lambda/\mu}$. Like the softmax operator in self-attention, the softmax operator in RPC-Attention is easy to differentiate through. Similarly, like the ReLU operator in feedforward layers, the shrinkage operator $S_{\lambda/\mu}$ is also easy to differentiate through. We do not encounter any instabilities with RPC-Attention, and the standard backpropagation algorithm is used to handle gradient propagation in RPC-Attention. As can be seen in Figure 1 (Right) in the attached PDF, the training loss and validation loss curve of the transformer with RPC attention is quite stable.

-----

We hope that our rebuttal has cleared your concerns about our work. We are glad to answer any further questions you have on our submission, and we would appreciate it if we can get your further feedback at your earliest convenience.

Meta-review: This work aims to understand attention mechanisms from kernel PCA. The presentation is clear, the problem is well-motivated, and the perspective is novel and insightful. More interestingly, a robust attention (RPC-attention) is further proposed. Major concerns include the performance stability and the computational cost. Overall the reviewers give positive scores after rebuttal discussions. AC agrees this is an interesting perspective on the understanding and design of the attention mechanisms and thus recommends acceptance.
Decision: Accept (poster)