Title: Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Self-attention

primal-dual representations

SVD

kernel method

asymmetry

transformer
Abstract: Recently, a new line of works has emerged to understand and improve self-attention in Transformers by treating it as a kernel machine. However, existing works apply the methods for symmetric kernels to the asymmetric self-attention, resulting in a nontrivial gap between the analytical understanding and numerical implementation. In this paper, we provide a new perspective to represent and optimize self-attention through asymmetric Kernel Singular Value Decomposition (KSVD), which is also motivated by the low-rank property of self-attention normally observed in deep layers. Through asymmetric KSVD, i) a primal-dual representation of self-attention is formulated, where the optimization objective is cast to maximize the projection variances in the attention outputs; ii) a novel attention mechanism, i.e., Primal-Attention, is proposed via the primal representation of KSVD, avoiding explicit computation of the kernel matrix in the dual; iii) with KKT conditions, we prove that the stationary solution to the KSVD optimization in Primal-Attention yields a zero-value objective. In this manner, KSVD optimization can be implemented by simply minimizing a regularization loss, so that low-rank property is promoted without extra decomposition. Numerical experiments show state-of-the-art performance of our Primal-Attention with improved efficiency. Moreover, we demonstrate that the deployed KSVD optimization regularizes Primal-Attention with a sharper singular value decay than that of the canonical self-attention, further verifying the great potential of our method. To the best of our knowledge, this is the first work that provides a primal-dual representation for the asymmetric kernel in self-attention and successfully applies it to modelling and optimization.

Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation

Abstract

Recently, a new line of works has emerged to understand and improve self-attention in Transformers by treating it as a kernel machine. However, existing works apply the methods for symmetric kernels to the asymmetric self-attention, resulting in a nontrivial gap between the analytical understanding and numerical implementation. In this paper, we provide a new perspective to represent and optimize self-attention through asymmetric Kernel Singular Value Decomposition (KSVD), which is also motivated by the low-rank property of self-attention normally observed in deep layers. Through asymmetric KSVD, i) a primal-dual representation of selfattention is formulated, where the optimization objective is cast to maximize the projection variances in the attention outputs; ii) a novel attention mechanism, i.e., Primal-Attention, is proposed via the primal representation of KSVD, avoiding explicit computation of the kernel matrix in the dual; iii) with KKT conditions, we prove that the stationary solution to the KSVD optimization in Primal-Attention yields a zero-value objective. In this manner, KSVD optimization can be implemented by simply minimizing a regularization loss, so that low-rank property is promoted without extra decomposition. Numerical experiments show state-of-theart performance of our Primal-Attention with improved efficiency. Moreover, we demonstrate that the deployed KSVD optimization regularizes Primal-Attention with a sharper singular value decay than that of the canonical self-attention, further verifying the great potential of our method. To the best of our knowledge, this is the first work that provides a primal-dual representation for the asymmetric kernel in self-attention and successfully applies it to modelling and optimization 2 .

1 Introduction

Transformers [1] have become ubiquitous nowadays with state-of-the-art results in various tasks, such as natural language processing [2, 3, 4], computer vision [5, 6, 7, 8], reinforcement learning [9, 10, 11], etc. In the remarkable success of Transformers, the self-attention blocks play a key role, where the complicated dependencies between the individuals in data sequences can be depicted by using the established queries, keys, and values. Despite the prevailing advantages, theoretical understandings towards Transformers seem yet lagged behind its unprecedented empirical performance.

cumula-

valu

1.0

Overviewoflayers[0,1,2,3]

1.0

1.0

11

10

DeiT-Small/16

-1.0

0.8

0.8

0.8

Layer index

８７６５４３２１

9

-0.8

tive singular

Normalized

0.6

0.6

0.6

0.6

DeiT-S/16

0.4

0.4

0.4

DeiT-S/16, L[11]

0.4

0.2

DeiT-S/16

0.2

DeiT-S/16

0.2

Primal.+

Primal.+

Primal.+

Primal.+, L[11]

0.2

0

0

0

0

0

50

100

150 196

0

50

100

150

196

0

50

100

150 196

0

50

100

150196

Singularvalueindex

Singularvalueindex

Singularvalueindex

Singularvalueindex

(a)

(b)

(c)

(d)

Recently, the kernel-based perspective has been proposed where the dot-product attention operation is shown as a kernel matrix [12]. This finding is quite encouraging by bridging kernels [13] with Transformers, as kernel methods have long been well studied with good interpretation ability. Following this spirit, different works have been proposed subsequently to improve self-attention, e.g., [14, 15, 16, 17, 18, 19]. However, in these works, the applied kernel techniques rely on Mercer kernels [20] requesting symmetry, which is inconsistent with the intrinsically asymmetric setups of self-attention. In [21], it analytically characterizes the attention by asymmetric kernels based on Reproducing Kernel Banach Spaces (RKBS) [22]. Nonetheless, neither the asymmetry property nor the related optimization is utilized for improvements. In [19], self-attention is derived with a primal-dual representation from the support vector regression, which still adopts the technique for Mercer kernels. Moreover, in the cast supervised task, the assumed ground-truth outputs of self-attention are practically non-existent, making it difficult to be applied in the optimization.

In this work, we provide a novel perspective to interpret self-attention with a primal-dual representation based on asymmetric Kernel Singular Value Decomposition (KSVD), which fills the gap of dismissing the asymmetry between theory and implementation. Specifically, in this unsupervised setup, we propose to remodel self-attention in the primal representation, namely, Primal-Attention, and to optimize it accordingly. Our method is driven by two major motivations. Firstly, we observe that attention matrices in Transformers can be low-rank, as shown in Figure 1(d), and this property becomes more significant towards deeper network layers. Secondly, the self-attention matrix is intrinsically an asymmetric kernel matrix [12, 21]. To this end, we propose KSVD for self-attention, which takes both low-rank and asymmetric properties into consideration. To the best of our knowledge, this is the first work that provides a primal-dual representation for the asymmetric self-attention and applies it to modelling and optimization. The contributions of this work are summarized as follows:

We characterize self-attention by KSVD with asymmetric kernels. Different from existing works employing symmetric kernel-based methods, we take asymmetry into account so as to be more consistent with the real setups in self-attention. (Section 2)

We derive a primal-dual representation for self-attention through KSVD, and propose a novel attention in the primal, named Primal-Attention, avoiding the expensive kernel computation in the dual. With KSVD, the values are interpreted as the projection weights that yield maximal variances of features, and the low-rank property can be pursued by confining the projection numbers. (Section 3, Section 4)

We prove that the stationary solution to the derived KSVD leads to a zero-value objective of the unconstrained primal problem. Therefore, the optimization of KSVD in Primal-Attention can be efficiently implemented by minimizing a regularization term added to the loss, with no need of extra decomposition operations. (Section 4)

In numerical experiments, Primal-Attention achieves state-of-the-art performance on various datasets together with efficiency advantages over the canonical self-attention. Moreover, we demonstrate that our deployed optimization from KSVD can regularize the attention with a

sharper singular value decay, hence promoting learning more low-rank features, which is shown in Figure 1. (Section 5)

2 Problem Statement: Self-attention with Asymmetric Kernel

Self-attention Let { x i ∈ R d } N i =1 be the input data sequence. In self-attention [1], the queries, keys and values output the linear projections of the input sequence, such that

where W q ∈ R d q × d , W k ∈ R d k × d , and W v ∈ R d v × d , commonly with the setup d q = d k . The attention scores are then given by a ( x i , x j ) = ⟨ q ( x i ) , k ( x j ) ⟩ / √ d k = ⟨ W q x i , W k x j ⟩ / √ d k . In the canonical self-attention, the 'softmax' activation is then applied to bring non-linearity and positives, yielding the attention weights:

̸

Similar to [12], the attention matrix, i.e., K := [ κ ( x i , x j )] ∈ R N × N , can be interpreted as a kernel matrix with entries κ ( x i , x j ) , where κ ( · , · ): R d × R d → R serves as the kernel function. Notice that in general, ⟨ W q x i , W k x j ⟩ ̸ = ⟨ W q x j , W k x i ⟩ , leading to an asymmetric kernel where K ij = K ji .

Then, the attention output o i ∈ R d v in each head is attained as:

In Transformers, multiple heads are commonly applied through the concatenation of all heads [1].

Asymmetric Attention Matrix In kernel methods, rigorous works have been presented with Mercer kernels that are symmetric and positive semi-definite [20] through the kernel trick from Reproducing Kernel Hilbert Spaces (RKHS) [13]. On the other hand in Transformer [1], the attention kernel matrix is asymmetric as shown in (2). Existing works leverage the kernel interpretation for improving self-attention [14, 17, 18, 19], however, their deployed kernel-based techniques all rely on Mercer kernels, which is inconsistent with the asymmetric nature. Instead, asymmetry is allowed in kernel tricks from Reproducing Kernel Banach Spaces (RKBS) [22] as in the following Definition 2.1.

Definition 2.1 (Definition 2 [21]; Theorem 2.1 [24];[25]) . For asymmetric kernels, the kernel trick from RKBS with the kernel function κ ( · , · ): X × Z → R can be defined by the inner product of two real measurable feature maps from Banach spaces B X , B Z on X , Z , respectively:

Based on Definition 2.1, the kernel matrix in self-attention can be characterized by the kernel trick from RKBS [21], providing an analytical tool from the aspect of kernel representer theorem.

SVD and Shifted Eigenvalue Problem SVD factorizes a given r -rank matrix A ∈ R N × M by two sets of orthonormal eigenbases: A = U Σ V ⊤ with Σ = diag { σ 1 , . . . , σ r } of positive singular values and the columns of U ∈ R N × r and V ∈ R M × r as the left and right singular vectors , respectively [26]. U , V reflect the subspace projections in relation to the columns and rows, as shown in (5), and contain different information residing in A due to the asymmetry. When A is squared and symmetric, SVD boils down to the eigendecomposition with U = V . In [27], a novel variational principle is proposed for SVD with Least Squares Support Vector Machines (LSSVM) [28], where the dual problem leads to a shifted eigenvalue problem in accordance to the decomposition theorem from Lanczos [29] regarding SVD, i.e., Theorem 2.2. This theorem is also of special importance in our work to the kernel extension of SVD in self-attention under the framework of LSSVM.

Theorem 2.2 (Lanczos [29]) . Any non-zero matrix A ∈ R N × M can be written as A = ˜ U ˜ Σ ˜ V ⊤ , where the matrices ˜ U , ˜ Σ , ˜ V are defined by the shifted eigenvalue problem:

where ˜ U ∈ R N × r and ˜ V ∈ R M × r satisfy ˜ U ⊤ ˜ U = I r and ˜ V ⊤ ˜ V = I r , and ˜ Σ ∈ R r × r is a diagonal matrix with positive numbers.

3 Primal-dual Representation of Self-attention based on Kernel SVD

In this section, we apply the kernel trick from RKBS to the asymmetric attention kernel, and derive self-attention with a primal-dual representation based on Kernel SVD (KSVD). Under this learning scheme, a new self-attention mechanism is proposed by remodelling the attention output in the primal representation, without explicit computation of the kernel matrix in the dual representation. With the stationarity conditions, we flexibly implement the optimization of KSVD through an additional loss term, which can regularize the model to improved low-rank properties without extra decomposition.

KSVD optimization problem in Primal and Dual By Definition 2.1 of RKBS, the kernel function in the dual for the asymmetric attention kernel K in self-attention can be formulated by K ij = κ ( x i , x j ) := ⟨ ϕ q ( x i ) , ϕ k ( x j ) ⟩ , with two feature maps ϕ q , ϕ k related to queries and keys. Recall the self-attention output in (3), the values { v ( x j ) } N j =1 resemble the dual variables projecting the kernel matrix in the dual representation of kernel methods, whereas the kernel involved is asymmetric. In this regard, the nonlinear version of SVD under the framework of LSSVM [27] well fits the self-attention setup, hence is set as the basis for the following work with asymmetric kernels built upon RKBS. Differently, we extend the matrix SVD setups in [27] to the case of asymmetric attention matrix with two input data sources as queries and keys. Moreover, we consider that in self-attention, values are input data-dependent, and thus generalize [27] with data-dependent projection weights.

Given the sequence { x i ∈ R d } N i =1 , we start from the primal optimization with KSVD:

where we have the data-dependent projection weights f ( X ) ⊤ W e =: W e | X ∈ R p × s , f ( X ) ⊤ W r =: W r | X ∈ R p × s relying on parameters W e , W r ∈ R N × s , the feature maps ϕ q ( · ) , ϕ k ( · ): R d → R p , the projection scores e i , r j ∈ R s , and the regularization coefficient Λ ∈ R s × s which is a positive diagonal matrix. The objective J in the primal optimization maximizes the projection variances of W ⊤ e | X ϕ q ( x i ) , W ⊤ r | X ϕ k ( x j ) regarding queries and keys, and also involves a regularization term coupling the projections. The corresponding solution in the dual is characterized by the right and left singular vectors, which captures the directions with maximal projection variances, w.r.t. rows (queries) and columns (keys) of the attention kernel matrix. Thus, with the formulated primal optimization in (6), the learning in self-attention is interpreted as a SVD problem on the attention matrix.

For clarity, we elaborate our primal optimization problem in (6) as follows. i) The projection weights are data-dependent, where f ( X ) =: F X ∈ R N × p denotes a transformation matrix containing the information of the sequence data X := [ x 1 , . . . , x N ] ⊤ ∈ R N × d . Notably, F X is a constant matrix once given X , and we let it linearly depend on X in experiments. Moreover, when F X is chosen as the identity matrix, it reconciles to the common setups in kernel methods in primal. ii) The feature maps related to queries and keys, respectively, are defined as ϕ q ( x i ) := g q ( q ( x i )) , ϕ k ( x i ) := g k ( k ( x i )) , where g q ( · ) : R d q → R p and g k ( · ) : R d k → R p denote the mappings composited on the linear projections q ( · ) and k ( · ) in (1) of queries and keys. Note that we leave the choice of g q and g k later explained in Remark 4.1. iii) By projecting ϕ q ( x i ) , ϕ k ( x j ) ∈ R p with weights W e | X , W r | X ∈ R p × s , we obtain the projection scores e i , r j ∈ R s along the s directions, usually s < p , which corresponds to the number of singular values of the induced kernel matrix in the dual optimization (7).

Remark 3.1 (Variance maximization objective) . In the formulated KSVD problem, the objective in the primal optimization (6) jointly maximizes the variances of the two projections e i , r j in the feature spaces determined by ϕ q , ϕ k along the directions W e | X , W r | X . Within this context, e i , r j learn to capture maximal information mutually residing in ϕ q and ϕ k regarding the queries and keys.

With Lagrangian duality and KKT conditions, we prove that the dual optimization problem to (6) leads to a shifted eigenvalue problem corresponding to the SVD on the asymmetric attention kernel K , given by Theorem 3.2. The proof is provided in the Supplementary Material.

Theorem 3.2 (Dual optimization problem of KSVD in self-attention) . With Lagrangian duality and the KKT conditions, the dual optimization problem of (6) leads to the shifted eigenvalue problem:

Dualrepresentation(canonicalself-attention)

Primalrepresentation(ours)

O(N2d,)

(sdN)Q

K = Φq(x)Φk(x)

batchdim

batchdim

NXN

Nxdy

NXP

pxN

Nxdy

NXP

×

px

S-

NXP

Welx

Wr1x

Φk(x)T

K:

i

=[K（xi,xj)]

v(x)

Φq(X)

v(x)

Φq(x)

Φk(x)

where Σ ∈ R s × s is a positive diagonal matrix, and H e = [ h e 1 , . . . , h e N ] ⊤ ∈ R N × s , H r = [ h r 1 , . . . , h r N ] ⊤ ∈ R N × s are the dual variables serving as the left and right singular vectors, respectively. The kernel trick to the asymmetric kernel matrix K is interpreted as K ij = ⟨ f ( X ) g q ( q ( x i )) , f ( X ) g k ( k ( x j )) ⟩ =: 〈 ϕ ′ q ( x i ) , ϕ ′ k ( x j ) 〉 .

As shown in Theorem 3.2, the solutions collect non-zero Λ in (6) such that Σ = Λ -1 . Based on Lanczos' decomposition in Theorem 2.2, we can then associate Σ with the non-zero singular values of the attention kernel K in (7), and H e , H r with the left and right singular vectors of K , such that K = H e Σ H ⊤ r . Therefore, formulas (6) and (7) provide the optimization problems of performing KSVD with the attention kernel matrix K in the primal and in the dual, respectively.

Self-attention as KSVD dual representation Firstly, we provide the primal-dual model representation of the derived KSVD problem. Secondly, we show that the dual representation of the model corresponds to the canonical self-attention. The derivation details involving the KKT conditions are provided in the Supplementary Material.

Remark 3.3 (Primal-dual representations of KSVD in self-attention) . In the KSVD formulations for the asymmetric kernel matrix in self-attention, with KKT conditions, the projection scores can be either represented in the primal using explicit feature maps or in the dual using kernel functions:

Remark 3.4 (Correspondence of KSVD and canonical self-attention output) . Recall the output o i of canonical self-attention (3) , it corresponds to the dual representation of the projection score e ( x ) in (8) , i.e., o i ≜ e ( x i ) . When the values { v ( x j ) } N j =1 in canonical self-attention are chosen as the dual variables { h r j } N j =1 , i.e., h r j := v ( x j ) , j = 1 , . . . , N , the values { v ( x j ) } N j =1 play the role of the right singular vectors of K .

From the perspective in Remark 3.4, the optimization goal in self-attention is interpreted to jointly capture the maximal variances of e i , r j as in (6), where the projection scores can be denoted as e i := e ( x i ) , r j := r ( x j ) through the representations (8). However, the canonical self-attention only outputs the e i -score ( o i ≜ e i ). In this sense, the output of the canonical self-attention only considers the projection scores involving the right singular vectors of the asymmetric attention kernel K .

4 Primal-Attention

Modelling It is quite remarkable that the attention output can be equivalently represented without the kernel expressions, avoiding the heavy computation of the kernel matrix. Within the context of KSVD, we further observe that there exists another set of projections r j regarding the left singular vectors in h e i as in (8), providing extra information residing in the asymmetric kernel matrix K .

We derive a novel attention mechanism by leveraging the primal representation of KSVD, namely, Primal-Attention, where two explicit feature maps ϕ q , ϕ k are adopted. To fully exploit the asymmetry in the kernel matrix of self-attention, Primal-Attention concatenates the two sets of projections using both left and right singular vectors, and thus formulates the attention outputs as follows:

In Primal-Attention, the projection weights W e | X , W r | X in the primal play the role as the counterparts of the values in the dual. Given f ( X ) =: F X an identity matrix, our KSVD problem in (6) boils down to the data-independent projection weight case as in [27], which can thereby be regarded as a special case of our derived KSVD in Primal-Attention. In this case, the kernel trick in the asymmetric attention kernel becomes K ij = ⟨ g q ( q ( x i )) , g k ( k ( x j )) ⟩ = ⟨ ϕ q ( x i ) , ϕ k ( x j ) ⟩ , i, j = 1 , . . . , N .

Remark 4.1 (Choices of ϕ q , ϕ k for non-linearity) . The canonical self-attention adopts softmax for introducing non-linearity to the attention score matrix; within our setups and kernel trick, it can be viewed as: κ ( x i , x j ) = ˆ D -1 ⟨ ϕ q ( x i ) , ϕ k ( x i ) ⟩ where ϕ q ( x ) := g ( q ( x )) , ϕ k ( x ) := g ( k ( x )) with g ( z ) := exp( -∥ z ∥ 2 / 2)(exp( w ⊤ 1 z ) , . . . , exp( w ⊤ p z )) , w i ∼ N (0 , I d q ) and d q = d k [14], and ˆ D := diag( ϕ q ( X )( ϕ k ( X ) ⊤ 1 N )) . The projection scores then correspond to e ( x ) = ˆ D -1 / 2 W ⊤ e | X ϕ q ( x ) and r ( x ) = ˆ D -1 / 2 W ⊤ r | X ϕ k ( x ) . In this case, two exponential feature maps need to be constructed and the normalization factor ˆ D to all samples needs to be computed. In this paper, we consider feature maps related to the Cosine similarity kernel on queries and keys, such that ϕ q ( x ) = g q ( q ( x )) := q ( x ) / ∥ q ( x ) ∥ 2 and ϕ k ( x ) = g k ( k ( x )) := k ( x ) / ∥ k ( x ) ∥ 2 in all experiments. It is easy to implement and able to bring both non-linearity and normalization to the feature maps.

Optimization The KSVD problem can either be optimized in the primal as a constrained optimization problem, or in the dual as a shifted eigenvalue problem (SVD on the kernel matrix K ). In Primal-Attention, we perform the optimization in the primal. In the following Lemma 4.2, we prove the zero-value property of the primal optimization objective J in (6) when it is evaluated at the stationary solution in (7). The proof is provided in the Supplementary Material.

Lemma 4.2 (A zero-value objective with stationary solutions) . The solutions of H e , H r , Σ to the shifted eigenvalue problem in the dual optimization (7) lead to the zero-value objective J in the primal optimization (6) .

With the property in Lemma 4.2, rather than solving SVD problem on the kernel matrix K in the dual, the optimization of Primal-Attention is realized by minimizing the primal objective to zero:

where L is the task-oriented loss, e.g., the cross-entropy loss for classification tasks, the summation term over l denotes the additive objectives J l of all attention blocks using the proposed PrimalAttention, where J l is implemented as the mean over all heads, and η > 0 is the regularization coefficient. Specifically, for each head in the self-attention using Primal-Attention, we have

where Λ is automatically determined by the optimization. KSVD optimization of Primal-Attention is easy to implement by adding a regularization loss. Hence, Primal-Attention not only represents selfattention with KSVD formulation in the primal, but utilizes the optimization of KSVD by regularizing the attention, satisfying the stationarity conditions of KSVD when reaching a zero-value objective.

5 Numerical Experiments

In this section, we verify the effectiveness of our Primal-Attention applied in Transformers on five well-established benchmarks: time series, long sequence modelling, reinforcement learning, image classification and language modelling. Notably, we consider two types of Transformers applied with our Primal-Attention, i.e., PrimalFormer (Primal.) and Primal. + . i) In PrimalFormer, PrimalAttention (9) is applied to all attention layers, regularized with the KSVD loss (11). This setup is

preferred when data shows relatively strong low-rank structure or the model redundancy is quite substantial. ii) Primal. + refers to the baselines from the Transformer family with the last layer replaced by our Primal-Attention, which is in favour in large-scale data and complicated tasks where less information compression is desired in the learning, especially in shallower layers. To specify the Transformer backbone, we denote our method by 'Primal. + Backbone' herein. Primal. + serves as a flexible variant combined with different Transformer backbones, and with KSVD optimization applied through an implicit low-rank regularization loss, advocating to learn more informative features, as in Figure 1. The two main hyper-parameters of our method are the coefficient η in (10) and the number of projection directions s of KSVD in (6). In data-dependent cases with f ( X ) , we take a subset of X by uniformly sampling n = min { s ∗ 10 , N } points from X for efficiency aspects, as the main patterns of a matrix can be retained with random linear projections shown by the Johnson-Lindenstrauss lemma [30]. Details are given in the Supplementary Material.

UEA Time Series Classification UEA Time Series Classification Archive [31] is the benchmark for the evaluation on temporal sequences. Following [11], we select 10 multivariate datasets with pre-processed data according to [32], and employ 2-layer Transformer as backbone with the hidden dimension 512 on 8 heads and the embedding dimension 64 for self-attention. The hyper-parameter search of our method is with η ∈ { 0 . 1 , 0 . 2 , 0 . 5 } , s ∈ { 20 , 30 , 40 } . Experiments are run on one NVIDIA Tesla P100 SXM2 16GB GPU.

Table 1 reports the test accuracy of the compared recent methods, where the best result is in bold. Both our PrimalFormer and Primal. + achieve comparable and better performance than the state-of-the-art results provided by Flowformer [11]. Notably, Primal. + yields the best accuracy with 1 . 2% overall improvement upon the canonical Transformer [1], i.e., by replacing the softmax-based self-attention with our Primal-Attention in the last layer. This shows the promising potential of Primal-Attention in enhancing temporal modelling capacity upon the canonical softmax self-attention. It is worth mentioning that our PrimalFormer applying the KSVD optimization with low-rank regularization to all layers also obtains good performance. This could be due to the fact that these datasets are relatively simple where model redundancy can exist, so that appropriately imposing the low-rank regularization in KSVD does not harm the model expressiveness for the data. We also compare the running time and memory consumption with the canonical Transformer and Flowformer, which is proposed recently with state-of-the-art performances and efficiency. Both our Primal. and Primal. + Trans. consistently lead to improved efficiency than the canonical Transformer. In Primal., all attention layers are implemented with the proposed Primal-Attention, while Primal. + Trans. applies the Primal-Attention

in the last layer, making Primal. a more efficient model. We note that our Primal. further surpasses Flowformer in most cases regarding both running time and memory.

Long-Range Arena Benchmark Long-Range Arena (LRA) [39] is a benchmark for the longsequence scenarios, including equation calculation (ListOps) [40], review classification (Text) [41], document retrieval (Retrieval) [42], image classification (Image) [43] and image spatial dependencies (Pathfinder) [44]. We follow the settings in [45] with PyTorch. The Transformer backbone is set with 2 layers, hidden dimension 128 with 2 heads and embedding dimension 64 with mean pooling, where Reformer (Re.) uses 2 hashes, Performer (Per.) has 256 random feature dimensions and Linformer (Lin.) uses a projection dimension of 256. Our hyper-parameter is set from η ∈ { 0 . 05 , 0 . 1 } , s ∈ { 20 , 30 } . Experiments are conducted on a single NVIDIA Tesla V100 SXM2 32GB GPU.

From the reported top-1 test accuracy in Table 3, our PrimalFormer shows better accuracy while achieves top efficiency (see Table 4) than several efficient self-attention counterparts including Reformer, Performer and Linformer. Notably, our model Primal. + Trans. achieves the state-ofthe-art accuracy of 60 . 4% , which is 1 . 6% higher than Transformer, and 1 . 2% higher than the currently best YOSO-E, showing that the deployed Primal-Attention is able to boost the performance upon canonical Transformer distinctively. On top of it, Primal. + Trans. has distinctively higher efficiency and require much less memory than the canonical Transformer as in Table 4, and our Primal. outperforms all compared variants of efficient Transformer. Compared to Table 1 with simpler data, Primal. + Trans. further achieves performance gain over PrimalFormer. The reason can be that the shallower (the first) layer needs more model capacity for depicting the data patterns in the learning, while the deep (the second) layer captures less detailed features and is regularized with more informative feature learning through our Primal-Attention. The efficiency of our Primal-Attention are further pronounced with longer sequences than that of the UEA benchmark in Table 2.

Reinforcement Learning We consider the offline reinforcement learning (RL) performance of our methods on D4RL benchmark [47] designed for continuous control tasks. We choose three different environments controlling the robot movement: HalfCheetah, Hopper and Walker. Our experiments are conducted on three datasets pre-collected under different policies: Medium-Expert, Medium and Medium-Replay. We follow the experimental settings in Flowformer [11], and also compare to Decision Transformer (DT) [10] which is commonly considered as the baseline with state-of-the-art performances based on the canonical attention. Note that offline RL is an auto-regressive task and our Primal-Attention is adapted to its causal version, as described in the Supplementary Material. We adopt the architecture of 3 layers, hidden dimension 256 with 4 heads, and the embedding dimension 64. Our hyper-parameters are set as η = 0 . 05 , s ∈ { 32 , 64 , 96 } . Each experiment is run with 3 different seeds on one NVIDIA Tesla P100 SXM2 16GB GPU.

Results in Table 5 demonstrate that our Primal. + outperforms all compared methods with a distinctly higher average reward. Specifically, our Primal. + DT reaches 4 . 0 points higher than the state-ofthe-art Flowformer [11]. Compared to the baseline Decision Transformer (DT), our Primal. + only replaces the self-attention in the top layer and keeps other structures the same and manages to improve the average reward by 5 . 3 points. This verifies the effectiveness of our Primal-Attention and the benefits of our low-rank regularization of KSVD for the generality of DT in offline reinforcement learning. We evaluate the efficiency with comparisons to DT and Flowformer in Table 6, which give distinctively better results than other baselines in Table 5. Our Primal. + DT achieves comparable time and memory efficiency as the most efficient baseline DT, while Flowformer shows significantly lower efficiency in both aspects. Recall that our Primal. + DT achieves an average reward of 77.5, which is 5.3 points higher than DT and 4.0 points higher than Flowformer.

Large-scale Experiments Weevaluate the capability of our Primal. + model with DeiT-Small/16 [7] as backbone on ImageNet-100 [48] and ImageNet-1K [23] for image classification task. We also experiment with the language modelling task on WikiText-103 [49]. On ImageNet, we train DeiTSmall/16 and our Primal. + DeiT-Small/16 from scratch following the same training protocols in [7] with 4 NVIDIA Tesla V100 SXM2 32GB GPUs. Our hyper-parameters are chosen as η = 0 . 05 , s ∈ { 32 , 64 , 96 } . On WikiText-103, we follow the setting in [50] where the sequence length is set as 512, the model consists of 6 decoder layers with 8 heads. We implement the causal version of Primal-Attention in the last layer, i.e., Primal. + Trans with η = 0 . 1 , s = 30 . Models are trained from scratch on 4 NVIDIA Tesla V100 SXM2 32GB GPUs for 150K updates after 6K-steps warm-up.

Table 7(a) provides the test accuracy, training time and memory on a single V100 GPU with batch size 256 on both ImageNet-100 and ImageNet-1K datasets. There is only one set of time and memory since models follow the same training protocols on both datasets. Our Primal. + achieves better accuracy than DeiT-Small/16 on ImageNet-100. It also achieves the same accuracy as baseline with less training time and memory on ImageNet-1K. On WikiText-103 in Table 7(b), our method with default setups significantly reduces the perplexity by 2.0 points than Transformer, and achieves comparable performances with the well-tuned Flowformer, a latest SoTA model, with enhanced efficiency. Results show that using Primal-Attention in the last layer helps the overall performance.

Ablation Study on Using Two Projections e ( x ) , r ( x ) We conduct an ablation study with (w/) and without (w/o) projection scores, i.e., r -scores, involving the left singular vectors, on LRA [39]. Table 8 shows that using both projections (w/ r -scores) helps boost performances, verifying the effectiveness of learning with the two sides of singular vectors on an asymmetric attention kernel.

(a) Test accuracy (%) and efficiency on ImageNet-100 [48] and ImageNet-1K [23].

(b) Results on WikiText-103 [49].

6 Related work

Since the pioneering work [12], the kernel-based approaches have become popular in Transformers, in which the kernel interpretation on the attention matrix has been shed light on. FourierFormer [17] treats the canonical self-attention as non-parametric regression with methodologies for symmetric kernels. [18] considers relative positional embedding with conditional positive definite kernel. [19] treats self-attention operation as support vector regression without considering the asymmetry in the deployed kernel methods, and the supervised regression is not applied in optimzing the attention either. [51] addresses the issue of asymmetry, however, it resorts to symmetrization by replacing the softmax attention with an approximated symmetric one, thereby still dismissing the asymmetry. These prior works deploy kernel-based techniques that are originally designed for symmetric kernels and request to suffice Mercer conditions, which is inconsistent with the asymmetric nature in self-attention, resulting in a nontrivial gap between the analytical understanding and numerical implementation towards revealing the rationale in Transformers. In [21], it leverages the kernel tricks from RKBS [22] that allows asymmetry and formulates attention as a binary kernel learning problem via empirical risk minimization. However, it is hard to find an explicit optimization accordingly in implementing Transformers. Nevertheless, [21] provides an analytical tool from the aspect of kernel representer theorem upon RKBS that allows asymmetry.

Much literature has also devoted to improving the efficiency of the attention computation through different approximation techniques. In the related works addressing the attention mechanism, Reformer [34] uses locally-sensitive hashing for sparse approximation. Performer [14] approximates self-attention matrix with random features. Linformer [46] considers low-rank approximation with the help of random projections. Nyströmformer [45] utilizes the Nyström method by down sampling the queries and keys in the attention matrix. [52] incorporates sparsity prior on attention. These works pose the focus on reducing the computation of the attention kernel matrix from the canonical self-attention. Hence, these works all address how to solve the problem in the dual form involving the kernel matrix, while we work in a significantly different way, that is, in the primal form.

7 Conclusion

In this paper, we interpret the self-attention in Transformers with asymmetric kernels and construct a learning framework with SVD on asymmetric kernels (KSVD) under the setups of LSSVM. Within the context of KSVD, a primal-dual model representation is formulated for self-attention and a novel attention mechanism (Primal-Attention) is proposed accordingly by leveraging the primal representation. It is quite significant that with Primal-Attention, not only the computation of the attention kernel matrix in the dual can be avoided, but also the cast unsupervised KSVD optimization can be efficiently incorporated into the training through an additional regularization loss for more informative low-rank property. The analytical derivations and numerical evaluations demonstrate our great potentials in bridging explicit model interpretability and state-of-the-art performances. Future works can include developing different variants with the low-rank property, e.g., robust Transformers, investigating more general applications of Primal-Attention to a wide range of architectures and tasks.

References

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT , pages 4171-4186, 2019.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(1):5485-5551, 2020.

Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6824-6835, 2021.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Proceedings of the International conference on machine learning , pages 10347-10357. PMLR, 2021.

Yingyi Chen, Xi Shen, Yahui Liu, Qinghua Tao, and Johan A.K. Suykens. Jigsaw-vit: Learning jigsaw puzzles in vision transformer. Pattern Recognition Letters , 166:53-60, 2023.

Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems , 34:12731286, 2021.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems , 34:15084-15097, 2021.

Haixu Wu, Jialong Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Flowformer: Linearizing transformers with conservation flows. In International Conference on Machine Learning , pages 24226-24242, 2022.

Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , 2019.

Vladimir N Vapnik. An overview of statistical learning theory. IEEE Transactions on Neural Networks , 10(5):988-999, 1999.

Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations , 2021.

Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. XCiT: Crosscovariance image transformers. Advances in neural information processing systems , 34:2001420027, 2021.

Tam Minh Nguyen, Tan Minh Nguyen, Dung DD Le, Duy Khuong Nguyen, Viet-Anh Tran, Richard Baraniuk, Nhat Ho, and Stanley Osher. Improving transformers with probabilistic attention keys. In Proceedings of the International Conference on Machine Learning , pages 16595-16621. PMLR, 2022.

Tan Nguyen, Minh Pham, Tam Nguyen, Khai Nguyen, Stanley Osher, and Nhat Ho. Fourierformer: Transformer meets generalized fourier integral theorem. Advances in Neural Information Processing Systems , 35:29319-29335, 2022.

Ta-Chung Chi, Ting-Han Fan, Peter J Ramadge, and Alexander Rudnicky. Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems , 35:8386-8399, 2022.

Tan Minh Nguyen, Tam Minh Nguyen, Nhat Ho, Andrea L. Bertozzi, Richard Baraniuk, and Stanley Osher. A primal-dual framework for transformers and neural networks. In International Conference on Learning Representations , 2023.

James Mercer. Functions of positive and negative type, and their connection with the theory of integral equations. Philosophical Transactions of the Royal Society A , 209(441-458):415-446, January 1909.

Matthew A Wright and Joseph E Gonzalez. Transformers are deep infinite-dimensional nonMercer binary kernel machines. arXiv preprint arXiv:2106.01506 , 2021.

Haizhang Zhang, Yuesheng Xu, and Jun Zhang. Reproducing kernel banach spaces for machine learning. Journal of Machine Learning Research , 10(12), 2009.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition , pages 248-255, 2009.

Rong Rong Lin, Hai Zhang Zhang, and Jun Zhang. On reproducing kernel banach spaces: Generic definitions and unified framework of constructions. Acta Mathematica Sinica, English Series , 38(8):1459-1483, 2022.

Pando G Georgiev, Luis Sánchez-González, and Panos M Pardalos. Construction of pairs of reproducing kernel banach spaces. In Constructive Nonsmooth Analysis and Related Topics , pages 39-57. Springer, 2013.

Gilbert Strang. Linear algebra and its applications. Belmont, CA: Thomson, Brooks/Cole, 2006.

Johan A.K. Suykens. SVD revisited: A new variational principle, compatible feature maps and nonlinear extensions. Applied and Computational Harmonic Analysis , 40(3):600-609, 2016.

Johan A.K. Suykens, Tony Van Gestel, Joseph De Brabanter, Bart De Moor, and Joos PL Vandewalle. Least Squares Support Vector Machines . World scientific, 2002.

Cornelius Lanczos. Linear systems in self-adjoint form. The American Mathematical Monthly , 65(9):665-679, 1958.

Joram Lindenstrauss and William B. Johnson. Extensions of lipschitz maps into a Hilbert space. Contemporary Mathematics , 26(189-206):2, 1984.

Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 , 2018.

George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 2114-2124, 2021.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning , pages 5156-5165, 2020.

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations , 2020.

Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020.

Zhanpeng Zeng, Yunyang Xiong, Sathya Ravi, Shailesh Acharya, Glenn M Fung, and Vikas Singh. You only sample (almost) once: Linear cost self-attention via bernoulli sampling. In International conference on machine learning , pages 12321-12332, 2021.

Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. In International Conference on Learning Representations , 2022.

Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. Soft: Softmax-free transformer with linear complexity. Advances in Neural Information Processing Systems , 34:21297-21309, 2021.

Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations , 2021.

Nikita Nangia and Samuel Bowman. Listops: A diagnostic dataset for latent tree learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop , pages 92-99, 2018.

Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies , pages 142-150, 2011.

Dragomir R Radev, Pradeep Muthukrishnan, Vahed Qazvinian, and Amjad Abu-Jbara. The acl anthology network corpus. Language Resources and Evaluation , 47:919-944, 2013.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. Advances in neural information processing systems , 31, 2018.

Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 14138-14148, 2021.

Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 , 2020.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations , 2016.

Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. In International Conference on Learning Representations , 2021.

Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang. Skyformer: Remodel self-attention with gaussian kernel and nyström method. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSelf-attention mechanisms serve as pivotal components in the domains of natural language processing and computer vision. Previous kernel function based self-attention variants, which are predicated on Mercer kernels, tend to overlook the inherent asymmetry of the attention matrix in the vanilla self-attention. In response to this oversight, this study introduces a novel approach, termed Primal-Attention. This approach leverages asymmetric Kernel Singular Value Decomposition (KSVD) to facilitate a low-rank approximation of the attention matrix, thereby incorporating an asymmetric kernel attention matrix. The empirical findings substantiate that Primal-Attention functions as a low-rank attention mechanism, exhibiting superior performance while maintaining low time complexity.\n\nSTRENGTHS:\n1. This paper introduces a novel perspective that interprets the self-attention mechanism as a KSVD optimization problem.\n\nWEAKNESSES:\n1. The exposition of methodologies in this paper lacks sufficient clarity. For instance, the crucial concept of primal optimization in self-attention with KSVD, as proposed in Equation 6, is introduced abruptly without a detailed discussion of the connection between KSVD and self-attention.\n2. Algorithm 1, outlined in Appendix B.1, details the entire process of Primal-Attention and employs random projection. However, the omission of a citation for the Johnson–Lindenstrauss lemma in the paper is inappropriate.\n\nQUESTIONS:\n1. Why should the primal optimization function, as depicted in Equation 6, be considered? On what basis is this function proposed? Furthermore, why should this function be maximized rather than minimized?\n2. Why is the formula for Primal-Attention in each head represented by Equation 11? The process of derivation appears to be missing.\n3. How do the authors handle the positive diagonal matrix $\\mathbf{\\Lambda}$? Given that $\\mathbf{\\Lambda}$ is a key component in Equations 6 and 11, can the authors provide a detailed explanation of the process?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

comments: SUMMARY:
This paper proposes a new understanding of self-attention in transformers via asymmetric Kernel Singular Value Decomposition (KSVD). In particular, the authors formulate a primal-dual representation of self-attention for maximizing the projection variances in the attention outputs and then derive a new attention mechanism, namely the Primal-Attention, to avoid direct computation of the kernel matrix. Using KKT conditions, they prove the Primal-Attention can obtain a zero-value objective. Experimental results are provided to justify the advantage of the Primal-Attention.

STRENGTHS:
1. The idea of formulating self-attention as solving an KSVD problem is novel and of potentially high-impact.

2. The derivation of self-attention from KSVD given in the paper is elegant and correct.

3. The paper is well-written, easy to follow, and enjoyable to read.

WEAKNESSES:
1.  More large-scale experiments, e.g., on full ImageNet or WikiText103 language modeling, should be conducted.

2. Empirical analysis, such as the efficiency analysis of the Primal-Attention vs. the baselines, should be provided to help provide better understanding of the proposed method.

QUESTIONS:
1. Can the proposed KSVD framework be used to explain other components in a self-attention unit, such as the residual connection, the layer normalization, and the feedforward network?

LIMITATIONS:
The authors have not clearly addressed the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

comments: SUMMARY:
This paper proposes a Primal-Attention method to realize self-attention blocks in Transformer with a kernel matrix. It firstly explains the relationships between self-attention and asymmetric kernel matrix. Secondly it formulates self-attention in the form of kernel SVD and derive its primal and dual representations. And eventually it proposes Primal-Attention, which uses not only the projection scores involving the right singular vectors of the asymmetric attention kernel K, but also the ones involving the left singular vectors of the asymmetric attention kernel K. Besides, it conducts several experiments on several tasks.

STRENGTHS:
1. The motivation to formulate a neural network structure into a kernel one is quite interesting and meaningful. 
2. The primal-dual representations are derivable.

WEAKNESSES:
1. The explanation of why the projection scores involving the left singular vectors of the asymmetric attention kernel K are used is not adequate. It lacks enough analyses to prove their importance.
2. The results of experiments show little improvements compared with baseline methods. And only LRA benchmark has efficiency analyses.

QUESTIONS:
1. Why are the projection scores involving the left singular vectors of the asymmetric attention kernel K so important? What does it mean physically by doing so?
2. Why do you conduct experiments of efficiency only on LRA benchmark? And why does Primal+ have less efficiency than Primal+Trans?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07