SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yingyi Chen *  
ESAT-STADIUS  
KU Leuven, Belgium  
yingyi.chen@esat.kuleuven.be
Francesco Tonin  
ESAT-STADIUS  
KU Leuven, Belgium  
francesco.tonin@esat.kuleuven.be
Qinghua Tao *  
ESAT-STADIUS  
KU Leuven, Belgium  
qinghua.tao@esat.kuleuven.be
Johan A.K. Suykens  
ESAT-STADIUS  
KU Leuven, Belgium  
johan.suykens@esat.kuleuven.be
Recently, a new line of works has emerged to understand and improve self-attention in Transformers by treating it as a kernel machine. However, existing works apply the methods for symmetric kernels to the asymmetric self-attention, resulting in a nontrivial gap between the analytical understanding and numerical implementation. In this paper, we provide a new perspective to represent and optimize self-attention through asymmetric Kernel Singular Value Decomposition (KSVD), which is also motivated by the low-rank property of self-attention normally observed in deep layers. Through asymmetric KSVD,  $i)$  a primal-dual representation of self-attention is formulated, where the optimization objective is cast to maximize the projection variances in the attention outputs;  $ii)$  a novel attention mechanism, i.e., Primal-Attention, is proposed via the primal representation of KSVD, avoiding explicit computation of the kernel matrix in the dual;  $iii)$  with KKT conditions, we prove that the stationary solution to the KSVD optimization in Primal-Attention yields a zero-value objective. In this manner, KSVD optimization can be implemented by simply minimizing a regularization loss, so that low-rank property is promoted without extra decomposition. Numerical experiments show state-of-the-art performance of our Primal-Attention with improved efficiency. Moreover, we demonstrate that the deployed KSVD optimization regularizes Primal-Attention with a sharper singular value decay than that of the canonical self-attention, further verifying the great potential of our method. To the best of our knowledge, this is the first work that provides a primal-dual representation for the asymmetric kernel in self-attention and successfully applies it to modelling and optimization $^2$ .
Transformers [1] have become ubiquitous nowadays with state-of-the-art results in various tasks, such as natural language processing [2, 3, 4], computer vision [5, 6, 7, 8], reinforcement learning [9, 10, 11], etc. In the remarkable success of Transformers, the self-attention blocks play a key role, where the complicated dependencies between the individuals in data sequences can be depicted by using the established queries, keys, and values. Despite the prevailing advantages, theoretical understandings towards Transformers seem yet lagged behind its unprecedented empirical performance.
2Our implementation is available at https://github.com/yingyichen-cyy/PrimalAttention
(c)
Figure 1: Spectrum analysis of the self-attention matrix on ImageNet-1K [23]. (a)-(c) plot the cumulative explained variance regarding the singular values of the attention matrix with mean and standard deviation of the chosen layers in pre-trained DeiT-Small/16 [7] and Primal.+DeiT-Small/16 (ours): the attention matrix attains sharper singular value decays in deeper layers, also shown in (d). Note that we also plot the cumulative explained variance curves of the self-attention matrix from the last layer, i.e., the 11-th layer denoted by "L[11]", of both models in (c). Our method shows an enhanced low-rank property of the attention matrix upon the baseline.
Recently, the kernel-based perspective has been proposed where the dot-product attention operation is shown as a kernel matrix [12]. This finding is quite encouraging by bridging kernels [13] with Transformers, as kernel methods have long been well studied with good interpretation ability. Following this spirit, different works have been proposed subsequently to improve self-attention, e.g., [14, 15, 16, 17, 18, 19]. However, in these works, the applied kernel techniques rely on Mercer kernels [20] requesting symmetry, which is inconsistent with the intrinsically asymmetric setups of self-attention. In [21], it analytically characterizes the attention by asymmetric kernels based on Reproducing Kernel Banach Spaces (RKBS) [22]. Nonetheless, neither the asymmetry property nor the related optimization is utilized for improvements. In [19], self-attention is derived with a primal-dual representation from the support vector regression, which still adopts the technique for Mercer kernels. Moreover, in the cast supervised task, the assumed ground-truth outputs of self-attention are practically non-existent, making it difficult to be applied in the optimization.
In this work, we provide a novel perspective to interpret self-attention with a primal-dual representation based on asymmetric Kernel Singular Value Decomposition (KSVD), which fills the gap of dismissing the asymmetry between theory and implementation. Specifically, in this unsupervised setup, we propose to remodel self-attention in the primal representation, namely, Primal-Attention, and to optimize it accordingly. Our method is driven by two major motivations. Firstly, we observe that attention matrices in Transformers can be low-rank, as shown in Figure 1(d), and this property becomes more significant towards deeper network layers. Secondly, the self-attention matrix is intrinsically an asymmetric kernel matrix [12, 21]. To this end, we propose KSVD for self-attention, which takes both low-rank and asymmetric properties into consideration. To the best of our knowledge, this is the first work that provides a primal-dual representation for the asymmetric self-attention and applies it to modelling and optimization. The contributions of this work are summarized as follows:
- We characterize self-attention by KSVD with asymmetric kernels. Different from existing works employing symmetric kernel-based methods, we take asymmetry into account so as to be more consistent with the real setups in self-attention. (Section 2)
- We derive a primal-dual representation for self-attention through KSVD, and propose a novel attention in the primal, named Primal-Attention, avoiding the expensive kernel computation in the dual. With KSVD, the values are interpreted as the projection weights that yield maximal variances of features, and the low-rank property can be pursued by confining the projection numbers. (Section 3, Section 4)
- We prove that the stationary solution to the derived KSVD leads to a zero-value objective of the unconstrained primal problem. Therefore, the optimization of KSVD in Primal-Attention can be efficiently implemented by minimizing a regularization term added to the loss, with no need of extra decomposition operations. (Section 4)
- In numerical experiments, Primal-Attention achieves state-of-the-art performance on various datasets together with efficiency advantages over the canonical self-attention. Moreover, we demonstrate that our deployed optimization from KSVD can regularize the attention with a
2 Problem Statement: Self-attention with Asymmetric Kernel
Self-attention Let  $\{\pmb{x}_i\in \mathbb{R}^d\}_{i = 1}^N$  be the input data sequence. In self-attention [1], the queries, keys and values output the linear projections of the input sequence, such that
$$
q \left(\boldsymbol {x} _ {i}\right) = W _ {q} \boldsymbol {x} _ {i}, \quad k \left(\boldsymbol {x} _ {i}\right) = W _ {k} \boldsymbol {x} _ {i}, \quad v \left(\boldsymbol {x} _ {i}\right) = W _ {v} \boldsymbol {x} _ {i}, \tag {1}
$$
where  $W_{q} \in \mathbb{R}^{d_{q} \times d}$ ,  $W_{k} \in \mathbb{R}^{d_{k} \times d}$ , and  $W_{v} \in \mathbb{R}^{d_{v} \times d}$ , commonly with the setup  $d_{q} = d_{k}$ . The attention scores are then given by  $a(\pmb{x}_i, \pmb{x}_j) = \langle q(\pmb{x}_i), k(\pmb{x}_j) \rangle / \sqrt{d_k} = \langle W_q\pmb{x}_i, W_k\pmb{x}_j \rangle / \sqrt{d_k}$ . In the canonical self-attention, the "softmax" activation is then applied to bring non-linearity and positives, yielding the attention weights:
$$
\kappa \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) = \operatorname {s o f t m a x} \left(\left\langle W _ {q} \boldsymbol {x} _ {i}, W _ {k} \boldsymbol {x} _ {j} \right\rangle / \sqrt {d _ {k}}\right), \quad i, j = 1, \dots , N. \tag {2}
$$
Similar to [12], the attention matrix, i.e.,  $K \coloneqq [\kappa(\pmb{x}_i, \pmb{x}_j)] \in \mathbb{R}^{N \times N}$ , can be interpreted as a kernel matrix with entries  $\kappa(\pmb{x}_i, \pmb{x}_j)$ , where  $\kappa(\cdot, \cdot) \colon \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}$  serves as the kernel function. Notice that in general,  $\langle W_q\pmb{x}_i, W_k\pmb{x}_j \rangle \neq \langle W_q\pmb{x}_j, W_k\pmb{x}_i \rangle$ , leading to an asymmetric kernel where  $K_{ij} \neq K_{ji}$ .
Then, the attention output  $\pmb{o}_i \in \mathbb{R}^{d_v}$  in each head is attained as:
$$
\boldsymbol {o} _ {i} = \sum_ {j = 1} ^ {N} v \left(\boldsymbol {x} _ {j}\right) \kappa \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) = \sum_ {j = 1} ^ {N} v \left(\boldsymbol {x} _ {j}\right) K _ {i j}, \quad i = 1, \dots , N. \tag {3}
$$
In Transformers, multiple heads are commonly applied through the concatenation of all heads [1].
Asymmetric Attention Matrix In kernel methods, rigorous works have been presented with Mercer kernels that are symmetric and positive semi-definite [20] through the kernel trick from Reproducing Kernel Hilbert Spaces (RKHS) [13]. On the other hand in Transformer [1], the attention kernel matrix is asymmetric as shown in (2). Existing works leverage the kernel interpretation for improving self-attention [14, 17, 18, 19], however, their deployed kernel-based techniques all rely on Mercer kernels, which is inconsistent with the asymmetric nature. Instead, asymmetry is allowed in kernel tricks from Reproducing Kernel Banach Spaces (RKBS) [22] as in the following Definition 2.1.
Definition 2.1 (Definition 2 [21]; Theorem 2.1 [24];[25]). For asymmetric kernels, the kernel trick from RKBS with the kernel function  $\kappa (\cdot ,\cdot)\colon \mathcal{X}\times \mathcal{Z}\to \mathbb{R}$  can be defined by the inner product of two real measurable feature maps from Banach spaces  $\mathcal{B}_{\mathcal{X}},\mathcal{B}_{\mathcal{Z}}$  on  $\mathcal{X},\mathcal{Z}$ , respectively:
$$
\kappa (\boldsymbol {x}, \boldsymbol {z}) = \left\langle \phi_ {x} (\boldsymbol {x}), \phi_ {z} (\boldsymbol {z}) \right\rangle , \forall \boldsymbol {x} \in \mathcal {X}, \phi_ {x} \in \mathcal {B} _ {\mathcal {X}}, \boldsymbol {z} \in \mathcal {Z}, \phi_ {z} \in \mathcal {B} _ {\mathcal {Z}}. \tag {4}
$$
Based on Definition 2.1, the kernel matrix in self-attention can be characterized by the kernel trick from RKBS [21], providing an analytical tool from the aspect of kernel representer theorem.
SVD and Shifted Eigenvalue Problem SVD factorizes a given  $r$ -rank matrix  $A \in \mathbb{R}^{N \times M}$  by two sets of orthonormal eigenbases:  $A = U\Sigma V^{\top}$  with  $\Sigma = \mathrm{diag}\{\sigma_1, \ldots, \sigma_r\}$  of positive singular values and the columns of  $U \in \mathbb{R}^{N \times r}$  and  $V \in \mathbb{R}^{M \times r}$  as the left and right singular vectors, respectively [26].  $U, V$  reflect the subspace projections in relation to the columns and rows, as shown in (5), and contain different information residing in  $A$  due to the asymmetry. When  $A$  is squared and symmetric, SVD boils down to the eigendecomposition with  $U = V$ . In [27], a novel variational principle is proposed for SVD with Least Squares Support Vector Machines (LSSVM) [28], where the dual problem leads to a shifted eigenvalue problem in accordance to the decomposition theorem from Lanczos [29] regarding SVD, i.e., Theorem 2.2. This theorem is also of special importance in our work to the kernel extension of SVD in self-attention under the framework of LSSVM.
Theorem 2.2 (Lanczos [29]). Any non-zero matrix  $A \in \mathbb{R}^{N \times M}$  can be written as  $A = \tilde{U} \tilde{\Sigma} \tilde{V}^{\top}$ , where the matrices  $\tilde{U}, \tilde{\Sigma}, \tilde{V}$  are defined by the shifted eigenvalue problem:
$$
A \tilde {V} = \tilde {U} \tilde {\Sigma},
$$
$$
A ^ {\top} \tilde {U} = \tilde {V} \tilde {\Sigma}, \tag {5}
$$
where  $\tilde{U} \in \mathbb{R}^{N \times r}$  and  $\tilde{V} \in \mathbb{R}^{M \times r}$  satisfy  $\tilde{U}^{\top}\tilde{U} = I_r$  and  $\tilde{V}^{\top}\tilde{V} = I_r$ , and  $\tilde{\Sigma} \in \mathbb{R}^{r \times r}$  is a diagonal matrix with positive numbers.
sharper singular value decay, hence promoting learning more low-rank features, which is shown in Figure 1. (Section 5)
3 Primal-dual Representation of Self-attention based on Kernel SVD
In this section, we apply the kernel trick from RKBS to the asymmetric attention kernel, and derive self-attention with a primal-dual representation based on Kernel SVD (KSVD). Under this learning scheme, a new self-attention mechanism is proposed by remodelling the attention output in the primal representation, without explicit computation of the kernel matrix in the dual representation. With the stationarity conditions, we flexibly implement the optimization of KSVD through an additional loss term, which can regularize the model to improved low-rank properties without extra decomposition.
KSVD optimization problem in Primal and Dual By Definition 2.1 of RKBS, the kernel function in the dual for the asymmetric attention kernel  $K$  in self-attention can be formulated by  $K_{ij} = \kappa(\pmb{x}_i, \pmb{x}_j) \coloneqq \langle \phi_q(\pmb{x}_i), \phi_k(\pmb{x}_j) \rangle$ , with two feature maps  $\phi_q$ ,  $\phi_k$  related to queries and keys. Recall the self-attention output in (3), the values  $\{v(\pmb{x}_j)\}_{j=1}^N$  resemble the dual variables projecting the kernel matrix in the dual representation of kernel methods, whereas the kernel involved is asymmetric. In this regard, the nonlinear version of SVD under the framework of LSSVM [27] well fits the self-attention setup, hence is set as the basis for the following work with asymmetric kernels built upon RKBS. Differently, we extend the matrix SVD setups in [27] to the case of asymmetric attention matrix with two input data sources as queries and keys. Moreover, we consider that in self-attention, values are input data-dependent, and thus generalize [27] with data-dependent projection weights.
Given the sequence  $\{\pmb{x}_i\in \mathbb{R}^d\}_{i = 1}^N$  , we start from the primal optimization with KSVD:
$$
\max  _ {W _ {e}, W _ {r}, \boldsymbol {e} _ {i}, \boldsymbol {r} _ {j}} J = \frac {1}{2} \sum_ {i = 1} ^ {N} \boldsymbol {e} _ {i} ^ {\top} \Lambda \boldsymbol {e} _ {i} + \frac {1}{2} \sum_ {j = 1} ^ {N} \boldsymbol {r} _ {j} ^ {\top} \Lambda \boldsymbol {r} _ {j} - \operatorname {T r} \left(W _ {e} ^ {\top} W _ {r}\right) \tag {6}
$$
$$
\boldsymbol {r} _ {j} = \left(f (X) ^ {\top} W _ {r}\right) ^ {\top} \dot {\phi_ {k}} (\boldsymbol {x} _ {j}), j = 1, \dots , N,
$$
where we have the data-dependent projection weights  $f(X)^\top W_e \eqqcolon W_{e|X} \in \mathbb{R}^{p \times s}$ ,  $f(X)^\top W_r \eqqcolon W_{r|X} \in \mathbb{R}^{p \times s}$  relying on parameters  $W_e, W_r \in \mathbb{R}^{N \times s}$ , the feature maps  $\phi_q(\cdot), \phi_k(\cdot) \colon \mathbb{R}^d \to \mathbb{R}^p$ , the projection scores  $e_i, r_j \in \mathbb{R}^s$ , and the regularization coefficient  $\Lambda \in \mathbb{R}^{s \times s}$  which is a positive diagonal matrix. The objective  $J$  in the primal optimization maximizes the projection variances of  $W_{e|X}^\top \phi_q(\boldsymbol{x}_i), W_{r|X}^\top \phi_k(\boldsymbol{x}_j)$  regarding queries and keys, and also involves a regularization term coupling the projections. The corresponding solution in the dual is characterized by the right and left singular vectors, which captures the directions with maximal projection variances, w.r.t. rows (queries) and columns (keys) of the attention kernel matrix. Thus, with the formulated primal optimization in (6), the learning in self-attention is interpreted as a SVD problem on the attention matrix.
For clarity, we elaborate our primal optimization problem in (6) as follows.  $i)$  The projection weights are data-dependent, where  $f(X) \coloneqq F_{X} \in \mathbb{R}^{N \times p}$  denotes a transformation matrix containing the information of the sequence data  $X \coloneqq [\pmb{x}_1, \dots, \pmb{x}_N]^\top \in \mathbb{R}^{N \times d}$ . Notably,  $F_{X}$  is a constant matrix once given  $X$ , and we let it linearly depend on  $X$  in experiments. Moreover, when  $F_{X}$  is chosen as the identity matrix, it reconciles to the common setups in kernel methods in primal. ii) The feature maps related to queries and keys, respectively, are defined as  $\phi_q(\pmb{x}_i) \coloneqq g_q(q(\pmb{x}_i))$ ,  $\phi_k(\pmb{x}_i) \coloneqq g_k(k(\pmb{x}_i))$ , where  $g_q(\cdot): \mathbb{R}^{d_q} \to \mathbb{R}^p$  and  $g_k(\cdot): \mathbb{R}^{d_k} \to \mathbb{R}^p$  denote the mappings composited on the linear projections  $q(\cdot)$  and  $k(\cdot)$  in (1) of queries and keys. Note that we leave the choice of  $g_{q}$  and  $g_{k}$  later explained in Remark 4.1. iii) By projecting  $\phi_q(\pmb{x}_i)$ ,  $\phi_k(\pmb{x}_j) \in \mathbb{R}^p$  with weights  $W_{e|X}, W_{r|X} \in \mathbb{R}^{p \times s}$ , we obtain the projection scores  $e_i, r_j \in \mathbb{R}^s$  along the  $s$  directions, usually  $s < p$ , which corresponds to the number of singular values of the induced kernel matrix in the dual optimization (7).
Remark 3.1 (Variance maximization objective). In the formulated KSVD problem, the objective in the primal optimization (6) jointly maximizes the variances of the two projections  $\mathbf{e}_i$ ,  $\mathbf{r}_j$  in the feature spaces determined by  $\phi_q$ ,  $\phi_k$  along the directions  $W_{e|X}$ ,  $W_{r|X}$ . Within this context,  $\mathbf{e}_i$ ,  $\mathbf{r}_j$  learn to capture maximal information mutually residing in  $\phi_q$  and  $\phi_k$  regarding the queries and keys.
With Lagrangian duality and KKT conditions, we prove that the dual optimization problem to (6) leads to a shifted eigenvalue problem corresponding to the SVD on the asymmetric attention kernel  $K$ , given by Theorem 3.2. The proof is provided in the Supplementary Material.
Theorem 3.2 (Dual optimization problem of KSVD in self-attention). With Lagrangian duality and the KKT conditions, the dual optimization problem of (6) leads to the shifted eigenvalue problem:
$$
K H _ {r} = H _ {e} \Sigma , \tag {7}
$$
$$
K ^ {\top} H _ {e} = H _ {r} \Sigma ,
$$
Figure 2: An illustration of Primal-Attention and canonical self-attention. Left: in canonical self-attention, the asymmetric attention matrix  $K$  can be induced by two feature maps  $\phi_q$ ,  $\phi_k$  through kernel trick from RKBS. The values  $v(X)$  serve as the dual variables projecting the kernel matrix  $K$  to the attention output. The time and space complexity are  $\mathcal{O}(N^2 d_v)$  and  $\mathcal{O}(N^2 + N d_v)$ . Right: in our Primal-Attention, we use the two feature maps  $\phi_q$ ,  $\phi_k$  in the primal to present the attention outputs, which are projected through the primal variables  $W_{e|X}$ ,  $W_{r|X}$ . The time and space complexity are  $\mathcal{O}(N p s)$  and  $\mathcal{O}(2 N p + 2 p s)$ . To align with the output size in the canonical self-attention, we add a linear map mapping from  $2s$  to  $d_v$  with negligible memory increase after Primal-Attention's output.
where  $\Sigma \in \mathbb{R}^{s\times s}$  is a positive diagonal matrix, and  $H_{e} = [h_{e_{1}},\dots ,h_{e_{N}}]^{\top}\in \mathbb{R}^{N\times s}$ ,  $H_{r} = [h_{r_{1}},\ldots ,h_{r_{N}}]^{\top}\in \mathbb{R}^{N\times s}$  are the dual variables serving as the left and right singular vectors, respectively. The kernel trick to the asymmetric kernel matrix  $K$  is interpreted as  $K_{ij} = \langle f(X)g_q(q(\pmb {x}_i)),f(X)g_k(k(\pmb {x}_j))\rangle =:\langle \phi_q'(\pmb {x}_i),\phi_k'(\pmb {x}_j)\rangle$
As shown in Theorem 3.2, the solutions collect non-zero  $\Lambda$  in (6) such that  $\Sigma = \Lambda^{-1}$ . Based on Lanczos' decomposition in Theorem 2.2, we can then associate  $\Sigma$  with the non-zero singular values of the attention kernel  $K$  in (7), and  $H_{e}, H_{r}$  with the left and right singular vectors of  $K$ , such that  $K = H_{e} \Sigma H_{r}^{\top}$ . Therefore, formulas (6) and (7) provide the optimization problems of performing KSVD with the attention kernel matrix  $K$  in the primal and in the dual, respectively.
Self-attention as KSVD dual representation Firstly, we provide the primal-dual model representation of the derived KSVD problem. Secondly, we show that the dual representation of the model corresponds to the canonical self-attention. The derivation details involving the KKT conditions are provided in the Supplementary Material.
Remark 3.3 (Primal-dual representations of KSVD in self-attention). In the KSVD formulations for the asymmetric kernel matrix in self-attention, with KKT conditions, the projection scores can be either represented in the primal using explicit feature maps or in the dual using kernel functions:
$$
\text {P r i m a l :} \quad \left\{ \begin{array}{l} e (\boldsymbol {x}) = W _ {\left. e \right| X} ^ {\top} \phi_ {q} (\boldsymbol {x}) \\ r (\boldsymbol {x}) = W _ {r | X} ^ {\top} \phi_ {k} (\boldsymbol {x}) \end{array} , \quad D u a l: \quad \left\{ \begin{array}{l} e (\boldsymbol {x}) = \sum_ {j = 1} ^ {N} h _ {r _ {j}} \kappa (\boldsymbol {x}, \boldsymbol {x} _ {j}) \\ r (\boldsymbol {x}) = \sum_ {i = 1} ^ {N} h _ {e _ {i}} \kappa (\boldsymbol {x} _ {i}, \boldsymbol {x}). \end{array} \right. \right. \tag {8}
$$
Remark 3.4 (Correspondence of KSVD and canonical self-attention output). Recall the output  $\mathbf{o}_i$  of canonical self-attention (3), it corresponds to the dual representation of the projection score  $e(\pmb{x})$  in (8), i.e.,  $\pmb{o}_i \triangleq e(\pmb{x}_i)$ . When the values  $\{v(\pmb{x}_j)\}_{j=1}^N$  in canonical self-attention are chosen as the dual variables  $\{\pmb{h}_{r_j}\}_{j=1}^N$ , i.e.,  $\pmb{h}_{r_j} := v(\pmb{x}_j)$ ,  $j = 1, \dots, N$ , the values  $\{v(\pmb{x}_j)\}_{j=1}^N$  play the role of the right singular vectors of  $K$ .
From the perspective in Remark 3.4, the optimization goal in self-attention is interpreted to jointly capture the maximal variances of  $e_i$ ,  $r_j$  as in (6), where the projection scores can be denoted as  $e_i := e(\boldsymbol{x}_i)$ ,  $r_j := r(\boldsymbol{x}_j)$  through the representations (8). However, the canonical self-attention only outputs the  $e_i$ -score ( $o_i \triangleq e_i$ ). In this sense, the output of the canonical self-attention only considers the projection scores involving the right singular vectors of the asymmetric attention kernel  $K$ .
4 Primal-Attention
Modelling It is quite remarkable that the attention output can be equivalently represented without the kernel expressions, avoiding the heavy computation of the kernel matrix. Within the context of KSVD, we further observe that there exists another set of projections  $\boldsymbol{r}_j$  regarding the left singular vectors in  $\boldsymbol{h}_{e_i}$  as in (8), providing extra information residing in the asymmetric kernel matrix  $K$ .
We derive a novel attention mechanism by leveraging the primal representation of KSVD, namely, Primal-Attention, where two explicit feature maps  $\phi_q,\phi_k$  are adopted. To fully exploit the asymmetry in the kernel matrix of self-attention, Primal-Attention concatenates the two sets of projections using both left and right singular vectors, and thus formulates the attention outputs as follows:
$$
\boldsymbol {o} _ {i} := \left[ \boldsymbol {e} _ {i}; \boldsymbol {r} _ {i} \right] = \left[ W _ {e | X} ^ {\top} \phi_ {q} \left(\boldsymbol {x} _ {i}\right); W _ {r | X} ^ {\top} \phi_ {k} \left(\boldsymbol {x} _ {i}\right) \right] = \left[ W _ {e} ^ {\top} f (X) g _ {q} \left(q \left(\boldsymbol {x} _ {i}\right)\right); W _ {r} ^ {\top} f (X) g _ {k} \left(k \left(\boldsymbol {x} _ {i}\right)\right) \right]. \tag {9}
$$