SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Primal-Attention, the projection weights  $W_{e|X}, W_{r|X}$  in the primal play the role as the counterparts of the values in the dual. Given  $f(X) \eqqcolon F_X$  an identity matrix, our KSVD problem in (6) boils down to the data-independent projection weight case as in [27], which can thereby be regarded as a special case of our derived KSVD in Primal-Attention. In this case, the kernel trick in the asymmetric attention kernel becomes  $K_{ij} = \langle g_q(q(\pmb{x}_i)), g_k(k(\pmb{x}_j)) \rangle = \langle \phi_q(\pmb{x}_i), \phi_k(\pmb{x}_j) \rangle$ ,  $i, j = 1, \dots, N$ .
Remark 4.1 (Choices of  $\phi_q, \phi_k$  for non-linearity). The canonical self-attention adopts softmax for introducing non-linearity to the attention score matrix; within our setups and kernel trick, it can be viewed as:  $\kappa(\pmb{x}_i, \pmb{x}_j) = \hat{D}^{-1}\langle \phi_q(\pmb{x}_i), \phi_k(\pmb{x}_i) \rangle$  where  $\phi_q(\pmb{x}) \coloneqq g(q(\pmb{x}))$ ,  $\phi_k(\pmb{x}) \coloneqq g(k(\pmb{x}))$  with  $g(z) \coloneqq \exp(-\|z\|^2/2)(\exp(\pmb{w}_1^\top z), \ldots, \exp(\pmb{w}_p^\top z))$ ,  $\pmb{w}_i \sim \mathcal{N}(0, I_{d_q})$  and  $d_q = d_k$  [14], and  $\hat{D} \coloneqq \mathrm{diag}(\phi_q(X)(\phi_k(X)^\top \mathbf{1}_N))$ . The projection scores then correspond to  $e(\pmb{x}) = \hat{D}^{-1/2}W_{e|X}^\top\phi_q(\pmb{x})$  and  $r(\pmb{x}) = \hat{D}^{-1/2}W_{r|X}^\top\phi_k(\pmb{x})$ . In this case, two exponential feature maps need to be constructed and the normalization factor  $\hat{D}$  to all samples needs to be computed. In this paper, we consider feature maps related to the Cosine similarity kernel on queries and keys, such that  $\phi_q(\pmb{x}) = g_q(q(\pmb{x})) \coloneqq q(\pmb{x}) / \|q(\pmb{x})\|_2$  and  $\phi_k(\pmb{x}) = g_k(k(\pmb{x})) \coloneqq k(\pmb{x}) / \|k(\pmb{x})\|_2$  in all experiments. It is easy to implement and able to bring both non-linearity and normalization to the feature maps.
**Optimization** The KSVD problem can either be optimized in the primal as a constrained optimization problem, or in the dual as a shifted eigenvalue problem (SVD on the kernel matrix  $K$ ). In Primal-Attention, we perform the optimization in the primal. In the following Lemma 4.2, we prove the zero-value property of the primal optimization objective  $J$  in (6) when it is evaluated at the stationary solution in (7). The proof is provided in the Supplementary Material.
Lemma 4.2 (A zero-value objective with stationary solutions). The solutions of  $H_{e}, H_{r}, \Sigma$  to the shifted eigenvalue problem in the dual optimization (7) lead to the zero-value objective  $J$  in the primal optimization (6).
With the property in Lemma 4.2, rather than solving SVD problem on the kernel matrix  $K$  in the dual, the optimization of Primal-Attention is realized by minimizing the primal objective to zero:
$$
\min  L + \eta \sum_ {l} J _ {l} ^ {2}, \tag {10}
$$
where  $L$  is the task-oriented loss, e.g., the cross-entropy loss for classification tasks, the summation term over  $l$  denotes the additive objectives  $J_{l}$  of all attention blocks using the proposed Primal-Attention, where  $J_{l}$  is implemented as the mean over all heads, and  $\eta > 0$  is the regularization coefficient. Specifically, for each head in the self-attention using Primal-Attention, we have
$$
\begin{array}{l} J \left(W _ {e}, W _ {r}, \Lambda\right) = \frac {1}{2} \sum_ {i = 1} ^ {N} e _ {i} ^ {\top} \Lambda e _ {i} + \frac {1}{2} \sum_ {j = 1} ^ {N} r _ {j} ^ {\top} \Lambda r _ {j} - \operatorname {T r} \left(W _ {e} ^ {\top} W _ {r}\right) \\ = \frac {1}{2} \sum_ {i = 1} ^ {N} \| \left(W _ {e | X} \Lambda^ {\frac {1}{2}}\right) ^ {\top} \phi_ {q} (\boldsymbol {x} _ {i}) \| _ {2} ^ {2} + \frac {1}{2} \sum_ {j = 1} ^ {N} \| \left(W _ {r | X} \Lambda^ {\frac {1}{2}}\right) ^ {\top} \phi_ {k} (\boldsymbol {x} _ {j}) \| _ {2} ^ {2} - \operatorname {T r} \left(W _ {e} ^ {\top} W _ {r}\right), \\ \end{array}
$$
where  $\Lambda$  is automatically determined by the optimization. KSVD optimization of Primal-Attention is easy to implement by adding a regularization loss. Hence, Primal-Attention not only represents self-attention with KSVD formulation in the primal, but utilizes the optimization of KSVD by regularizing the attention, satisfying the stationarity conditions of KSVD when reaching a zero-value objective.
In this section, we verify the effectiveness of our Primal-Attention applied in Transformers on five well-established benchmarks: time series, long sequence modelling, reinforcement learning, image classification and language modelling. Notably, we consider two types of Transformers applied with our Primal-Attention, i.e., PrimalFormer (Primal.) and Primal+.i) In PrimalFormer, PrimalAttention (9) is applied to all attention layers, regularized with the KSVD loss (11). This setup is
Table 1: Test accuracy (\%) on the UEA time series classification archive benchmark [31] with comparisons to canonical Transformer (Trans.), Linear Transformer (Linear.), Reformer (Re.), Longformer (Long.), Performer (Per.), cosineFormer (Cos.), Flowformer (Flow.), YOSO-E and SOFT.
Dataset / Model	Trans. [1]	Linear. [33]	Re. [34]	Long. [35]	Per. [14]	YOso-E [36]	Cos. [37]	SOFT [38]	Flow. [11]	Ours
Primal.	Primal.+Trans.
EthanolConcentration	32.7	31.9	31.9	32.3	31.2	31.2	32.3	33.5	33.8	33.1	35.4
FaceDetection	67.3	67.0	68.6	62.6	67.0	67.3	64.8	67.1	67.6	67.1	63.8
HandWriting	32.0	34.7	27.4	39.6	32.1	30.9	28.9	34.7	33.8	29.6	28.7
HeartBeat	76.1	76.6	77.1	78.0	75.6	76.5	77.1	75.6	77.6	76.1	77.1
JapaneseVowels	98.7	99.2	97.8	98.9	98.1	98.6	98.3	99.2	98.9	98.4	98.9
PEMS-SF	82.1	82.1	82.7	83.8	80.9	85.2	83.2	80.9	83.8	89.6	90.2
SelfRegulationSCP1	92.2	92.5	90.4	90.1	91.5	91.1	91.1	91.8	92.5	92.5	92.5
SelfRegulationSCP2	53.9	56.7	56.7	55.6	56.7	53.9	55.0	55.6	56.1	57.2	56.1
SpokenArabicDigits	98.4	98.0	97.0	94.4	98.4	98.9	98.4	98.8	98.8	100	100
UWaveGestureLibrary	85.6	85.0	85.6	87.5	85.3	88.4	85.6	85.0	86.6	86.3	88.4
Average Accuracy	71.9	72.4	71.5	72.0	71.9	72.2	71.5	72.2	73.0	73.0	73.1
Table 2: Efficiency comparisons on running time and memory consumption on the UEA benchmark [31] where running time (s/Epoch), the peak training memory usage (GB) are given. We report how much faster and how less memory each model uses than Transformer.
Dataset / seq. length	Ethanol. 1751	Face. 62	Hand. 152	Heart. 405	Jap. 26	PEMS-SF 144	SCP1 896	SCP2 1152	Arabic. 83	UWave 315	Avg. Time (s/Epoch)	Memory (GB)
Trans. [1]	4.3 (1×)	10.7 (1×)	0.3 (1×)	0.7 (1×)	0.5 (1×)	0.6 (1×)	1.8 (1×)	1.8 (1×)	3.7 (1×)	0.3 (1×)	2.5 (1×)	10.9 (1×)
Flow. [11]	2.4 (1.8×)	9.8 (1.1×)	0.3 (1.0×)	0.7 (1.0×)	0.6 (0.8×)	0.7 (0.9×)	1.4 (1.3×)	1.3 (1.4×)	4.4 (0.8×)	0.3 (1.0×)	2.2 (1.1×)	2.8 (3.9×)
Primal.+Trans.	3.3 (1.3×)	5.7 (1.9×)	0.3 (1.0×)	0.7 (1.0×)	0.5 (1.0×)	0.7 (0.9×)	1.6 (1.1×)	1.6 (1.1×)	4.0 (0.9×)	0.3 (1.0×)	1.9 (1.3×)	6.5 (1.7×)
Primal.	2.3 (1.9×)	6.9 (1.6×)	0.4 (0.8×)	0.4 (1.8×)	0.5 (1.0×)	0.8 (0.8×)	1.4 (1.3×)	1.3 (1.4×)	4.5 (0.8×)	0.4 (0.8×)	1.9 (1.3×)	2.7 (4.0×)
preferred when data shows relatively strong low-rank structure or the model redundancy is quite substantial. ii) Primal.+ refers to the baselines from the Transformer family with the last layer replaced by our Primal-Attention, which is in favour in large-scale data and complicated tasks where less information compression is desired in the learning, especially in shallower layers. To specify the Transformer backbone, we denote our method by "Primal.+Backbone" herein. Primal.+ serves as a flexible variant combined with different Transformer backbones, and with KSVD optimization applied through an implicit low-rank regularization loss, advocating to learn more informative features, as in Figure 1. The two main hyper-parameters of our method are the coefficient  $\eta$  in (10) and the number of projection directions  $s$  of KSVD in (6). In data-dependent cases with  $f(X)$ , we take a subset of  $X$  by uniformly sampling  $n = \min\{s*10, N\}$  points from  $X$  for efficiency aspects, as the main patterns of a matrix can be retained with random linear projections shown by the Johnson-Lindenstrauss lemma [30]. Details are given in the Supplementary Material.
UEA Time Series Classification UEA Time Series Classification Archive [31] is the benchmark for the evaluation on temporal sequences. Following [11], we select 10 multivariate datasets with pre-processed data according to [32], and employ 2-layer Transformer as backbone with the hidden dimension 512 on 8 heads and the embedding dimension 64 for self-attention. The hyper-parameter search of our method is with  $\eta \in \{0.1, 0.2, 0.5\}$ ,  $s \in \{20, 30, 40\}$ . Experiments are run on one NVIDIA Tesla P100 SXM2 16GB GPU.
Table 1 reports the test accuracy of the compared recent methods, where the best result is in bold. Both our PrimalFormer and Primal.+ achieve comparable and better performance than the state-of-the-art results provided by Flowformer [11]. Notably, Primal.+ yields the best accuracy with  $1.2\%$  overall improvement upon the canonical Transformer [1], i.e., by replacing the softmax-based self-attention with our Primal-Attention in the last layer. This shows the promising potential of Primal-Attention in enhancing temporal modelling capacity upon the canonical softmax self-attention. It is worth mentioning that our PrimalFormer applying the KSVD optimization with low-rank regularization to all layers also obtains good performance. This could be due to the fact that these datasets are relatively simple where model redundancy can exist, so that appropriately imposing the low-rank regularization in KSVD does not harm the model expressiveness for the data. We also compare the running time and memory consumption with the canonical Transformer and Flowformer, which is proposed recently with state-of-the-art performances and efficiency. Both our Primal. and Primal.+Trans. consistently lead to improved efficiency than the canonical Transformer. In Primal., all attention layers are implemented with the proposed Primal-Attention, while Primal.+Trans. applies the Primal-Attention
Table 3: Test accuracy (\%) on the LRA benchmark [39] with comparisons to canonical Transformer (Trans.), Reformer (Re.), Performer (Per.), Linformer (Lin.), Nyströmformer (Nyström.), Longformer (Long.) and YOSO-E.
Dataset (seq. length)	Trans. [1]	Re. [34]	Per. [14]	Lin. [46]	Nyström. [45]	Long. [35]	YOSO-E [36]	Ours
Primal.	Primal.+Trans.
ListOps (2K)	37.1	19.1	18.8	37.3	37.2	37.2	37.3	37.3	37.3
Text (4K)	65.0	64.9	63.8	55.9	65.5	64.6	64.7	61.2	65.4
Retrieval (4K)	79.4	78.6	78.6	79.4	79.6	81.0	81.2	77.8	81.0
Image (1K)	38.2	43.3	37.1	37.8	41.6	39.1	39.8	43.0	43.9
Pathfinder (1K)	74.2	69.4	69.9	67.6	70.9	73.0	72.9	68.3	74.3
Average Accuracy	58.8	55.1	53.6	55.6	59.0	59.0	59.2	57.5	60.4
Table 4: Efficiency comparisons on running time and memory consumption on LRA [39].
Model	Time (s/1K-steps)	Memory (GB)
ListOps	Text	Retrieval	Image	Pathfinder	ListOps	Text	Retrieval	Image	Pathfinder
Trans. [1]	194.5 (1×)	694.8 (1×)	1333.7 (1×)	334.5 (1×)	405.5 (1×)	5.50 (1×)	21.24 (1×)	18.72 (1×)	5.88 (1×)	5.88 (1×)
Nyström. [45]	68.4 (2.8×)	120.9 (5.7×)	235.5 (5.7×)	179.5 (1.9×)	221.2 (1.8×)	0.89 (6.2×)	1.69 (12.6×)	3.29 (5.7×)	1.93 (3.0×)	1.93 (3.0×)
Lin. [46]	63.4 (3.1×)	116.5 (6.0×)	226.2 (5.9×)	158.5 (2.1×)	204.0 (2.0×)	1.73 (3.2×)	3.45 (6.2×)	6.33 (3.0×)	3.45 (1.7×)	3.45 (1.7×)
Per. [14]	83.8 (2.3×)	157.5 (4.4×)	320.6 (4.2×)	211.4 (1.6×)	278.1 (1.5×)	1.67 (3.3×)	3.34 (6.4×)	6.28 (3.0×)	3.34 (1.8×)	3.34 (1.8×)
Re. [34]	87.0 (2.2×)	168.5 (4.1×)	339.9 (3.9×)	223.7 (1.5×)	286.7 (1.4×)	1.64 (3.3×)	3.29 (6.5×)	6.09 (3.1×)	3.29 (1.8×)	3.29 (1.8×)
Primal.+Trans.	113.4 (1.7×)	367.6 (1.9×)	546.5 (2.4×)	212.1 (1.6×)	263.2 (1.5×)	5.24 (1.1×)	20.7 (1.0×)	18.59 (1.0×)	5.35 (1.1×)	5.35 (1.1×)
Primal.	56.5 (3.4×)	93.6 (7.4×)	185.3 (7.2×)	142.9 (2.3×)	180.0 (2.3×)	0.69 (7.9×)	1.37 (15.5×)	2.99 (6.3×)	1.39 (4.2×)	1.52 (3.9×)
in the last layer, making Primal. a more efficient model. We note that our Primal. further surpasses Flowformer in most cases regarding both running time and memory.
Long-Range Arena Benchmark Long-Range Arena (LRA) [39] is a benchmark for the longsequence scenarios, including equation calculation (ListOps) [40], review classification (Text) [41], document retrieval (Retrieval) [42], image classification (Image) [43] and image spatial dependencies (Pathfinder) [44]. We follow the settings in [45] with PyTorch. The Transformer backbone is set with 2 layers, hidden dimension 128 with 2 heads and embedding dimension 64 with mean pooling, where Reformer (Re.) uses 2 hashes, Performer (Per.) has 256 random feature dimensions and Linformer (Lin.) uses a projection dimension of 256. Our hyper-parameter is set from  $\eta \in \{0.05,0.1\}$ ,  $s \in \{20,30\}$ . Experiments are conducted on a single NVIDIA Tesla V100 SXM2 32GB GPU.
From the reported top-1 test accuracy in Table 3, our PrimalFormer shows better accuracy while achieves top efficiency (see Table 4) than several efficient self-attention counterparts including Reformer, Performer and Linformer. Notably, our model Primal.+Trans. achieves the state-of-the-art accuracy of  $60.4\%$ , which is  $1.6\%$  higher than Transformer, and  $1.2\%$  higher than the currently best YOSO-E, showing that the deployed Primal-Attention is able to boost the performance upon canonical Transformer distinctively. On top of it, Primal.+Trans. has distinctively higher efficiency and require much less memory than the canonical Transformer as in Table 4, and our Primal. outperforms all compared variants of efficient Transformer. Compared to Table 1 with simpler data, Primal.+Trans. further achieves performance gain over PrimalFormer. The reason can be that the shallower (the first) layer needs more model capacity for depicting the data patterns in the learning, while the deep (the second) layer captures less detailed features and is regularized with more informative feature learning through our Primal-Attention. The efficiency of our Primal-Attention are further pronounced with longer sequences than that of the UEA benchmark in Table 2.
Reinforcement Learning We consider the offline reinforcement learning (RL) performance of our methods on D4RL benchmark [47] designed for continuous control tasks. We choose three different environments controlling the robot movement: HalfCheetah, Hopper and Walker. Our experiments are conducted on three datasets pre-collected under different policies: Medium-Expert, Medium and Medium-Replay. We follow the experimental settings in Flowformer [11], and also compare to Decision Transformer (DT) [10] which is commonly considered as the baseline with state-of-the-art performances based on the canonical attention. Note that offline RL is an auto-regressive task and our Primal-Attention is adapted to its causal version, as described in the Supplementary Material. We adopt the architecture of 3 layers, hidden dimension 256 with 4 heads, and the embedding dimension 64. Our hyper-parameters are set as  $\eta = 0.05$ ,  $s \in \{32, 64, 96\}$ . Each experiment is run with 3 different seeds on one NVIDIA Tesla P100 SXM2 16GB GPU.
Table 5: Rewards on D4RL [47] datasets. We report mean and variance for 3 seeds. A higher reward and a lower deviation indicates better performance. We consider Decision Transformer (DT), Linear Transformer (Linear.), Reformer (Re.), Performer (Per.), cosFormer (Cos.) and Flowformer (Flow.).
Dataset	Environment	DT [10]	Linear. [33]	Re. [34]	Per. [14]	Cos. [37]	Flow. [11]	Ours Primal.+DT
Medium -Expert	HalfCheetah	83.8±3.3	78.2±3.2	81.5±1.6	85.1±2.1	85.5±2.9	90.8±0.4	77.8±22.1
Hopper	104.0±2.5	107.2±0.9	104.2±9.8	93.5±13.9	98.1±7.4	109.9±1.0	111.5±0.2
Walker	107.7±0.6	67.2±27.3	71.4±1.8	72.6±2.4	100.5±14.5	108.0±0.4	108.9±0.1
Medium	HalfCheetah	42.4±0.1	42.3±0.2	42.2±0.1	42.1±0.2	42.1±0.3	42.2±0.2	43.0±0.0
Hopper	64.2±1.1	58.7±0.4	59.9±0.7	59.7±7.5	59.8±3.8	66.9±2.5	74.5±0.6
Walker	70.6±3.2	57.9±10.6	65.8±4.9	63.3±10.7	71.4±1.2	71.7±2.5	77.9±7.8
Medium -Replay	HalfCheetah	34.6±0.6	32.1±1.5	33.6±0.7	31.7±0.9	32.8±3.6	34.7±1.5	38.9±0.4
Hopper	79.7±7.4	74.3±7.0	66.1±2.6	64.6±24.2	59.3±16.5	75.5±14.5	88.5±12.5
Walker	62.9±5.0	62.1±7.4	50.1±3.5	61.3±6.7	60.5±9.9	62.0±3.1	76.8±10.3
Average Reward	72.2±2.6	64.4±6.5	63.9±2.9	63.8±7.6	67.8±7.6	73.5±2.9	77.5±6.0
Table 6: Efficiency comparisons on running time (s/1K-steps) and memory (GB) on D4RL [47].
Model	Medium-Expert	Medium	Medium-Replay
Time	Memory	Time	Memory	Time	Memory
DT (average reward: 72.2)	20.8	0.3	20.8	0.3	20.8	0.3
Flow. (average reward: 73.5)	54.4	1.5	54.4	1.5	54.3	1.5
Primal.+DT (average reward: 77.5)	23.5	0.3	23.4	0.3	23.3	0.3
Results in Table 5 demonstrate that our Primal.+ outperforms all compared methods with a distinctly higher average reward. Specifically, our Primal.+DT reaches 4.0 points higher than the state-of-the-art Flowformer [11]. Compared to the baseline Decision Transformer (DT), our Primal.+ only replaces the self-attention in the top layer and keeps other structures the same and manages to improve the average reward by 5.3 points. This verifies the effectiveness of our Primal-Attention and the benefits of our low-rank regularization of KSVD for the generality of DT in offline reinforcement learning. We evaluate the efficiency with comparisons to DT and Flowformer in Table 6, which give distinctively better results than other baselines in Table 5. Our Primal.+DT achieves comparable time and memory efficiency as the most efficient baseline DT, while Flowformer shows significantly lower efficiency in both aspects. Recall that our Primal.+DT achieves an average reward of 77.5, which is 5.3 points higher than DT and 4.0 points higher than Flowformer.
Large-scale Experiments We evaluate the capability of our Primal.+ model with DeiT-Small/16 [7] as backbone on ImageNet-100 [48] and ImageNet-1K [23] for image classification task. We also experiment with the language modelling task on WikiText-103 [49]. On ImageNet, we train DeiT-Small/16 and our Primal.+DeiT-Small/16 from scratch following the same training protocols in [7] with 4 NVIDIA Tesla V100 SXM2 32GB GPUs. Our hyper-parameters are chosen as  $\eta = 0.05$ ,  $s \in \{32, 64, 96\}$ . On WikiText-103, we follow the setting in [50] where the sequence length is set as 512, the model consists of 6 decoder layers with 8 heads. We implement the causal version of Primal-Attention in the last layer, i.e., Primal.+Trans with  $\eta = 0.1$ ,  $s = 30$ . Models are trained from scratch on 4 NVIDIA Tesla V100 SXM2 32GB GPUs for 150K updates after 6K-steps warm-up.
Table 7(a) provides the test accuracy, training time and memory on a single V100 GPU with batch size 256 on both ImageNet-100 and ImageNet-1K datasets. There is only one set of time and memory since models follow the same training protocols on both datasets. Our Primal.+ achieves better accuracy than DeiT-Small/16 on ImageNet-100. It also achieves the same accuracy as baseline with less training time and memory on ImageNet-1K. On WikiText-103 in Table 7(b), our method with default setups significantly reduces the perplexity by 2.0 points than Transformer, and achieves comparable performances with the well-tuned Flowformer, a latest SoTA model, with enhanced efficiency. Results show that using Primal-Attention in the last layer helps the overall performance.
Ablation Study on Using Two Projections  $e(x), r(x)$  We conduct an ablation study with (w/) and without (w/o) projection scores, i.e.,  $r$ -scores, involving the left singular vectors, on LRA [39]. Table 8 shows that using both projections (w/  $r$ -scores) helps boost performances, verifying the effectiveness of learning with the two sides of singular vectors on an asymmetric attention kernel.
Table 7: Results on large-scale experiments including image classification and language modelling.
(a) Test accuracy  $(\%)$  and efficiency on ImageNet-100 [48] and ImageNet-1K [23].
Model	ImageNet-100 (Top-1 Acc.)	ImageNet-1K (Top-1 Acc.)	Time (s/1K-steps)	Memory (GB)
DeiT-Small/16	74.2	79.8	2425.5	14.2
Primal.+DeiT-Small/16	75.7	79.8	2330.2	14.0
(b) Results on WikiText-103 [49].
Model	Perplexity (↓)	Time (s/1K-steps)	Memory (GB)
Trans. [1]	33.0	3108.4	9.0
Flow. [11]	30.8	3998.4	10.5
Primal.+Trans.	31.0	3104.0	8.9
Table 8: Ablation on  $r$  -scores involving left singular vectors on LRA [39] with test accuracy (%).
Model	r-scores	ListOps	Text	Retrieval	Image	Pathfinder	Avg. Acc
Primal.	w/o	36.8	52.4	58.2	30.5	50.2	45.6
w/	37.3	61.2	77.8	43.0	68.3	57.5
Primal.+Trans.	w/o	37.1	65.1	79.2	42.8	72.8	59.4
w/	37.3	65.4	81.0	43.9	74.3	60.4
Since the pioneering work [12], the kernel-based approaches have become popular in Transformers, in which the kernel interpretation on the attention matrix has been shed light on. FourierFormer [17] treats the canonical self-attention as non-parametric regression with methodologies for symmetric kernels. [18] considers relative positional embedding with conditional positive definite kernel. [19] treats self-attention operation as support vector regression without considering the asymmetry in the deployed kernel methods, and the supervised regression is not applied in optimizing the attention either. [51] addresses the issue of asymmetry, however, it resorts to symmetrization by replacing the softmax attention with an approximated symmetric one, thereby still dismissing the asymmetry. These prior works deploy kernel-based techniques that are originally designed for symmetric kernels and request to suffice Mercer conditions, which is inconsistent with the asymmetric nature in self-attention, resulting in a nontrivial gap between the analytical understanding and numerical implementation towards revealing the rationale in Transformers. In [21], it leverages the kernel tricks from RKBS [22] that allows asymmetry and formulates attention as a binary kernel learning problem via empirical risk minimization. However, it is hard to find an explicit optimization accordingly in implementing Transformers. Nevertheless, [21] provides an analytical tool from the aspect of kernel representer theorem upon RKBS that allows asymmetry.
Much literature has also devoted to improving the efficiency of the attention computation through different approximation techniques. In the related works addressing the attention mechanism, Reformer [34] uses locally-sensitive hashing for sparse approximation. Performer [14] approximates self-attention matrix with random features. Linformer [46] considers low-rank approximation with the help of random projections. Nyströmformer [45] utilizes the Nyström method by down sampling the queries and keys in the attention matrix. [52] incorporates sparsity prior on attention. These works pose the focus on reducing the computation of the attention kernel matrix from the canonical self-attention. Hence, these works all address how to solve the problem in the dual form involving the kernel matrix, while we work in a significantly different way, that is, in the primal form.
In this paper, we interpret the self-attention in Transformers with asymmetric kernels and construct a learning framework with SVD on asymmetric kernels (KSVD) under the setups of LSSVM. Within the context of KSVD, a primal-dual model representation is formulated for self-attention and a novel attention mechanism (Primal-Attention) is proposed accordingly by leveraging the primal representation. It is quite significant that with Primal-Attention, not only the computation of the attention kernel matrix in the dual can be avoided, but also the cast unsupervised KSVD optimization can be efficiently incorporated into the training through an additional regularization loss for more informative low-rank property. The analytical derivations and numerical evaluations demonstrate our great potentials in bridging explicit model interpretability and state-of-the-art performances. Future works can include developing different variants with the low-rank property, e.g., robust Transformers, investigating more general applications of Primal-Attention to a wide range of architectures and tasks.