PolaFormer: Polarity-aware Linear Attention for Vision Transformers
Authors: Weikang Meng
Yadan Luo
Xin Li
Dongmei Jiang
Zheng Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Linear attention
polarity-aware attention
efficient transformer
Primary area: foundation or frontier models, including LLMs

Abstract: Linear attention has emerged as a promising alternative to softmax-based attention, leveraging kernelized feature maps to reduce complexity from quadratic to linear in sequence length. However, the non-negative constraint on feature maps and the relaxed exponential function used in approximation lead to significant information loss compared to the original query-key dot products, resulting in less discriminative attention maps with higher entropy. To address the missing interactions driven by negative values in query-key pairs, we propose a polarity-aware linear attention mechanism that explicitly models both same-signed and opposite-signed query-key interactions, ensuring comprehensive coverage of relational information. Furthermore, to restore the spiky properties of attention maps, we provide a theoretical analysis proving the existence of a class of element-wise functions (with positive first and second derivatives) that can reduce entropy in the attention distribution. For simplicity, and recognizing the distinct contributions of each dimension, we employ a learnable power function for rescaling, allowing strong and weak attention signals to be effectively separated. Extensive experiments demonstrate that the proposed PolaFormer improves performance on various vision tasks, enhancing both expressiveness and efficiency by up to 4.6%.

Published as a conference paper at ICLR 2025
POLAFORMER: POLARITY-AWARE LINEAR ATTENTION FOR VISION TRANSFORMERS
Weikang Meng Yadan Luo",, Xin Li,, Dongmei Jiang", Zheng Zhang'*
1 Harbin Institute of Technology, Shenzhen, China
2 Pengcheng Laboratory, China
3 UQMM Lab, University of Queensland, Australia
zacharymengwk @ gmail. com darrenzz2198 gma com
ABSTRACT
Linear attention has emerged as a promising alternative to softmax-base atten- tion, leveraging kernelized feature maps to reduce complexity from quadratic to linear in sequence length. However, the non-negative constraint on feature maps and the relaxed exponential function used in approximation lead sig- nificant information loss compared to the original query-key dot products, re- sulting in less discriminative attention maps with higher entropy. To address the missing interactions driven by negative values in query-key pairs, we pro- linear attention mechanism that explicitly models both same-signed and epposite-signed query-key interactions, ensuring comprehen- sive coverage of relational information. Furthermore, to restore the spiky prop- erties of attention maps, we provide theoretical analysis proving the exis- tence of a class element-wise functions (with positive first and second deriva- tives) that reduce entropy in the attention distribution. For simplicity, and recognizing the distinct contributions each dimension, we employ a learn- able power function for rescaling. allowing strong and weak attention sig- nals to be effectively separated. Extensive experiments demonstrate that the proposed PolaFormer improves performance ing both expressiveness and efficiency by up to 4.6%. Code is available at https: /tls:///iit //gpt/gtttu///t// com /wtpaaaaaaaaaaaaaaaaaaaaa /ool//ooooo loala aFormer.
pose polarity-
various vision tasks, enhanc-
Original Image
Softmax Attention
Linear Attention
Flatten
PolaFormer
Figure 1: Attention weight visualization. Unlike prior linear attention approaches (KKatharopoulos el al., 2020) the 3rd and (Han el al., 2023a) 4th plots) that generate uniform responses, the proposed PolaFormer captures a more accurate query-key interaction with lower entropy, closely resembling softmax while maintaining linear complexity.
INTRODUCTION
Transformers have demonstrated remarkable success across a broad range vision tasks (Yuan et al., b021b; Cai et al., 2022). The core component, dot-product attention with softmax normaliza- tion, enables transformers to capture long-range dependencies effectively. However, this comes at the cost of quadratic complexity O(N2) in relation to the sequence length N. resulting in consider- able computational overhead particularly when processing long-sequence videos or high-resolution
"Correspondence Zheng Zhang <darrennz2196) @gmail.cmmm>>>>

---

Published as a conference paper at ICLR 2025
Blementwise Multipication
V
mwweyy
Pos tive Part
K
V
Negative Part
Concat
Compute Same signed
V
Vo
Part to Compute Opposte signee
The overall framework of PolaFormer. Our framework explicitly separates query-key pairs based on their polarity into two distinct streams, with scaled outputs controlled by the learnable sign-aware matrices G' and Go for same-signed and opposite-signed components, respectively. A channel-wise power function with the learnable exponent p is employed to learn the rescaling process, capturing the sharpness characteristic of softmax
Figure 2:
images. This limits their efficiency resource-constrainee environments, making practical deploy- ment difficult in such scenarios.
To mitigate this challenge, various methods have been proposed to accelerate attention computa- tion. Techniques such localized or sparse attention reduce the number of tokens key-value pairs by restricting attention to smaller windows or sparser patterns, thereby lowering the overall computational costs. While effective, these methods often sacrifice important contextual informa- tion, leading to unstable convergence behaviors and performance degradation. As more principled solution, linear attention (Katharopoulos et al., 2020) replaces the Softmax operation in the query- key dot product with kernalized feature maps, effectively reducing time and space complexity from O(N'd) to ((Nd).. where denotes the feature map dimension. Recent advances in linear atten- tion have centered on designing two key components. i.e., (1) non-negative feature maps such as ELU+1 (Katharopoulos al., 2020) and ReLU (Qin al., 2022) and (2) kernel functions includ- ing Gaussian kernels (Chen et al., 2021), Laplace kernels (Verma, 2021) and polynomial kernels (Kacham al., 2024), to preserve the core properties of the original Softmax function while im- proving computational efficiency.
Despite the efficiency gains, linear attention still falls short in expressive capacity compared to softmax-bast attention: As illustrated in Figure 1, it often yields more uniform attention weights across query-key pairs, thus resulting reduced specificity. For instance, when querying a particular region like bird wing. linear attention tends to activate key tokens from unrelated areas (e.g., poles) cqually, introducing noise that disrupts downstream vision tasks. Our analysis identifies two primary causes for this shortfall, both stemming from information loss during the Softmax approximation:
(1) Loss of Negative Values. Linear attention models that rely on non non-negative feature maps, such ReLU, fail to maintain consistency with the original query-key dot product. These feature maps retain only positive-positive interactions, while crucial negative-negative and positive-negative in- teractions completely dropped. This selective representation limits the model's ability to capture a comprehensive range of relationships, leading to diminished expressiveness and reduced discrim- inative power in the resulting attention maps.
(2) Loss of Attention Spikeness. Without the exponential scaling of softmax, linear attention leads more uniform weight distributions and lower entropy. This uniformity weakens the model's ability to distinguish between strong and weak query-key pairs, impairing its focus on important features and reducing performance in tasks requiring fine detail.
In this work, we propose a polarity-aware linear attention (PolaFormer) mechanism, designed to address the limitations of prior linear attention models by incorporating the previously omitted negative interactions. Unlike traditional approaches that only preserve positive-positive query-key interactions, PolaFormer explicitly separates query-key pairs based on their polarity- a full spectrum of same-signed (positive-positive, negative-negaatvv) and opposite-sinned (positive- negative, negative possiive- interactions as shown in Figure 2. These interactions are processed in two streams, allowing for more accurate reconstruction of the original softmax attention weights. To avoid unnecessary complexity, we split the value vector along the channel dimension, handling both types of interactions without introducing additional learnalle parameters. The outputs are then
2

---

Published as a conference paper at ICLR 2025
concatenated and scaled with learnable sign-aware matrix, ensuring faithful reconstruction of query-key relationships.
To mitigate the issue of uniform attention weights commonly observed in lincarized attention, provide a theoretical foundation showing that an element-wise function can rescale the query-key responses to reduce entropy, provided the function has positive first and second derivatives. This insight helps clarify why previous feature maps such as ReLU and ELU tend to elevate entropy, leading to overly smoothed attention distributions. For simplicity, we employ channel-wise learn- able power function for rescaling, which retains the sharpness of the exponential function inherent in Softmax. This enables the model capture spiky attention peaks, improving its ability to dis- tinguish between strong and weak responses. Together, these enhancements offer a more robust solution to bridging the gap between linearized and softmax-based attention. We conduct exten- sive experiments on various vision tasks and the Long Range Arena benchmark (Tay et al., 2021), demonstrating that our model enhances performance by up to 4.6% while preserving a superior balance between expressive capability and efficiency.
2 RELATED WORK
Efficient Vision Transformers. By cutting images into smaller patches and processing them as a sequence, Vision Transformers (ViT) (Dosovitskiy et al., 2021) successfully transfer transformer models (Vaswani et al., 2017) from language tasks vision tasks, and have achieved remarkable results. However, the quadratic complexity of the self-attention mechanism in ViT makes it expen- sive train. Existing works have made various improvements VIT for computational efficiency. Trin-Transformer (Liu et al., 2021) introduces a shifted windows scheme to limit self-attention computation to local windows. Pyramid Vision Transformer (PVT) (Wang et al., 2021) uses pro- gressive shrinking pyramid to reduce the computations of feature maps. Deit (Touvron et al., 2021) enables models to achieve competitive performance without pretraining on large datasets by utiliz- ing designed tokenization mechanisms and training strategies. However, these improvements do not solve the bottleneck of the self-attention mechanism, and quadratic complexity, thus the train- ing cost is still unaffordable the model scale increases. To address this issue, VMamba (Liu et al., 2024) extracts the information of the picture based on the spatial state model (SSM) encod- ing through serializing and scanning the picture, at the same time it inherits the linear complexity of SSM. VHeat (Wang et al., 2024) conceptualize image patches as heat sources and simulate the conduction process, and utilize DCT and IDCT operations to reduce the complexity These methods have just been proposed and are not yet as widely validated and deployed at scale as Transformers, model performance is also not significantly higher than the other models.
((N1.5).
Linear Attention. quaddatiiiic transformers focus on alleviating the inefficiency of the standard self-attention mechanism due to the softmax function and its quadratic complexity. A preferable solution is to use kernel-based similarities to reduce the complexity by approximating the softmax operator. The initial linear attention (Katharopoulos et al., 2020) proposes to substitute the Softmax function with linear dot-product of kernel feature maps, which facilitates reducing the complex- ity from (NN2) to O(N). Following this Softmax-free scheme, some variants of linear attention have been proposed by employing different kernel functions, such as ReLU(-)) (Shen et al., 2021) and 1 ELU(-) (Katharopoulos et al., 2020). Moreover, to fulfill the non-negative and distribution properties of attention matrix, Cosformer (Qin et al., 2022) combines the ReLU function and cos- based re-weighting mechanism to enhance the self-attention weighs with locality inductive biases. FLatten Transformer (Han et al., 2023a) extends ReLU(-)) with power operation to maintain both properties of attention weights, ie., non-negative and low-entropy. It is a practical way to use power function calculate the inner product to approximate exp. which is similar the use of power function to approximate max- maxx-ooolig proposed in R-MAC (Tolias et al., 2016). Recently, Agent Attention (Han et al., 2023b), a claimed generalized linear attention, introduces n agent tokens to aggregate features based on combination of Softmax and linear attention with O(Nnd) complex- ity. As both N and n increase simultaneously with the model size, the complexity of the generalized linear attention is not absolutely linear with respect to N. Notably, the balanced performance still relies on the softmax operator and additional agent tokens, which violates the original premise of linear attention, í.e., softmax-fre and linear complexity. Current kernel functions either suffer from performance degradation or introduce excessive computational overhead. We observed significant information loss in comparison original query-key dot products due to the non-negative constraint
3

---

Published as a conference paper at ICLR 2025
on attention weights and the intricate kernel designs aimed at achieving low entropy. This issue will be further addressed in the following sections of this work.
PRELIMINARY
In this section, we first highlight the inefficiency of the standard self self-attention mechanism, followed by a discussion of the variants of existing linear attention methods.
3.1 Low EFFICIENCY OF SELF-ATTENTION MECHANISM
Consider a sequence x E R of token length N and dimension D. x is devided into heads, the dimension of each head is d. In each head, tokens at various positions are collectively attended to capture long-range dependencies. The output O = {or E RN can be formulated as:
QKT va
Σ explq.kk /Vd) V o2 ΣE= exp(q.k) /vd) Vi-
O = Softmax
Here, the query, key, and value vectors of dimension d are obtained by linearly projecting the inputs with three learnable matrices Q {q: (q K = [k.] V = [vv E R.. For each head, the complexity of self-attention is O(N making the mechanism inefficient for long sequences.
3.2 KERNEL-BASED LINEAR ATTENTION
To mitigate the efficiency bottlenecks of standard self-attention, kernel-based linear attention mech- anisms (Katharopoulos et al., 2020) have been proposed, which decompose the similarity function into dot products of feature maps. Following the notations in (Choromanski et al., 2021; Chen al., 2021), I), we define SM(q. k) = cxp(q:k)) as the softmax kernel function. Mathemattcalll, linear attention aims use (kk))(())) T approximate SM(:,), where the feature map Φ(-) Hd + Rd is applied row-wise to the query and key matrices. As a result, the t-th row of attention output or can be rewritten as,
Σ ov Σ ((q))((k))
(qq) (k;) (q.)) Σf kjkk)))))))
(2)
By leveraging the associative property of matrix multiplication, the complexity per head is reduced to ONN'", which scales linearly with the sequence length.
Choices of Feature Map db-)) The primary distinction between various linear attention methods lies in the choice of feature maps ((-). Considering SM(-.-) -)) is PSD kernel function and the chosen feature map must satisfy two properties:
1. Non-negativity. To preserve the non- non-negaevve values in the approximation SM, previous methods utilize activation functions like (x) = 1 + ELU(x) (Katharopoulos et al., 2020) or <(x) ReLU(x) (Qin et al., 2022; Han al., 2023a). Other approaches connect SM
with Gaussian kernel that uses (xx) exp( |x incorporating trigonometric random positive features.
2. Low Entropy. It has been observed the attention-weights distribution in standard Trans- formers tends to be more "spiky" in linear ones, exhibiting lower entropy (Zhang et al., 2024a). To rescale the query-key dot products back the original magnitudes. techniques such as Taylor expansion (Keles et al., 2023) or higher norms on the numerical value of query and key (Han al., 2023a) have been employed.
However, using non-negative feature maps inherently results in the loss of information from the original negative values, which may carry important information in the original dot product calcu- lation. This leads to discontinuities the linear attention map compared the standard attention. Furthermore, existing rescaling strategies (Han et al., 2023a) manually select fixed norm across all dimensions, i.e., Φxx) = (ReLU(x))) where (x) optimal across different datasets.
||x | This fixed norm p may not be
PROPOSED APPROACH
In this section, we present a novel polarity-aware attention mechanism that accurately captures query-key interactions without incurring additional computational overhead. Our method incorpo-
4

---

Published as a conference paper at ICLR 2025
rates a learnable dimension- wise power function that dynamically rescales the magnitudes of same- and ayposite-signed components, effectively reducing entropy the linear attention.
4.1 POLARITY-WWAR ATTENTION
The key idea behind polarity-aware attention is to address the limitations of existing linear attention mechanisms, which often discard valuable information from negative components. We start by decomposing the query vector q = (+: he[] E Rd and key vector k = (hi Ne(d E IRd element- wise into their positive and negative components:
q = q 9 k=kt-
where q. max(g: 0) and q; max -4, 0), representing the positive and negative parts of q, respectively, and similarly for k. Substituting these decompositions into the inner product of q and k gives:
(q. = (a kt) + (9, k (9t,k-) (
naglected nagstives
The first two terms capture the similarity between same-signed components, while the latter two terms represent interactions between opposite-sige signed components. Previous linear attention ap- proaches, such as ReLU-basee feature maps, eliminate negative components by mapping them to zero, resulting in significant information loss when approximating query-key dot products.
To address this, polarity- aware attention mechanism separates query-key pairs based their polarity, computing their interactions independently. The attention weights are calculated as follows:
SM(q. k =xp((ak)
(5)
(ø(9") "o(""))(+))))) + piq (((k))((k)) )T) (ø(a oø(k ) o(q (okk))
This formulation recovers the information embedded in both positive and negative components.
Learnable Polarity-aware Mixing, While this formulation captures key information carried by both same-signed and opposite-signed components, directly subtracting opposite-signes similarities can violate non-teeaayvyyy constraints, leading to unstable training and suboptimal performance. To avoid the pitfalls of subtractive operation, we instead resort learnable mixing mechanism that weighs the contributions of sime-signed and opposite-signed query-key similarities.
More concretely, we split each value vector E Nxd along the d dimension into two halves to separately handle same- and opposite-signed response, i.e., = [v"; vo), where both v and v have a dimensionality of d/2. The output attention is then computed as:
pbla: q. D olk k, vZ o,
:(q. q. 1) (K: G
G
o(q :q. 9. Σ ((K) :, 1) I)
:(q. :q :4 E ( (Kjk:k)) :k,1) 1)
where [.--] denotes concatenation operation. G E Nx and G E Nx are two learn- able polarity-aware coefficients matrices ap- plied with element-wise multiplication, which are expected to learn the complementary re- lationship between same-signed and opposite- signed values. As shown in Figure 3, there is a clear negative correlation and value dis- crepancy between the weights learned G and Go, which evidences our learnable mixing strategy compensates for the relaxed subtraction operation in Equation (5).
21
25
2000
(b) Weight Distribution Figure 3: Visualizations of weights in G and G
(a) Pearson Correlation
Low-Rank SM. Previous theoretical work (Verma, 2021) has shown that SM inherently low- rank, particularly in higher layers where the spectrum distribution becomes more skewed. This property can lead to degenerate solutions when learning value vectors, especially when compact representations are required to accommodate polarity-aware information in our case. We explore various techniques such depthwise and deformable convolutions increase the rank, which can refer to the ablation study in Section 5.4.
5

---

Published as a conference paper at ICLR 2025
4.2 REDUCING ENTROPY IN LINEAR ATTENTION VIA LEARNABLE POWER FUNCTIONS
fettmax--free linear attention mechanisms often exhibit higher entropy compared softmax-based attention, leading to less sharp value vector attention, which is detrimental to tasks requiring precise attention. To recover the low entropy characteristics observed in softmax-hase terpret each row in (M k as generalized unnormalized positive sequence x (Z1,..., #N) and analyze its entropy using our proposed positive sequence entropy (PSE) measure, defined as:
attention, we rein-
Definition (Positive Sequence Entropy (PSE)), Let a sequence x (Z1,..,,N in which x, 2 0, N
i 1I...,,. N, and s = -i= 24 >
O. Then the entropy of this positive sequence is defined by:
N
PSE(x) =
log
3G.
With PSE(.) defined, we now seek a function g(-) that can be applied element-wise to o(q') and <()) = (ook))..... ((k*) )] such that the PSE of the i-th row of the linear attention map is reduced. The following theorem formalizes the conditions under which this reduction in PSE can be achieved.
Theorem 1. Let y E for = 1,...N, N, and let g [0, +20) + [0. +x0) be differentiable function satisfying the condition g'(x) > 0 and (xxx) > 0 for all x > 0. Then, there exists such
function g such that the PSE of the transformed sequence is strictly less than that of the original sequence. Specifically, we have:
PSE((g(x), 90) ().)))))((((x))......)))))))))") M)) )'' ,xx,,))),,,,,""))
(8)
Proof and supporting lemmas are provided in Section A.I.
This theorem also provides insights into why commonly used feature maps & such as ReLU or ELU+1 fail to reduce entropy effectively, they do not satisfy the necessary conditions of having both a positive first and second derivative across their entire domain.
To select suitable function g. There exists wide variety of functions g that meet these condi- tions. However, for the sake of model simplicity and efficiency, we opt for the most straightforward choice: a power function with an exponent greater than 1. Additionally, as different dimensions may contribute unequally to the similarity computation, we design learnable exponents to capture the varying importance of each dimension, formalized as follows:
p + a sigmoid(w),.. (w., wa),, g(x: (z
where is hyper-paramest scaling factor and [wi...., wa] are learnable parameters. There- fore, the feature map in our linear attention can be expressed as Ø(x*) = g(RcLU(x):p) and ((x = g(ReLU(-XX) p), where refers either q orrk k.
Complexity Analysis. We now analyze the complexity complexity of PolaFormer and demonstrate its linear complexity. Let d denote the number of channels, d' the dimensionality after kernelized, and k the kernel size of convolution (d' = d since 9() does just a clement-wise mapping). The computational cost for query, key, value, coefficients G* and G and outputs projections is d. Performing matrix multiplication for (Q, K. V) across each head requires dd'. The convolution operation contributes k² while the Dlement-wise multiplication of polarity Go and Go requires Nd computations. Summarizing these components, the total complexity of PolaFormer given Equation (10), confirming its linear complexity w.r.f. sequence length N.
coefficients
Ω 5Nd + 4Ndd + 22 + Nd
(10)
proj
Pola Aton
Conv
coeff
5 EXPERIMENTS
In this section, we evaluate our PolaFormer model on three tasks: image classification on ImageNet- IK (Deng et al., 2009), object detection and instance segmentation on COCO (Lin et al., 2014), and semantic segmentation on ADE20K (Zhou et al., 2019), comparing its performance with previous efficient vision models. Additionally, we assess PolaFormer on the Long Range Arena (LRA) task (Tay et al., 2021) to compare against other linear attention models. We first train PolaFormer from scratch on the image classification task, then fine-tune the pre-trained model on ADE20K dataset for segmentation and COCO dataset for detection. The models were pretrained on NVIDIA A800 GPUs and fine-tuned on 8 NVIDIA RTX A6000 and 8 NVIDIA RTX 3090 GPUs.

---

Published as a conference paper at ICLR 2025
Table 1: Comparison various linear attention methods relative to the original models (DeiT-T and Swin-T) the ImageNet- IK dataset, with the best results highlighted in boldface.
METHOD
RESO
PARAMS
FLOPs
Acc(%)
DeiT (Touvron et al., 2021)
224
5.7M
1.1G
72.2
SeIT-EfficientAAre (Shen al., 2021)
2242
5.7M
1.1G
70.2
DeiT--ydrrAAtn (Bolya et al., 2022)
224
5.7M
1.1G
68.3
DeiT-EahanceLAtta (Cai ct al., 2022)
224-
5.8M
1.1G
72.9
DeiT-AngularAttt (You et al., 2023)
224
5.7M
1.1G
70.8
DeiT-FLattenAttn (Han et al., 2023a)
2242
6.1M
1.1G
74.1
DeiT-MobiAttn (Yao et al., 2024)
224
5.7M
1.2G
73.3
DeiT-PolaFormer
224
6.1M
1.2G
74.6
Swin (Liu al. 2021)
224
28M
4.4G
81.2
Swin-HydraAtth (Bolya et al., 2022)
224
29M
4.5G
80.7
Swin-F.fficiettAtte (Shen et al., 2021)
2242
29M
4.5G
81.0
Awin-LibeerrAnguularAAtt (You et al., 2023)
2242
29M
4.5G
79.4
Swin-EahanccdlAtta (Cai al., 2022)
224
29M
4.5G
81.8
(Han et al., 2023a)
224
29M
4.5G
82.1
Swin- ForaFoeer
224
29M
4.5G
82.6
+:1.4
133
Valan Mambo
76
Noriaa
Soin (V Pff
Suth
PXI
WT
Salt
25
5-
75
10B
125
353
175
10
1703 1:25 1.00
1:75
2:00
7:75
7:5
155
20
2A
Figure 4: Efficiency analysis with Accuracy vs. FLOPs and Accuracy vs. Runtime curves on the ImageNet- IK dataset.
5.1
IMAGENET-IK CLASSIFICATION
The ImageNet-IK 1K (Deng et al., 2009) dataset is the widely used dataset for image classification tasks, containing 1,000 categories and over 1.2 million training images. We comprehensively as- sess our model's performance using Top-1 accuracy, and compare against recent state-of-the-art efficient Vision Transformer (ViT) models. Specifically, we selected four representative ViT back- bones: DeiT (Touvron et al., 2021), PVT (Wang al., 2021), PVTv2 (Wang al., 2022) and Tran-Transformer (Liu et al., 2021). We replaced their self aelf-attention modules with the our proposed polarity-aware attention module and trained these Pola-variants from scratch on ImageNet-
Results. The experimental results are presented in Table and Table 5, consistently showing that our model outperforms the baseline models. For instance, in Table 1, our DeiT-T-PolaFormer surpasses other DeiT variants from 0.5% to 6.3%. In Table 5, the WYT-T/S-Polaaorre 3.7% and 2.1% comparing with the corresponding baseline with comparable FLOPs. Additionally, method integrated in Swin and PVTv2 achieves better balance between performance and ef- ficiency. These results demonstrate that the PolaFormer enhances the expressive capability of the attention mechanism and can be widely applied in various attention-based models.
obtain an increase of
Efficiency Analysis. We visualize the efficiency comparison between the proposed PolaFormer and other linear attention approaches with similar FLOPs in the first two plots of Figure 4. The results show that our model can achieve comparable performance with significantly less compu- tation. Furthermore, PVT-PolaForme and Swin-PolaFormes on RTX3090 and RTXA6000 platforms, as shown in the third and forth plots of Figure PVT-PolaFormec achieves 1. and 1.12x faster inference speed and
evaluate the inference speed of PolaFormer. To be specific, test the
achieves 1.32x and 1.29x faster, both with comparable or higher accuracy. These figures highlight the excellent trade-off between accuracy and latency that our model provide.
5.2 ОBJECT DETECTION AND INSTANCE SEGMENTATION
We further validate the effectiveness of the proposed approach across various vision tasks, including object detection task the COCO dataset (Lin et al., 2014), which contains over 118K training images and 5K validation images. We integrate Pola-Swin and Pola-PVT separately as the back- bone into Mask-RCNN (M) (He et al., 2017), RetinaNet (R) (Lin al., 2017) and Cascade Mask

---

Published as a conference paper at ICLR 2025
Table 2:
Object detection and instance segmentation results on the COCO dataset. The "Type" column specifies the detector used: R represents RetinaNet, M for Mask R-CNN, and C for Cascade Mask R-CNN. For semantic segmentation on the ADE20K dataset (last column), two encoder types employed: S corresponds to Semantic FPN, and U refers to UperNet.
DETECTION AND INSTANCE SEGMENTATION
SEMANTIC SEG
МETHOD
SCH.
TYPE
AP
AP
AP
Apm
AP
AP
TYPE MioU($)
PVT-T
1x
R
36.7
35.7
1x
M
36.7
59.2
39.3
35.1
56.7
37.3
R
PVT-T-Flatten
S
37.2
1x
M
38.2
61.6
41.9
37.0
57.6
39.0
PVT-T-PolaPoreer
1x
R
39.5.22
60.1
42.1
S
38.3
,0.4,a.7 ...7
62.4 3.2
43.9
,2.3
59.4..7
40.3..a.0
R
40.4
PVT-S
S
39.8
1x
M
40.4
62.9
43.8
37,8
60.1
40.3
ForaFeerr
IX
R
43.2...a
64.1
46.4
41.0,1.2
M
43.5
66.1.,32
47.9..4.1
40.2,
63.1,,3.0
43.0..2.7
M
43.7
66.6
47.7
39.8
63.3
42.7
Swin-T
3x
M
46.0
68.1
50.3
41.6
65.1
44.9
44,5
3x
c:
50.4
69.2
54.7
43.7
66.6
47.3
M
44.2
67.3
48.5
T-Fateen
3x
M
46.5
68.5
50.8
42.1
65.4
45.1
44.8
3x
C
50.8
69.6
55.1
44.1
67.0
48.1
1x
44.8,.1.1
67.6
49.14..a
40.5
64.1,00
43.5...7
Swin - PolaFormer
3x
M
47.0
68.9 -o.s
51.5
42.3
66.0
45.8
.0.9
45.8
3x
C
51.1.0.
70.0
55.6,0.9
44.4.0.7
67.3
48.3....0
R-CNN (C) (Cai Vasconcelos, 2021) implementations and evaluate their performance based on the ImageNet-II IK pretrained weights. As shown in Table (left), our model consistently outper- forms the original backbones under all settings, achieving notable improvements in all metrics. For instance, our PVT-T-PolaFormer tested with both and M detectors, surpasses the baselines from 2.3% to 4.6%. Additionally, our Swin-T-PolaFormm achieves 49. 19.1% in APÊ showing a 1.4% improvement compared the original Swin-T with detector. We additionally evaluate PVT-S- PolaFormer with R and M detectors, and Swin-T-PolaFormer with M and C detectors using 1 x and 3x schedule. Compared to classification tasks, our model delivers more substantial performance gains detection, which demands fine-grained attention maps for accurate localization of bound- ing boxes. Our model captures previously omitted interactions involving negative values and better restores attention maps with appropriate scales, effectively distinguishing between similar and dis- similar query-key relationships.
5.3 SEMANTIC SEGMENTATION
similar phenomenon was observed when fine-tuning our pre-trained model for pixel-wise se- mantic segmentation tasks the ADE20K dataset. ADE20K (Zhou et al., 2019) provides a di- verse set of annotations for scenes, objects, and object parts, containing 25,000 images of complex scenes with various objects in natural spatial environments. We integrate Pola-Swin and Pola-PVT with ImageNet- K pre-trained weights into two segmentation models, SemanticFPN (Kirillov et al., 2019) and UperNet (Xiao al., 2018), using mIoU as the evaluation metric. The results. shown in Table 2 (right), demonstrate a performance improvement in mloU ranging from 1.2% to 2.6%. These findings further highlight the versatility of our model, showing that can be effectively fine-tuned and adapted to a wide range of vision tasks.
5.4 ABLATION STUDY
Table 3: Ablation study on each module using DeiT-T on ImageNet-1K.
Impact of Components. We evaluate the effectiveness of each component in PolaFormer. As shown in Table 3, to address the low-rank issue the attention map, we exam- ine the impact of incorporating deformable convolutions (DCN) and depth-wise convolutions (DWC) row I and row 4, respectively. DWC demonstrates better adaptabil- ity, achieving an accuracy of 74.6%. It is important to note that our model is agnostic to the choice of convolution modules. Furthermore, adopting polarity coefficients Go and Go yields 1.8% improvement in row 3 and 4, indicating that the model effectively learns the complementary rela- tionship between same-signed and opposite-signed values.
POLARITY G
DWC DCN ACC. (%)
61. 127
6.5
72.8
74.6
8

---

Published as a conference paper at ICLR 2025
Table 4: Comparisons (%) between the proposed PolaFormer and other linear attention models on LRA, with the best results are highlighted in boldface.
MODEL
TEXT
LISTOrs
RETRIEVAL
PATHFINDER
IMAGE
AVERAGE
Transformer
61.55
38.71
80.93
70.39
39.14
58.
LocalAttn
52.98
15.82
53.39
66.63
41.46
46.06
Linear Trans.
65.90
16.13
53.09
75.30
42.34
50.55
Reformer
56.10
37.27
53,40
68.50
38.07
50.67
Performer
65.40
18.01
53.82
77.05
42.77
51.41
Synthesizer
61.68
36,99
54.67
69.45
41.61
52.88
Longformer
62.85
35.63
56.89
69.71
42.22
53.46
Informer
62.13
37.05
79.35
56.44
37.86
54.57
Bigbird
64.02
36.05
59.29
74.87
40.83
55.01
Linformer
57.29
36.44
77.85
65.39
38.43
55.08
Kernelized
60.02
38.46
82.11
69.86
32.63
56.62
Cosformer
63.54
37.2
80.28
70.00
35.84
57.37
Nystrom
62.36
37.95
80.89
69.34
38.94
57.90
Skyformer
64.70
38.69
82.06
70.73
40.77
59.39
Hodgehog
64.60
37.15
82.24
74.16
40.15
59.66
PolaFormera
73.06
37.35
80.50
70.53
42.15
60.72
PolaFormer..ss 72.33
38.76
80.37
68.98
41.91
60.47
PolaFormer 71.93
37.60
81.47
69.09
42.77
60.57
Comparison with Other Linear Attention. To compare with other linear attention models, evaluate our PolaFormer on the Long Range Arena (LRA) (Tay et et., 2021) task, which composed with five tasks: ListOps (Nangia & Bowman, 2018), Text Classification (Maas et al., 2011), Document Retrieval on AAN dataset (Radev et al., 2013), Pathfinder (Linsley al., 2018), and Image Classification on CIFAR-10 (Krizhevsky, 2009). The sequence length in both TEXT and RETRIEVAL tasks is 4k, in LISTOPS is 2k and in PATHFINDER and IMAGE 1k. Following the setup Skyformer (Chen et al., 2021), we adopt a comparable number of parameters and train the entire model end-to-end with the task-specific losses. Results for different scaling factors a of PolaFormer are shown in the bottom rows of Table PolaFormer, obtains highest accuracy 73.06% in Text Classification task, PolaFormer.es has achieved the state-of-the- art results in ListOps task for 38. 78%76% accuracy and the PolaFormer gains the best performance in Image Classification task with accuracy of 42.77% It is worth mentioning that PolaFormer achieves the highest overall scores on LRA benchmark, with all variants outperforming other linear attention models. Our model achieves better scores in linear complexity relying its extraction ability and higher rank, showing great potential in both NLP and CV tasks.
IMDb review dataset
Impact of Learnable Scaling, In Equation (9), introduce the scaling factor a in our learnable power function, and analyze the effect of exponent p on the model performance. The value of primarily depends on the model size and context length. As the model size increases, a larger is required to effectively select the most relevant tokens from long sequences, thereby reducing information entropy. We evaluate the model with 3, 5, on the LRA task, shown the bottom of Table 4. Although PolaFormer,=3 achieves the best performance, in practice, for classification tasks, the results are relatively insensitive to variations a, with difference of no more than 2%.
CONCLUSION
In this work, we presented PolaFormer, novel efficient transformer with linear complexity. Our PolaFormer built on two properties of the original softmax attention: (i) making each element of the attention weight neg-negt and (ii) making attention weight spikier. To fulfill these properties, we computed the similarity a polarity-awarc form to avoid neglecting negatives; theoretically, we proposed family of element-wise functions to lower the entropy and employ learnable power function for simplicity and rescaling. Besides, we used convolution to alleviate the problem of de- generate solutions caused by the low-rank property of SM and introduced polarity-aware coefficient matrices to learn the complementary relationship between same-signed and opposite-signed values. We validated the effectiveness of the proposed PolaFormer's in series of vision tasks and additionally benchmarked on the LRA testbed to fairly compare with mainstream linear attention models. The experimental results demonstrated that our model has good compatibility with most attention-based models and measures up to better balance between performance and efficiency.

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of classification results on the ImageNet-IK IK dataset. The default input resolu- tion 224², except for the last row, which reports results for variants using a resolution of 384?.
MODEL
RESO
PARAMS
FLOPS
Acc(%)
SBCFormer-B (Lu et al., 2024)
221
14M
1.6G
80.0
SBCFormer-L (Lu et al., 2024)
224
19M
2.7G
81.1
CAS-ViT-1- (Zhang et al., 2024b)
224
22M
3.5G
82.3
VisionMamba--1 (Zhu et al., 2024)
224
7M
1.1G
76.1
VisionMamba-S (Zhu et al., 2024)
224
26M
3.7G
80.6
VisionMamba-B (Zhu et al., 2024)
221
98M
13.7G
81.9
T2T-14 (Yuan et al., 2021a)
224
21.5M
4.8G
81.5
T2T-19 (Yuan et al., 2021a)
224
39.2M
8.5G
81.9
T2T-24 (Yuan et al., 2021a)
224
64.1M IM
13.8G
82.3
CvT-13 (Wu et al, 2021)
224
20M
4.5G
81.6
CvT-21 (Wu et 2021)
2242
32M
7.1G
82.5
CvT-13 (Wu et al., 2021)
3842
20M
16.3G
83.0
CvT-21 (Wu et al., 2021)
3842
32M
24.9G
83.3
HiViT-T (Zhang et al., 2023)
2212
19M
4.6G
82.1
HiViT-S (Zhang al., 2023)
224
38M
9.1G
83.5
HiViT-B (Zhang et al., 2023)
224
66M
15.9G
83.8
PVT-T-FLatten (Han et al., 2023a)
224
11M
1.9G
77.8
PVT-S-FLatten (Han et al., 2023a)
224
22M
4.0G
81.7
PVTv2-b0-FLatten (Han et al., 2023a)
224
3.2M
0.6G
71.1
PVTv2-b0-MobiAtt (Yao et al., 2024)
224
3.5M
0.6G
71.5
PVTv2-b1-FLatten (Han et al., 2023a)
224
13M
2.2G
79.5
Swin-S-FLateen (Han et al., 2023a)
2212
51M
8.7G
83.5
Swin-B-FLatten (Han et al., 2023a)
224
89M
15.4G
83.8
PVT-T et al., 2021)
2242
13M
1.9G
75.1
PVT-T-PolaFormer
2242
12M
2.0G
78.8 +-3.7
PVT-S (Wang al., 2021)
2242
25M
3.8G
79.8
PVT-S-PolaFormer
2212
21M
4.1G
81.9+2.1 +2.1
PVTv2-b0 (Wang et al., 2022)
224
3.7M
0.5G
70.5
PVIV2260-PolaFormer
2242
3.4M
0.6G
72.3 1.8
PVTv2-b1 (Wang et al., 2022)
224
13M
2.1G
78.7
PVTv2-bl-PolaFormer
2242
13M
2.2G
80.2 +1.5
Swin-S (Liu et al., 2021)
2242
50M
8.7G
83.0
Swin-S-PolaFormer
2242
50M
8.7G
83.6
Swin-B (Liu et al., 2021)
2242
88M
15.4G
83.5
Swin-B-PolaFormee
2212
88M
15.4G
83.8 +0.3
Swin-S (Liu et al., 2021)
3842
50M
25.2G
84.3
Swin-B (Liu et al., 2021)
3842
88M
47.0G
84.5
Swin-S-PolaForme
3842
50M
25.5G
84.7+0..
+0.4
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research is partially supported by National Natural Science Foundation of China (Grant No. 62372132), Shenzhen Science and Technology Program (Grant No. RCYX30210800255077 and Australian Research Council (DE240100105, DP240101814, DP230101196), We would like to express our gratitude Huawei funding and valuable discussions with Dr. Wei Zhou.
REFERENCES
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman. Hydra attention: Efficient attention with many heads. In Computer Vision ECCV 2022 Workshops Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII, pp. 45.49. Springer, 2022.
Han Cai, Chuang Gan, and Song Han. Efficientvit: Enhanced linear attention for high-resolution low-computation visual recognition. arXiv preprint arXiv: 14756, 2022.
Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: high quality object detection and instance segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 43(5):1483-1499) 2021.
Yifan Chen, Qi Zeng. Heng Ji, and Yun Yang. Skyformer: Remodel self-attention with gaussian kernel and nystrom method. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems (NeurIPS), pp. 2122-2135, 2021.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sariós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with per- formers. In 9th International Conference on Learning Representations (ICLR). OpenReview.net, 2021.
MMCV Contributors. MMCV: OpenMMLab computer vision foundation. https://gftthb.
com/ opt//mpen-mmm ab/ ab/mmcv, /mmv, 2018
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations (ICLR). OpenReview.net, 2021.
Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song. and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In IEEE International Conference on Computer Vision (ICCV), pp. 5938-5948, 2023a.
Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Shiji Song, and Gao Huang. Agent attention: On the integration of softmax and linear attention. CoRR, abs/2312.08874, 2023b.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN, In IEEE In- ternational Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp- IEEE Computer Society, 2017.
Praneeth Kacham, Vahab Mirrokni, and Peilin Zhong. Polysketchformer: Fast transformers via sketching polynomial kernels. In Forty-first International Conference on Machine Learning (ICML), 2024.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th Interna- fional Conference on Machine Learning (ICML), volume 119, pp. 5156-5165, 2020.

---

Published as a conference paper at ICLR 2025
Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegde. On the computational complexity of self-attention, In 34th International Conference on Algorithmic Learning Theory, volume 201, pp. 597-619, 2023.
Alexander Kirillov, Ross Girshick. Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6399-6408, 2019.
A Krizhevsky, Learning multiple layers of features from tiny images. Thesis, University of Tront, 2009.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects context. In European Conference Computer Vision (ECCV), volume 8693, pp. 740-755, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE International Conference Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 2999-3007. IEEE Computer Society, 2017.
Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. In Advances in Neural Information Processing Systems (NeurIPS), pp. 152-164, 2018.
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. CoRR, abs/2401. 2024.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Hierarchical vision transformer using shifted windows. IEEE International Conference on Computer Vision (ICCV), pp. 9992-10002, 2021.
Swin transformer:
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations (ICLR), 2019.
Xiangyong Lu, Masanori Suganuma, and Takayuki Okatani. Sbeformer: Lightweight network capa- ble of full-size imagenet classification at FPS on single board computers. In /EEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1112-1122. IEEE, 2024.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In The 49th Annual Meeting of the Associ- ation for Computational Linguistics: Human Language Technologies (ACL-HLT), pp. 142-150, 2011.
Nikita Nangia and Samuel R. Bowman. Listops: A diagnostic dataset for latent tree learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics (NAACL.), pp. 92-99. Association for Computational Linguistics, 2018
Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. Cosformer: Rethinking softmax in attention. In The Tenth International Conference on Learning Representations (ICLR). OpenReview.net, 2022.
Dragomir R. Radev, Pradeep Muthukrishnan, Vahed Qazvinian, and Amjad Abu-Jbara. The ACL anthology network corpus. Lang. Resour. Evaluation, 47(4))919-944 2013.
Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexitiss. In JEEE Winter Conference on Applications of Computer Vision ( (ACV)) pp. 3538. IEEE, 2021.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient trans- formers. In 9th International Conference on Learning Representations (ICLR). OpenReview.net, 2021.
12

---

Published as a conference paper at ICLR 2025
Giorgos Tolias, Ronan Sicre, and Hervé Jégou. Particular object retrieval with integral max-pooling of CNN activations. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Hugo Touvron, Matthieu Cord, Mathiis Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139, pp.
10347-10357, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NeurIPS), pp.
2017.
Madhusudan Verma. Revisiting linformer with a modified self-attention with linear complexity. CoRR, abs/2101.10277, 2021. URL https::/ / /aaaaaxivvvv.o/..//aaa org/ /too/2201.10277
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In IEEE/CVF International Conference Computer Vision (ICCV), pp. 548-558. IEEE, 2021.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. PVT v2: Improved baselines with pyramid vision transformer. Comput. Vis. Media, 8(3):115-244, 2022.
Zhaozhi Wang, Yue Liu, Yunfan Liu, Hongtian Yu, Yaowei Wang, Qixiang Ye, and Yunjie Tian. vheat: Building vision models upon heat conduction. CoRR, abs/2405. 16555, 2024.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In 2021 IEEE/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 22-31. IEEE, 2021.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 418-434, 2018.
Zhiyu Yao, Jian Wang, Haixu Wu, Jingdong Wang, and Mingsheng Long. Mobile attention: Mobile- friendly linear-attention for vision transformers. In Forty-first International Conference Ma- chine Learning (ICML), 2024.
Haoran You, Yunyang Xiong, Xiaoliang Dai, Bichen Wu, Peizhao Zhang, Haoqi Fan, Peter Vajda, and Yingyan Celine Lin. Castling-vit: Compressing self-attention via switching towards linear- angular attention at vision transformer inference. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 14431- 14442. IEEE, 2023.
Li Yuan, Yunpeng Chen, Tao Wang. Weihao Yu, Yujun Shi, Zihang Jiang, Francis E. H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on im- agenet. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 338-547. IEEE, 2021a.
Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang- Hrformer: High-resolution transformer for dense prediction. CoRR, abs/21 10.09408, 2021b.
Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher Ré. The hedgehog & the poreu- pine: Expressive linear attentions with softmax mimicry. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024a.
Tianfang Zhang, Lei Li, Yang Zhou, Wentao Liu, Chen Qian, and Xiangyang Ji. Cas-vit: Con- volutional additive self-attention vision transformers for efficient mobile applications. CoRR, Abs/2408.0370x 2024b.
13

---

Published as a conference paper at ICLR 2025
Xiaosong Zhang, Yunjie Tian, Lingxi Xie, Wei Huang. Qi Dai, Qixiang Ye, and Qi Tian. Hivit: A simpler and more efficient design of hierarchical vision transformer The Eleventh Inter- national Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. Int. J. Comput. Vis. (IJCV), 127 (3):302-321, 2019.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation leannnn with bidirectional state space model. Forty-first International Conference Machine Learning (ICML), 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
This Appendix provides proof and supporting lemma for Theorem followed by implementation details for various vision tasks. The source code is available in the supplementary material for reference.
Proof. A.1: The mathematical proof and supporting lemmas of Theorem
Implementation Details. A.2: Training settings for all experiments
Long Sequence Efficiency
Comparison of the results of models with different G initializations
Visualization of Attention Probability Distribution's Entropy
Visualization of Attention Maps
A.1 PROOF OF THEOREM 1
Theorem. Let x, y" E Rdd for n. =..=,.... I....., and dimensions are independently distributed. Given that g : [0, +00)
[0, +20) is differentiable function satisfying the condition g'(x) 0 and g"(x) > 0 for all æ > 0. Then, there exists such function such that the PSE of the transformed
sequence is strictly less than that of the original sequence. Specifically, have:
PSE((g(x), g( "("))))((")),,)))))) "),..,,,................ (x(x),
PSE((x, "x,,,,,,,,,")
(11)
Proof. We establish two lemmas facilitate the proof of the main theorem.
Lemma 1. Let be a function induced by g [0, +oo) + j0, +oo) with the conditions of g g((x) and g" (x) > 0, for all x > O, defined as:
5((x. g(x))
(12)
where x, Rd+ ,9(x) (o(x1)....., g(xa)) Then f(x) > f((x) > 0 and (") 0, for all x 2 0.
Proof. Consider the element-wise function for pairs of y) with dimension d:
g(x) = (g(x)) ((xa)) g(y) (g(v)),... ... (va))
(13)
Then, the inner-produc, between gxx) and g(y) is given by,
(g(x), Σ g(z))g(us).
(14)
Because of the independence across dimensions, we apply Jensen's inequality, leveraging g((x) > and g" (ax) 0, yielding:
E[f((q. k)) ((o(g)) 9kk)] E ((a))(()))
d
Eig(a:)((k)) Σ Elg(k))F[((k))
(15)
((E(a)))(B(k:)) (Jensen's Inequality)
(g (E(q)). (E(k)) = (((g)) f(El k)))
where E[q] = Ejga)) denotes vector. Consequently, we have the following results, i.e.,
E[S((q, k))] < (IB( k))),
(16)
15

---

Published as a conference paper at ICLR 2025
indicating that is concave function having positive second derivative. Also, according to the definition x and y, f is obviously mapping from [0, +00) to [0, +00) with a positive first derivative.
Lemma 2. Given nwo positive values (a, b), and function : 10, +o0) + [0, +00) with the condi- tions of (z) > 0 and (x) > 0, we have PSE(f(a), f(6)) S (a,E(aa,))
Proof. Consider the case N (extendable N > 2). Without loss of generality, we assume > b, == 8. then > 1, and PSE(a, b) can be calculated as
H1 - a
log(
log( a a+ b 1 c+
+ a+b" a a+b
-( c+1 c+1 c+1
c
log(
log(
(17)
log(e+1) c+1 log(c)
Then, we apply the kernel function f on (a, b), and it is mapped to (f(a), f(b). Then, we define f(a) and it is easy prove that d >>
d by d 7bJ
Followed by Eq. (17), can compute
F(E( as:
d
H₂ log(d 1) d log(d)
(18)
Through defining h(x) = log(æ + +1 1))
x I+I log(z), x > 1, we have
log(rr
l1(x) (a - 1)²
(19)
(1(x) S 0, x x> 1
indicating that H1 h(c) H2 = h(c) for all 1, ie.. H2 < H1. Therefore, all functions that satisfy the conditions have the effect of entropy decrease.
Now come back to the theorem. Firstly, we define / induced by g that
f((x, y)) (((x)).(y))
(20)
From Lemma 1, we know that f is a function with positive first and second derivative. Then by using Lemma 2. we have,
PSE(F((x, f(x,)
PSE((x,))) (x.y?))
(21)
Therefore, the scaling effect can be achieved by the element- wise computation based on a function g with positive first and second derivative. This allows for the removal of the softmax function, enabling linear complexity and lower entropy in the attention mechanism.
A.2 IMPLEMENTATION DETAILS
Classification. In this task, we use the AdamW optimizer (Loshchilov Hutter, 2019) to train all of our models for 400 epochs, including 20 epochs for linear warm-up. The basic learning rate is set 1e-3 for 1024 batch size. Additionally, we use a weight decay of 5 2. The training framework developed on the top of the official Swin Transformer implementation made by Microsoft.
Object Detection. In this task, we utilize pretrained PVT models and Swin models on as the backbone and connect them to various detectors. Specifically, for the PVT model, we select from RetinaNet and Mask R-CNN as detectors, with the schedule set to 1x. For the Swin model, we choose the detector from Mask R-CNN and Cascade Mask R-CNN as detectors, where models ing Mask R-CNN are experimented with under both 1x and 3x schedule settings, while models using Cascade Mask R-CNN case are trained under the 3x schedule. All experiments follow the
16

---

Published as a conference paper at ICLR 2025
mnev-detection (Contributors, 2018) project. The training epoch is set 12 per schedule and we use the AdamW optimizer with a learning rate of 1e 4 and weight decay of le 4.
Semantic Segmentation.
employ pretrained PVT models and Swin models two rep- resentative segmentation models, SemanticFPN and UperNet. The task is conducted based imcc--semennattin (Contributors, 2018) project. The training interation is set to 40000 for PVT-SEPN models, 160000 for Swin-UperNet models by using Adam optimizer with a learning rate of 2e and weight decay of le 3.
Long Range Arena. We evaluate the PolaFormer based on the official implementation of Skyformer (Chen al., 2021). For Listops and Text Classification, rate. For Pathfinder, we set batch size to 128 with 5e 4 learning rate. For Image Classification, we batch size to 256 with le learning rate. For Retrieval sub-task, we set batch size 16 with 2e 4 learning rate. All models are trained from scratch using the Adam W optimizer.
set batch size to 32 with le learning
A.3 LONG SEQUENCE EFFICIENCY
To evaluate the scalability of our model in such settings, performed experiments on the Long- Range Arena (LRA) benchmark. These results demonstrate PolaFormer's efficiency and scalability for both high-resolution vision tasks and oongeeeuennce NLP applications.
Table 6: Throughput and Peak Memory of various models. A denotes the accuracy. denotes the throughput of each model and M denotes the peak memory cost.
Softmax
Kernelized
Nystrom
Linformer
Informer
Skyformer
Pola(ours)
39.14
32.63
38.94
38.43
37.86
40.77
42.15
T
736.36
862.32
1251.28
1613.19
85.85
923.04
1340.89
(Ik)
M
9645
13013
5941
3471
5357
8091
4505
A
70.39
69.86
69.34
65,39
56.44
70.73
70.53
T
691.67
811.59
1125.08
1057.03
748.98
1065.63
(Ik)
M
4831
65155
2980)
1745
2687
4055
2286
A
38.71
38.46
37,95
36.44
37.05
38.69
37.35
List
T
402.06
496.48
834.85
528.52
305.53
627.
949.80 80
(2k)
M
4473
6084
1186
881
1712
1151
61.55
60.02
62.36
$7.29
62.13
64.7
73.05
Text (4k)
T
252.06
327.27
1330,68
970.90
521.16
949,80
876,74
M
17122
11720
2043
1742
5736
3082
11555
A
80.93
82.11
80.89
77.85
79.35
82.06
80.5
Retri (4k)
T
116.30
144.83
496.48
424.18
142.94
348.60
344.93
M
8947
10699
2011
1649
3399
2987
1139
A
58.14
56.62
55.08
54.57
59 4.
60.72....5e
Avg
T
439.69
528.50 1.20
2.22
918.77x2.00
271 20.62
915. 22.08
M
9003.6
9606.2.
2832.2x6.31
1897.6,0.21
3983.2x2044
3985.4x0.44
2047.2xx0.2
A.4 COMPARISON OF THE RESULTS WITH DIFFERENT INITIALIZATIONS OF COEFFICIENTS MATRICES
To assess the impact of G initialization on downstream tasks, we conducted additional experiments using five distinct initialization methods. These experiments were performed on text classification (TEXT) task in Long Range Arena (LRA) with sequence length of 4k, maintaining the same exper- imental setup as described in Table 4. The initialization strategies tested included Kaiming uniform, zero initialization, normal distribution (N(0, 1)), uniform distribution (24(0, 1)), and constant ones. The results are summarized in the table below:
Init Comparison
Kaiming Uniform
Zeros Normal(O,1)
Uniform(0,1)
Ones
Acc
73.06
72.17
74.30
74.40
70.70
17

---

Published as a conference paper at ICLR 2025
Liner 2e Il:122
Pdaterer
mag Mhyy
11.mm-230 1Loppe222
Figure Visualization of different attention probability distributions
Original Image
Softmax Attention
Linear Attention
Flatten
PolaFormer
Figure Visualization of the attention maps.
A.5 VISUALIZATION OF ATTENTION PROBABILITY DISTRIBUTION ENTROPY
We compute the entropy of standard self-attention, linear attention (Katharopoulos et al., 2020) and PolaFormer. Additionally, we visualize the distribution of one row of the Attention Score matrix, as shown in Figure 5. clear that our model has lower entropy than linear attention.
A.6 VISUALIZATION OF ATTENTION MAPS
To further show the characteristics of accurate similarity calculation and low information entropy of our model. We visualize more examples shown in Figure 6. Thanks to the superiority of our designed kernel function, PolaFormer can calculate the similarity more accurately and focus on more relevant places.
18


Reviews:
[{"review_id": "35TyI0mThG", "reviewer": "Reviewer_KHpw", "summary": "Linear attention replaces the softmax over query-key dot products with a kernel function that makes the attention operation linear in kernel space, which breaks the quadratic complexity complexity in the sequence length. \nThis paper proposes a variant of linear attention where (1) the positive and negative components of the query and key vectors are separated and (2) the individual components are made more \"peaky\" by applying a per-component $x \\leadsto x^p$ transformation where the powers $p>1$ are learned.", "questions": "please clarify how G^s and G^o can be trained", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1.  The method presented in this paper is simple and well justified\n\nS2. Paper generally easy to read \n\nS3. results outperform the full attention baseline (I merely expected it to be a good approximation)", "weaknesses": "W1. The main dawback of the experiments is that there is not experiment on truly high resolution image, where this method would fully benefit from the linear complexity\n\nW2. Some parts are not clear (see below) \n\n\nUnclear points: \n- it would be useful to specify from the intro that the method is applied with full training -- it is unclear until the experiments that this is not fine-tuning and not a drop-in replacement for softmax attention at inference time\n\nL133: better = more accurate or faster? \n\nL162: relationship between d and D ?\n\nL248 it is unclear how G^s and G^o can be trained since they depend on the batch size N -- or is N assumed to be fixed? \n\neq (7) maybe not necessary to define what entropy is... Also the argument about better accounting for negative correlation is repeated 4-6x in the paper. By shaving these repetitions you could make room to move the proof to the main paper.\n\nL300 IIUC d' = d since g() does just a pointwise mapping -- please mention this explicitly\n\nMaybe some useful related work on the power kernels : [Tolias et al, Particular Object Retrieval With Integral Max-Pooling of CNN Activations, ICLR'16]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hpDVBVzxgU", "reviewer": "Reviewer_KD6g", "summary": "To overcome the shortcomings of current linearized self-attention, the paper proposes polarityaware linear attention mechanism that attends to both both same-signed and opposite-signed query-key interactions. the latter on is oftern ignored by existing methods. Besides, the proposed polarity-aware attention can also addresses the loss of attention spikeness.", "questions": "See weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and organized.\n2. The paper provides solid motivation for the method and the proposed approach is well justified.\n3. The experimental results show promising performance.", "weaknesses": "1. After reading the paper, I am still not sure how the non-negativity constraint is perserved. Especially when a learnable matrix $G$ is applied. Could authors provide more explanation on how the learnable matrix $G$ can perserve the non-negativity constraint?\n\n2. Some latest baselines are missing in the paper. For instance, authors should consider incorporating the latest work [1] for comparsion. This baseline also proposes a new linear self-attention to achieve both high expressiveness capacity and low computation complexity.  The comparison on image classification or object detection would be helpful for a comprehensive assessment of the propsoed method.\n\n3. One minor error in line 98. Negative-negative should be Negative-positive.\n\n[1] Han, Dongchen, et al. \"Agent attention: On the integration of softmax and linear attention.\" ECCV 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VZSuyRJYWo", "reviewer": "Reviewer_veYX", "summary": "This paper introduces PolaFormer, a new attention mechanism designed for vision transformers that aims to improve both expressiveness and efficiency. Traditional linear attention methods reduce computational complexity by approximating the softmax function but often lose critical information by ignoring negative query-key interactions and producing less discriminative attention maps with higher entropy. PolaFormer tackles this issue by explicitly modeling both positive and negative query-key interactions, ensuring a more comprehensive capture of relational information. Additionally, it employs a learnable power function to adjust the attention distribution's entropy, restoring the sharpness characteristic of softmax-based attention. The authors back their approach with theoretical analysis and demonstrate its effectiveness through extensive experiments across various vision tasks, showing notable performance improvements while maintaining linear computational complexity.", "questions": "1. Did introducing the learnable power function and the polarity coefficients \\( G_s \\) and \\( G_o \\) affect training stability? For example, did you encounter issues like vanishing or exploding gradients? If so, how did you mitigate them?\n2. How sensitive is the model's performance to the initialization and learning of the polarity coefficients \\( G_s \\) and \\( G_o \\)? Did you notice any significant performance drops or instability when varying these parameters?\n3. Have you tried applying PolaFormer to NLP tasks like machine translation or language modeling? Since attention is crucial in these areas too, it would be interesting to see if your method provides benefits there.\n4. You theoretically show that your method reduces entropy in the attention distribution. Did you measure the entropy empirically in your experiments to confirm this? It would be interesting to see a plot or some data showing how the entropy changes with your method compared to others.\n5. The attention weight visualizations in Figure 1 are helpful. Could you provide more examples, perhaps showing how PolaFormer focuses on relevant parts of the input in different tasks or layers? This could provide more intuitive insight into how your method works.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The key strength is the introduction of polarity-aware attention. By decomposing the query and key vectors into positive and negative components (as in Equation (3)), they capture all possible interactions: positive-positive, negative-negative, positive-negative, and negative-positive. This is a significant departure from traditional methods that only consider positive interactions due to non-negative feature maps.\n2. They provide solid theoretical analysis. For example, in Theorem 1, they prove that using a function \\( g \\) with positive first and second derivatives (like their learnable power function) reduces the entropy of the attention distribution. This mathematical proof gives credibility to their claim that their method restores the \"spikiness\" of the attention weights.\n3. The experimental results are impressive. On ImageNet-1K classification, their PolaFormer variants outperform the baselines by significant margins. For instance, DeiT-T-PolaFormer improves Top-1 accuracy by up to 6.3% over other DeiT variants. In object detection tasks on COCO, they achieve improvements ranging from 2.3% to 4.6% in AP scores. These aren't just marginal gains—they're substantial improvements that demonstrate the practical value of their method.\n4. Despite the added complexity of handling negative interactions and using a learnable power function, they manage to keep the computational complexity linear with respect to sequence length \\( N \\), as shown in their complexity analysis (Equation (10)). They also report faster inference speeds compared to other models with similar FLOPs, which is a big deal for real-world applications.", "weaknesses": "1. Introducing a learnable power function and additional parameters like the polarity coefficients \\( G_s \\) and \\( G_o \\) could introduce training challenges, such as sensitivity to initialization or convergence issues. The paper doesn't discuss whether they encountered any of these problems or how they addressed them.\n2. Since attention mechanisms are also fundamental in NLP, it would have been interesting to see PolaFormer's performance on language tasks. The paper focuses solely on vision tasks, so we don't know if the benefits carry over to NLP applications.\n3. While they do perform an ablation study (as shown in Table 3), it could be more comprehensive. For instance, they could explore how different values of the scaling factor in the learnable power function affect performance, or how sensitive the model is to the choice of convolutional modules used to increase the rank of the attention map.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LImsXAhS6o", "reviewer": "Reviewer_EBEw", "summary": "This paper aims to improve the state of the art linear transformers with linear complexity O(N) with N the sequence length. It focus on addressing two issues as identified: 1) Loss of negative values, and 2) loss of attention spikiness. The proposed method, PolaFormer, is based on the idea of separating the query-key pairs with their plarity into two branches, one for positive and one for negative. Other improvement includes making some previously hand-designed static parameters to be learnable. Experiments are conducted on image classification, object detection and segmentation, and the long range arena tasks.", "questions": "Please check the weakness parts", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Good and clear writing, with good background and context, detailed description of equations, and transformer mechnisam, and the literature review of previous linear transformers. \n\nVisualisation is easy to understand\n\nThe key idea is delivered clearly, with the assistance of well designed charts and graphs.\n\nThe experiments show good margin as compared to previous alternative methods.", "weaknesses": "First point unclear to me is about the definition of negative values (Line 201). As clearly stated in softmax kernel function (Line 16), they operate in the output end of softmax function where it is deemed that all values are not negative. It is not about the input end of softmax where values can be either negative or positive. Under this context, I do not see anywhere negative values are lost or overlooked as kernel based linear attention is dealing with the softmax-ed space where no negative values exist at all. \n\nEven if one wants to have negative values, what is the fundamental challenge with just changing the corresponding component, such as instead of using ReLU, one can simply use other activation functions allowing negative values such as leaky ReLU, Tanh etc. No discussion on this simple baseline choice is provided. This also implies that the motivation of going for more complex design is thus not strong. That being said, for being actionable, it is suggested that the authors compare their approach against baselines using activation functions that allow negative values, such as leaky ReLU or tanh or shift sigmoid with both negative and positive. This would help clarify the advantages of their more complex design over simpler alternatives.\n\nIt is hard to relate Eq (4) with the loss of negative values as discussed before such as Lines 190-194. Why ReLU based mapping will just operate on latter 3 terms and leave out the first term? This seems to be ad-hoc assumption. I would suggest that the authors provide a more detailed explanation or proof of how ReLU-based mappings specifically affect the terms in Eq (4), and why this leads to information loss. \n\nThe other learnable parts to replacing static hand-designed values is good bit but hard to be claimed as addressing major challenges in this context.\n\nAs this work is about scalability of transformer along with the length of input sequences, what is the range of each experiment. In general, the sequences are not that long for vision tasks like image classification, object detection and segmentation. They are not the best suitable test applications. LRA may give longer sequences which is thus better for test. I would suggest the authors to provide specific sequence lengths for each experiment. Additionally, including experiments on tasks with longer sequences would be helpful to better demonstrate the scalability of this approach.\n\nAlso, the scalability of the proposed model along with the increase of sequence length is not evaluated, which should be a key aspect for this problem. Including an experiment or analysis that explicitly shows how this model's performance and efficiency scale with increasing sequence length, compared to baseline methods, would be important and insightful.\n\nSimilarly, the ablation study is best done with long sequence based tasks, but not short ones like image-net classification.\n\nTable 4: while $\\alpha$ in Eq (9) is learnable, why in this table it looks like the value is set to 3/5/7? Or I misunderstand? Besides, please show what are the learned value of $\\alpha$ for each other experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)