Title: Bridging the Divide: Reconsidering Softmax and Linear Attention
Venue: NeurIPS 2024
Accepted: yes
Keywords: Softmax attention

linear attention
Abstract: Widely adopted in modern Vision Transformer designs, Softmax attention can effectively capture long-range visual information; however, it incurs excessive computational cost when dealing with high-resolution inputs. In contrast, linear attention naturally enjoys linear complexity and has great potential to scale up to higher-resolution images. Nonetheless, the unsatisfactory performance of linear attention greatly limits its practical application in various scenarios. In this paper, we take a step forward to close the gap between the linear and Softmax attention with novel theoretical analyses, which demystify the core factors behind the performance deviations. Specifically, we present two key perspectives to understand and alleviate the limitations of linear attention: the injective property and the local modeling ability. Firstly, we prove that linear attention is not injective, which is prone to assign identical attention weights to different query vectors, thus adding to severe semantic confusion since different queries correspond to the same outputs. Secondly, we confirm that effective local modeling is essential for the success of Softmax attention, in which linear attention falls short. The aforementioned two fundamental differences significantly contribute to the disparities between these two attention paradigms, which is demonstrated by our substantial empirical validation in the paper. In addition, more experiment results indicate that linear attention, as long as endowed with these two properties, can outperform Softmax attention across various tasks while maintaining lower computation complexity. Code is available at https://github.com/LeapLabTHU/InLine.

Bridging the Divide: Reconsidering Softmax and Linear Attention

Abstract

Widely adopted in modern Vision Transformer designs, Softmax attention can effectively capture long-range visual information; however, it incurs excessive computational cost when dealing with high-resolution inputs. In contrast, linear attention naturally enjoys linear complexity and has great potential to scale up to higher-resolution images. Nonetheless, the unsatisfactory performance of linear attention greatly limits its practical application in various scenarios. In this paper, we take a step forward to close the gap between the linear and Softmax attention with novel theoretical analyses, which demystify the core factors behind the performance deviations. Specifically, we present two key perspectives to understand and alleviate the limitations of linear attention: the injective property and the local modeling ability. Firstly, we prove that linear attention is not injective, which is prone to assign identical attention weights to different query vectors, thus adding to severe semantic confusion since different queries correspond to the same outputs. Secondly, we confirm that effective local modeling is essential for the success of Softmax attention, in which linear attention falls short. The aforementioned two fundamental differences significantly contribute to the disparities between these two attention paradigms, which is demonstrated by our substantial empirical validation in the paper. In addition, more experiment results indicate that linear attention, as long as endowed with these two properties, can outperform Softmax attention across various tasks while maintaining lower computation complexity. Code is available at https://github.com/LeapLabTHU/InLine.

1 Introduction

Recent years have witnessed the unprecedented success of Transformer and attention [32] in the field of computer vision [7]. Softmax attention, also known as dot-product attention, has demonstrated remarkable expressive power, leading to state-of-the-art performance across various vision tasks [30, 2, 23, 35]. However, applying Softmax attention in vision also faces challenges. The quadratic complexity of Softmax attention results in prohibitively high computational cost when applied with a global receptive field. Previous works [33, 19, 36, 6, 45] have strived to reduce the computation complexity by restricting receptive fields or introducing sparsity. Although effective, these approaches inevitably compromise Softmax attention's ability for long-range modeling and scalability.

The nature of Softmax attention forces to compute the dot-products between queries and keys QK ⊤ ∈ R N × N at first, and then aggregates values V ∈ R N × C by the normalized score Softmax( QK ⊤ / √ d ) , which accounts for the quadratic O ( N 2 ) complexity w.r.t. the sequence length N . On the contrary, linear attention relaxes the similarity score between Q and K from Softmax to other functions

which can be decomposed into kernels, i.e. , the linear attention replaces the original score function Sim( Q,K ) = Softmax( QK ⊤ / √ d ) with Sim( Q,K ) = ϕ ( Q ) ϕ ( K ) ⊤ , where ϕ ( · ) is the kernel function. This substitution enables a change in the computation order from ( ϕ ( Q ) ϕ ( K ) ⊤ ) V to ϕ ( Q ) ( ϕ ( K ) ⊤ V ) based on the associative law of matrix multiplication, reducing the complexity from O ( N 2 ) to O ( N ) w.r.t. the sequence length N . Nevertheless, every coin has two sides. Linear attention proves to be less effective than Softmax attention [3, 27, 40], whose poor expressive power limits its practical application. Although many pieces of research [27, 28, 22, 9, 40] have attempted to alleviate this issue in different ways, we still do not have a complete understanding of the key factors that contribute to the gap between linear and Softmax attention.

In this paper, we delve into the fundamental differences between linear and Softmax attention, offering two insightful perspectives to demystify the topic: the injective property and local modeling capability. Firstly, we consider attention as a function that maps a query to an attention score. We find that the injectivity of this attention function greatly affects the performance of the model. Specifically, if the attention function is not injective, different queries will induce identical attention distributions, leading to severe semantic confusion within the feature space. Our rigorous analysis has demonstrated that the Softmax attention function is an injective function, whereas the linear attention function is not. Therefore, linear attention is vulnerable to the semantic confusion problem, which largely leads to its insufficient expressiveness. Secondly, our analysis of the attention weight distribution has confirmed that the success of Softmax attention is not solely dependent on its strong long-range modeling capabilities. Effective local modeling is also crucial to achieving optimal outcomes.

To validate our analyses, we present two simple yet effective methods to endow linear attention with the injective property and the local modeling ability, respectively. The widely employed Swin Transformer [19] architecture is used to validate our findings. The results highlight the importance of both properties in the gap between linear and Softmax attention. Moreover, comprehensive experiments demonstrate that linear attention, endowed with these two properties, outperforms the widely used Softmax attention across diverse tasks.

Our main contributions and takeaways are summarized as follows: (1) Injectivity is a key disparity between linear and Softmax attention. While Softmax attention is injective, the non-injective nature of linear attention causes semantic confusion and severely impairs model performance. To the best of our knowledge, our work is the first to conceptualize attention as a mapping function and prove the vital importance of its injective property. (2) Local modeling is still essential to the effectiveness of the attention mechanism, even though it is renowned for its large receptive field and outstanding long-range modeling ability. (3) We challenge the viewpoint that linear attention is inferior to Softmax attention and demonstrate that with the above two properties, linear attention can outperform Softmax attention while maintaining lower computation complexity.

2 Related Works

Vision Transformer and Softmax Attention. Vision Transformer [7] is the pioneer work that introduces self-attention to vision. Since then, attention has found success in various vision tasks [7, 2, 24]. The widely used attention mechanism is Softmax attention [32], also known as dot-product attention, which computes the similarity between all query-key pairs. Although effective, its quadratic computation complexity leads to unmanageable cost when processing global feature maps. Therefore, various approaches [19, 34, 36, 14, 45, 12] have been proposed to reduce the computational overhead of Softmax attention. PVT [33] employs downsampling of keys and values to reduce computational complexity. Swin Transformer [19] restricts the receptive field by introducing window attention pattern. NAT [14] mimics convolution and calculates attention within the neighborhood of each feature, and DAT [36] presents an input-dependent sparse attention pattern.

Linear Attention. As opposed to Softmax attention, Linear attention is another attention paradigm with a natural linear complexity of O ( N ) . Linear attention replaces Softmax with kernel functions, thereby reducing computational complexity to linear through a change in computation order. Nonetheless, prior studies [27, 3, 29, 40, 11] have demonstrated that linear attention performs markedly worse than Softmax attention. CosFormer attributed this discrepancy to the efficient re-weighting mechanism of Softmax attention and proposed cosine re-weighting to enhance linear attention. Nyströmformer [38] and SOFT [22] use matrix decomposition to further approximate Softmax operation. Efficient Attention [28] applies Softmax function to queries and keys. TransNormer [26] identifies

Softmax

Attention

𝐀𝐭𝐭𝐧 𝒒𝟏

𝐀𝐭𝐭𝐧 𝒒𝟐

𝐀𝐭𝐭𝐧 𝒒𝟑

𝐀𝐭𝐭𝐧 𝒒𝟒

Linear

Attention

InLine

Attention

(Ours)

(a)

𝝓(·) = 𝐑𝐞𝐋𝐔(·)

Softmax

Attention

Linear

Attention

InLine

Attention

(Ours)

(b)

𝝓(·) = 𝐑𝐞𝐋𝐔(𝑨 · +𝒃)

Legend

0.31

0.23

0.36

0.22

0.41

0.20

0.44

0.19

0.25

0.20

0.25

0.17

0.24

0.15

0.23

0.13

0.32

0.23

0.32

0.23

0.32

0.23

0.32

0.23

0.25

0.19

0.25

0.19

0.25

0.19

0.25

0.19

k1

k3

k4

0.31

0.23

0.35

0.22

0.39

0.21

0.42

0.21

0.25

0.20

0.26

0.17

0.26

0.14

0.26

0.12

1.0

0.8

0.36

0.25

0.37

0.23

0.39

0.21

0.41

0.19

0.6

0.21

0.18

0.22

0.17

0.24

0.16

0.25

0.14

0.4

0.31

0.24

0.31

0.24

0.31

0.24

0.31

0.24

0.2

0.25

0.21

0.25

0.21

0.25

0.21

0.25

0.21

0.0

0.35

0.23

0.37

0.23

0.40

0.23

0.45

0.22

0.24

0.18

0.24

0.16

0.24

0.13

0.24

0.10

q1

q2

q3

b

k1

k3

k4

91

q2

Eb

q4

k1

k3

k4

that unbounded gradients and attention dilution harm linear attention. FLatten Transformer [9] introduces a focused function to address the over-smoothing issue. MLLA [10] draws inspiration from Mamba [8] to improve linear attention.

Despite their elegant outcomes, the fundamental reason for the disparity between linear attention and Softmax attention remains unclear. In this work, we perform an in-depth analysis of the disparities between linear and Softmax attention, identifying two crucial properties of high-performance Softmax attention: injectivity and local modeling ability. We present both theoretical proofs and experimental verification to validate our findings.

3 Preliminaries

Attention Formulation . Let x ∈ R N × C be an input of N tokens. In each self-attention head, x is transformed into Q = xW Q , K = xW K , V = xW V through projection matrices W Q/K/V ∈ R C × d , where C and d are the channel dimension of module and each head. Therefore, we have Q,K,V ∈ R N × d , and Q i , K i , V i ∈ R d . Based on this, Softmax attention [32] computes the attention weights and calculates the output as the weighted sum of value:

For simplicity, we omit √ d in exp( Q ⊤ i K j / √ d ) since we can equivalently renormalize Q and K . This attention paradigm has been highly successful in modern vision Transformers. However, it should compute the similarity between all query-key pairs, resulting in O ( N 2 ) complexity. Consequently, employing Softmax attention with a global receptive field results in overwhelming computation cost.

Linear attention [15] was proposed to efficiently handle the computation challenge with linear complexity of O ( N ) . Specifically, exp( Q ⊤ i K j ) is replaced by ϕ ( Q i ) ⊤ ϕ ( K j ) , where ϕ is kernel function. In this way, linear attentions reformulate eq. (1) as:

The form of O L i suggests that explicitly computing attention weights L i is unnecessary. Instead, we can change the computation order from ( ϕ ( Q ) ϕ ( K ) ⊤ ) V to ϕ ( Q )( ϕ ( K ) ⊤ V ) based on the associative property of matrix multiplication. By doing so, the computation complexity is reduced to O ( N ) .

̸

Injective Property . Let f : A → B be a mapping function. We call f an injective function if and only if ∀ x, y ∈ A,x = y , it holds that f ( x ) = f ( y ) .

̸

4 Analysing the Gap between Linear and Softmax Attention

Due to its linear computation complexity, linear attention is considered a promising solution to address the computational challenges of Softmax attention in high-resolution scenarios. However, previous works [27, 3, 40] have shown that linear attention's expressive power is significantly lower than that of Softmax attention, rendering it impractical for real-world applications. In this section, we conducted an in-depth analysis of the gap between linear and softmax attention from two perspectives: injective mapping and local modeling capability, and offer both theoretical proofs and experimental verification to enhance understanding of the key disparities between these two attention types.

4.1 Injectivity of Attention Function

We first define the function of Softmax and linear attention as follows:

where Q i denotes the query, and S i , L i are the attention scores in eq. (1) and eq. (2). Given keys K ∈ R N × d , S K , L K can be viewed as the function of query q , mapping each q to its corresponding Softmax and linear attention scores, S K ( q ) and L K ( q ) . Then the final outputs of Softmax and linear attention corresponding to q can be formulated as O S = S K ( q ) ⊤ V and O L = L K ( q ) ⊤ V .

̸

Injective property . In this work, we identify that the injective property of the attention function significantly impacts model performance, which may largely contribute to the gap between linear and Softmax attention. Specifically, we prove that under mild assumptions, the Softmax attention function S K is injective, whereas linear attention function L K is not (Proposition 1 and 2. Please refer to Appendix for complete proof). As a consequent, for two different queries p and q ( p = q ), Softmax attention should produce different attention distributions S K ( p ) = S K ( q ) , while linear attention may yield the same linear attention values L K ( p ) = L K ( q ) . Since different queries p = q typically represent distinct semantics, the non-injective property of linear attention actually leads to semantic confusion, i.e. L K ( p ) = L K ( q ) and O L p = L K ( p ) ⊤ V = L K ( q ) ⊤ V = O L q , making the model unable to distinguish certain semantics.

̸

̸

Proposition 1 (Softmax attention is injective) Given K ∈ R N × d with rank( K ) = d and rank([ K, 1 N × 1 ]) = d +1 . ∀ p, q ∈ R d , p = q, we have S K ( p ) = S K ( q ) .

̸

̸

Proposition 2 (Linear attention is not injective) Let ϕ : R d → R d be a continuous function. ∃ p, q ∈ R d , p = q, s.t. L K ( p ) = L K ( q ) .

We provide an example to better understand the injective property and confusion problem. As shown in Fig. 1(a), there are four collinear vectors with different lengths. Benefiting from injectivity, Softmax attention ensures that each of these four queries obtains distinct attention scores, producing more focused attention distributions for longer queries. Nevertheless, with kernel function ϕ ( · ) = ReLU( · ) , linear attention fails to distinguish the same semantics with different intensities, i.e. collinear queries with varying lengths, resulting in identical attention scores for all these four queries. Consequently,

̸

linear attention is unable to yield more focused attention scores for stronger semantics, which may explain the lack of focus ability discussed in [9]. When using kernel functions with stronger nonlinearity, linear attention encounters more pronounced confusion issues. For instance, in Fig. 1(b), employing kernel function ϕ ( · ) = ReLU( A · + b ) , linear attention assigns exactly the same attention scores to four queries with different directions and lengths. This serious semantic confusion can directly impair model's performance.

̸

Confusion problem in real models . While Fig. 1 illustrates the concept of confusion, it is also crucial to verify if this issue occurs in real models. Therefore, we conduct statistical analysis based on Deit-T. We count the occurrences of confusion for each image (i.e., p = q but S K ( p )=S K ( q ) or L K ( p )=L K ( q ) ) during inference on the ImageNet [5] validation set. As it is rare for two vectors to be strictly equal in floating-point representation, we consider them approximately equal if the L2 norm of their difference is less than 1e-3. The results are provided in Fig. 2. Almost all samples did not encounter confusion on model employing Softmax attention, whereas a large number of samples encountered confusion more than 2 5 times on linear attention model. This proves the existence of confusion problem with linear attention in real models.

The importance of injectivity . We further verify the importance of injectivity by inducing confusion in Softmax attention. To achieve this, we apply additional non-injective map-

0

1

2 1

2 2

2 3

2 4

2 5

2 6

2 7

2 8

2 9

2 10

2 11

2 12

Confusion Occurs

0.0

0.2

0.4

0.6

0.8

Percentage

Softmax

Linear

InLine (Ours)

ping functions to each query before the Softmax attention calculation, i.e., introducing Q i = f ( Q i ) prior to eq. (1), where f is a non-injective function. Specifically, we use f 1 ( q ) = ReLU( q ) ∥ ReLU( q ) ∥ to simulate the confusion observed in linear attention using the kernel function ϕ ( · ) = ReLU( · ) , as depicted in Fig. 1(a), and employ f 2 ( q )= ReLU( Aq + b ) ∥ ReLU( Aq + b ) ∥ to replicate the confusion in Fig. 1(b). As shown in Tab. 1, introducing confusion leads to an obvious decrease in performance, underscoring the crucial role of the attention function's injective property. Therefore, the non-injectivity of linear attention is likely a key factor leading to its limited expressive capacity.

̸

Make linear attention injective . We propose a simple yet effective solution to make linear attention an injective function. The proof of Proposition 2 (see Appendix) demonstrates that ∀ α = 0 , αϕ ( p ) obtains identical scores in linear attention due to the omission of α in division, resulting in noninjectivity. Hence, we simply transform the normalization of linear attention from division to subtraction, presenting our injective linear attention (InLine) as follows:

and the attention output corresponding to Q i can be written as O I i = InL K ( Q i ) ⊤ V . This modification ensures that the attention weights still sum up to 1, while transforming the attention function into an injective one (see Proposition 3). Thus, injective linear attention can distinguish different queries, akin to Softmax attention, and it no longer suffers from confusion problem.

̸

Proposition 3 (InLine attention is injective) Let ϕ : R d → R d be an injective map. Given K ∈ R N × d with rank( ϕ ( K ))= d and rank([ ϕ ( K ) , 1 N × 1 ]) = d +1 . ∀ p, q ∈ R d , p = q, ⇒ InL K ( p ) = InL K ( q ) .

̸

Additionally, similar to linear attention, InLine attention can be calculated with O ( N ) complexity by changing the computation order:

Image

Linear Attn

InLine Attn

Image

Linear Attn

InLine Attn

Since we can compute ∑ N j =1 ϕ ( K j ) and 1 N ∑ N j =1 V j once and reuse them for every query, the overall complexity of InLine attention is 2 Nd 2 + Nd = O ( Nd 2 ) . Accounting for multi-head, the complexity becomes O ( NCd ) , where C and d are the channel dimension of the model and each head.

4.2 Local Modeling Capability

Attention mechanism is famous for its large receptive field and outstanding long-range modeling capability. However, we find that effective local modeling is crucial for the effectiveness of attention.

In Fig. 3, we compute the sum of attention values assigned to local 3 × 3 neighborhoods for each query using DeiT-T. With a total of 14 × 14+1=197 tokens in each attention layer of DeiT-T, if attention scores are randomly assigned, the expected sum of attention for a 3 × 3 neighborhood would be 9 197 . The result shows that all three attention paradigms tend to pay more attention to the neighborhoods of each query, revealing local bias, especially in shallow layers. Notably, Softmax attention allocates a substantial amount of attention to local windows, suggesting a stronger local modeling ability compared to the

1

2

3

4

5

6

7

8

9

10

11

12

Layer

0.0

0.2

0.4

0.6

Local Attn Score

Linear Attn

InLine Attn (Ours)

9/197

other two attention paradigms. Visualizations are provided in Fig. 4 to further confirm this finding.

We speculate that Softmax attention's superior performance stems from robust local priors and strong local modeling capabilities. To validate this hypothesis, we employ attention masks to mask out tokens from various positions and assess their effect on model performance.The results are presented in Tab. 2. Two key observations emerge: 1. Masking out local tokens significantly decreases model performance, while randomly masking out the same number of tokens has a minor impact on results. 2. Softmax attention's performance suffers more severely than InLine attention when local tokens are masked out. These findings demonstrate the significance of local modeling for both attention types and prove that Softmax attention's advantage over InLine attention primarily attributes to its stronger local modeling ability.

Based on our analysis, increasing local bias may enhance the expressive power of InLine attention. In light of this, we employ a MLP to predict additional local attention residual for InLine attention.

Specifically, the output corresponding to Q i is defined as:

where InL K ( · ) denotes InLine attention function, x is the average of input tokens, r is the predicted local attention residual, and V N ( i ) j represents the value in the 3 × 3 neighborhood of Q i . In this way, we explicitly enhance InLine attention's local bias by introducing local attention residual term. We refer to InLine attention with local attention residual, i.e. eq. (6), as InLine attention module . As the local residual term introduces little computational cost Nd + d 2 +9 Nd , the InLine attention module still maintains a linear complexity of O ( N ) .

5 Empirical Study

In Sec. 4, we analyzed two core factors behind the performance gap between Softmax and linear attention, proposing possible remedies. In this section, we conduct empirical verification to fully validate the importance of these two properties and the effectiveness of our methods.

5.1 Implementation

We utilize the popular Swin Transformer architecture [19] to investigate the effects of injectivity and local modeling capability. Specifically, we substitute the original Softmax attention in SwinT with linear attention to establish the baseline model. Subsequently, we introduce the injective property and local bias in turn to assess their respective impacts. To fully verify the effectiveness of InLine attention module, we further apply it to four advanced and representative Transformer models including DeiT [30], PVT [33], Swin [19], CSwin [6] and offer broad comparisons with various state-of-the-art methods using Softmax attention.

5.2 Datasets and Experiment Details

ImageNet classification . The ImageNet-1K [5] recognition dataset contains 1.28M training images and 50K validation images with a total of 1,000 classes. For a fair comparison, we train our model using identical settings as the corresponding baseline model. We use AdamW [21] optimizer to train all our models from scratch for 300 epochs, employing cosine learning rate decay with 20 epochs of linear warm-up. The initial learning rate is 1 × 10 -3 , and the weight decay is 0.05. Augmentation and regularization strategies consist of RandAugment [4], Mixup [42], CutMix [41], and random erasing [43]. Following CSwin [6], EMA [25] is used in the training of InLine-CSwin models.

COCO object detection . COCO [18] object detection and instance segmentation dataset has 118K training and 5K validation images. We follow the training and testing strategies of the corresponding baseline model and employ pretrained InLine backbones to conduct experiments.

ADE20K semantic segmentation . ADE20K [44] is a well-established benchmark for semantic segmentation which encompasses 20K training images, 2K validation images and 150 semantic categories. The same setting as baseline model is adopted.

5.3 Empirical Analysis of Injectivity and Local Modeling

Injective property . As shown in Tab. 3, we adopt four different kernel function ϕ ( · ) to validate the effect of injectivity. As discussed in Sec. 4.1, with kernel function ϕ ( · ) = ReLU( · ) , linear attention fails to distinguish the same semantics with different intensities. Addressing this issue with InLine attention leads to a 2.5 increase in accuracy. When using ϕ ( · ) = ReLU( A · + b ) , linear attention faces more severe semantic confusion, and introducing injective property results in a significant accuracy boost of 9.8, from 70.2 to 80.0. These obvious improvements fully prove the significance of injectivity and validate the effectiveness of our injective linear attention. We also employ two kernel functions that do not ensure non-negativity. Consistent with the findings in [27], linear attention fails to converge without non-negativity assurance. We attribute this to extreme semantic confusion. For example, with ϕ ( · ) = Identity( · ) , linear attention is unable to distinguish completely opposite semantics, assigning identical attention scores to q and -q .

Local modeling capability . Tab. 4 highlights the importance of local modeling capability. In the left table , we apply pure InLine attention to Swin-T and gradually increase the window size from 7 2 to 56 2 . Due to the linear complexity of InLine attention, we can adopt different window sizes while preserving identical computational cost. Larger window sizes lead to larger receptive fields, typically associated with improved performance. However, the results show that the model performance does not improve with increasing window sizes. We believe this can be attributed to the insufficient local modeling capability: a small window size restricts the receptive field but introduces strong local bias, enhancing local modeling, while a large window size enlarges the receptive field but further diminishes local modeling ability. To validate this, we apply InLine attention with local residual and present the results in the right table . Significant improvements can be observed upon the introduction of the local residual term. Additionally, the increase in window size leads to steady performance improvement after introducing local residual, which strongly supports our analysis.

(a)

(b)

(c)

5.4 Main Results and Broad Comparisons

As shown in Tab. 4, InLine-Swin-T with local residual achieves better results than the Swin-T baseline, increasing from 81.3 to 82.4. Therefore, we wonder whether InLine attention module can perform better than the widely adopted Softmax attention in various scenarios. To validate this, we further apply it to several representative Transformers and conduct comprehensive comparisons on image classification, object detection, and semantic segmentation.

ImageNet classification. Firstly, We apply our InLine attention module to DeiT [30], PVT [33], and Swin Transformer [19], presenting the results in Tab. 5. It can be seen that substituting Softmax attention with our method results in notable improvements. For example, InLine-PVT-S outperforms PVT-L with 30% of the parameters and 40% of the FLOPs. Subsequently, We apply our module to the advanced Transformer design, CSwin Transformer [6], and offer a broad comparison with various state-of-the-art models on ImageNet-1K. As depicted in Tab. 6, our InLine-CSwin model not only yields better results than CSwin, but also surpasses various SOTA CNN and Transformer designs. These results demonstrate that our InLine attention module tends to be a superior alternative to the widely used Softmax attention.

Inference throughput analysis. We offer real speed measurements in Fig. 5. As shown in Fig. 5(a), InLine models achieve an obviously better trade-off between accuracy and latency. In Fig. 5(b), we increase the window size from 7 2 to 56 2 . Due to the quadratic complexity of Softmax attention, Swin-T's speed drops sharply as window goes larger. On the contrary, InLine-Swin-T with linear complexity even exhibits higher speed with larger windows. This may be due to the reduction in the latency caused by the window partition. With a global receptive field, InLine benefits from both high performance (see Tab. 4) and fast speed. Furthermore, Fig. 5(c) shows the significant computational advantage of InLine in high-resolution scenarios.

COCOobject detection. Tab. 7 shows that InLine attention consistently improves the results in object detection tasks. For instance, InLinePVT-S outperforms PVT-T with 6.7 box AP under similar FLOPs, and InLine-PVT-L surpasses PVT-M by 3.4 box AP with fewer FLOPs, showing the advantage of InLine attention's linear complexity in high-resolution scenarios.

ADE20K semantic segmentation. We employ our model on two representative segmentation models, SemanticFPN [16] and UperNet [37]. As depicted in Tab. 8, benefited from injectivity and effective local modeling ability, Our InLine achieves better results under all settings with obviously lower computational cost.

Comparison with SOTA linear attention designs . As shown in Tab. 9, our simple InLine attention design outperforms various linear attention methods without bells and whistles. Additionally, our Inline attention can possibly integrate with previous designs to achieve better results, which we leave for future work. For instance, the advanced focused function in FLatten [9] can also be employed in InLine attention.

5.5 Ablation Study

The effectiveness of our two key designs has been verified and detailed analyzed in Sec. 5.3. In Tab. 10, we offer additional results to validate the impact of different kernel functions. It is shown that our InLine attention can effectively work with different kernel functions, further validating the effectiveness of our method. The ReLU and Exponential functions achieve slightly better results. In this paper, we use Identity( · ) as default for simplicity.

6 Conclusion

In this paper, we shed some light on the core factors leading to the performance gap between linear and Softmax attention. We identify and validate two fundamental disparities between these two attention paradigms: injective property and local modeling capability. Injectivity implies that the attention function assigns distinct attention scores to queries with varying semantics, reflecting the ability to distinguish different semantics. Using different kernel functions, linear attention's non-injectivity results in various semantic confusions. Furthermore, despite being recognized for their robust long-range modeling capability, attention mechanisms heavily depend on effective local modeling for impressive results. Thorough empirical validation unequivocally supports our analyses. Our findings also demonstrate that with the above two properties, linear attention can outperform Softmax attention with lower computation complexity.

References

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman. Hydra attention: Efficient attention with many heads. In ECCVW , 2022. 10

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV , 2020. 1, 2

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In ICLR , 2021. 2, 4

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In CVPRW , 2020. 7

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009. 5, 7, 15

Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR , 2022. 1, 7, 8, 9

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021. 1, 2

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 , 2023. 3

Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In ICCV , 2023. 2, 3, 5, 10

Dongchen Han, Ziyi Wang, Zhuofan Xia, Yizeng Han, Yifan Pu, Chunjiang Ge, Jun Song, Shiji Song, Bo Zheng, and Gao Huang. Demystify mamba in vision: A linear attention perspective. In NeurIPS , 2024. 3

Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Shiji Song, and Gao Huang. Agent attention: On the integration of softmax and linear attention. In ECCV , 2024. 2

Yizeng Han, Dongchen Han, Zeyu Liu, Yulin Wang, Xuran Pan, Yifan Pu, Chao Deng, Junlan Feng, Shiji Song, and Gao Huang. Dynamic perceiver for efficient visual recognition. In ICCV , 2023. 2

Ali Hassani and Humphrey Shi. Dilated neighborhood attention transformer. arXiv preprint arXiv:2209.15001 , 2022. 8

Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In CVPR , 2023. 2, 8

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In ICML , 2020. 4

Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR , 2019. 10, 17

Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In CVPR , 2022. 8

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014. 7, 15

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV , 2021. 1, 2, 7, 8, 9, 14, 16

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR , 2022. 8

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR , 2018. 7

Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. Soft: Softmax-free transformer with linear complexity. In NeurIPS , 2021. 2

Zanlin Ni, Yulin Wang, Renping Zhou, Jiayi Guo, Jinyi Hu, Zhiyuan Liu, Shiji Song, Yuan Yao, and Gao Huang. Revisiting non-autoregressive transformers for efficient image synthesis. In CVPR , 2024. 1

Zanlin Ni, Yulin Wang, Renping Zhou, Rui Lu, Jiayi Guo, Jinyi Hu, Zhiyuan Liu, Yuan Yao, and Gao Huang. Adanat: Exploring adaptive policy for token-based image generation. In ECCV , 2024. 2

Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization , 1992. 7

Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. arXiv preprint arXiv:2210.10340 , 2022. 2

Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. In ICLR , 2022. 2, 4, 8

Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In WACV , 2021. 2, 10

Weixuan Sun, Zhen Qin, Hui Deng, Jianyuan Wang, Yi Zhang, Kaihao Zhang, Nick Barnes, Stan Birchfield, Lingpeng Kong, and Yiran Zhong. Vicinity vision transformer. TPAMI , 2023. 2

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML , 2021. 1, 7, 8, 9, 16

Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV , 2021. 8

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017. 1, 2, 3

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV , 2021. 1, 2, 7, 9, 16

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media , 2022. 2, 8

Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In CVPR , 2024. 1

Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In CVPR , 2022. 1, 2

Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV , 2018. 10

Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. In AAAI , 2021. 2

Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. In NeurIPS , 2021. 8

Haoran You, Yunyang Xiong, Xiaoliang Dai, Bichen Wu, Peizhao Zhang, Haoqi Fan, Peter Vajda, and Yingyan Celine Lin. Castling-vit: Compressing self-attention via switching towards linear-angular attention at vision transformer inference. In CVPR , 2023. 2, 4, 10

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV , 2019. 7

Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR , 2018. 7

Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI , 2020. 7

Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV , 2019. 7, 15

Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, and Rynson WH Lau. Biformer: Vision transformer with bi-level routing attention. In CVPR , 2023. 1, 2

Appendix

A Mathematical Proof

This section offers mathematical proofs for the three propositions outlined in the main paper.

A.1 Proof of Proposition 1

Proposition 1 (Softmax attention is injective) Given K ∈ R N × d with rank( K ) = d and rank([ K, 1 N × 1 ]) = d +1 . ∀ p, q ∈ R d , p = q, we have S K ( p ) = S K ( q ) .

̸

̸

̸

Proof. To arrive at a contradiction, assume ∃ p, q ∈ R d , p = q, s.t. S K ( p ) = S K ( q ) . Then we have:

Consider the following two cases:

c = 0 . Then eq. (8) ⇒ p ⊤ K j = q ⊤ K j ⇒ K ( p -q )=0 . As rank( K ) = d , we have p = q , which contradicts the assumption that p = q .

̸

̸

c = 0 . Then eq. (8) ⇒ p ⊤ K j + c = q ⊤ K j ⇒ K ⊤ j p + c = K ⊤ j q ⇒ K ⊤ j ( p -q ) /c =1 . Therefore, we have K ⊤ j ( p -q ) /c = 1 ⇒ K ( p -q ) /c = 1 N × 1 , which contradicts the fact that rank([ K, 1 N × 1 ]) = d +1 and equation Kx = 1 N × 1 does not have a solution.

Both cases arrive at a contradiction, which proves the original proposition.

Notably, the two assumptions rank( K ) = d and rank([ K, 1 N × 1 ]) = d +1 are easy to satisfy in real models. In practice, the number of tokens N is usually much larger than head dimension d . For instance, in the first stage of Swin Transformer [19], N = 56 2 = 3136 and d = 32 . Therefore, we have K ∈ R N × d , N > d . If rank( K ) < d , it indicates that the N key tokens lie in a low-dimension subspace of R d . If rank([ K, 1 N × 1 ]) < d +1 , it means that the N key tokens are in a hyperplane of R d . Given that N is much greater than d , it is highly likely that the N key tokens have various directions in R d , instead of locating in a low-dimension subspace or hyperplane. In this case, it holds that rank( K ) = d and rank([ K, 1 N × 1 ]) = d +1 .

A.2 Proof of Proposition 2

Lemma 1 (Existence of collinear features in linear attention) Let ϕ : R d → R d be a continuous injective function. ∃ p, q ∈ R d , p = q, ∃ α ∈ R , α = 0 , s.t. ϕ ( q ) = αϕ ( p ) .

̸

̸

̸

Proof. Let f : U → S, f ( x ) = ϕ ( x ) ∥ ϕ ( x ) ∥ , U = { x | ϕ ( x ) = 0 , x ∈ R d } , S = { x | ∥ x ∥ = 1 , x ∈ R d } ϕ is a continuous function, so f is continuous on U . ϕ is an injective function, so it has no more than one zero point. Therefore, U = R d or { x | x = x 0 , x ∈ R d } is an open subset of R d , where x 0 is the zero point of ϕ . f ( U ) ∈ S is not an open set, as every point of f ( U ) is not an interior point.

̸

Assume f is injective. Then f : U → S is a continuous injective map. According to Brouwer Invariance of Domain Theorem , U is an open set ⇒ f ( U ) is also open, which contradicts the fact that f ( U ) is not an open set.

̸

Therefore, f is not injective. ⇒∃ p, q ∈ R d , p = q, s.t. f ( p ) = f ( q ) .

̸

̸

Proposition 2 (Linear attention is not injective) Let ϕ : R d → R d be a continuous function. ∃ p, q ∈ R d , p = q, s.t. L K ( p ) = L K ( q ) .

Proof. Consider the following two cases:

̸

ϕ is not injective. ∃ p, q ∈ R d , p = q, s.t. ϕ ( p ) = ϕ ( q ) ⇒ L K ( p ) = L K ( q ) .

̸

ϕ is injective. Lemma 1 ⇒∃ p, q ∈ R d , p = q, ∃ α ∈ R , α = 0 , s.t. ϕ ( q ) = αϕ ( p ) .

̸

A.3 Proof of Proposition 3

Proposition 3 (InLine attention is injective) Let ϕ : R d → R d be an injective map. Given K ∈ R N × d with rank( ϕ ( K ))= d and rank([ ϕ ( K ) , 1 N × 1 ]) = d +1 . ∀ p, q ∈ R d , p = q, ⇒ InL K ( p ) = InL K ( q ) .

̸

̸

̸

The subsequent proof mirrors Proposition 1.

Similar to the analysis in Sec. A.1, rank( ϕ ( K )) = d and rank([ ϕ ( K ) , 1 N × 1 ]) = d +1 are easy to satisfy.

B Complete Experimental Results

We provide complete experimental results on ImageNet-1K classification [5], COCO object detection [18], ADE20K semantic segmentation [44] in Tab. 11, Tab. 12 and Tab. 13. The results demonstrate that InLine attention module consistently outperforms Softmax counterparts across all settings, strongly supporting our analyses and fully validating its effectiveness.

C Model Architectures

We summarize the detailed architectures of four Transformer models used in the main paper, including InLine-DeiT, InLine-PVT, InLine-Swin and InLine-CSwin in Tab.14-18.

D Limitations

In this paper, we shed some light on the core factors leading to the performance gap between linear and Softmax attention. We identify and validate two fundamental and essential disparities between these two attention paradigms: injective property and local modeling capability. Firstly, we prove that linear attention is not injective, which is prone to assign identical attention weights to different query vectors, thus adding to severe semantic confusion problem. Secondly, we confirm that effective local modeling is important for the success of Softmax attention, in which linear attention falls short. The aforementioned two essential differences significantly contribute to the disparities between these two attention paradigms, which is unequivocally proved by our thorough empirical validation in the paper. However, there may be other differences between Softmax and linear attention, and this paper is not exhaustive.

̸

̸

stage

output

InLine-PVT-M

InLine-PVT-L

InLine Block

PVT Block

InLine Block

PVT Block

res1

56

×

56

Conv4×4, stride=4, 64, LN









win 56

×

56

dim 64

head 1









×

3

None









win 56

×

56

dim 64

head 1









×

3

None

res2

28

×

28

Conv2×2, stride=2, 128, LN









win 28

×

28

dim 128

head 2









×

3

None









win 28

×

28

dim 128

head 2









×

8

None

res3

14

×

14

Conv2×2, stride=2, 320, LN









win 14

×

14

dim 320

head 5









×

18

None









win 14

×

14

dim 320

head 5









×

27

None

res4

7

×

7

Conv2×2, stride=2, 512, LN









win 7

×

7

dim 512

head 8









×

3

None









win 7

×

7

dim 512

head 8









×

3

None

stage

output

InLine-CSwin-T

InLine-CSwin-S

InLine-CSwin-B

InLine Block

InLine Block

InLine Block

res1

56

×

56

Conv7×7, stride=4, 64, LN

Conv7×7, stride=4, 64, LN

Conv7×7, stride=4, 96, LN









win 56

×

56

dim 64

head 2









×

2

None









win 56

×

56

dim 64

head 2









×

3

None









win 56

×

56

dim 96

head 4









×

3

None

res2

28

×

28

Conv3×3, stride=2, 128, LN

Conv3×3, stride=2, 128, LN

Conv3×3, stride=2, 192, LN









win 28

×

28

dim 128

head 4









×

4

None









win 28

×

28

dim 128

head 4









×

6

None









win 28

×

28

dim 192

head 8









×

6

None

res3

14

×

14

Conv3×3, stride=2, 256, LN

Conv3×3, stride=2, 256, LN

Conv3×3, stride=2, 384, LN

None









win 7

×

14

dim 256

head 8









×

18

None









win 7

×

14

dim 256

head 8









×

29

None









win 7

×

14

dim 384

head 16









×

29

res4

7

×

7

Conv3×3, stride=2, 512, LN

Conv3×3, stride=2, 512, LN

Conv3×3, stride=2, 768, LN

None









win 7

×

7

dim 512

head 16









×

1

None









win 7

×

7

dim 512

head 16









×

2

None









win 7

×

7

dim 768

head 32









×

2

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work in Sec. D.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper provides a full set of assumptions and a complete, correct proof for the theoretical results, i.e. , the propositions, in Sec. A of the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully discloses all the information needed to reproduce the main experimental results of the paper in Sec. 5.2.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is available at https://github.com/LeapLabTHU/InLine.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all the training and test details in Sec. 5 and Sec. 5.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not reported because it would be too computationally expensive.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide information on the computer resources in Sec. 5.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics, in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work focuses on attention mechanism and has no direct societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks as it is a foundational research focusing on attention paradigm.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use all assets properly according to their licenses, and give credits to the creators in Sec. 5.2.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the computational inefficiency of Softmax attention in Vision Transformers, particularly when handling high-resolution inputs. The authors provide a theoretical analysis showing that the injectivity and local modeling capabilities of attention mechanisms significantly impact performance. They demonstrate that linear attention, which has linear complexity, is not injective and thus performs poorly compared to Softmax attention. To address this, the authors propose modifications to make linear attention injective, resulting in InLine Attention, which improves performance in vision tasks while maintaining computational efficiency. Experiments on high-resolution vision tasks show that InLine attention exhibits comparable performance to Softmax attention.\n\nSTRENGTHS:\n- The paper presents a solid theoretical analysis explaining the performance gap between linear and Softmax attention, focusing on injectivity and local modeling capabilities.\n- The proposed modifications to linear attention are simple yet improve the performance and computational efficiency of Vision Transformers.\n\nWEAKNESSES:\n- The only novelty in the paper is the analysis of the injective property of Softmax and linear attention. The overall novelty and contribution of the paper are limited.\n- The analysis of the local modeling capability of Softmax and linear attention was originally presented by [1]. The authors do not mention this work in their manuscript.\n- The authors do not mention or provide any ablation study of the different embedding functions that can be used with InLine attention. Some works suggest that the exponential function is more beneficial than ReLU [2]. If the authors can show that InLine attention has similar performance for both ReLU and exponential functions, it could justify the strength of the method.\n- There is a lack of experiments on language models. Generally, language models are much harder to train with linear attention than Vision Transformers. Vision tasks mostly require learning local interactions where language exhibits more long-range dependencies between tokens. If the authors can show their method works on language models, it could improve the soundness of this work.\n\nQUESTIONS:\nNormalizing linear attention by its mean (centering) as suggested in eq. 4 should result in negative attention scores. To be strictly positive, it requires all numbers to be greater than $-1/N$. Can the authors provide some analysis or ablation study showing the effect of negative values on the attention on performance?\n\n[1] The Devil in Linear Transformer\n\n[2] Linear Log-Normal Attention with Unbiased Concentration\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-28"}

comments: SUMMARY:
This paper invetigate the linear attention in the Vision task

STRENGTHS:
1. The paper is well written, the motivation is clear.

2. The findings is this work is 1) inear attention is not injective, which is prone to assign identical attention weights to different query vector. 2) effective local modeling is essential for the success of Softmax attention, which linear attention deos not have. Thoses findings may be improtant for design efficient linear transformer.

3. The experimental results are good.

WEAKNESSES:
1. The source code is not avaliable, the reproductability is unclear at this time.

2. how this linear attetnion compare with Vision Mamba since Mamba is efficient.

3. The experimental results are mainly on CV domain, is this algorithm adaptive to NLP or time series domain?

QUESTIONS:
1. The source code is not avaliable, the reproductability is unclear at this time.

2. how this linear attetnion compare with Vision Mamba since Mamba is efficient.

3. The experimental results are mainly on CV domain, is this algorithm adaptive to NLP or time series domain?

LIMITATIONS:
The experimental results are mainly on CV domain, is this algorithm adaptive to NLP or time series domain?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to solve the computational challenges of Softmax attention in vision tasks due to its quadratic complexity with respect to sequence length. Linear attention as an alternative, reduces complexity to linear time by altering the similarity function from Softmax to kernel functions. However, the authors argue linear attention’s poor expressive power and non-injective nature can lead to semantic confusion. The authors propose two methods to enhance linear attention: enforcing injective properties and improving local modeling capabilities. Using the Swin Transformer architecture, they validate these methods, showing that linear attention can match or exceed Softmax attention’s performance while maintaining lower computational costs. The main contributions are highlighting the importance of injectivity and local modeling in attention mechanisms and demonstrating that linear attention, with these enhancements, can outperform traditional Softmax attention.\n\nSTRENGTHS:\n1.\tThis paper thoroughly analyzes the shortcomings of linear attention in vision tasks compared to Softmax attention, identifying non-injective properties and attention confusion as potential root causes. The authors validate these issues through quantitative and qualitative experiments, demonstrating that they contribute to performance drops. The claims seem well-founded, and the verification process appears robust.\n\n2.\tTo address these issues, the authors propose a simple yet effective modification: using subtraction in the normalization of linear attention instead of division, creating a method they call injective linear attention (InLine).\n\n3.\tThe proposed InLine method achieves competitive performance on ImageNet 1k classification and various downstream tasks.\n\nWEAKNESSES:\n1. Although the authors' hypothesis and claims seem reasonable, the performance of the proposed method is not remarkable. This paper also lacks the comparison to some related works. For example, another linear attention based method VVT [A] achieves the Top-1 Acc(%) of 84.1 on ImageNet1k with 61.8M Param and 10.8 GFLOPs. The proposed method, InLine-CSwin-B has a Top-1 Acc(%) of 84.5 on ImageNet1k with 73M Param and 14.9G FLOPs. Although InLine-CSwin-B is higher regarding accuracy by 0.4%, it uses 20% more Params and 40% more GFLOPs. This largely weakens the authors' claims. \n\n2. The analysis of local modeling capability (L264-L275) indicates correlation rather than causation. The authors gradually increase the window size and find it does not lead to better performance. Based on this observation, they claim “the insufficient local modeling capability: a small window size restricts the receptive field but introduces strong local bias, enhancing local modeling, while a large window size enlarges the receptive field but further diminishes local modeling ability”. They find that adding a residual connection can solve the problem and thus claim that the local modeling capability of linear attention is problematic. However, as a common practice, a larger receptive field usually requires a different learning rate to ensure the network converges sufficiently. The effect of the residual connection here may not be as the authors claim, but rather just stabilizing the gradient.\n\n\n[A] Sun, W., Qin, Z., Deng, H., Wang, J., Zhang, Y., Zhang, K., Barnes, N., Birchfield, S., Kong, L. and Zhong, Y., 2023. Vicinity vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10), pp.12635-12649.\n\nQUESTIONS:\nOverall the reviewer is quite concerned about the performance of the proposed method, especially given the fact it loses the direct comparison to a very related and comparable linear attention model.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWhile linear attention reduces the quadratic complexity of softmax attention, it often suffers from inferior performance. The authors analysed the reason behind it and identified two crucial properties which linear attention lacks: 1) injectivity where different queries in linear attention may have the same attention scores, increasing semantic confusion; 2) local modeling where linear attention can’t capture local patterns well. To bridge the gap, the authors proposed injective linear attention (InLine) with local enhancement, which achieves comparable and even better performance than softmax attention across several models and benchmarks.\n\nSTRENGTHS:\n1) The analysis of injectivity and locality includes both theoretical understanding and empirical evidence;\n2) InLine achieves competitive performance to softmax attention, and performs better than several previous linear models.\n\nWEAKNESSES:\n1) The locality issue of linear attention has been discussed in-depth before;\n2) The motivation and formulation of InLine are very similar to FLatten, and doesn’t show substantial quality gain to FLatten;\n3) It would be great to have language modeling experiments;\n\nQUESTIONS:\n1) This is not the first paper discussing the locality problem of linear attention. Check [1] for more details. \n2) The intuition of InLine is very similar to FLatten. For example, FLatten removes the division operation so it satisfies the injectivity and Flatten adopts depthwise convolution, enhancing locality modeling. Based on Table 9, InLine doesn’t outperform FlAtten significantly. The authors should give a more comprehensive analysis on the similarity and differences compared to Flatten and why InLine is preferred.\n\n[1] Qin et al., 2022; The Devil in Linear Transformer.\n\nLIMITATIONS:\nThe authors discussed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}