Title: DuSA: Fast and Accurate Dual-Stage Sparse Attention Mechanism Accelerating Both Training and Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: Efficient Attention Mechanism

Sparse Attention Mechanism

Transformer
Abstract: This paper proposes the Dual-Stage Sparse Attention (DuSA) mechanism for attention acceleration of transformers. In the first stage, DuSA performs intrablock sparse attention to aggregate local inductive biases. In the second stage, DuSA performs interblock sparse attention to obtain long-range dependencies. Both stages have low computational complexity and can be further accelerated by memory acceleration attention mechanisms directly, which makes DuSA faster than some extremely fast attention mechanisms. The dual-stage sparse attention design provides a lower error in approximating vanilla scaled-dot product attention than the basic single-stage sparse attention mechanisms and further advances the basic sparse attention mechanisms to match or even outperform vanilla scaled-dot product attention. Even in some plug and play situations, DuSA can still maintain low performance loss. DuSA can be used in both training and inference acceleration. DuSA achieves leading performance in different benchmarks: long range arena, image classification, semantic segmentation, object detection, text to video generation, and long context understanding, and accelerates models of different sizes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDuSA introduces a novel Dual-Stage Sparse Attention method to accelerate both the training and inference of vision transformers. The main idea is to split the attention computation into two stages. Both stages implement sparse attention: the first stage is designed to capture local inductive biases within individual blocks, while the second stage aggregates cross-block information. Since both stages can work with the existing attention acceleration techniques, and sparse attention has low computational complexity, DuSA achieves a significant speedup over the vanilla scaled-dot product attention (VSA). Experiments on different benchmarks demonstrate the effectiveness of the proposed DuSA method.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n\t- The proposed method is compatible with existing efficient attention implementations, allowing it to be easily integrated into the existing vision transformer models to get better performance. The experimental results also show the real speedup, which makes the method solid and practical.\n- Weakness\n\t- This paper spends a considerable amount of space analyzing the upper bound of the approximation error, but this bound is not used to guide the design of sparse attention. Additionally, as the author noted, this upper bound is relatively rough and does not seem helpful for sparse attention analysis and design right now (though that could change later). Maybe it’d be better to move this section to the appendix.\n\t- Many discussions and ablation studies are in the appendix. It'd be better to have them in the \"Experiments and Results\" session to help readers understand.\n\t- In the abstract, the paper claims that DuSA maintains low performance loss in plug-and-play scenarios. But this topic is only discussed in Section A2, with evaluations limited to ImageNet. The results show a significant drop in accuracy.\n\nQUESTIONS:\n- There's been a lot of recent works on sparse attention for LLMs. Can DuSA be applied to LLMs — for both prefill and decoding?\n- DuSA is similar to a recent paper XAttention [1], they both use block sparsity and diagonal sparse patterns. From the results of XAttention, antidiagonal patterns outperform diagonal ones, which is leveraged in DuSA. Could you compare DuSA and XAttention in more detail?\n\n[1] Ruyi Xu, et al. XAttention: Block Sparse Attention with Antidiagonal Scoring. arXiv:2503.16428.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes DuSA, which is a Dual-Stage Sparse Attention (DuSA) mechanism to accelerate the attention part of the vision transformers. A theoretical upper bound using DuSA to approximate VSA is derived.  Experiments demonstrated that the proposed DuSA has a lower approximation error than the single-stage design, despite the higher theoretical upper bound. DuSA can be used in both training and inference acceleration.

STRENGTHS AND WEAKNESSES:
Strengths:

1. Theoretical derivation and analysis of the proposed attention mechanism 
2. The plug and play nature of the technique at both training and inference 
3. Overall good results


Weakness:

1. More insights on the experimentally shown good results given the slightly weaker theoretical analysis would be better
2. More analysis on memory consumption patterns during training would be more insightful
3. The plug-and-play's accuracy drops might be a little too large

QUESTIONS:
1. Could the authors provide more explanation on larger theoretical upper bound but experimentally better results?
2. Any insights of the proposed techniques on general transformer architectures? Or is there any thing unique making the proposed techniques especially effective for Vits?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a new multi-stage sparse attention method to reduce the computational complexity compared to naïve full self attention, while keeping the approximation error to a minimum. The efficacy is validated across different vision and long-sequence experiments.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n**Originality & Significance:**  \n- Novel two-stage method that improves upon other sparse single-stage ones while being still competitive in terms of efficiency/throughput\n- Authors demonstrate the applicability via improvements across five different tasks\n\n**Quality:**  \n- Contributions are generally well-placed within the wider area of research\n- Theoretical derivations provided for the underlying concepts\n\n**Clarity:** \n- Contributions and underlying motivation clearly outlined in the intro \n- Explanation of key-concepts well-supported through a good mix of figures \n-\tThe paper is mostly well written and easy to read and follow\n\n## Weaknesses  \n- Clarity and insights provided to the reader could be improved -- see questions\n- Since the proposed DuSA aims to approximate/replace vanilla scaled dot-product attention, at least one ‘simpler’ architecture that exclusively uses this type of attention on a global scale (e.g. ViT) would provide valuable insights -- see questions\n- Somewhat limited experiments in terms of variation in model scale – see questions\n\nQUESTIONS:\n**[Q1]:** The models used for DuSA are CSWin and Swin – both models that use convolutional down-sampling in their multi-stage architecture;   \n$\\textrightarrow$  I would be interested if the authors have tried to use their approach in a conventional ‘monolithic’ ViT? Since this keeps the resolution of the feature maps consistent throughout the architecture, I’d assume this provide even more opportunity to improve efficiency given that quadratic complexity of VSA?   \n$\\textrightarrow$  It could also provide ‘cleaner’ insights given the simple structure of the architecture.\n\n\n**[Q2]:** The models across all experiments seem to be around 50-60M parameters; Have the authors tried different scales of the architectures to validate that the attention works similarly across these, given that larger models can sometimes perform different internal processing steps than small models? How does it work for very small model? \n\n**[Q3]:** While I do appreciate thorough derivations of the proposed methods, I feel like the derivation of the upper bound somewhat stands ‘by itself’ in this paper. After the derivation, the theoretical results are not really used to further analyse or interpret the empirical findings; While there are some empirical findings placed in the appendix in terms of error, they don’t really back up the theoretical ones; Rather, the authors themselves comment that the bounds are probably not tight enough.   \n$\\textrightarrow$  Have the authors thought about better combining these theoretical results with some empirical validation regarding whether the lower approximation errors actually are reflected in the results on actual tasks?  \n$\\textrightarrow$  If not, I’d suggest shortening the bound derivation and using the space instead to provide further insights to the reader in terms of the method, e.g. by including some of the visualisations/analyses currently placed in the appendix – e.g. Fig A2, or the swapped execution of stages (Tab A4), or similar.\n\n\n**Further Comments / Corrections**:  \n- While Strategy 1 and 2 are introduced, it is often unclear which one has been used -- and why there are two in the first place; It might be worth providing further analysis in the main paper, including some visualisations that are currently in the appendix (Fig. A2)!\n- The authors use ‘hard-aware’ multiple times in their manuscript – I assume they mean “hardware-aware”?  \n- In Method 3.1 (VSA) l123: The statement “Since m >> c, …” is incorrect in two ways:   \n  1) The number of patches being much larger than the dimension is not a required condition for the attention-matrix to grow quadratically with m (it does this always, given the nature of its computation)\n  2) The statement itself is not necessarily true: Typical ViT-style VSA on ImageNet results in a sequence length of 196+1 tokens ((224/16) x (224/16)) +cls token; but the number of channels is typically of the same scale or larger (192 up to 1280)\n\nLIMITATIONS:\nSome limitations are discussed in the appendix of the paper, but these relate mainly to the tightness of the derived bound. I’d like to recommend the authors to think about some additional limitations inherent to their current approach in terms of applicability and/or other aspects that might be of interest to the reader/practitioner.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes the Dual-Stage Sparse Attention (DuSA). DuSA performs two stage sparse attention mechanism where in the first it performs intra block to learn local inductive biases and in second stage they learn interblock attention mechanism to learn long range dependencies. this two stage mechanism reduces the error in vanilla scaled-dot product attention compared to basic single-stage sparse mechanisms. The first stage  obtains  the local inductive biases and the second stage aggregates all blocks according to their similarities  to aggregate  long-range dependencies.

STRENGTHS AND WEAKNESSES:
Strengths:
- DuSA has low error in approximating vanilla scaled-dot product attention
- Both stages have low computational complexity and are compatible with memory efficient methods, and can be further accelerated.  DuSA can further improve and accelerate some advanced sparse attention mechansims
- the paper clearly explained the math behind the approximation process of the two stage process in DuSA.
- DuSA is compared with state-of-the-art attention mechanisms in LRA.  DuSA conducted experiments in image classification, semantic segmentation and object detection performance

Weaknesses:
- ELFATT performance is same or better than DuSA in tables 3 and 4. ELFATT has less number of FLOPs compared DuSA

QUESTIONS:
Questions:
- even though DuSA has less memory, DuSA has equal FLOPs or more compared to existing methods like ELFATT.
- Can DuSA be extended auto-regressive based methods?

LIMITATIONS:
Limitations are discussed in the paper
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03