Beyond 2:4: Exploring V:N:M Sparsity for Efficient Transformer Inference on GPUs

Authors: Kang Zhao, Tao Yuan, Han Bao, Zhenfeng Su, Chang Gao, Zhaofeng Sun, Zichen Liang, Liping Jing, Jianfei Chen
Primary area: infrastructure, software libraries, hardware, systems, etc.
Keywords: V:N:M sparisity, Transformer inference acceleration, accuracy-speedup tradeoffs, channel permutation

Abstract:
To date, 2:4 sparsity has stood as the only sparse pattern that can be accelerated
using sparse tensor cores on GPUs. In practice, 2:4 sparsity often possesses low
actual speedups (≤ 1.3) and requires fixed sparse ratios, meaning that other ratios,
such as 4:8, 8:16, or those exceeding 50% sparsity, do not incur any speedups on
GPUs. Recent studies suggest that V:N:M sparsity is promising in addressing
these limitations of 2:4 sparsity. This sparsity divides a weight matrix into mul-
tiple V×M blocks, pruning (M-4) columns within each block and applying 2:4
sparsity to the remaining columns. V:N:M sparsity inherently encompasses 2:4
sparsity but allows for higher and more flexible pruning ratios, typically resulting
in greater practical speedups. However, regarding accuracy, the effects of V:N:M
sparsity on broader Transformer models, such as vision Transformers and large
language models (LLMs), are largely unexamined. Moreover, Some specific is-
sues related to V:N:M sparsity, such as how to select appropriate V and M values,
remain unresolved. In this study, we thoroughly investigate the application of
V:N:M sparsity in vision models and LLMs across multiple tasks, from pretaining
to downstream tasks. We propose three key approaches to enhance the applica-
bility and accuracy of V:N:M-sparse Transformers, including heuristic V and M
selection, V:N:M-specific channel permutation and three-staged LoRA training
techniques. Experimental results show that, with our methods, the DeiT-small
achieves lossless accuracy at 64:2:5 sparsity, while the DeiT-base maintains ac-
curacy even at 64:2:8 sparsity. In addition, the fine-tuned LLama2-7B at 64:2:5
sparsity performs comparably or better than training-free 2:4 sparse alternatives on
downstream tasks. More importantly, V:N:M-sparse Transformers offer a wider
range of speedup-accuracy trade-offs compared to 2:4 sparsity. Overall, our explo-
ration largely facilitates the V:N:M sparsity to act as a truly effective acceleration
solution for Transformers in cost-sensitive inference scenarios.

Decision: reject
Reviewer scores: 6, 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730074658379

thread_type:
official_review

<thread_object>
creator: reviewer_qk6w

<content>
summary:
This paper studies accuracy and speedup tradeoff of V:N:M sparse Transformers. It introduces a heuristic method for parameter selection, a channel permutation approach for low budget training and a three-stage LoRA training procedure applicable to various scenarios. Experimental results show that the proposed methods significantly accelerate the V:N:M sparse Transformers while maintaining lossless accuracy compared to 2:4 sparsity.

strengths:
1. The authors propose three training approaches to enhance the applicability and accuracy of V:N:M sparse Transformers. They present a heuristic V and M selection method to find the tradeoff between accuracy and speedup for V:N:M sparse Transformers. To improve the accuracy within limited training budgets, they introduce a channel permutation approach. Finally, they introduce a Dense-Dynamic-Fixed mask training procedure to maintain the model accuracy.

2. The paper is well-written and easy to follow up. The introduction and motivation are clear and well-organized. The experiments are comprehensive.

weaknesses:
1. Why is the paper claimed that only a 2:4 sparse pattern can be accelerated by sparse tensor cores? I have some doubts about it. One could write a custom CUDA kernel and accelerate based on the indices of the weights.
2. How did you implement the kernel using sparse tensor core? For higher sparsity (like 2:12, 2:24), the observed speedup falls short of the theoretical speedup, possibly due to inefficiencies in the implementation. Moreover, would you consider using cusparse/cublas or other CUDA kernels to compare speedup with the proposed method at highly sparsity?
3. In Fig.4(a), the authors compare accuracy improvement under similar speedup conditions. In my view, more parameters generally lead to higher accuracy, so it would be fairer to compare accuracy at the same sparsity. Fig.4(a) shows how V and M selection impacts the accuracy and speedup of V:N:M sparse Transformers. However, the V:N:M setting in Fig.8 differs. Does this imply that V and M selection is not applied in Fig.8?

questions:
Please see the above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730554382102

thread_type:
official_review

<thread_object>
creator: reviewer_dgv5

<content>
summary:
The conventional 2:4 sparsity has been the only sparse pattern capable of accelerating GPU sparse tensor cores, and its performance was well-established. 
However, the paper claims that 2:4 sparsity has limitations, such as low actual speedups and support for only fixed sparsity ratios. 
To address this, the paper adopts the V:N:M approach. 
V:N:M sparsity combines n:m pruning with structured pruning and claims to overcome the fixed 50% pruning ratio limitation of traditional 2:4 sparsity.

strengths:
1. Figure 1 is well-drawn and easy to understand.
2. The experiments conducted on the ViT model and LLM model are excellent.

weaknesses:
1. [Novelty] I am not sure about the differences between this paper and the Venom paper [1]. Wasn't V:N:M proposed in the Venom paper [1]? How does it differ from Venom [1]?
2. The Venom paper proposed a GPU kernel for speedup sparse model on GPUs. how does this paper handle processing on the GPU? Does it propose a dedicated GPU kernel? If so, what are the differences compared to the Venom [1] GPU kernel?
3. The title of the paper includes "Transformer"—what specific characteristics of Transformers does it take into account?
4. Looking at Table 5, the caption lacks sufficient explanation. For instance, a clear description of "$\Delta$" is necessary in the caption.


[1] Venom: A vectorized N:M format for unleashing the power of sparse tensor cores. SC 2023

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650039063

thread_type:
official_review

<thread_object>
creator: reviewer_wkmr

<content>
summary:
The paper addresses the limitations of 2:4 sparsity, by thoroughly investigating the application of V:N:M sparsity in vision models and LLMs across multiple tasks. The authors propose methods for selecting optimal V and M values and introduce techniques like V:N channel permutation and a three-stage LoRA training method, designed to enhance the accuracy and applicability of V:N Transformers, particularly in vision and language models.

strengths:
1. The paper goes beyond theoretical contributions by proposing practical methods to improve the deployment of V:N:M
sparsity. Techniques like heuristic selection of V and M values, channel permutation, and three-staged LoRA training are thoughtfully designed to optimize accuracy-speedup trade-offs, even with limited training resources.
2. The authors validate their approach across multiple benchmarks, including vision Transformers and large language models (LLMs), demonstrating that V:N:M sparsity can achieve high speedups with minimal accuracy loss.

weaknesses:
1. The heuristic method for selecting V and M values may not work for extensive Transformers models. My concerns are that even in the paper the proposed methods can achieve lossless performance on specific model DeiT-base and LLama2-7B. But what are the expected results on other models?
2. The novelty and originality may not be enough. This paper is more like an engineering implementation technique. Authors may need to explain originality more.

questions:
1. The heuristic method for selecting V and M values may not work for extensive Transformers models. My concerns are that even in the paper the proposed methods can achieve lossless performance on specific model DeiT-base and LLama2-7B. But what are the expected results on other models?
2. The novelty and originality may not be enough. This paper is more like an engineering implementation technique. Authors may need to explain originality more. 
3. "V:N:M-specific channel permutation, which improves the accuracy of V:N:M-sparse Transformers". Why channel permutation can improve accuracy? 
4. "a higher MD leads to better Transformer accuracy" Is this claim from extensive experiments or other places?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691780626

thread_type:
official_review

<thread_object>
creator: reviewer_w4dx

<content>
summary:
This paper explores applying V:N:M sparsity to the weight parameters of transformer-based models, which can well retain accuracy while harvest inference speedup from sparse tensor cores in advanced GPUs. To achieve this, the authors propose three techniques to convert a dense transformer into a V:N:M sparse model, including a heuristic method for selecting V and M values, V:N:M-specific channel permutation, and a three-stage LoRA training approach for LLMs. They conduct extensive experiments on mainstream transformers, including DeiT, Swin Transformer, and Llama2-7B, across multiple tasks, demonstrating that their method outperforms naive 2:4 sparsity versions. Overall, this paper could be a good empirical contributions, paving the way for V:N:M sparsity compression for Transformers.

strengths:
This paper is well-written and easy to follow. It integrates multiple techniques to accelerate V:N:M-sparse transformers. While the novelty of each technique is somewhat limited, the resulting performance improvements are substantial.
For theory strengths, the paper formulates the selection of V and M value as an optimization problem, balancing accuracy with speedup constraints. To solve this efficiently, they propose a two-phase sifting process to identify the optimal (V, M) combinations; Moreover, it adapts channel permutation to V:N:M sparsity, applying it to both the input and output of weight matrices to enhance accuracy for low training budgets. Finally, to ensure stable training of V:N sparse transformers, the authors integrate a three-stage LoRA training approach, consisting of Dense LoRA, Sparse LoRA with a dynamic mask, and Sparse LoRA with a fixed mask.

weaknesses:
1. While it's intuitive that a smaller M results in lower sparsity in the sparse transformers, the process for excluding certain (V, M) combinations is unclear. Additionally, there is insufficient evidence in the paper demonstrating that mask diversity (MD) improves transformer accuracy.
2. In Definition 1, the objective is to maximize accuracy under speedup constraints s. However, there should be experimental evidence demonstrating that the chosen V and M values meet these constraints.
3. In Section 5.4, although V:N:M sparse Llama2 outperforms RIA and Wanda, it still suffers significant accuracy loss compared to the dense counterpart. For instance, HellaSwag accuracy drops from 57.23 to 42.88, and ARC-C from 43.3 to 34.3.
4. Equation (3) lacks the description of $\overline{M}_{t-1}$
5. In section 4.3, the definitions of $Mv$, $Wv$, $Bv$ and $ Av$ are unclear. Also, it would better to disclose detailed LoRA configurations in the three-staged LoRA training experiments (section 5.4) , such as dynamic sparse mask initialization, the interval of updating sparse masks, the actual training iteration assignment two different stages etc.
6. Figure 3 does not intuitively illustrate "in the absence of regularization, frequent updates to the masks can negatively impact the gradient flow of V:N:M sparse Transformers during fine-tuning". Additionally, adding a comprehensive empirical study would make it more convincing to draw the conclusion that "the iterations for the first two stages should not exceed 10% of the total iterations."

questions:
1. For large language models (LLMs), latency is not solely determined by FLOPs; it is typically constrained by memory bandwidth and access patterns. In Definition 1, is the measured speedup based on a batch size of 1? Furthermore, when varying the batch size, will this lead to different V and M sparsity values? 
2. Can channel permutation enhance the performance of SR-STE training or LoRA training?
3. How is the regularization coefficient $\lambda$ in equation (3) tuned to improve training stability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721563087

thread_type:
official_review

<thread_object>
creator: reviewer_apm6

<content>
summary:
This paper investigates the use of V:N:M sparsity in Transformer models and proposes three key methods to enhance its performance: heuristic V and M selection, V:N:M channel permutation, and a three-stage LoRA training technique. The experimental results demonstrate that V:N:M sparsity offers a broader range of speedup-accuracy trade-offs compared to traditional 2:4 sparsity.

strengths:
1. This paper introduces a comprehensive framework that generates highly accurate V:N:M-sparse Transformers under different constraints, allowing users to balance the system performance and model accuracy.

2. The paper addresses key challenges in applying V:N:M sparsity by introducing three innovative techniques: heuristic V and M selection, V:N:M channel permutation, and a three-stage LoRA training technique. 

3. Experiments show superior performance of proposed methods.

weaknesses:
1. The paper lacks a comparison with the latest related works, which limits its ability to contextualize the advantages and innovations within the current research landscape, such as [1]. Moreover, please also report the model performance without sparsity in Figure 4,5,7,8,9 to show the influence of introducing model sparsity. 

2. The experiments in this paper are limited to small-scale models, such as ViT, DeiT, and Llama-7B. It does not extensively explore the performance on larger models, like ViT-Huge or LLaMA models with higher parameter counts, where the effectiveness of the sparsity method might vary.

3. The work would be stronger if it provided performance benchmarks for Llama2-7B on pretraining benchmarks, such as MMLU or GSM-8k, to give a more comprehensive evaluation of the model's capabilities across a broader set of important benchmarks. If computational resources are limited, consider performing post-training on Llama2-7B to demonstrate the applicability of this paper's methods.

[1] Yang H, Yin H, Shen M, et al. Global vision transformer pruning with hessian-aware saliency[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 18547-18557.

questions:
This paper propose three techniques, including heuristic V and M selection, V:N:M channel permutation, and a three-stage LoRA training technique. Please further explain the relationship between these techniques, are they complementary, sequential, or something else?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732389769926

thread_type:
official_comment

<thread_object>
creator: area_chair_pstf

<content>
title:
Engage in Discussions Before Nov 26 (AoE)

comment:
Dear Reviewers,

First, let me thank you for your invaluable contributions to the ICLR review process. Your constructive feedback plays a key role in enhancing the quality of submissions.

---

As we approach the final days of the discussion phase (ending **Nov 26, 2024, AoE**), I kindly remind you to:

- Please take a moment to review the authors' responses to your comments. This is an opportunity to clarify any remaining questions, acknowledge misunderstandings, and refine your evaluation.

- If you need further clarification, don't hesitate to post your comments as soon as possible.

- If the authors' responses address your concerns or provide new insights, please consider updating your score to reflect this.

---

Your thoughtful participation during this phase is especially valuable for borderline papers, where additional input can be critical to ensuring a fair decision-making process.

I understand how busy this time of year can be and truly appreciate the time and care you dedicate to this important role. Your efforts make a tangible impact on the success of ICLR.

Thank you once again for your dedication.

Best regards,

Area Chair, ICLR 2025
</content>
</thread_object>

</thread>
<thread>
creation date:
1733237699745

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal (Many Thanks to All Reviewers, Area Chairs, and Program Chairs)

comment:
We sincerely thank all reviewers for their thoughtful evaluations. We are encouraged by their recognition of our clear and well-organized motivation (QK6w), comprehensive evaluations across both vision Transformers and LLMs (apM6, WKMR, QK6w, Dgv5), substantial performance improvements (W4Dx), and the property of "easy to follow" (W4Dx, Dgv5, QK6w). Below, we briefly re-emphasize some key points:

## Distinction from Initial Work
The initial work for V:N:M sparsity focused on GPU kernel acceleration, while our research centers on algorithmic design aimed at enhancing the accuracy and applicability of V:N:M-sparse Transformers. Our work and the initial study operate at different levels of V:N:M sparsity.

## Novelty of Our Work
We introduce three innovative methods that significantly improve the accuracy of various V:N:M-sparse Transformers under different constraints: **a heuristic V and M selection method**, **a V:N:M-specific channel permutation (CP) method**, and **a three-staged LoRA training method**. Each of these methods is supported by our insights into V:N:M sparsity and addresses challenges that have never before been explored in this domain.

## Key Insights in Our Work
1. We demonstrate that, contrary to common assumptions, higher parameter counts do not necessarily correlate with improved accuracy for V:N:M-sparse Transformers. Both the sparse granularity defined by V and the retained parameter counts dictated by M significantly impact accuracy.
2. We reveals that CP for V:N:M sparsity differs fundamentally from that used for 2:4 sparsity, as both input and output channel permutations affect retained parameter norms for V:N:M-sparse weight matrices.
3. We empirically prove for the first time that brief dynamic mask changes following dense LoRA initialization can greatly enhance LoRA training effectiveness.

## Contributions of Our Work
1. Our proposed framework within the three novel methods allows V:N:M-sparse Transformers to achieve comparable or superior accuracy relative to 2:4 sparsity. Notably, with our efforts, the DeiT-small and DeiT-base models maintain lossless accuracy at 64:2:5 and 64:2:8 sparsity, respectively. Given that V:N:M-sparse Transformers achieve higher speedups and offer a broader spectrum of speedup-accuracy trade-offs than 2:4 sparsity, our research promotes the widespread implementation of V:N:M sparsity as an effective strategy for Transformer compression. **We establish that V:N:M sparsity outperforms 2:4 sparsity in compressing highly redundant Transformers**.
2. More broadly, our proposed three methods are applicable to neural architectures where linear layers play predominant roles, such as ResMLP.

Additionally, we supplement extensive experiments during the rebuttal phase, including the following (note that the figure and table numbers refer to those in the rebuttal webpage for your convenience):
- Results for large-scale V:N:M-sparse Transformers (ViT-large in [Figure 3.1](https://anonymous.4open.science/r/conf_dat_Anonymous-8055/downstream_vit-large.pdf), ViT-huge in [Figure 3.2](https://anonymous.4open.science/r/conf_dat_Anonymous-8055/downstream_vit-huge.pdf), and Llama2-13B in [Figure 3.3](https://anonymous.4open.science/r/conf_dat_Anonymous-8055/downstream_Llama2-13B.pdf)).
- V and M selection results across various batch sizes (Table 2.5).
- An ablation study on the proportion of the first two stages in our three-staged LoRA training method (Table 2.4).
- Comparisons of loss changes between fixed and dynamic LoRA training ([Figure 2.4](https://anonymous.4open.science/r/conf_dat_Anonymous-8055/lora_loss_change.pdf)).
- The impact of CP on LoRA training (Tables 2.6, 2.7, and 2.8).
- Results showing V:N:M-sparse Transformers with more parameters but lower accuracy (Table 3.5).
- Results for V:N:M-sparse ResMLP with our three proposed methods ([Figure 4.1](https://anonymous.4open.science/r/conf_dat_Anonymous-8055/downstream_resmlp-b24.pdf), Table 4.3).

Moreover, we provide clarifications on the following aspects:
- Details on the motivation and principles of mask diversity (A1 to W4Dx, A4 to WKMR).
- Evidence that selected V and M values meet optimization objectives (A2 and A7 to W4Dx).
- Hyperparameters used in the three-staged LoRA training method (A5 to W4Dx).
- The distinctions between our work and the initial study regarding V:N:M sparsity (A4 to WKMR, Part 1 to Dgv5).
- Specific GPU cores employed to accelerate different sparse patterns (A1 to QK6w).
- The importance of training token counts and resources for achieving high absolute accuracy in sparse LLMs (Explanation 2 to apM6, A3 to W4Dx).

All significant experiments and clarifications have been integrated and highlighted in our revised paper, which also addresses several typographical errors.

Best wishes,

authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper explores the use of V:N:M sparsity to accelerate ML models, addressing limitations of the conventional 2:4 sparsity format. The authors propose three techniques to enhance the accuracy and applicability of V:N:M sparsity pattern: (a) a heuristic to select V and M values to balance between accuracy and speedup, (b) a V:N:M-specific channel permutation technique to improve performance under limited training budgets, (c) a three-staged LoRA training to maintain accuracy during fine-tuning.

## Strengths

- The proposed framework applies to different architectures, including both vision models and LLMs, which broadens its potential impact (Reviewer WKMR, Reviewer QK6w).
- The introduction of V and M selection using mask diversity (MD) contributes to the understanding of accuracy-speedup trade-offs (Reviewer WKMR).
- The channel permutation method and three-staged LoRA training approach add practical value for fine-tuning sparse models under limited resources (Reviewer W4Dx, Reviewer QK6w).
- The authors run extensive evaluations on various benchmarks, demonstrating the potential of V:N:M sparsity to achieve competitive results in different scenarios (Reviewer QK6w, Reviewer W4Dx).


## Weaknesses 

- V:N:M sparsity underperforms significantly in Llama models, as highlighted by Reviewer apM6. Despite achieving a 1.5x speedup, the accuracy drop relative to the dense baseline cannot be overlooked. This suggests that V:N:M sparsity may not generalize well to LLMs.
- Reviewer apM6 specifically noted:

> Based on the post-training results for the Llama models, it seems that V:N:M sparsity is not well-suited for Llama model training. While it does show improvements compared to the baseline, the accuracy loss cannot be neglected relative to the 1.5x speedup it provides.

- Several reviewers (Dgv5, WKMR) questioned the originality of the contributions, noting that the work appears to build on existing studies like Venom without sufficiently distinguishing itself. Reviewer Dgv5 stated: 

> I think there are still 'lack of novelty' of this paper.

- The heuristic nature of V and M selection raises doubts about its scalability to larger models, as pointed out by Reviewer WKMR. Furthermore, the experiments do not adequately validate whether the proposed methods consistently meet the claimed speedup and accuracy trade-offs under different configurations.

- The work heavily relies on the Venom GPU kernel for speedup, which detracts from the technical contributions of the paper (Reviewer Dgv5, Reviewer QK6w).

---

The primary reason for recommending **rejection** is the inconsistent results and limited generalizability of V:N:M sparsity, particularly for LLMs. As Reviewer apM6 noted, the accuracy loss for Llama models is substantial and cannot be justified by the modest speedup achieved. This limitation directly challenges the claim of broad applicability and undermines the practical impact of the proposed framework. 

While the paper introduces some promising ideas, the inconsistent results and lack of sufficient contributions convinced me to recommend rejection. Future work could focus on addressing the limitations in LLMs, improving the scalability of the techniques, and establishing clearer distinctions from prior work.