NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM
Authors: Connor Holmes
Minjia Zhang
Yuxiong He
Bo Wu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Efficient Inference Methods
Model Compression
Sparsification
Transformer
Natural Language Processing

Abstract: Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained Transformer networks. However, these models often contain hundreds of millions or even billions of parameters, bringing challenges to online deployment due to latency constraints. Recently, hardware manufacturers have introduced dedicated hardware for NxM sparsity to provide the flexibility of unstructured pruning with the runtime efficiency of structured approaches. NxM sparsity permits arbitrarily selecting M parameters to retain from a contiguous group of N in the dense representation. However, due to the extremely high complexity of pre-trained models, the standard sparse fine-tuning techniques often fail to generalize well on downstream tasks, which have limited data resources. To address such an issue in a principled manner, we introduce a new learning framework, called NxMTransformer, to induce NxM semi-structured sparsity on pretrained language models for natural language understanding to obtain better performance. In particular, we propose to formulate the NxM sparsity as a constrained optimization problem and use Alternating Direction Method of Multipliers (ADMM) to optimize the downstream tasks while taking the underlying hardware constraints into consideration. ADMM decomposes the NxM sparsification problem into two sub-problems that can be solved sequentially, generating sparsified Transformer networks that achieve high accuracy while being able to effectively execute on newly released hardware. We apply our approach to a wide range of NLP tasks, and our proposed method is able to achieve 1.7 points higher accuracy in GLUE score than current best practices. Moreover, we perform detailed analysis on our approach and shed light on how ADMM affects fine-tuning accuracy for downstream tasks. Finally, we illustrate how NxMTransformer achieves additional performance improvement with knowledge distillation based methods.

NxMTTranffrrmr: Semi-Structured Sparsification for Natural Language Understanding via ADMM
Connor Holmes
Minjia Zhang Microsoft Bellevue, WA 98004 minjiazdmicrosoft...om com
Colorado School of Mines Golden, CO 80401 cholmes@mines... edu
Yuxiong He Microsoft Bellevue WA, 98004 yuxhe@microsoft..o com com
Bo Wu
Colorado School of Mines Golden, CO 80401 bwudmines edu
Abstract
Natural Language Processing (NIP) has recently achieved great success by using huge pre-trained Transformer networks. However, these models often contain hundreds of millions or even billions of parameters, bringing challenges to online deployment due latency constraints. Recently, hardware manufacturers have introduced dedicated hardware for NxM sparsity to provide the flexibility of un- structured pruning with the runtime efficiency of structured approaches. NxM sparsity permits arbitrarily selecting M parameters to retain from contiguous group of N in the dense representation. However, due the extremely high com- plexity of pre-trained models, the standard sparse fine-tuning techniques often fail generalize well on downstream tasks, which have limited data resources. To address such an issue in principled manner, we introduce new learning framework, called NxMTransformer, to induce NxM semi-structured sparsity on pretrained language models for natural language understanding to obtain better performance. In particular, we propose to formulate the NxM sparsity as a con- strained optimization problem and use Alternating Direction Method of Multipliers (ADMM) to optimize the downstream tasks while taking the underlying hardware constraints into consideration. ADMM decomposes the NxM sparsification prob- lem into two sub- oroblrosssms that can be solved sequentially, generating sparsified Transformer networks that achieve high accuracy while being able to effectively execute on newly released hardware. We apply our approach to a wide range NLP tasks, and proposed method is able to achieve 1.7 points higher accuracy in GLUE score than current best practices. Moreover, we perform detailed analysis our approach and shed light on how ADMM affects fine-tuning accuracy for downstream tasks. Finally, we illustrate how NxMTransformer achieves additional performance improvement with knowledge distillation based methods.
Introduction
Large-scale Transformer networks have achieved remarkable success for wide variety of natural language tasks, including natural language inferencing, sentiment analysis, question answering, and others. The state-of-the-art of these NLP models employs transfer learning paradigm which contains two stages: a cupi-super pre-training stage that trains a masked language modeling on massive web text, followed by fine-tuning stage where the pre-trained model is adapted specific
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

downstream tasks with much smaller datasets. The size of these language models has dramatically increased in recent years; even relatively small models [ 227 consist of hundreds of millions of parameters while larger models [23] 25 24 stretch well into multi-billions. The large model size brings challenges for both deployment and training costs. While training large-scale model often requires significant time even on large training clusters, the trained model also incurs significant challenges in deployment due to latency and capacity constraints.
These challenges motivate techniques to compress and accelerate these models, even in datacenter environments where hardware resource limitations are at their smallest. These techniques include but are not limited to model quantization [35]
B. low rank decomposition [19, knowledge 4 These compression techniques can often be combined to maximize performance gains 37
28
distillation [T 27. and model sparsification
Among different techniques, sparsification attempts to identify parameters that can be removed from the model without significantly compromising model accuracy. Sparsification techniques typically fall under two broad categories: unstructured and structured. Unstructured techniques will remove the individual parameters based on their importance (e.g., weight magnitude), which often yield the best accuracy but are unfriendly to modern hardware. Structured sparsification techniques remove parameters in groups (e.g., entire rows or columns), which result in models that retain their dense structure but can also add constraints that limit the expressiveness of the model.
Recently, hardware manufacturers introduced support for NxM stritstructured sparsity to provide the benefits of both structured and unstructured sparsity. In NxM semi-structured sparsity, model may preserve M parameters from each contiguous group of N original parameters. This relatively weak constraint on sparsification allows for sparse representations similar in flexibility to those of unstructured approaches but also permits efficient hardware implementation as well. Consider the 4x2 semi-structured sparsity implementation found on NVIDIA GPUs based on the Ampere architecture [.. The Ampere architecture introduces a small set of multiplexers that select values from the input matrix corresponding the retained values in the weight matrix 221. The output of this operation remains compatible with the efficient Tensor Cores for dense matrix-matrix operations. While the Ampere GPUs are the first to market with this capability, the matrix multiplication accelerators within them are similar to those used by other accelerators [5] which should enable other vendors to provide support for this type of sparsity as well.
To induce semi-structured sparsity, a small set of approaches have been offered. ASP (1 proposes training the dense network until convergence, using single-shot magnitude-base pruning to induce sparsity conformant to the NxM constraints, and repeating the original training procedure to recover accuracy. Zhou, et al. (36] uses sparse-refined straight-through estimator (SR-STE) to introduce the sparsity throughout the entire training process. Both of these techniques pose some challenges for large-scale pre-trate pre-trained Transformer networks in particular. ASP can require a second costly sparse pre-train of the model and the single-shot magnitude-bast transferrability to different downstream tasks. SR-STE on the other hand sparsifies from the random model initialization, which avoids the costly sparse retraining but also necessitates performing the pre- tre-training process with only a sparse representation in mind. Since NxM sparse hardware is not yet ubiquitous, maintaining compatibility with a single dense pretrained representation is valuable so the costly pre-training process does not need to be performed for both sparse and dense networks.
pruning might hurt the knowledge
In this work, we propose to effectively induce NxM semi-structured sparsity for large-scale Trans- former networks to leverage newly released Sparse Tensor Core hardware by making the following contributions. (1) We introduce a principled method, NxMTransformer (See Figure D to obtain Transformer networks with NxM semi-structured sparsity for different downstream tasks using Alter- nating Direction Method of Multipliers (ADMM), a technique designed for large-scale non-convex optimization problems with constraints. Such a method allows us to alternate promoting the NxM sparsity of the network and optimizing the fine-tuning performance. (2) We conduct comprehensive experiments and demonstrate that NxMTransformer achieves 1.7 points higher accuracy than state-of- the-art techniques to introduce NxM sparsity for natural language processing tasks. (3) We perform detailed analysis on our approach and shed light on how ADMM affects fine-tuning accuracy for downstream tasks. (4) Finally, we show that NxMTransformer is complimentary to alternative model compression techniques such as knowledge distillation and can further sparsify distilled models while preserving accuracy.
2

---

Query
Self Attention
Attention Output
Classifier Network
Embeediings
Key
PFNI
FFN2
Transformer Layer (x12)
Compress for Execution
ADMM
Dense Pretrained
Dense Representation of
Non-zero
Model
Sparse Finetuned Model
Values
Figure 1: The layers sparsified by NxMTransformer are highlighted in blue in the block structure of a BERT model. As shown for FFNI, NxMTransformer simultaneously finetunes the pretrained representation while inducing NxM stri-structured sparsity using ADMM. This sparse model be trivially converted to the deployment format for compatible hardware.
2 Background and Related Work
Compression Techniques for NLP Models. Due to the prominence of large-scale language mod- els, there has been significant interest in compressing these models. Sparsification has been shown to be a promising approach to improving model inference speed and reducing memory cost earlier in computer vision [T)] Sparsifying pre-trained language models turns out to be challenging task be- cause the semi-supervised models tend to have less intrinsic sparsity than CNN-based models. Some prior studies try to understand and apply sparsification for large-scale language model compression. Chen, et al. 4 extend the Lottery Ticket Hypothesis 7 to pre-trained models, finding that winning tickets for the pre-training tasks do transfer universally to downstream tasks. Gordon, al. [X] find that the level of sparsity discovered during the pretraining process represents most of the sparsity that can be discovered in the model. Guo, al. [ show that a proximal pruning strategy achieves higher accuracy than competing lasso regularization methods and iterative magnitude pruning. Most of these studies still focus on unstructured sparsity, which encounters difficulty to obtain large speedups on modern hardware. As structured technique, Michel, et al. 20 observe that many transformer heads themselves are redundant and can be pruned with minimal accuracy loss. While structured technique, this technique provides limited acceleration benefits since much of the compute time for short-to-mediune sequence lengths spent in the intermediate layers rather than the attention heads themselves. In addition to parameter sparsification, BERT models can also realize compression
gains with model guantization B31 2X and knowledge distillation [27 29 30 14 These techniques
complimentary to proposed method and can be combined to achieve more effective overall
compression, as in [10].
Semi-structured Sparsity. There have been few studies to induce NxM sparsity for DNN models. Among them, SR-STE [36] is the most comparable work to this paper. SR-STE single stage compression technique that uses extended Straight-through Estimator and sparse-refined term to improve the induction of NxM sparsity in the network. SR-STE demonstrated across image and neural machine translation tasks and shows improved accuracy over NVIDIA's ASP 21.. Unlike this work, SR-STE is designed for training from random initialization, not a pretrained model.
From performance perspective, CUSPARSEIt 21 (the NVIDIA linear algebra library for NxM sparsity) demonstrated 1.3X 1.6X performance uplift for the matrix multiplications prevalent in BERT models using sparse NxM Tensor cores Nvidia A100 GPUs. Higher speedups may be

---

achieved by increasing batch size, sequence length. or model dimensionality with a maximum of 1.8X improvement for fpl6 and 1.9X for int8. Realized speedups are smaller than the theoretical 2X improvement of 50% sparsity primarily due to memory bandwidth constraints. The performance provided by cuSPARSHIt is a reasonable expectation for what can achieved with NxMTransformer.
Models including NxM sparsity may be deployed for serving using optimized frameworks like TensorRT 2. which include optimized layer operators that take advantage of the hardware's features. Moving forward, inference-driver machine learning compilers like TVM S can also introduce support for the semi-structured sparse Tensor operators using the same mechanisms as were used to provide support for dense Tensor cores, which use similar API.
ADMM for Neural Networks. Alternating Direction Method of Multipliers (ADMM) has been used in previous works primarily for compressing convolutional neural networks. Ye, et al. [4 explore an iterative approach to ADMM that shortcuts full ADMM convergence with a masked retraining operation. Ren, et al. [26 use ADMM for both unstructured sparsification and quantization of the model while combining the technique with a heuristic for determining whether speedup will be achieved given the achieved compression of layer. Ma, al. [ use domain knowledge of CNN filter structure to perform a structured pruning operation for mobile hardware efficient compression. While the above techniques all use ADMM the sparsifying mechanism, none examine in-depth applicability to pre-trained language models for NxM sparsity.
3 Methodology
In this section, we formulate ADMM for NxM sparsity, describe the specific aspects of pre-trained models that define the optimization pipeline, and describe the high-level optimization schedule.
3.1 Problem Definition.
Consider adapting a pre-trained large-scale language model & with Transformer layers (e.g.. BERT) to natural language understanding tasks, such sentiment analysis, entailment, question and answering, etc, where the training instances are inputs (often text phrases) and target pairs: {zt, y4 J. Assume the collection of pre-trained model weights is W (Wo. the goal of NxMTransformer is to load T and fine-tune it to W' such that (W satisfies the constraints of at most M weight parameters having non-zero values out N consecutive weights, while achieving similar performance in comparison to fine-tuning the task-specific objective function [((W))) (e.g., cross-entropy for classification) using Wo but without constraints.
3.2 NxMTransformer
Different from conventional DNN training objectives, the above problem is non-convex with com- binatorial constraints. Therefore, it cannot be directly solved by gradient-based methods such as stochastic gradient descent. To address this issue, adopt the alternating direction method of multipliers (ADMM), which is reliable method for large-scale constrained optimization (c.g., with combinatorial constraints). In particular, modify the objective function of the NxM sparsity problem as
min f((.)) + Σ g (Z.) subject to W, Z,,i L....,L (w.)
where f(-) the fine-tuning objective function, and gi(-) an added penalty function and Z, are auxiliary variables. To apply ADMM, we define the penalty function as
W.)
{ oo
(2)
otherwise
For convenience's sake, we will omit the notation of the bias since it is not relevant to the task of sparsifica-
tion.

---

where S; represents the constraint set S; = (W; that have at most M weight parameters having non-zero values out of N consecutive weights).
Choice of S,. Not all weights in pre-trained Transformer models need to satisfy this NxM spar- sity constraint. BERT and similar pretrained models typically consist of three major components: embeddings. Transformer blocks, and classifiers (See Figure M. For NxM semi-structured sparsity, we solely consider weights in Transformer blocks. Take BERT as an example, each Transformer block consists of 6 fully connected sub-layers: the query Q, key K, value V layers. the attention output layer Attn., and two feed-for corddfo network FFN1 and FFN2. Each of the fully connected layers can take advantage of NxM semi-structured sparsity; furthermore, these layers constitute the vast majority of inference wall-time. Of the six fully connected layers, FFN1 and FFN2 are particularly important sparsify, alone requiring more than half of the inference wall-time for Transformer block. Note that attention head pruning techniques 20 are unable to sparsify FFN1 and FFN2. The self-attention mechanism itself does not include any trained parameters and is unable to be sparsified using this technique. We exclude the embedding layer since the lookup operation associated with the embedding layers is incompatible for acceleration with Tensor Cores. The classifier is composed of fully connected layers. For a given task, the classifier weights are randomly initialized the beginning of the fine-tuning process. We find that sparsifying these matrices using ADMM will unnecessarily harm accuracy. Since the execution of these layers is typically under 2% of the inference wall-time, the runtime cost is minimal for doing so.
Decomposing the minimization problem into sub-problems. Once we define St we apply the augmented Lagrangian, which decomposes equation into two sub-problems on W and Z:
Sub-problem cettffommmnne-prmmottii min f((W,) + {W.)
Σ 2
W. z U (3)
L
P f[ Z, + U (4)
Sub-problem (NxM perrstty-rooooting):
min (Z.)
g (Z.)
The first sub-problem solves the performance promoting problem, which consists of two terms. The first term is the standard objective function for fine-tuning the task, and the second term is a Le regularization term. The regularization target Z U is dynamically updated, based U.S which is the dual variable (i.c., the Lagrange multiplier). Since the L2 term is convex, the complexity of solving sub-problem 1 (e.g., via ADAM (16)) is the same as minimizing f(-). The second sub- problem solves the sparsity promoting problem. Since it optimizes the sparse constraints separately, it can be solved analytically as the solution of the Euclidean projection of W + U onto our constraint. For the case of NxM semi-structures sparsity, this is accomplished by retaining the M largest values of the contiguous group of N values (See Figure D.. which can be solved in linear time. Finally, we need to update the dual variable U as U == - + W Z to guarantee that the dual feasibility condition is satisfied in each ADMM iteration.
Sparsity-indueing based fine-tuning, The typical ADMM pipeline fully trains a model to con- vergence before introducing ADMM (26 This two-step process is necessary since the primary objective of ADMM is to optimize the existing network to conform to the constraints; ADMM will only introduce small changes to parameters retains in the model. However, for pre-trained language models, the primary purpose of the fine-tune is to adapt the classifiers for the specific downstream with minimal disturbance to the parameters of the pre-trained representation. As result, we apply the three aforementioned steps (i.e., two sub-problems and the update of the dual variable) while fine-tuning the model. particular, we perform the three steps in an alternating manner, i.e., performing some number of fine-tuning steps with Adam to solve the first sub- uuo-probeem, solving the Euclidean projection for each weight matrix for the second sub-problem, and finally updating the auxiliary variable. This sequence will be referred to as one ADMM iteration. The optimization proceeds until the W and Z variables have converged, at which point we have a sparsified network compliant with our NxM constraint.
5

---

Evaluation
In this section, we evaluate NxMTransformer and show its effectiveness in compressing Transformer networks over a wide range of NLP tasks.
Implementation. NxMTransformerr is implemented PyTorch compatible library for spar- sifying models with NxM semi-structure sparsities. Furthermore, HuggingFace Transformers B3] compatible Trainer implemented enable easy integration with their model collection and training scripts. Our approach supports different NxM sparse patterns (e.g., 4:1, 8:4) so long as the weight's input dimension is a multiple of N. For evaluation, we focus on evaluating 4:2 sparsity since it is supported in commodity hardware. We use pretrained model checkpoints for both BERT and DistiBBERT provided by the HuggingFace model repository. All models were fine-tuned on an Intel Xeon 2630 v4 server with 2x NVIDIA Titan V running Ubuntu 18.04. PyTorch version 1.7.1 was used alongside Transformers 4.3.2. Finetuning these models required between minutes (RTE) and 5 hours (MNLI) depending on task size. For the set of training hyperparameters used for training NxMTransformer, see Table B
Dataset. We evaluate NxMTransformer and our baselines using the the General Language Under- standing Evaluation (GLUE) benchmark [3]] a collection of NLP tasks varying in data availability and complexity. We report the Spearman correlation for STS-B, the Fl score for MRPC, Matthews correlation for CoLA, and accuracy for all remaining tasks. The reported average the geometric mean of reported scores.
4.1
Main Results
BERT (6:
This is the BERT bass model from publicly available checkpoint.
Inline with ASP practices [T], we perform one-shot magnitude-based masked pruning on the pretrained model. This baseline is considered best practices for a large pretrained language representation for semi-structured sparsity.
ASP:
ADMMunttuuturdd To measure the accuracy cost of semi-structured accuracy specifically, we create another baseline that uses ADMM but induces unstructured sparsity at 50% per-layer (rather than global) sparsity.
6 the authors only report the development results on a few tasks. Therefore, produce the BERT bascline results. We fine-tune BERT for 5 epochs each downstream task. We perform a grid search of batch sizes 16 and 32, and learning rates 1e-5, 3e-5, and 5e-5 for SST-2, QNLI, and MNLI, due their high training cost. Learning rates of 7e-5 and 9e-5 are additionally used for the remaining tasks. For masked fine-tune, the model was fine-tuned with learning rates le-5, 3e-5, 5e-5, 7e-5, and 9e-5 across batch sizes 16 and 32. ADMMrnammnnred is trained using the same hyperparameters sweeps as Nr.MTransormerr. For all configurations, we set the fine-tune to have 5 epochs, and the best observed result the validation set reported.
Hyperparameters.
We report the evaluation results for BERT in Table and make the following key observations.
First, the pruning based method sparsifies weights of Transformer blocks but cannot explicit satisfy the underlying hardware constraints, e.g., the 4:2 sparsity. Although preserving the highest accuracy on downstream tasks (81.3 vs. 81.8 on average), the obtained sparse weights have random structure of non-zero weights, which inefficient to execute in modern hardware systems. result, the performance benefit with these unstructured sparsity based approaches is negligible, even when the pruning rate is high (e.g., 95%) 32
Second, when it comes to NxM sparsity, NxM'Transformer achieves an average score of 80.4, outperforming ASP by 1.6 points. In particular, we observe that for large tasks (MNLI, QNLI, SST-2), NxMTransformer performs comparably to ASP. However, NcMMTransformer dramatically outperforms ASP for the small tasks (CoLA, STS-B, MRPC, RTE), increasing accuracy by 2.9 points on average. Since the smaller tasks can be more sensitive to random seed performance, we also performed a random seed sweep across random seeds. NxMTransformer still outperformed
Apache 2.0 License
http:://tuggingffcccooo/ssstsss http://aaaacccccot/ttaaaaaaeatat www.eeerttttseeeaaeeaeeeeeeeeeee uncased Apache 2.0 License

---

Table The dev set results on the GLUE benchmark. The results show that NxMTransformer is able to achieve higher accuracy than ASP for NxM sparsity, especially when the downstream tasks have low data resources.
Model
Task
Average
MNLI (m/mm)
SST-2
QNLI 108k
CoLA 8.5k
STS-B 5.7k
MRPC
RTE 2.5k
Samples
392k
67k
3.5k
Baseline (BERT base)
84.5/84.8
92.5
91.6
56.7
89.6
91.7
70.7
81.8
ADMMunstssturs
84.0/84.7
92.5
91.0
57.5
89.6
90.5
68.2
81.3
ASP
83.3/83.4 82.3/83.4
91.9
90.6
51.7
88.7
88.1
63.9
78.8
NxMTransformer
92.3
90.4
55.3
89.3
90.8
68.6
80.5
0.89
1.2
NoMTranssformer
ASP
Validation Accuracy
0.88
Training Loss
0.86
0.84
NoMMTranffommr
2000
Epoch
Training Step
(a) Validation accuracy of NxMTTrnsformer and ASP networks best configuration STS-B.
(b) Training loss of the NrMTransformer and ASP networks hest configurations of STS-B.
Figure 2: ASP and NxMTransffrmer on STS-B
ASP by 1.9 points across the tasks with fewer than 10,000 training examples when median was our metric of accuracy rather than maximum. This pattern suggests that while more data allows less principled mechanism to recover accuracy, an explicit optimization approach that take the sparsity constraints into account would yield much better accuracy results when the downstream tasks have low data resources. As a result, Nx MTMrrasffrrmer retains 99% of the accuracy of the unstructured sparsity (ADMMunsmnnne)) and 98.4% of the uncompressed model (BERT Different from ADMMünsncaueh which suffers from expensive irregular memory accesses, our NxMTransformer method can effectively leverage the underlying Sparse Tensor Core and achieves inference speedups even with 50% overall sparsity.
4.2 Analysis Results
In this section, further investigate the performance gain of NxMTransformer and its impact the downstream task accuracy with NxM sparsity.
Validation accuracy improvement. We first compare the model trained with ASP and NxMTTran-- former. To evaluate NxMrransformer, we perform a hard prune of small weights at the end of every epoch, so we evaluate the model as has already sparsified. As we can see Figure 2 NxMTransformer converges slower than ASP in the beginning of the fine-tuning. This is presumably because the initial model weights are heavily violating the hardware constraints, causing significant degradation when performing the pruning action. As the training moves forward, NxMTransformer able catch up and outperform ASP at around epoch This is because by using ADMM, UxMTransformer trains the dense model to gradually converge to a sparse model that satisfies the provided constraints, so pruning weights would gradually have a smaller impact to model accuracy. Since then, the validation accuracy of both ASP and NxMTrannformer are still increasing, but ASP tends to plateau after epochs, whereas NxMTransformer continues to increase, outperforming ASP by 0.5 point in the end. On the other hand, also observe that ASP has slightly lower training loss

---

towards the end (as shown in Figure 26 indicating that ASP might be overfitting on the dev set (potentially due the small amount of data).
Dynamism of sparse subnetwork masks. The penalty parameter controls the balance between maximizing the accuracy of the model and reaching a sparse model. The larger that is, the greater the influence of the sparsity-inducing regularizer and the more quickly the model converges to sparse solution. The trade-off represented by tuning p manifests itself by values moving into and out of the sparse subnetwork mask between ADMM iterations. Since the sparsity is induced more slowly by a small a parameter is more likely to be included when the second sub sub-problem is solved. Empirically, find that frequently moving values into and out of the sparse subnetwork mask results degrades the ultimate sparse networks accuracy. In Figure 3 is tuned to achieve different values of similarity, which is calculated the fraction of values that remain in the sparse subnetwork mask from one ADMM iteration to the next. For both CoI.A and QNLI a clear correlation between average similarity and accuracy exists until approximately 99% similarity, where the strength of the regularizer is over-weighted against the training loss and accuracy begins to degrade.
0.900
0.54
x
Matthews Correlation
0.675
X
0.850
0.52
Accuracy
0.825
0.800
0.50
0.775
x
0.750
x
0.725
Average Similarity
Average Similarity
(a) Similarity Matthews Correlation CoLAA
(b) Similarity vs Accuracy on QNLI (108k samples). Learning: 3e-5, Batch size: 32
(8.5k). Learning rate: 9c-5, Batch size: 32
Figure 3:
Analysis of mask similarity and accuracy metrics for subset of GLUE tasks. Different similarities are achieved through p tuning.
Inspecting the values of weights that undergo swapping illustrates why higher dynamism in the sparse subnetwork mask incurs an accuracy penalty. Figure g shows a parameter outside of the sparse subnetwork mask for just a single iteration will decrease in magnitude by approximately 15% from its initial magnitude. A second iteration further increases this penalty to 25%. This is in contrast to parameters that remain the sparse mask for the entirety of the optimization process and retain all of their magnitude. The philosophy behind training from pretrained model is to retain the information of that process. Large changes in parameter magnitude are destructive to that pretrained information because the parameter only partially reflects that learned information.
4.3 When NxM Sparsity Meets Knowledge Distillation
Knowledge distillation has been proven to be another promising technique to compress a large model and also yield models with regular and dense structures. However, there have been few studies on the sparsity of knowledge distilled models, especially in the context of transfer learning and pre-trained language models. On first sight, it may seem that once distilling a large model into a smaller model, there will be less redundancy in the model, where sparsification might hurt model accuracy significantly. In this section, we investigate how NxMTransformer affects KD compressed models. We apply NxMTransformer to student model obtained through DistilBERT (7)) which is a 6-layer BERT model with 768 hidden dimension size. The results are shown Table and we make the following observations.
First, despite DistilBERT and NxMTrransformer have the same number parameters, NxMTrans- former achieves 2 points higher accuracy average than DistilBERT, which indicates that removing

---

Average Parameter Magnitude
0.05
Pretraining Magnitude
Post Training Magnitude
0.04
0.03
0.02
0.01
0.00
Appearances in Sparse Subnetwork Mask
Figure 4: Comparison of average parameter value before and after fine-tuning on 10-epoch STS-B experiment (learning rate: 5e-5, batch size: 16, pt 1e-3) based on the number of times was present in the sparse subnetwork mask.
Table 2: The dev set results on the GLUE benchmark with knowledge distillation. The results show NxMTransformer retains 97.6% of the DistilBERT model.
Model
Purams
Average
MNLI (m/mm)
SST-2
QNLI
CoL.A
STS-B
MRPC
RTE
NrMrransformer (4:2 (ERT;;)
66.6M
82.3/83.4
92.3
90.4
55.3
68.6
80.5
DistiBBRR (BERTg)
66.6M
82,4/82.5
90.9
89.1
53.4
86.6
89.6
63.5
78.5
45.3M
80.7/81,2
90.5
87.5
50.1
87.1
88.7
59.2
76.6
Transformer layers from the BERT model more detrimental to the model accuracy and NxMTrans- former's semi-structured approach captures redundancy (intra-layer) much more efficiently.
Second, NxMTTaanffrrnr retains 97.6% of the aceuracy of the dense DistilBERT model. While slightly worse than the retained accuracy ratio for BERT (98.4%) itself, this indicates that while the depth of dimensionality of the model may be reduced, the relatively low amounts of sparsity exploited by semi-structured sparsity are still prevalent in fully connected layers. The result also seems to suggest the potential existence of winning ticket even highly compressed BERT model 4
More recently, knowledge distillation techniques such as TinyBert [4 and MiniLM 32 leverage fine-grained knowledge transfer to help student better mimic teacher's behavior. As method largely orthogonal to how knowledge gets transferred between teacher and student, we expect the effectiveness NxMTTransormmer witnessed DistilBERT should apply models distilled through these more advanced techniques as well and will leave more extensive studies as future work.
5 Conclusion
Semi-structured sparsity can improve runtime resource efficiency without large penalties in model performance. This work demonstrates the effectiveness low-overhead ADMM approach to introduce NxM semi-structured sparsity for large pretrained natural language models. Furthermore, NxMTransformer is an orthogonal optimization to existing compression techniques, such as knowl- edge distillation and reduced precision inference. However, NxMTransformer is limited in that is not a lossless compression technique and does introduce an accuracy gap. Furthermore, it is untested emerging pretrained Transformer representations for vision tasks and it is unclear how it would transfer to this emerging domain.
Negative Societal Effects
NxMTransformer exposes two key avenues for negative societal impacts. First, since NxMTrans- former designed to inherit from a pretrained model representation, it inherits any societal-level biases that may be embedded in the parent model. Previous work has identified that BERT models do

---

encode both gender bias [7 and bias against people with disabilities [3] NxMTransformer does not specifically attempt combat these biases and downstream tasks fine-tuned with NxMTrrnsffomer will inherit them as well. The second potential source of negative societal impacts is due to the act of pruning itself. Hooker, al. [12 identify that for convolutional neural networks, pruning can disproportionately reduce accuracy of lower frequency output examples. Although the model design for CNNs different from that of Transformers, it is unlikely this alone would mitigate this source of network bias. These sources of bias can introduce real-world harms fine-tuned natural language models are increasingly used for online content moderating, brand sentiment, career matching, and other human human-facing algorithms that can affect livelihoods.
Acknowledgments and Disclosure of Funding
We thank the anonymous NeurlPS reviewers for their constructive comments. This work was supported by the National Science Foundation under grants CCF-1823005 and NSF CAREER Award (CNS-1750760).
10

---

References
[1] Nvidia a100 tensor core gpu architecture, 2020.
[2] Accelerating inference with sparsity using the nvidia ampere architecture and nvidia tensorrt, 2021.
[3] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael R. Lyu, and Irwin King. Binarybert: Pushing the limit of BERT quantization. CoRR, abs/2012.15701, 2020.
[4] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks, 2020.
[5] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang. Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. Tvm: An automated end-to-end optimizing compiler for deep learning. In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, OSDI' 18, page 579-594, USA, 2018. USENIX Association.
[6] Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[7] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
[8] Mitchell Gordon, Kevin Duh, and Nicholas Andrews. Compressing BERT: Studying the effects of weight pruning on transfer learning. In Proceedings of the 5th Workshop Representa- tion Learning for NLP,, pages 143-155, Online, July 2020. Association for Computational Linguistics.
[9] Fu-Ming Guo, Sijia Liu, Finlay S. Mungall, Xue Lin, and Yanzhi Wang. Reweighted proximal pruning for large-scale language representation. CoRR, abs/ 19099.9499,,,,.. 2019.
[10] Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark Horowitz, and Bill Dally, Deep compression and EIE: efficient inference engine on compressed deep neural network. In 2016 IEEE Hot Chips 28 Symposium (HCS), Cupertino, CA, USA, August 21-23, 2016, pages 1-6. IEEE, 2016.
[11] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
[12] Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. Characteris- ing bias in compressed models, 2020.
[13] Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Craig Denuyl. Social biases in nip models as barriers for persons with disabilities. In Proceedings of ACL 2020, 2020.
[14] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TERy Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163 4174, Online, November 2020. Association for Computational Linguistics.
[15] Norman P. Jouppi, Cliff Young. Nishant Patil, David Patterson, Gauray Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin,
11

---

Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium Computer Architecture, ISCA '17, page 1-12, New York, NY, USA, 2017. Association for Computing Machinery.
[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
[17] Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166-172, Florence, Italy, August 2019, Association for Computational Linguisties.
[18] Xiaolong Ma, Fu-Ming Guo, Wei Niu, Xue Lin, Jian Tang, Kaisheng Ma. Bin Ren, and Yanzhi Wang. Pconv: The missing but desirable sparsity in dnn weight pruning for real-time execution on mobile devices. Proceedings of the AAAI Conference on Artificial Intelligence, 14(04):5117-51244 Apr. 2020.
[19] Yihuan Mao, Yujing Wang. Chufan Wu, Chen Zhang, Yang Wang, Quanlu Zhang, Yaming Yang. Yunhai Tong, and Jing Bai. Ladabert: Lightweight adaptation of BERT through hybrid model compression. In Donia Scott, Núria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages Linguistics, 2020.
International Committee on Computational
[20] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[21] Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks, 2021.
[22] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Ve Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8035. Curran Associates, Inc., 2019.
[23] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
[24] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
[25] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. ArXiy, October 2019.
[26] Ao Ren, Tianyun Zhang, Shaokai Ye, Jiayu Li, Wenyao Xu, Xuehai Qian, Xue Lin, and Yanzhi Wang. Admm- nn: An aegorithm-hardwaan co-design framework of dans using alternating direction methods of multipliers. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '19 page 925-938, New York, NY, USA, 2019. Association for Computing Machinery.
[27] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01100 2019.
12

---

[28] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Q-BERT: hessian based ultra low precision quantization of BERT. AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second
In The Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8815-8821. AAAI Press, 2020.
[29] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 4322-4322 4331. Association for Computational Linguisties, 2019.
[30] Zhiqing Sun, Hongkun Yu, Xiaodan Song. Renjie Liu, Yiming Yang, and Denny Zhou. Mobile- bert: a compact task-agnostic bert for resource-limited devices. In ACL (2020), 2020.
[31] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNIP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics.
[32] Wei Wen, Chumpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2074-2082 2016.
[33] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Languuge Processing: System Demonstrations, pages 38 48, Online, October 2020. Association for Computational Linguistics.
[34] Shaokai Ye, Tianyun Zhang, Kaiqi Zhang, Jiayu Li, Kaidi Xu, Yunfei Yang, Fuxun Yu, Jian Tang, Makan Fardad, Sijia Liu, Xiang Chen, Xue Lin, and Yanzhi Wang. Progressive weight pruming of deep neural networks using ADMM. CoRR, abs/1 810.07378, 2018.
[35] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8BERT: quantized 8bit BERT. CoRR, abs/1910.06 188, 2019.
[36] Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n.m fine-grained structured sparse neural networks from scratch. In International Conference on Learning Representations, 2021.
[37] Neta Zmora, Guy Jacob, Lev Zlotnik, Bar Elharar, and Gal Novik, Neural network distiller: python package for DNN compression research. CoRR, 0ttp/190..12222. 2019.
13


Reviews:
[{"review_id": "zzEovhLpE-h", "reviewer": "Reviewer_KcKK", "summary": "This paper describes a ADMM-based approach to NxM sparsification in large-scale language models (e.g. BERT). Semi-structured sparsification, such as the 4:2 sparsity implementation of Ampere GPUs, allows for a best of both worlds: the weak constraint (picking M weights out of a contiguous N) enables sufficient expressivity to not overly degrade accuracy, while still allowing for an efficient hardware implementation. The authors compare against a prior approach (ASP) on a range of GLUE benchmarks (MNLI, SST-2, QNLI, CoLA, STS-B, MRPC, and RTE), outperforming ASP by 2.9 points on average for tasks with small datasets (and comparably on those with larger datasets). The authors also provide insight into the fine-tuning accuracy, and show that the approach is complementary to a (recent but not quite state-of-the-art) knowledge distillation method. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The target problem is interesting with high potential impact, especially given the rising prevalence large language models. \n\nThe novelty is not particularly high, given that ADMM has already been applied to the problem of compressing neural networks, and similar previous approaches have been proposed for NxM sparsification, albeit for randomly initialized models (rather than pretrained ones). \n\nThe authors performed quite extensive evaluation on GLUE benchmarks, but the reported improvements were generally quite modest. It was interesting to see empirically that this approach is in fact orthogonal to a form of knowledge distillation. \n\nThe quality of writing was not overly high, making it difficult to understand the paper in places. In addition to clarifying explanations, the authors should clean up various editing issues:\n(1) Remove \"(needs CCITE)\" from the first sentence of the introduction.\n(2) \"a masked language modeling\" -> \"model\"\n(3) \"NxMTransformer(See Figure 1\") -> Add space\n(4) \"Finally, we show that NxMTransformer is complimentary to\" -> \"complementary\"\n(5) Many strange citations of the form \"[LASTNAME], et al.\" -> remove the comma here\n(6) \"for the specific downstream with minimal disturbance\" -> \"downstream task\"\n(7) \"from publicly available checkpoint\" -> \"from a publicly available checkpoint\"\n(8) \"Inline with\" -> \"In line with\"\n(9) \"The baseline is considered best practices\" -> \"best practice\"\n(10) \"using the same hyperparameters sweeps\" -> \"hyperparameter\"\n(11) \"Different from\" -> \"Unlike\"\n(12) \"moving values into and out of the sparse subnetwork mask results degrades the ultimate sparse networks accuracy\" -> remove \"results\" and \"networks\" -> \"network's\"\n(13) \"despite DistilBERT and NxMTransformer have the same number of parameters\" -> \"having\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I was satisfied with the authors' statement on limitations and societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "RZqZVjR465i", "reviewer": "Reviewer_NSKn", "summary": "The paper proposed to apply the recently developed NxM semi-structured sparsity by hardware manufacturers to compress BERT, with an ADMM-based sparsifying mechanism. Specifically, the authors used 4x2 semi-structured sparsity pattern since it's supported by NVIDIA GPUs, which halves the number of parameters. In the experiments, the paper showed that the ADMM-based approach outperforms the previous ASP method, which requires a second costly sparse pre-train of the model. The proposed model also demonstrates better performance compared to DistilBERT under the same number of parameters setting.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Advantages:\n1. The proposed method show good empirical performance when compared to other baseline models such as ASP or DistilBERT.\n2. The paper is the first to apply ADMM-based sparsifying mechanism to pre-trained language models.\n\nWeakness:\n1. Given that both NxM semi-structured sparsity and ADMM-based sparsifying mechanism are already well-developed, this paper only combines these two techniques and thus lack novelty.\n2. One of the advantages of NxM semi-structured sparsity compared to unstructured sparsity is its superior inference speed. However, the paper didn't present any experimental results w.r.t. inference speed or latency.\n3. In \"When NxM Sparsity Meets Knowledge Distillation\", the paper might also need to present a DistilBERT model with 45.3M parameters to show the performance gain of DistilBERT-NxMTransformer. Otherwise, it's unclear how DistilBERT-NxMTransformer is better than other baselines under the same number of parameters setting.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "7shoQUqTpVo", "reviewer": "Reviewer_x2gA", "summary": "This paper addresses sparsifying Transformers for improving efficiency on NLP tasks. In particular, they use ADMM to alternate between optimizing the weights of the model to perform well on a fine-tuning task, and inducing \"NxM\" sparsity (out of N contiguous parameters, only M are non-zero).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: There is some previous work that aims to induce this kind of semi-structured sparsity using an alternating approach in a neural network. There is much work on inducing a wide variety of sparsity constraints using alternating approaches like ADMM for more classical models (e.g., linear models), and I would recommend the authors include more of these citations in their related work. It is clear how this work differs from previous work, though appropriate baselines are missing.\n\nQuality: The submission is technically sound, this is an appropriate use of ADMM. The datasets evaluated are standard, though have drawbacks; a number of the GLUE tasks are quite small and have high variance even when only varying the random seed, and so the results on validation sets for MRPC, CoLA, etc. should be taken with a grain of salt. The authors claim on line 80 that, \"We conduct comprehensive experiments and demonstrate that NxMTransformer achieves 1.7 points higher accuracy than state-of-the-art techniques to induce NxM sparsity\", but without more appropriate baselines this claim is not well supported. I don't think this is enough to warrant rejection, but it should be addressed.\n\nClarity: The paper is well-written. I have a few suggestions for how to make it more clear: \nWhen introducing the decomposition into sub-problems, the doal variable U is introduced, but not described in much detail. Equations 3 and 4 are otherwise fairly clear, though I had to go use external sources to read up on ADMM to remind myself how this worked, and this shouldn't be necessary.\nIn Table 1, say how sparse the resulting models are. \nIn Section 4.2, the paragraph lines 252-265, say the dataset(s) being evaluated.\nLine 188 is missing the word \"task\".\nNice job including the information in lines 203-209! That's very helpful.\nLine 221 is one of the first times I see it noted that you're aiming for 50% sparsity -- this should be much earlier. If it's already mentioned earlier as well, it should be highlighted more.\n\n\nSignificance: My main complaints are about baselines, and variance from random seeds. \nFor example, Table 1 doesn't show variance from random seeds, but it should, because the variance for tasks like RTE and MRPC can be larger than the \"gains\" presented here.\nLine 241 claims on the small tasks have the best results, and while it's possible that's a reproducible result, 2.9 points is not a large difference for those tasks.\nIn Table 1, you bold your \"MNLI mm\" result, even though it's the same as ASP; this is overclaiming a bit.\nThe baseline you use in Table 1 is \"ASP\", and the description on line 217 sounds as though this involves taking a pretrained model, performing one-shot magnitude-based pruning, then fine-tuning the pruned model on a task. To ablate the choice of using the ADMM algorithm, you should also compare against pruning after fine-tuning. Please let me know if I'm understanding this correctly (and make this more clear in the paper). As it stands, you don't have a good baseline justifying why you need to use the more complicated algorithm you introduce.\nFigure 2 (b) isn't very informative, you can remove it or put it in the appendix to save space.\nTable 2 results are just not informative without a baseline. Your main claim is that your ADMM approach for sparsifying a model is useful even when applied to distilled models, but to support that claim you need to compare against baselines not using ADMM (so, just magnitude pruning before fine-tuning, and magnitude pruning after fine-tuning). As a reader, if I wanted to sparsify a distilled model, Table 2 without a baseline doesn't provide enough evidence for me to choose your ADMM approach over something more simple.\n\nWhile I do argue for better baselines in this paper, I still think it's a valuable contribution and worth publishing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper sufficiently addresses limitations and societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "-KVkFqbNBZg", "reviewer": "Reviewer_k8jW", "summary": "This paper introduces a new method called NxMTransformer to induce NxM semi-structured sparsity on pertained language models, for recent hardware dedicated for efficient NxM sparsity (e.g. newly released Sparse Tensor Core). NxMTransformer fromulates the NxM sparsity as a constrained optimization problem and use ADMM to alternately promote the NxM sparsity constraint and finetuning performance. Experiments on GLUE datasets show that the proposed approach can beat the best NxM sparsity method by 1.7 points.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* Developing parameter-efficient NLP models is an important research direction. It is especially cool that this paper considers efficient NLP methods optimized for specific hardwares.\n* This paper introduces a novel method to induce NxM sparsity for recently released hardware and beats the state-of-the-art NxM sparsity method on GLUE datasets.\n\nCons:\n* The experimental results can be stronger\n    - Currently, it only evaluates on GLUE validation dataset with BERT-base models, and it only considers one parameter-efficient baseline ASP. It can potentially evaluate its method on GLUE test sets with BERT-large models, and consider more parameter-efficient methods as baselines (even if they are not all NxM sparsity). \n     -  Many parameter-efficient NLP methods (e.g. see missing references) can achieve <1% sparsity (compared to 50% sparsity in this paper) on GLUE datasets without losing much accuracy -- even though those methods are not NxM, the paper will benefit if authors can compare and justify its method's advantages over those methods.\n    -  This paper currently beats ASP mostly on the small datasets. Those small datasets are very small and can have high variance even with hyper-parameters tuning. It will be good to justify its advantages on more datasets.\n* Missing background and importance of NxM structured sparsity\n   - What is NxM sparsity? How does NxM sparsity hardware work? \n   -  What hardware will it exactly be useful for (newly released Tensor Cores?), and why are that hardware important? \n   - What are the practical benefits to satisfy NxM sparsity? Does it improve inference speed, by how much?\n\nQuestion\n        - Can a similar approach (ADMM pipeline) be used to optimize for another type of sparsity / hardware?\n\nMissing References \n  * Mahabadi et al, Parameter-efficient Multi-task Fine—tuning for Transformers via Shared Hypernetworks\n  * Guo et al, Parameter-efficient transfer learning with Diff Pruning\n  * Mahabi et al, Compactor: Efficient Low-Rank Hypercomplex Adapter Layers\n  * Zaken et al, BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models\n\nUpdate: I've decided to keep the scores after reading the latest responses. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper uses ADMM to induce structured weight sparsity during neural net finetuning. 

It is specifically targeting a very local sparsity pattern (N nonzeros in every block of M weights) which is supported in recent nvidia hardware and libraries with near-linear speedups. This is a different constraint scenario than most structured sparsity approaches, and ADMM does seem like a good fit.

As such, reviewers find the contribution valuable but also highlight some weaknesses. The experimental methodology seems susceptible to noise; multiple runs would help; however, simply getting comparable performance seems sufficient when sparsity is the goal. Reviewers and possibly all readers are left wanting speed comparisons; authors should place more emphasis on the library+hardware support for NxM-sparsity to better clarify that this is established independent of this work. The writing and presentation should be polished as well.
Decision: Accept (Poster)