Minimum Variance Unbiased N:M Sparsity for the Neural Gradients
Authors: Brian Chmiel
Itay Hubara
Ron Banner
Daniel Soudry
Venue: ICLR 2023 notable top 25% (2023)
Keywords: pruning
compression
structured sparsity
acceleration

Abstract: In deep learning, fine-grained N:M sparsity reduces the data footprint and bandwidth of a General Matrix multiply (GEMM) up to x2,  and doubles throughput by skipping computation of zero values. So far, it was mainly only used to prune weights to accelerate the forward and backward phases. We examine how this method can be used also for the neural gradients (i.e. loss gradients with respect to the intermediate neural layer outputs). To this end, we first establish a tensor-level optimality criteria. Previous works aimed to minimize the mean-square-error (MSE) of each pruned block. We show that while minimization of the MSE works fine for pruning the weights and activations, it catastrophically fails for the neural gradients. Instead, we show that accurate pruning of the neural gradients requires an unbiased minimum-variance pruning mask. We design such specialized masks, and find that in most cases, 1:2 sparsity is sufficient for training, and 2:4 sparsity is usually enough when this is not the case. Further, we suggest combining several such methods together in order to potentially speed up training even more. A reference implementation is supplied in the supplementary material.

Published as a conference paper at ICLR 2023
MINIMUM VARIANCE UNBIASED N:M SPARSITY FOR THE NEURAL GRADIENTS
Brian Chmiel fo+ Itay Hubara fo- Ron Banner
Daniel Soudry"
HHabana Labs An Intel company, Caesarea, Israel,
DDeaarmeett of Electrical Engineering Technion, Haifa, Israel
bochmel
ihubara
rbanner @habana.
caniel soudry gmail
ABSTRACT
In deep learning, fine-grained N:M sparsity reduces the data footprint and band- width of General Matrix multiply (GEMM) up x2, and doubles throughput by skipping computation zero values. So far, it was mainly only used to prune weights to accelerate the forward and backward phases. We examine how this method can used also for the neural gradients (i.e., loss gradients with respect to the intermediate neural layer outputs). To this end, we first establish tensor- level optimality criteria. Previous works aimed to minimize the mean-square-error (MSE) of each pruned block. We show that while minimization of the MSE works fine for pruning the weights and activations, it catastrophically fails for the neural gradients. Instead, show that accurate pruning of the neural gradients requires an unbiased minimum-varinn and find that in most cases, 1 1:2 sparsity sufficient for training, and 2:4 sparsity usually enough when this is not the case. Further, we suggest combining several such methods together order to potentially speed up training even more. A reference implementation is supplied in the supplementary material.
pruning mask. We design such specialized masks,
INTRODUCTION
Pruning Deep Neural Networks (DNNs) is one of the most effective and widely studied methods to improve DNN resource efficiency. Since DNNs
waramerraetri,,,, most researchers focused weights pruning, Yet, recently researchers suggested that sparsity of activations (Jaszczur et al. 2021 Kurtz al. 2020) and gradients (Chmiel
2021b) could be exploited as well. However, these types of instructured pruning only reduce the memory footprint (Frankle & Carbin 2018 Evei et al. 2020). possible to also reduce the compute footprint by enforcing some structure on the pruning mask, such as block sparsity (Wen et al. 2016),, filter sparsity (Li et al. 2017). or N:M fine-grained sparsity (Nvidia 2020
Hubara et al. [222 Mishra et al.) 2021).
We focus on N:M fine- fineigraieed sparsity, in which, N out of every M contiguous elements would be pruned, for at least one of the two matrices involved in the matrix multiplication. Nvidia's sparse tensor cores (Nvidia 2020 Mishra et al. [202I) I) can use N:M fine-grained sparsity to accelerate matrix multiplication. Specifically, NNvidia (2020) used 2:4 format to accelerate inference up to x2. They suggested using three-step scheme: (a) train dense model, (b) prune weights to obtain 2:4 fixed mask, and (c) use the original training regime to retrain with the masked weights.
Following works suggested methods to accelerate different parts of this scheme. First, Zhou et al. (202T) was able to omit steps (a) and (b) by training with an N:M mask from scratch using a straight- through estimator (STE) and additional regularization. Specifically, they keep dense copy of the weights and set different weight decays rates to the masked and unmasked weights. Next, Hubara al. (202T) focused on accelerating the remaining step (c), i.e., sparse training, as do here.
"Equal contribution.

---

Published as a conference paper at ICLR 2023
Table Exploring fine-grained sparsity on different training phases with different sampling methods (MVUE, MSE). While previous methods aim to accelerate the forward and backward phases, we focus on accelerating the update phase. The combination of all methods allows us to accelerate all training phases.
Tensor
Training T-Weights + Gradients
Phase
Weights Nvidia
T- Weights
Gradients (ours)
2020
Hubara et al.
2021
Forward
V(MSE)
/ (MSE)
V(SSE)
Backward
X
(MSE)
V(MSE)
Update
X
X
/MVUEE
(MVUEE
Recall that in each training step Backpropagation, which has three phases. Generally, each phase requires a General Matrix Multiplication (GEMM) for each DNN layer l:
[Forward] = Wih-1;; h1 = f(())
(1)
[Backward] gi Diag( (x)) 19++1
(2)
aC
[Update] aw, gihff11
(3)
where C' is the loss function, and in each layer 1, fi is a non-linear activation function, Wi represents the weights, 2 the aecvviiivaionn, h; the post-activations and 9i 0s
DC is the neural gradient.
Nvidia suggested accelerating only the inference phase (i.e., the forward pass in eq. Equation D.. while the backward and update passes were kept dense. Noting that the backward phase uses the transposed (sparse) weight matrix, Hubara et al. [2021) used transposable mask, i.e., mask that can be transposed and still match the N:M fine grniggr structure. This enabled the acceleration of the backward phase. Although Hubara et al. (202T) suggested different methods to find the optimal transposable mask efficiently, they did not suggest how to accelerate the update phase.
In this work explore different methods to accelerate the update phase well using N:M sparsity. We need to decide Equation (3) if we want to prune the activations (h+) or the neural gradients (9t). In order to avoid a mismatch with the forward phase in Equation (I). where the activations are not pruned, we decided in this work to focus on the neural gradient for the update phase. To that end, we examine gradients with fine meaigra pruning and establish tensor-level optimality criteria. So far, N:M sparsity in the weights was obtained by minimizing the Mean Square Error (MSE). We explain (Section 3 that, while this MSE criterion can also be used for the N:M sparsity in activations (which can be useful for inference, we discuss in Section 6). for N:M sparsity in the neural gradients it is better to use a Minimum Variance Unbiased Estimate (MVUE).
We develop (in 4 such MVUE pruning methods for 1:2 and 2:4 sparsity in the neural gradients. Our experiments (in Section 5 suggest that while the traditional minimum MSE method crashed, our MVUE method with 1:2 sparsity is usually sufficient for training, and 2:4 sparsity is enough when this is not the case. Moreover, we suggest to combine several such methods together (fine-graa time-grained sparse neural gradients and sparse transposable fine- fine-grained weights) in order to potentially speed up training even more and be able to accelerate all training phases with N:M fine-grained sparsity. In Table [ present all the N:M fine-grained structured sparsity methods, which part of the network they accelerate, the relevant optimality criteria we use, and the configurations we use to fully accelerate training.
In summary, this paper makes the following contributions:
We developed unbiased minimum variance optimality criteria for pruning neural gradients with
N:M structured sparsity.
We propose 1:2 and 2:4 unbiased minimum variance methods prune the neural gradients and demonstrate that they achieve small or no degradation, where previous methods failed.
We combine these methods with previous methods for N:M structured sparsity in the weights, and observe small or no degradation. Thus, the GEMMs in all training phases can potentially be accelerated by x2.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORKS
Pruning has been extensively investigated in the last few years. Most of the pruning methods focus on pruning the weights (Evci et al.) 2020; Frankle & Carbin 2018 Janowsky 1989; Liu et al. 2018). Unstructured pruning methods achieved impressive sparsity ratio with minimal or no accuracy degradation, e.g. [Renda et al. (2020) achieved over 80% sparsity in ResNet50 over the ImageNet dataset without sacrificing accuracy. Despite this impressive achievement, the ability of unstructured pruning methods to actually reduce computational resources of modern hardware is limited (Nvidia 2020
Mishra et al. (021).
Structured pruning methods vary between boarse-grained and fine- tiee-graindd methods. Coarse-graines methods such as filter-wise or layer-wise pruning (Li et al. [017]] Luo et al. (017]) Wen et al. (016) naturally supported by hardware and software but these methods were only able maintain the test accuracy for sparsity ratio significantly lower than 50%. Recently, Nvidia introduced the
Ampere GPU architecture (Nvidia 2020, MMshra et al. (02T) hardware with software support for N:M fine-grained structured sparsity. Specifically, they showed that 2:4 fine-grained structured sparsity, where two of every four contiguous elements are zero, achieves up x2 improvement in the GEMM operation. They suggested a three-step scheme to accelerate inference. Later, Zhou al. (202I) accelerated their method by avoiding the first two steps. Next, /Hubara al.] (2021) accelerated the remaining training step by suggesting transposable mask, which accelerates both the forward and backward phases & of the training). Stosic & Stosic (2021) further demonstrated the transposable mask can accelerate training with minimal accuracy degradation on 20 different models for various tasks and datasets. Pool & Yu (2021) suggested permuting the weight matrices to improve accuracy of sparse models for inference. (Sun et al. (202I) suggested a mixed layer-wise N:M sparsity schemes to improve the uniform sparsity scheme with similar complexity constraints. Holmes et al. (202T) suggests new learning framework improve the performance of N:M sparse NLP models on downstream tasks.
Beyond pruning the weights, recent work also focuses on unstructured sparsity of the activations or neural gradients. Kurtz et al. (2020) suggested a parametrized activations function called Forced- A Rectified Linear Unit (FATReLU) which increases the naturally sparse of ReLU with any accuracy loss. Jaszczur et al. (2021) studied the sparsification of the activations in Transfomer-he based models. "MeProp" (Sun et al. 2017) prunes the K smallest aisolute-valuee entries of the neural gradients on the fly, using the top-k algorithm. |Aamir Raihan & Aamodi (2020) used top-k pruning on the copies of weights and activations used the backpropagation, Ye al. (2019). suggested "stochastic pruning". reaching higher sparsity levels on the neural gradient. CCmiel et al. (20215) improved their results with a lognormal distribution approximation for the neural gradient achieving more than 80% sparsity the neural gradients without accuracy degradation.
In parallel to our work, two additional works suggested to use N:M structured sparsity to be able to accelcrate training: In the first, McDanel et al. (2022) suggested a method to use N:M structured data pruning for the neural gradients to accelerate the backward phase, which was also accelerated in
Hubara et al. (202T). In Appendix [B.3] we show the degradation of applying McDanel et al.) (2022) method also in the update phase. In the second, WWixiang et al. (022) suggested to use the spatial similarity in vision models fix the loss of information after applying the N:M mask. Their mask is applied on the weights and activations, while keeping the neural gradients in full precison. However, this spatial similarity can not be exploited in other domains such as natural language processing. As far as we know, no previous work suggested using N:M fine-grained sparsity to accelerate the update phase, by pruning the neural gradients.
WHICH OPTIMALITY CRITERIA TO USE?
When pruning weights during training, we require local (tensor level) criterion to select which weights to prune. A popular criterion is minimizing the Mean Square Error (MSE). For a deterministic vector a and a random pruning operator 0 we can write the MSE of pruning as
MSE (a)) = (aa a|l² (a) - aa) + +| [[])((a)) al|² A Var[0(a)] + Bias ((a)))
where F: denotes expectation over the randomness of ((a).
3

---

Published as a conference paper at ICLR 2023
Recently, Chmiel al. (2021a) investigated which optimality criteria to use, but in the context of quantization (i.e., there ((a) was a quantizer). They found that, when quantizing the weights or activations, should indeed minimize the MSE of the quantization error. In contrast, for the neural gradients, they found that it is critical to use unbiased quantization (i.e., (ia)) = 0) such as stochastic rounding. Specifically, CChiiel et al. (2021a) showed that unbiasedness in the neural gradient leads to an unbiased estimator of the weight mini-batch gradient, which enables proper convergence of SGD, according standard SGD analysis (e.g. [ottoou et al. (2018). Therefore, we suggest to apply N:M fine- fine-grained sparsity on the neural gradients using the same optimality criteria and focus on finding an unbiased estimator. From all the possible unbiased estimators, we will prefer the one that reduce the MSE. Since we focus on an unbiased estimator (i.e., (aa)] 0), all that remains to minimize the variance Var(0(a)). Therefore, we conclude that the Minimum Variance Unbiased Estimator (MVUE) is optimal for the neural gradients.
4 MINIMUM VARIANCE UNBIASED ESTIMATOR FOR N:M SPARSITY
In this section, we propose two unbiased estimators with minimum variance: one for the 1:2 case and then another to the 2:4 case. Given block of entries (i.e. vector) a, each estimator produces another block (aa) with the relevant N .M sparsity pattern.
4.1 MINIMUM VARIANCE UNBIASED ESTIMATOR FOR 1:2 SPARSITY
For block [2), one entry needs to be pruned, so any 1:2 method has the following form
[01 0] w.p. p [0, 02) w.p. -
(a)
(4)
We wish to design an unbiased estimator for this pruning method, so
0 (a)) [2]-02]-
(5)
To find unbiased estimator which minimizes the total block variance, using equations 4 and 5
calculate the total variance of a block, as the sum of its element variances:
Varg [0(a)]
Var [a))
> E [a)]
(a)()) vpp of + 2 (1 p) ab (6)
Using equations 4 and together obtain the following expression for the total variance in the block, as function of en alone (for more information, see Appendix II
a3 -1 az
Var [0(a)] = 01 0j - af
Since we wish to minimize this quantity, differentiate equation with respect to 01, and equate to zero to obtain following unbiased estimator, which has the lowest variance of all unbiased estimators (full details in Appendix A.I):
(sign(ar) (lax) + (a2l), 0] [0. sign(az) (layl + [a2l)] w.p.
w.p.
0 (a)
Let us calculate the mean MSE of this unbiased method. By substituting into Equation 7 the optimal solution for v1, the optimal estimator Equation g has a variance of 2a102. Therefore, since the method is unbiased we obtain MSE Bias² + Var = 0 2aja2 2aja2 In Figure we present an illustration of the proposed MVUE 1:2. Table compares different methods for 1:2 structured pruning of the neural gradients, ResNet18 Ci dataset. Notice, the proposed MVUE method has the best accuracy, although it does not minimize the MSE, as done by the 'greddy' method.
4.2 OPTIMALITY CRITERIA FOR 2:4
We now extend the results from the previous section to 2:4 pruning. With block e [a,, a2 az aa), we construct an unbiased 2:4 block pruning method (aa) with minimum variance. First, we note the method must satisfy the following condition
a, with probability P;
0, (a)
(9)
pi
4

---

Published as a conference paper at ICLR 2023
Table 1:2 sparsity on the neural gradients of ResNet18 cifar10 dataset. 'Greedy' the traditional
minimum MSE method of choosing the smallest element for each block. 'Biased" refers to the case
[01, 2/2] [a1. a2] in Equation (4) for p | + l22 'Uniform' refers to uniform sample.
Le. p 0.5, [vi, 22] a11, a2). "Unbiased" refers to unbiased uniform sampling, i.e 0.5.
[vi, U2] = 12a1, 20a].
'MVUE' refers to the minimimum variance unbiased estimator Equation
Method
Baseline
Greedy
Biased
Uniform
Unbiased
MVUE (Ours)
Accuracy (%)
90.02
85.5
71.8
85.8
87.2
89.8
Greedy
Unbiased
MVUE (Ours)
1.00
1.00
1.00
0.75
0.75
0.75
6 0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
0.00 0.25 0.50 0.75 1.00
0.00
0.25 0.50 0.75 1.00
0.00
0.25 0.50 0.75
a1
a2
a2
(a)
(b)
(C)
Figure 1: Fine-graine greengraindd 1:2 Sparsity for blocks located on the first quarter of the unit circle. The
blocks [a1, [2] (represented by red dots) sampled 100 times each, and then averaged (green dots) using one of three methods: (a) greedy is the traditional method that generates the block [0, a2) a1 (22 or (a1: O] otherwise. In this method, all 100 samples the same for each block, resulting in a biased average. (b) unbiased each block [01, 42] is equally likely to be pruned to [2a 0J or [0, 2az). Although the average of the 100 samples is unbiased, it does not have minimum variance.
(c) Our unbiased method with minimum variance (Equation 88, has a smaller spread here than in (b).
since then, and only then, we get an unbiased estimate:
(a)) ps P: + 0- (1 -p:)pp)--- az, Vi (1,2,3,4)
(10)
In this case, the variance of each element in the pruned block is:
Var [0)() E (a) [a)) = [
P: + 0² (1-p:)
-a²
0,
(11)
Pt
Pt
Then, the total variance in the pruned block is
Varg [0(a)] A Σ Var [0, (a)]
Σ
(12)
Pi
We wish to minimize this quantity under the following equality and inequality constraints
ΣM-2-0 p 130 Vi (1,2,3,4)
(13)
Therefore, to find pi, we need apply the KKT conditions on the following Lagrangian:
L L= Σ 3
a²
+
- + Σ - -2).
(14)
Pj
Differentiating the Lagrangian with respect P., obtain
0L =
a²
Op:
p.
2
+
+
Vi (1,2,3,4),
(15)
5

---

Published as a conference paper at ICLR 2023
where, for each i, the constant A could be zero positive. Using Equation 15 for the case X we get that p. = a:/ vp. This, coupled with the normalization constraint (E, Pi = 2) implies that
2a, Σ,g
P.
Vi E 21,
(16)
Turning to the case where A > 0 for some specific i. we have = because of the complementary slackness condition KKT. The normalization constraint (,,, 2) therefore guarantees that
Exe; Ph = This implies all other pa (for io + i) are in the range [o, 1], so the constraint ps S is
slack, and therefore Ås= = 0 for every k + i. Therefore, from equation 15 we have that
0L
+ pk
vk Xi
(17)
8pk
Pk
Since Exzi Pie = 1 we conclude that the optimality criterion is
ag
:i:p-= =1
and
(18)
Exys
Thus, a 2:4 fine- fine-grained pruning method can be optimal only it always satisfies either Equation 18 or T6 We provide such method in Appendix A.2 This method allows to sample pairs of
elements for a 2:4 policy that always satisfies one of the criteria stated in Equations 18 and 16
4.3 COMPARISON OF THE OPTIMAL 1:2 AND OPTIMAL 2:4 METHODS
Given a block [01, ag, a3 an), we can either apply optimal 2:4 method directly on that block
(2) or we can break it into two sub-blocks [01, 42] and [ag, 01), and apply optimal 1:2 method twice i... (1:2( a2]) and (1:2([as ([az, agl). We can show (proof in Appendix A.33 that the former alternative is preferable and introduces less variance, i.e.,
Var(02-a(a)) & Var (1:201:(((11: [2]] + Var
(as, a.l)]
(19)
4.4 APPROXIMATELN OPTIMAL 2:4 METHOD
2.0
As shown in Table 3 in terms of time complex- ity, the optimal 2:4 method might not be fea- sible. Using insights gained from the optimal solution, we now present a simple npar-optimal 2:4 method called aVPEox-MVUE. The idea is simple. We first remove from the block one element a, where i is chosen with probability
1.8
Variance ratio
1.6
1.4
P: 01 02 + 03 + 04
1.2
In order select second element, repeat the same procedure for the three remaining elements with probability
1.0
Ratio between the variance (Equa- tion (12) of the -VPRox-MVUR: 2:4 and MVUE 2:4 when scanning (step size 0.005 all possible values of a block [a1, 22 03, 1]. where 0 S a1 02, ag S
Figure 2:
Pj + a2 +
03 a. a4
Thus, each element is chosen with probability:
1. Notice that ratio is bounded below 2. The maximum is achieved near the left edge, when
lasl
lak
a max(ai, 02 a3).
P: =
E,layl
+
Reght
a
Σ
jot
aayl
(20)
The effect of this approximated method on the variance of the estimator is presented in Figure by the ratio of the variance between the two methods:
Var 02:4
Var(O"pprox (xpprex)
Var (arpt)

---

Published as a conference paper at ICLR 2023
where both variances calculated analytically using Equation for a block [a1, 02 dg. a1] where d1 S 02 S 0g I ag we set a1 = 1 and scan with small steps all combinations of a1 d2, dg. The scan suggests the variance ratio is bounded below twd and therefore the approximate method is a 2-approximation of the old method. As can be seen in Table 3 the approximated method reduces the complexity time of MVUE 2:4 by 70%, in non-optimized implementation. Moreover, in Appendix B we give additional details on this experiment and compare the number of operations required for finding MVUE mask with the computation gain achieved. Based on that we derive a simple rule to decide for each layer when neural gradient pruning is efficient.
Without loss of generality
EXPERIMENTS
In this section, we demonstrate the effective- ness of our proposed method over several vision and language models. First we show the ef- feet of the proposed method for the fine-grained N:M structured sparsity the neural gradi- ents. Then we combine this method with the fine-grained N:M transposableweiighe method (Hubara et al., 2021), allowing the accelera- tion with N:M structured sparsity in all training GEMM. Moreover, we show the combination of N:M structured sparsity in all training GEMM with 8-bit quantization achieving non or small accuracy degradation. Experimental details ap- pear in Appendix A.4
Table 3: Overhead of different algorithms for find- ing the required masks: ratio of their running time over regular training (ResNet50). Notice the over- head reduction in the Approx-MVUR 2:4 in com- parison to MVUE 2:4. All experiments were run in FP32 without sparse-tensor cores and in non- optimized implementation.
Method
Overhead (%)
MVUE 1:2
%
MVUE 2:4
95 %
Approx-MVUE 2:4
3 %
Notice that, while the Nvidia A100 tensor core supports sparse GEMM operation with 2:4 struc-
tured pruning, their software only supports for inference (weights pruning) and not for training. Since there is currently no support for our method in any Al accelerator, we cannot show an actual training time reduction. Therefore, in Appendix B we attempt to estimate when neural gradient pruning is efficient by comparing the number of operations required for finding MVUE mask with the computation gain achieved. We note, that this is the common practice in the neural network compression literature, where the algorithms often appear before the hardware that support them. For example, though we can find FP8 training publications since 2019 (Sun et al., 2019), only recently did Nvidia announce their first GPU that supports the FP8 format (H100).
N:M structured sparsity on the neural gradients In Table we show the results of applying the suggested N:M structured sparsity for various models and datasets. The 1:2 results refer to the MVUE method (Section 4.1 while the 2:4 results refer to the approximate-MVUL Notice the comparison with the traditional greedy method of keeping the largest elements in each block. While the greedy method has a very significant degradation, the proposed method achieved small or no degradation with the proposed methods.
method (Section (44).
Accelerating all training phases In Table 5 we showed the results of the combination between the proposed N:M MVUE for the neural gradients and the N:M transposable weights presented in Hubara et al.] (20TT). The combination between both methods allows to be able to accelerate with N:M structured sparsity all training GEMM operations with minimal or no accuracy degradation. Moreover, in Table 6 we show the combination of N:M sparsity in all training GEMM with 8-bit quantization. For the guantization we used INT8 (Choi et al. 2018a in the weights and activations and FP8 (Chmiel et al. 2021b) in the neural gradients.
'The largest values are near the left edge of the scan, which represents the limit where a4 > max(a1, a2 aa). Near this edge, we additionally checked with very small (logarithmically spaced) step sizes that the variance ratio is bounded below two.
7

---

Published as a conference paper at ICLR 2023
Table 4: Effect of applying the proposed MVUE 1:2 and approx-MVUE 2:4 on the neural gradients for different models and datasets. Notice that in most MVUE 1:2 achieved full precision accuracy and when it did not, the traditional method keeping the N largest elements in each block (minimum MSE) which suffers from a significant degradation.
2:4 method closed the gap. "Greedy' refers to the
Model
Dataset
FP32
MVUE 1:2
Approx-211 2:4
Greedy
ResNet18 8
ImageNet
70.6 %
70.58 %
70.6 %
48.2 %
ResNe150
ImageNet
77.2 %
76.4 %
77.12 %
59.3 %
ResNext50
ImageNet
77.61 %
76.05
77.55 %
60.7
DenseNet-121
ImageNet
74.4 %
74.1 %
74.4 %
70.3 %
ViT-B16
Cifar10
98.8 %
98.4 %
98.7 %
96.7
79.38 (EM)
78.55
79.15
66.2
Bert finetune
Squad
87,03 (F1)
86.41
86.82
70.2
Bert pretrain
Wiki
0.72 (MLM)
0.718
0.72
0.68
Transformer
WMT En-De
27.5 (BLUE)
27.32
27.44
25.55
Table 5: Effect of applying N:M structured sparsity all training phases. We combine the suggested
MVUE 1:2 and approx-MVUE 2:4 for the neural gradients in the update phase with the transposable weights Hubara et (2021) the forward and backward phases.
Model
Update (G)
Forward (W)
Backward (W2)
Accuracy
FP32
FP32
FP32
70.6
FP32
2:4
2:4
70.5 %
ResNet18
MVUE 1:2
2:4
2:4
70.4 %
Approx-2:4
2:4
2:4
70.6 %
FP32
FP32
FP32
77.2 %
FP32
2:4
2:4
77.1
ResNet50
MVUE 1:2
2:4
2:4
75.6 %
Approx-2:4
2:4
2:4
77.1
FP32
FP32
FP32
77.61 %
FP32
2:4
2:4
77.4
ResNext50
MVUE 1:2
2:4
2:4
75.88 %
Approx-2:4 -2:4
2:4
2:4
77.37 %
FP32
FP32
FP32
27.5 (BLUE)
FP32
2:4
2:4
27.5
Transformer
MVUE 1:2
2:4
2:4
27.19
Approx-244 2:4
2:4
2:4
27.35
Table 6: Effect of the combination of N:M structured sparsity in all training phases with 8 bit quantization on ResNet18/51 in ImageNet datasets. For the quantization we used INT8 (Choi et al. 2018a) in the weights and activations and FP8 (Chmiel et al. 2021b) in the neural gradients.
Model
Update (G)
Forward (W)
Backward (W")
Accuracy
FP32
FP32
FP32
70.6
ResNet18
Approx-2:4
8-bit
2:4 + 8-bit
2:4 8-bit
70.3 %
FP32
FP32
FP32
77.2 %
ResNet50
Approx-2:4 8-bit
2:4 + 8-bit
2:4 8-bit
76.48 %
DISCUSSION
Conclusions In this work, we studied the effect of N:M structured sparsity the neural gradients
to accelerate the update phase. Based on a previous work (Chmiel et al. (0216), which showed the importance of unbiasedness of the neural gradients in quantization, variance method for pruning the neural gradient using 1:2 and 2:4 structured sparsity. Since the optimal 2:4 method may not be feasible in term of complexity, suggest approximate method
suggest unbiased minimum
8

---

Published as a conference paper at ICLR 2023
which increases the variance only by a factor of (making it a 2-approximation) We showed that our methods achieved small or no degradation while the traditional greedy method completely failed. Moreover, combine our method with a previous method for transposable weights (Hubara et al.) 2021). This enables a potential acceleration by x2 of all GEMMs in the training process using only N:M fine grained sparsity. In the following paragraphs we will diseuss additional aspects of N:M structured sparsity acceleration including the benefits of pruning both matrices involved in a single matrix multiplication and the potential improvement in the inference phase (Equation T.)
Should we prune both matrices involved in the matrix multiplication? So far, fine-graine grained sparsity papers focused on pruning only one matrix in each phase (Nvidia 2020; (Zhou et al. [02T Hubara et al., 2,2T), achieving up x2 acceleration in the corresponding phase. Since pruning the weights is the most common approach, an interesting question is what would happen if we prune two matrices in one GEMM, such both the weight and activation matrices in the forward phase? Can this accelerate the computation? Next, we explain why pruning both matrices cannot further accelerate computation (i.e., by x4) in modern accelerators, but that it reduces the required bandwidth and thus simplify the hardware design.
We start by analyzing what is the expected acceleration when both matrices (that are involved in the matrix multiplication) follow N:M fine-grained sparsity. Assuming we have two N:M fine-grained blocks bw and by with masks Mew and Mog. the number of Multiply and Accumulate operations (MACs) required for multiplying and accumulating the blocks may vary from zero to N. For example, for 2:4 fine-grained sparsity, there are (t) 6 possible mask configurations. Thus, the expected number of MACs in block, assuming uniform distribution on the non-zeros in the blocks, would be:
E B[#MA bw)] E [[[ MACs((b. bw) [Mull =
6 - 0
1
6 6
2+ 12.6..1 1-1
4
(21)
6
Thus average, for each block we get N/2 MACs. While some architectures (such CPU) can avoid all unnecessary multiplications, architectures with a systolic array at the core of their matrix multiplication engine, as modern hardware accelerators, must always assume the worst case. Therefore, for these types of architectures, we cannot achieve an additional compute reduction by pruning both matrices involved in the matrix multiplication. Yet, the bandwidth reduction for both matrices is the same. This property helps support sparse and dense matrix multiplication without creating dedicated hardware which has twice the bandwidth to one of the matrices involve in the GEMM. It is specifically important when targeting higher sparsity. For instance, if only the activations obey 1:4 fine- grained structure then the weights bandwidth is x4 higher than the activations bandwidth as for every single block we bring one activations and four weights to the engine. In summary, pruning both matrices cannot accelerate modern accelerator computation, but reduces the required bandwidth, and thus simplifies the hardware design. Next, we explain how to use such scheme for inference acceleration by pruning both weights and activations.
Inference acceleration with activation pruning Can we accelerate the network using N:M sparsity on additional tensors? So far, fine-grained sparsity was applied in the forward pass (Equation D only to the weight matrix. Next, we first show the effect of pruning the activations and then we combine both weights and activations pruning to improve the inference phase.
In Appendix |A.5 we demonstrate that one can also accelerate inference by applying N:M sparsity on the activations. Specifically, in Appendix Table 7 we experimented with greedy N:M fine-graines sparse activations on ResNet18 and ResNer50 over ImageNet dataset, wherein for each block of size M we keep the M-N larger elements. Note that in CNNs the activations memory footprint is much larger than the weights footprint (especially for the first set of layers), so in term of memory reduction activations pruning is more effective than weights pruning. Throughout our experiments. we did not change the training regime and pruned the activations from scratch. As can be seen, applying only fine-grained sparse activations results in notable accuracy degradation. However, a simple fix is to apply ReLU before the fine- -neigrnided sparse activations; this results in on-par accuracy for both ResNet18 and ResNet50. In Appendix Table 8 we experimented with fine-grained N:M structured sparsity both weights and activations. To compete with the latest inference acceleration results based on quantization-avee aware techniques, we further quantize the weights and activation to 4-bit and show better results in terms of bit-operations (BOPS) (Wang et al. 2020) to accuracy than 2-bits inference methods. Notably, While using 2-bit for both weights and activations has the potential of x4 acceleration, in modern hardware it would probably be only up to x2 is simplifies the design and reduces the die area, see Nvi where 16bit GEMM has 800 TFLOPS and 8bit GEMM has 1600 TFLOPS). Thus we argue that our 4-bit with sparse weights and activations has a similar potential.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
The research of DS was Funded by the European Union (ERC, A-B-C-Deep, 101039436). Views and opinions expressed are however those of the author only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them. DS also acknowledges the support of Schmidt Career Advancement Chair in AI.
REFERENCES
Greaphcore
ipu.
URL
httpe://dos..rabhoeeaaa//oooofft
tensor -ser- 1a http:///waaaaaao///////////////a
Habana URL httpa///aaaaaaa/ooaa/aaaaaaaaaaa
Nvidia h100.
http:://eea..geer...iiaaaaaa.aa. nvi www.llleehoee--aaaateeeereeeeee in-thtth
URL
Tesla
dojo
technology.
URI. https://teslo-cchhthrreee com/
Sratic/SBYYBB
22tesloo-dbojo-tccchooooooggggg
Md Aamir Raihan and Tor M. Aamodt. Sparse weight activation training. arXiv preprint
arXiv:2001.01969, 2020. /oLL/tttt.:/axxl.oot//////
9199
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In NeurlPS, 2018.
Léon Bottou, Frank Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-31) 311, 2018.
Brian Chmiel, Ron Banner, Elad Hoffer, Hilla Ben Yaacov, and Daniel Soudry. Logarithmic unbiased quantization: Simple 4-bit training in deep learning. ArXiv, abs/2112.1076) 2021a.
Brian Chmiel, Liad Ben-Uri, Moran Shkolnik, E. Hoffer, Ron Banner, and Daniel Soudry. Neural gradients are lognormally distributed: understanding sparse and quantized training. In ICLR, 2021b.
Jungwook Choi, P. Chuang, Zhuo Wang, Swagath Venkataramani, V. Srinivasan, and K. Gopalakrish- nan. Bridging the accuracy gap for 2-bit quantized neural networks (qnn). ArXiv, abs/1807 0bs/1807.00666 069964, 2018a.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, P. Chuang, V. Srinivasan, and K. Gopalakrish- nan. Pact: Parameterized clipping activation for quantized neural networks. ArXiv, abs/1805.0600. 2018b.
Matteo Croci, Massimiliano Fasi, Nicholas Higham, Theo Mary, and Mantas Mikaitis. Stochastic rounding: implementation, error analysis and applications. Royal Society Open Science, 9, 03 2022. doi: 10.1098/rsos..111311
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra Modha. Learned step size quantization. arXiv preprint 1xXv::9022.08133 2019.
Utku Evci. Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICI.R 2018.
10

---

Published as a conference paper at ICLR 2023
Connor Holmes, Minjia Zhang, Yuxiong He, and Bo Wu. Nxmtransformer: Stri-strecturrd sparsifi- cation for natural language understanding via admm. ArXiv, abs./2110.157m 2021.
Mark Horowitz. 1.1 computing's energy problem (and what do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp- 10-14. IEEE, 2014.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Seffi Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method find M transposable masks. In NeurIPS, 2021.
S. A. Janowsky. Pruning versus clipping in neural networks. Physical Review A, 39/ 19(11)660006603, 1989. URL htt.tttts//iinkksss..oo
39. 6600
Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser, Wojeiech Gajewski, Henryk Michalewski, and Jonni Kanerva. Sparse is enough in scaling transformers. 2021.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4350-4359, 2019.
Mark Kurtz, Justin Kopinsky, Rati Gelashvili. Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 5533 5543. PMLR, 13-18 18 Jul 2020. URL httthtttp//oocceeed/ee
mlr.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. ffev. hhttps://githui http:///ooo///bfbcoeeeeeeeee 2022. commit XXXXXXX.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In /CLR, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell, Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270, 2018.
Jian- Hao Luo, Jianxin Wu, and W. Lin. Thinet: A filter level pruning method for deep neural network compression. 2017 IEEE International Conference Computer Vision (ICCV), pp. 5068-5076, 2017.
Stefan Mach, Fabian Schuiki, Florian Zaruba, and Luca Benini. Fpnew: An open-source multi- format floating-point unit architecture for svergy-proportional transprecision computing. IEEE Transactions Very Large Scale Integration (VLSI) Systems, 29(4):774-787, 2020.
Bradley McDanel, Helia Dinh, and J. R. Magallanes. Accelerating dnn training with structured data gradient pruning. ArXiv, abs/2202.00774, 2022.
Asit K. Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosie, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. ArXiv, abs/2104.08378, 2021.
Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhski,, Ron Banner, Alex M. Bronstein. and Avi Mendelson. Loss aware post- training quantization. arXiv preprint 1xX1v:911..07190
2019. htIp:ttt::
org abs /1111 11111119111111111111111111111111
Nvidia.
a100 tensor core gpu architecture.
2020.
URL http://httpa
nvi mwt.ee-aappeearrcccfttteeeeeeeee
Jeff Pool and Chong Yu. Channel permutations for sparsity. In NeurlPS, 2021.
11

---

Published as a conference paper at ICLR 2023
A. Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. ArXiv, 0aos/2003.02389, 2020.
Darko Stosic and Dusan Stosic. Search spaces for neural model training. ArXiv, abs/2105.119200 2021.
Wei Sun, Aojun Zhou, Sander Stuijk, Rob G. Wijnhoven, Andrew Nelson, Hongsheng Li, and Henk Corporaal. Dominosearch: Find layer-wise fine- fine-grained n.m sparse schemes from dense neural networks. In NeurIPS, 2021. URL http:///eepeeeeeeeeeeeeeerrrrrrt
Xiao Sun, Jungwook Choi, Chia-Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In NewrlPS, 2019.
Xu Sun, Xuancheng Ren, Shuming Ma, and Houfeng Wang. meprop: Sparsified back propagation for accelerated deep learning with reduced overfitting. In ICML, 2017.
Ying Wang, Yadong Lu, and Tijmen Blankevoort. Differentiable joint pruning and quantization for hardware efficiency. In European Conference on Computer Vision, pp. 259-277. Springer, 2020.
Xu Weixiang, Xiangyu He, Ke Cheng, Peisong Wang, and Jian Cheng. Towards fully sparse training: Information restoration with spatial similarity. In Association for the Advancement of Artificial Intelligence (AAAI), 2022.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Xucheng Ye, P. Dai, J. Luo, X. Guo. Y. Qi, Jianlei Yang, and Yiran Chen. Accelerating cnn training by pruning activation gradients. arXiv preprint 1rXiv:1908.0011 2019. URL http //arxiv. www.iv.org//bss/10000
00173
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning nam fine-grained structures sparse neural networks from scratch. In ICLR, 2021.
12


Reviews:
[{"review_id": "t0y3Kc9JrM5", "reviewer": "Reviewer_e1dv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for a stochastic unbiased masking of the gradients such that half of them are 0 and they can be used to accelerate the matrix multiplications in an accelerator such as a GPU. In particular, the method induces a form of sparsity called N:M sparsity where N out of M consecutive (in memory) elements are 0. The authors propose to use a masking such that the gradients are still unbiased but also such that they have the minimum variance. Thorough experiments show that using their approximate 2:4 sparsity algorithm as well as the exact 1:2 algorithm, allows training a variety of neural network architectures on both images and text with minimal loss in final performance, if any. Moreover, the authors show that their algorithm can be combined with similar algorithms for the forward and backward pass as well as quantization without significant drop in performance.", "summary_of_the_review": "The method is very intuitive and straightforward. It may be slightly incremental work, however I believe it is a clear contribution.", "strength_and_weaknesses": "Strengths\n------------\n\n- The theoretical analysis is well written and intuitive\n- Accelerating GEMMs is a very important topic of research as it affects all modern neural network architectures\n- Thorough experimental evaluations show that the method indeed does not hurt performance and can be used to train a variety of models\n\nWeaknesses\n--------------\n\n- The main weakness of the paper is the lack of real world performance improvements due to lack of custom kernels or support from currently available accelerators. The speedup that is mentioned is a best case scenario and makes the method seem more useful than it is probably.\n- Another weakness is the lack of comparisons with MSE based sparsity for the gradients. Even though previous work has shown that unbiased estimators are more important for the gradients than the weights, it would strengthen the paper significantly if instead of the greedy estimator the paper also compared with MSE optimal biased estimators.\n\nTypos\n------- \n\n- eq. 8 typo it should be $sign(a_2) \\left(\\|a_1\\| + \\|a_2\\|\\right)$\n- Fig. 2 typo $a_4 >> \\max(a_1, a_2, a_3)$", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KjLbB3pNhxE", "reviewer": "Reviewer_B9mF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper develops a new scheme for pruning the gradients passed through a network during back-propagation. In particular, the proposed method designs a stochastic pruning operator which is unbiased and provides the minimum variance output. Both 1:2 (2 inputs, 1 non-zero output) and 2:4 (4 inputs, 2 non-zero outputs) variants of this operator were developed. In testing, networks pruned with the proposed operator demonstrated little-to-no degradation in performance across a range of network architectures and classification problems.\n", "summary_of_the_review": "The paper provides a simple gradient pruning procedure that could accelerate future hardware systems. While the form of the pruning operator could use further motivation, I'm overall in favor of this paper's publication.", "strength_and_weaknesses": "# Strengths\n\nThe paper provides a novel method for pruning the back-propagation gradients.\n\nThe proposed method does not hurt network performance, whereas naive greedy baselines hurt performance significantly.\n\nThe paper is well-written and reasonably easy to follow.\n\nThe paper provides a significant discussion of the implications the paper's results have on hardware acceleration and design\n\n# Weaknesses\nThe paper doesn't make a strong case for why the purning operator needs to be unbiased and only points vaguely to Chmiel et al.'s finding that \"for the neural gradients... it is critical to use unbiased quantization (i.e., Bias[θ(a)] = 0).\"\n\nFor completely understandable reasons (no hardware support for pruning during training), the paper is unable to demonstrate any actual training time reductions. Hopefully this changes as new hardware is developed.\n\n# Suggestion\nThe introduction of the MSE of the pruning operation describes $a$ as scalar, rather than a vector. The resulting discussion is then confusing as there's no stochasticity in $a$ and one is left asking why not just let $\\theta(a)=a$. I suggest introducing $a$ as a vector from the start, as this will make the resulting discussion easier to follow -- there are many choices for the pruning operator which would produce different MSEs. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "4cG3mdSALj", "reviewer": "Reviewer_ccBg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to accelerate training by pruning the calculated gradients. More specifically, the authors propose 1:2 and 2:4 minimum-variance unbiased estimators and show their effectiveness on a number of tasks, where accuracy is not severely affected by the proposed pruning scheme.", "summary_of_the_review": "A well written paper with interesting and relevant results for practical applications of DL.", "strength_and_weaknesses": "The paper is well written and proposes very attractive improvements to training speed without affecting the accuracy.\nGiven a relatively general title, I'd be curious to see some results on using the approximate method in setups with different sparsity setups, e.g, 4:8, and also higher sparsity, e.g., 1:4.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%