Towards Structured Dynamic Sparse Pre-Training of BERT
Authors: Anastasia S. D. Dietrich
Frithjof Gressmann
Douglas Orr
Ivan Chelombiev
Daniel Justus
Carlo Luschi
Venue: ICLR 2022 Submitted (2022)
Keywords: sparsity
natural language processing
pre-training
computational efficiency

Abstract: Identifying algorithms for computational efficient unsupervised training of large language models is an important and active area of research. 
In this work, we develop and study a straightforward, dynamic always-sparse pre-training approach for BERT language modeling, which leverages periodic compression steps based on magnitude pruning followed by random parameter re-allocation. 
This approach enables us to achieve Pareto improvements in terms of the number of floating-point operations (FLOPs) over statically sparse and dense models across a broad spectrum of network sizes. 
Furthermore, we demonstrate that training remains FLOP-efficient when using coarse-grained block sparsity, making it particularly promising for efficient execution on modern hardware accelerators.

Under review as a conference paper at ICLR 2022
TOWARDS STRUCTURED DYNAMIC SPARSE PRE- TRAINING OF BERT
Anonymous authors Paper under double-bling review
ABSTRACT
Identifying algorithms for computational efficient unsupervised training of large language models is an important and active area of research. In this work, we develop and study straightforward, dynamic always-sparse pre-training approach for BERT language modeling, which leverages periodic compression steps based on magnitude pruning followed by random parameter re-allocation. This approach enables us to achieve Pareto improvements in terms of the number of floating- point operations (FLOPs) over statically sparse and dense models across a broad spectrum of network sizes. Furthermore, we demonstrate that training remains FL.OP-efficient when using coarse-ge -rannee block sparsity, making it particularly promising for efficient exccution modern hardware accelerators.
INTRODUCTION
The increasing task performance gains of large, pre-trained language models have fueled interest in computationally efficient unsupervised training (Kaplan et al.] (202). In recent years, sparsity
has regained popularity as a technique for improving the computational efficiency of deep learning
models (Hoeller et al. 202T). Current sparsity methods can be distinguished into approaches that
impose sparsity on the weights of neural networks via weight sparsity (Frankle & Carbin 2019 Gale
et al. 2019 Bellec et al. 2017: Mostafa & Wang [019 EEci et al.] [2019] Dettmers & Zettlemoyer
2019 Mocanu et al. 2018] Jayakumar al.) 22220) or techniques that dynamically route activations
to only interact with subset of the network weights via conditional sparsity (Shazeer et al. 2017
Lepikhin et al. 2020 Fedus et al., 2021
FeduuFedus
Lewis et al. 202T).
In weight sparse training (Frankle & Carbin 2019 Gale et al. 2019), the number of network
parameters is reduced by imposing sparsity patterns on the network weights. As a result, weight sparse training can lead to significant savings in FLOPs, making it promising for scaling to larger network architectures for a given compute budget. One of the most promising candidates for weight sparse training is dynamic sparsity (DynSparse), which reduces FLOPs while only requiring training of sparse subsets of the over- peer-paramettrizee network (Bellec et al. 2017; Mostafa & Wang 2019
Evei al., 20099 Dettmers & Zettlemoyer) 2019 Mocanu et al. 201 11 Jayakumar et al. (022002)) Liu
et al.. 2222la. la). In DynSparse approaches, the sparsity pattern imposed on the weights is continuously modified during training using pruning and re-allocation strategies. This evolution leads to a joint
exploration of both network topology and parameters, which has been shown to outperform static
sparsity baselines (Bellec et al., 2017 Mostafa & Wang. 2019 Zettlemoyer 2019).
Evei et al. 2019: Dettmers &
However, far, the limited performance on language modeling task (Evci et al. 2019 has resulted in
DynSparse training not seeing wide adoption for large-scale language modeling tasks despite recent advances (Jayakumar et al., 2020). Given the high cost and energy consumption of unsupervised training of large-scale language models (Strubell et al., 2019 Patterson et al.) [02I]. dynamic sparsity bears the potential to make pre-training more efficient and affordable. To this end, we adopt and investigate DynSparse training techniques (Dettmers & Zettlemoyer, 2019 Evei et al. 2019) for wre-training of BERT bidirectional language encoder (Devlin et al. 2018) based on the highly scalable Transformer architecture (Vaswani et al. 2017).
Our work achieves Pareto improvements versus the dense baseline using both structured and unstruc- tured DynSparse training of BERT.

---

Under review as a conference paper at ICLR 2022
1.1 CONTRIBUTIONS
Investigating dynamic always-sparse training for BERT pre-training. We adapt the DynSparse training algorithm to BERT pre-training (Section 2. In particular, we find that gradient-based re-allocation (Evei al. 2019) results in collapse of the explored network parameters (Figure which we mitigate through the use of random parameter re-allocation.
Achieving scalable FLOPs efficient dynamic sparse training. We compare dense and sparse methods for a given FLOPs budget and demonstrate both algorithmic scalability and Pareto improvement on the FLOPs scale, as shown in Figure
Adapting dynamic always- sparse training to block structures. We extend the unstructured DynSparse training towards block-sparse structure (Section 3.2). In particular, we find that the choice of metric during block pruning has a strong influence on the task performance, as shown in Figure
Pareto improvements for structured DynSparse training We show that the resulting structured DynSparse training of BERT without structured regularization gives Pareto improvements com- pared to the dense BERT baseline, as shown in Figure
In the following section, we report the results of explorative experiments conducted to motivate our study DynSperse training of BERT. The rest of the paper then concentrates on DynSparse training, with methodology discussed in Section 2 and results presented in Section 3
1.2 IDENTIFYING SUITABLE ANGLE OF ATTACKS FOR SPARSE PRE-TRAINING
Sparse training of unsupervised language models is relatively under-explored. compared to sparse training of the supervised fine-tuning objective (Radiya-Diee -Dxitt & Wang 2020 Chen et al. 2020 Sanh ct al. 2020). Consequently, we design two explorative experiments assess whether DynSparse training is a suitable sparse training algorithm for pre-training of BERT.
Firstly, analyze the importance of trainable parameters by keeping random pattern of weights non-trainable (constant non-zero) or zero-valued throughout training. This experiment allows us to disentangle the role of 'zero' vs. 'untrainable" weights in the connectivity patterns, to shed light on the parameter dependence of BERT pre-training. Like zeroed weights, the constant weights are unable to encode new information about the task. Still, they might promote the propagation of signals or gradient-flow through the network, which has been considered core aspect of some sparse training algorithms in vision (Evci et al., 2020 Lubana & Dick 2021 Tessera et al. (02T). Non-zero parameters might also lead to better util utilization of the remaining trainable parameters. However, as shown in Figure aa), we find that none of these effects plays a relevant role the training dynamics, since the task performance of the network with sparsified weights (dashed orange line) matches the one with the same fraction of untrained weights (solid blue line). Different from vision models that are often based on convolutions, the transformer architecture contains large dense matrices and multiplicative interaction (Jayakumar et al. 2019). While zeroing parameters is not expected to affect the training dynamics, the performance remains bounded by the number of trainable parameters.
Secondly, we would like to analyze the effect of sparsification at different stages of the training process. For this purpose, we keep random subset of the network parameters untrainable in the first half of the pre-training, before making them trainable in the second half (and vice versa). Unlike magnitude pruning, which eliminates learned information, freezing and unfreezing parameters ensures symmetry between the different phases (ignoring the linearly decaying learning rate schedule). The agreement in the task performance towards the end of training in Figure [b) indicates that representation is continuousl built up during training, with no particular effect of when the sparsification is applied. This lack of preference is interesting, given that pre-training has been found to lead to reduction of the intrinsic dimension (Li et al., 2018) with respect to downstream tasks (Aghajanyan et al.) (2020). Our results suggest that sparse pre-training does not necessarily profit from an initial dense training phase. Therefore, we can distribute computation in way that is both algorithmic and computationally beneficial. In DynSparse training, the network representation is always--parse ie. it does not rely on the representation of the underlying dense network, making the approach suited for sparse training of large language modeling architectures. Consequently, we believe that BERT pre- pre-training is well suited for DynSparse training.
2

---

Under review as a conference paper at ICLR 2022
3.2
type
n-
zero
125 late untrained
untrained
1--0.25 early untrained
3,0
MLM loss
2.8
2.6
(a)
(b)
2.4
10
10-1
0.0 0.5
1.0 1.5 2.0
2.5
3.0
3.5
4,0
Fraction of trainable parameters
Sequences
168
Figure 1: (a) MLM validation loss of BERT-Small with random subset of parameters set to (solid blue curve) or kept untrained (dashed orange). (b) training loss curves of BERT-Small during pre-training of 10 epochs (757k steps) fixing random subset of the parameter either early (orange dashed) or late (blue dash-dotted) during the training, as well as for the dense baseline (solid black). The vertical line indicates the unfreeze (freeze) event location, where untrainable parameters are made trainable (or trainable parameters are frozen). We pick the best learning rate for each experiment using a grid search over 0.000087 2 with m 0, 1..., 5, given in Table A.15
2 METHODOLOGY
Throughout this work, we study the self-supervieed pre-training objective from the original BERT model (Devlin et al. 2018), which consists of the Masked Language Model (MLM) loss, correspond- ing to the task performance in predicting a random subset of masked tokens, and the noisier Next Sentence Prediction (NSP) loss for binarized next sentence prediction. We focus on a single phase of pre-training with sequence length of 128, using the Adam optimizer. All hyperparameters are given Appendix A for a training length of 10 epochs.
2.1 ADAPTING UNSTRUCTURED DYNSPARSE ALGORITHM TO BERT
Figure 2: Schematic illustration of pruning and re-allocation step in a typical DynSparse training algorithm leading to an evolution of the network representation in parameter space. The dynamic evolution of the sparsity pattern allows the DynSparse training algorithm explore a larger fraction of the network parameters compared to static sparsity. while remaining "always sparse". For unstructured DynSparse training, the granularity of the sparsity pattern is of block size I 1x1, while for structured DynSparse training. the block size is chosen between 4 4x4, 8x 8 and 16x 16.
Subset 2
Paramete
Subsen
Sparaity uposte Prune (pruning ratioupr) the paramener re-allocate parameters
In the present work, we first study and adapt the unstructured DynSparse training schematically shown in Figure to pre-training of the BERT language models. Specifically, we initialize the sparsity pattern randomly with the same fixed sparsity ratio all fully connected encoder weights (non-embedding weights). The weights are initialized using a truncated normal distribution (see also Figure 9). During update step of DynSparse training (see Algorithm [] we magnitude pruning to remove a time / dependent fraction pr(() of the network parameters. The same fraction of parameters re-allocated elsewhere in the weight tensor. To complete the sparsity update step, all newly allocated parameters and their corresponding first and second-order moments of the Adam optimizer are initialized to zero. Given that DynSparse training has been primarily developed for vision architectures (Dettmers & Zettlemoyer 2019 Evci et al. 2019) and did not show competitive performance on the language tasks, we find it necessary to reassess some of the algorithm choices for BERT. In particular, during the re-allocation step of DynSparse training, use random re-allocation of pruned weights instead of gradient- based techniques as RigL (Evci et al. 2019). For one, this avoids potential issues from a collapse of the explored parameter space (compare Figure [T].. More importantly, the absence dense gradient computation makes our approach always-sparse, such that the full dense model is never actually instantiated. We found that the cosine decay of the pruning
3

---

Under review as a conference paper at ICLR 2022
ratio introduced in Evei al. (2119) outperforms constant pruning schedules and leads to reduction of the changes in network topology during training. We refer to the maximum pruning ratio pr simply as "pruning ratio" throughout the paper. All DynSparse hyperparameters optimized for sparsity ratio 0.9 (for more details, refer to Appendix A.1
2.2 BLOCK-SPARE DYNSPARSE ALGORITHM
Extending unstructured DynSparse training towards using structured sparse computation requires modification to both the prune and update steps in Figure Magnitude pruning be justified as a simple compression algorithm resulting in unstructured sparsity. However, there no unique way to extend the magnitude pruning metric to blocks of parameters. Choosing good metric for block pruning is essential, as magnitude pruning has been surprisingly successful in preserving the task performance of sparse networks (Gale et al. 2019). In the following, we evaluate the selection of norms consisting of L Lo-norm and L L--norm as different criteria for estimating the importance of blocks of parameters. For a block of weights B W (reel(na))))) taken from weight tensor W indexed by B, the L"-norm is given by
1/2
L²(B)= = Ei
where the exponent p. e-g. p = 00, 2, 1, controls the relative importance individual parameters of a block according to their magnitude. In the limit of block size 1 x 1, the block pruning according to Eq. ( reduces to magnitude pruning. allowing us to investigate the task performance with increasing block sizes. For small values of p - 0, each parameter in the block contributes equally towards the importance of the block element as | + 1, while for large values of p - oc the importance of the block collapses towards the parameter with the largest magnitude, with L (B) - max([B)..
Therefore, the pruning metric for blocks controls the extent to which the magnitude of each of the parameters in block contributes to the importance of the block itself.
Algorithm 1: Structured DynSparse training
Input: total number of training step T, total number of sparsity updates n 1, pruning ratio p, (t) at time sparsity ratio s, block size B;
Initialize: Impose random block-sparse pattern on non-embedding weights with uniform constant sparsity ratio across all layers. Initialize weights sampled from truncated normal distribution;
for k = 1 to n do
Train network with static sparsity pattern for T/n steps;
For each weight tensor:
prune fraction (t) of blocks with smallest - defined in Eq. [
re-allocate the same fraction pr (t) of blocks; re-allocated parameters, first and second-ord -odder
moments Adam optimizer are all initialized to zero
end
Structured regularization Group Lasso regularization is commonly used as a structured sparsity- inducing regularizer (Wen et al. 2017: Narang et al. 2017). We introduce the Group Lasso regu- larization in the update A W of weight tensor W following the decoupling of weight decay and Adam optimizer from Loshchilov Hutter (017)). More specifically, the entry (i,j) of the parameter
update A AW is adjusted to AW;"" as
Wij
AWT AW, Ir(t): Agroup Wald VB
W2
+
(na))3(()))
where (i..j) denotes the set of weight indices that belong to the same block as the (i. weight element and 7tt)) is the linearly decaying learning rate (see Appendix A.. The remaining coefficients are the Group Lasso coefficient Agroup. and the small constant 10 for numerical stability. The
extra pre-factors watd 0.02 and /B corresponding to the standard deviation of the weights
4

---

Under review as a conference paper at ICLR 2022
initialization (see Appendix A and the square-roo of the block size respectively chosen such as to ensure that the regularization coefficients of weight decay and group lasso are comparable in magnitude.
2.3 PARETO CURVE ASSESSMENT OF FLOPS EFFICIENCY
A recent review by Hoefter et al. 2021 pointed out the need for a rigorous framework for comparing sparse training algorithms. In the present work, we introduce methodology for comparing the sparse task performance on the full BERT-family Pareto curve (Turc et al. 22119) beyond the Same Capacity Sparse vs. Dense Comparison approach introduced by Tessera et al. (202T). Comparing different algorithms using a Pareto curve allows to perform multi-objective assessment under competing constraints, e.g., the desire to use little compute and achieve a high task performance. This multi-objectivy assessment is particularly useful for assessing the generality and scalability of different training algorithms. Furthermore, the use of Pareto curves allows us to systematically assess algorithmic differences by comparing DynSparse training with dense and static baselines an equal FLOPs budget.
Choosing optimal learning rates for sparse and dense models of various sparsity ratios and model sizes essential to ensure a fair comparison of different methods. Naive grid search optimization of the hyperparameters for full Pareto investigation quickly becomes intractable. To mitigate this, have identified and tested the applicability of scaling rules for learning rates across model sizes and sparsity ratios.
The dense and sparse BERT-family learning rates are obtained from a grid search for 0.0001 2 with m 0, 1, ..., 5 shown in Figure 12 (see Appendix A..4). Interestingly, the results indicate that for given number of parameters, the optimal learning rate of the sparse model is significantly larger than the learning rates of dense models (Figure (3). To reduce the number of hyperparameter sweeps for large model sizes, we generalize the learning rate scaling with sparsity as
7)(5) 77(5 0) exp (1.969s² + 0.2905s),
where r)ss 0) is the optimal learning rate obtained for a dense model of given size. We tested the prediction of the unstructured static sparse learning rate fit from Eq. (3) using DynSparse training with block sparsity 16x 16 across both model sizes and sparsity ratio, and obtained good agreement between the predicted optimal sparse learning rate obtained from this rule and values obtained through grid search, shown in Figure 13] We also found that the learning rate rule generalizes from static to unstructured DynSparse, as shown in Table A. 14] Identifying the mechanism allowing sparse models to profit from larger learning rates than dense models with the same number of parameters (see Figure 13) is left as an exciting direction for future research.
3 RESULTS
3.1 ADAPTING DYNSPARSE TRAINING TO BERT
In order establish general improvement of the dynamic sparse training algorithm with both FLOPs and memory, we study dynamic sparse training of BERT family across multiple model sizes. We analyze the scaling behavior of our DynSparse models with model size (for fixed sparsity ratio) and sparsity ratio (for a fixed model size).
We find that the DynSparse training algorithm with random re-allocation and sparsity s 0.9 leads to Pareto improvements compared to the dense BERT-family (see Figure 3). The improvements of DynSparse training over the dense baseline remain for range of model sizes, indicating that DynSparse training can achieve more efficient utilization of FLOPs or network parameters at any scale. Furthermore, we find that these performance advantages are due to the continued updates of the sparsity pattern. We do not observe any improvements of the static baseline in FLOPss efficiency of larger models when the randomly initialized sparsity pattern is kept constant. In fact, for large model sizes, static sparsity almost perfectly matches the dense baseline. This indicates that the sparse network architecture itself brings no performance advantages. Any improvements are therefore expected to arise from continuous compression and evolution of the network representation. For DynSparse BERT-Base. we achieve an improvement in FLOPs efficiency by a factor of 0.48 compared to an interpolation of the dense BERT family, as indicated by the horizontal black arrow in Figure 3
5

---

Under review as a conference paper at ICLR 2022
X
Dynäparse Static
Dynäparsc
MLM loss
10"
U.S
0.75
Nini
MIM
Small
Medium
(a)
Bast
2x10
(b)
2x 105
10
10P
101t
Non-embedding FLOPs
Non-embedding FLOPs
Figure Pareto curve of the BERT family (Ture al. 2019), comparing validation MLM loss of unstructured DynSparse training (orange dotted line) with static spar- sity (solid blue line) and the dense baseline (black dashed line, the standard deviation not visible at this scale) as function of FLOPs. All sparsity results are obtained for pre- tre-training with sparsity ratio 0.9, n = 160 pattern up- dates, and optimal pruning ratio p, 0.5 (see Figure S). The black arrow indicates a reduction of FL.OPs for the same MLM loss by a factor of 0.48.
Figure 4: Comparing validation MLM loss of DynSparse training of BERT- Medium with various sparsity ratios (in- dicated by color and marker style and joint by orange dotted line) with dense training of BERT family (black dashed line) as a function of non-embedding FLOPs. For all sparsity ratios, the hyperparameters optimized for spar- sity ratio 0.9.
We observe task performance improvements across range of sparsity ratios (see Figure 4. However, since the results used hyperparameters tuned for sparsity 0.9, performance for other sparsity ratios could potentially be further improved with additional tuning. In sum, we find that DynSparse training leads to more efficient utilization of parameters and FLOPS for all model sizes.
weights
0.10
p.- 0.5
2.46
80
40
2.45
0.5
0.25
0.09
160
2.45
0.08
MLM loss
MLM loss
2.44
1.0
160
0.06
2.43
2.43
0.05
R:
2.42
2.42
0.04
(b)
0.03
(c)
0.75
2.41
0.8
1.0
removed new weights
rablo DDF covered
Figure 5: Characterization of the DynSparse pre-training of BERT-Medium with sparsity ratio 0.9. All layer-wise averages shown correspond to maximum value obtained during training. (a) MLM loss as function of the fraction of explored network parameters (DOF) with changing number of sparsity pattern updates (b) MI.M loss as a function of the ratio of removed, new weights with changing pruning ratio pr- (c) Joint effect of pruning ratio pr (solid line) on the ratio of removed, new weights, and DOF covered during DynSparse training. The best performing values (n 160,
pr = 0.5) from (a) are marked by a circle.
To improve understanding of the sparse training dynamics, extract measures that help to explain the efficiency of specific hyperparameter choices (see Appendix [. Given that the DynSparse task performance advantage arises from the continual update of the sparsity pattern, we begin by quantifying the amount of parameter exploration. While the DynSparse models have only tiny fraction of parameters available at any given time, the pattern update means that they can explore all network parameters throughout training and thus increase the effective weight space. We measure the effectively covered space by tracking the fraction of network weights of the corresponding dense network that have been activated at any point during the training and compare with the parameter count of the equivalent dense network to obtain the total explored degrees of freedom (DOF All
A similar quantity has been independently studied in Liu al. 202 "in-time over-parametrrizatonn

---

Under review as a conference paper at ICLR 2022
2,300
2.05
M
MLM loos
2,500
MLM 163
2.515
2.02
2,510
2,505
2.500
group lasso
2.59
weight decey
2,495
10-2 regularization coefficient
16-1
2t-rorm
1-- Form Block Metric
1" norm
Figure 6: MLM validation loss of DynSparse BERT-Medium with sparsity 0.9 for block size B = 16 as function of the regularization coefficient for Group Lasso regularization (solid blue) or weight decay (orange dashed). The eтror bars correspond to the standard deviation over three runs. Number of updates n = 80, pruning ratio p, = 0.5.
Figure 7: Block metric dependence of DynSparse training of BERT-Medium with
sparsity s = 0.9 for block size B = 16.
Confidence interval estimated by
calculating the standard deviation over
three datapoints with their numerical values given in Table A.7
quantities shown in the following correspond to averages taken over all layers, we did not observe a systematic layer dependence of these quantities.
The total explored degrees of freedom monotonically increases during training, starting at the fraction of non-zero at the beginning of training and then saturating at an algorithm and hyperparameter dependent value toward the end of training (see Figure 11 for typical shape). We observe that the maximal number of explored DOF can be controlled through the pruning ratio and the number of sparsity pattern updates n (Figure 5). An increase in the update frequency leads simultaneous saturation in both task performance and the number of explored degrees of freedom (Figure SJa)). On the other hand, the pruning ratio pr- reaches an optimal value and strongly influences the performance with different fraction of removed, new weights (Figure b)). Notably, we find that the best pruning ratios are reached once the ratio of DOF approaches 1, corresponding to almost complete exploration Further increases in remove trainable weights that have
of all network parameters (Figure(5jo) just been initialized in the previous update step and lead to deterioration the task performance. Overall, we note that the best task performance is obtained by balancing the DOF while avoiding wasted compute in the form of parameters that are being allocated and immediately removed (as demonstrated in Figure 5). Given these findings, postulate that ideal training outcomes require exploration of all available parameters as well as an only moderate amount of noise injection.
3.2 BLOCK-SPARSE DYNSPARSE TRAINING
In structured DynSparse training, the block pruning done according to the Lo-norms from Eq. of the respective blocks of parameters. For the T--norms in Table we obtain the best performance using the -nomm, which corresponds to the sum of the parameter magnitudes. Moreover, all block parameters contribute toward the block's importance, given that -noor outperforms other norms that assign larger importance to dominating weights in block.
studied (p 1,2, oo), shown
Next, we evaluate the use of structured regularization applied to sparse weights during DynSparse training with block size 16x 16. To compare potential advantages from using a structured regular- ization against unstructured regularization method, we have also evaluated the task performance for tuning weight decay coefficient instead the Group Lasso coefficients. As shown in Figure [ we obtain the best task performance using weight decay. The regularization coefficients are only tuned for the sparse, non-embedding weights. Other sources of unstructured regularization such as dropout (and in the case of Group Lasso also weight decay) are set to zero. While our results are in agreement with the competitiveness of block pruning versus the Group Lasso experiments in Narang al. (2017). we have not tested more advanced regularization methods (Yang et al. 2019 Mummadi et al. 2019). We find that the structured regularization does not lead to any performance ad advantages tuning weight decay.
7

---

Under review as a conference paper at ICLR 2022
Table Task performance of DynSparse training of BERT-Base with sparsity 0.9 for various block sizes B. compared to dense BERT-Small with similar number of FLOPs and linear interpolation of baseline values ("Matched") with exactly the same number of FLOPs.
0.75
0.63
Reduction
Hyperparameters
specifically tuned for B 16
(number of updates n = 80, pruning ratio p, = 0.5). See
0.55
Appendix Table A.8 for block size dependence. The standard deviation is estimated over three runs.
0.50
1.0
12
14
16
Block size
Model
ML.M
FLOPs
Figure Block size dependence of the reduction in FLOPs for DynSparse training compared to (interpolation) of the dense BERT family for given task performance. Values correspond to the block sparse DynSparse training of BERT-Bas Base given in Table [
Small (dense)
2.310 + 0.002
10.07
10
Matched (dense)
2.350
8.33 10"
Base (s = 0.9)
16
2.311 = 0.01
8.33 109
Base (s 0.9)
8
2.295 = 0.002
8.33 10
Base (s 0.9)
2.272 0.01
8.33
Base (s 0.9)
1
2.160 # 0.002
8.33 109
Next, we compare structured DynSparse training against the baseline using both horizontal and vertical slice of Pareto plot. For a vertical slice, e.g., constant FLOPs comparison, demonstrate that DynSparse training can preserve some task performance advantages when block sparsity of size 4 4, 8,8, and 16x 16 is used (Table [ For horizontal slice (see horizontal arrow in Figure 3 measuring the reduction in FLOPs for given task performance, we achieve reduction between 0.5 for unstructured sparsity and 0.83 for block size 16, as shown in Figure X The inverse of the FL.OPs efficiency improvements gives the maximum relative execution time of FI.OPs for sparse compared to dense computation to preserve Pareto improvements in terms of wallclock time and needs to be below factor of 2 for unstructured sparsity and 1.2 for block sparsity for DynSparse training (see Appendix A.3) This compute efficiency makes DynSparse training promising for practical applications that seek further benefit from the higher computational efficiency of block computation.
RELATED WORK
Lottery ticket and pruning initialization Weight sparsity has been traditionally viewed as a technique for compressing network representation leading to reduced FLOPs and trainable param- eters. The lottery ticket hypothesis (Frankle Carbin 2019 Frankle et al. 2020) postulates that through iteratively pruning and re-initialization, is often possible to identify smaller subnetworks at initialization or early on during training that can be trained to full model performance of a large over- parametrized network. Since then, there has been significant amount of work studying techniques for identifying sparsity distributions at initialization (Lee et al.
2019 Wang et al.] [020 Tanaka et al.,
2020. Zhang & Stadie 2020 Lee al. 20200 Frankle et al. 2021 Su al. (020). Recently, the
identification of lottery tickets early during training has allowed achieving time-to-train savings after
short, dense training phase (Chen
2021) by pruning attention heads and neurons early during
the pre-training phase (Chen et al. 2021
Dynamic sparsity In DynSparse (Mocanu et al. 2018 Bellec et al. 2017 Liu et al. 2019 Mostafa
& Wang 2019 Dettmers & Zettlemoyer 2019 Evei et al. 2019 Liu et al. 2021a). the sparse con-
nectivity pattern is evolved during training. Most DynSparse algorithms currently rely magnitude pruning to remove unimportant network parameters. However, the algorithm show large differences in the exact re-allocation criteria, which range from random re-allocation (Bellec et al. 2017 Mo-
canu et al. 2018 Liu et al., [2019] (222aa) to a directed evolution based on momentum (Dettmers & Zettlemoyer 2019) or gradients (Evci et al. 2019).
Compute-efficient sparse training Complementary to viewing weight sparsity as a compression technique of dense networks, sparsity allows increasing network dimensions, potentially resulting in an augmentation of the effective model capacity for given amount of compute and memory (Gray et al. 2017). However, most investigations into sparse training currently impose algorithmic con-
8

---

Under review as a conference paper at ICLR 2022
straints through the of pre-defined sparsity patterns (Vooturi et al. 2020 Zhou et al. 2021) soarse-grained sparsity structures (Gray et al. 2017) or even result in increased compute and memory compared to dense training through the use of masking.
In the present work, we contribute towards compute-efficiet training from an algorithmic point of view, by extending DynSparse training towards structure. Additionally, we leverage the 2nd generation of Graphcore's Intelligence Processing Unit (IPU) (Graphcore, 2021) to dynamically train large, structured DynSparse models using Graphcore's 's DynSparse library
Structured sparsity Simple unstructured sparse training algorithms based magnitude pruning heuristic have shown remarkable ability to preserve the task performance of over wver-parametrized neural networks (Gale al., 2019). Nevertheless, on the execution side, unconstrained magnitude pruning results in unstructured sparsity patterns, which remain challenging to support on traditional hardware accelerators (Naang al. 2017). Using coarser- -eraiied sparsity structures resulting in contiguous memory access can mitigate this problem. Nevertheless, the resulting gains in execution efficiency are often achieved the cost of deterioration in task performance (Narang al. 2017
Mostafa & Wang 2019). Approaches to improve the task performance of structured sparsity during training range from structured regularization (Wen et al. 2017 Narang et al., 2017; Yang al. Mummadii et al. [019;; Louizos et al. 2018). threshold-basce pruning using representation based on block sparsity (Narang al. 2017), network slimming (Liu et al., 2017) 7 and low-rank factorization (Wang et al. 2019) to frequently changing sparsity pattern with distinct granularity varying from block (Hadifar et al. 20200, channel (Gao et al. (0188) to full layers (Fan et al., 2020).
Conditional sparsity and models with large number of parameters Unlike dynamic sparsity, conditional sparsity does not reduce the number of trainable parameters that define the model. The task performance of semi-supervised language models generally improves with model size under appropriate scaling of total computation time and dataset size (Kaplan et al., (222).00 In conditional sparse training (Shazeer et al., 2017; Lepikhin et al., 2020 Fedus et al. [202] Lewis al., 202I). activations are dynamically routed to subsets of the network weights distributed over a large number of hardware accelerators. Conditional sparse training leverages increases in the number of network parameters to improve the task performance for constant FLOPs budget (Fedus et al. 2021).
5 CONCLUSION & FUTURE WORK
In this work, we demonstrated that DynSparse training of BERT leads to a more FLOP-efficient utilization of the trainable parameters. Our experimental work has focused on BERT MLM pre- training with sequence length 128, and further research is needed to evaluate the performance of pre-training with larger sequence lengths and fine-tuning to downstream tasks.
An important direction stems from the practical opportunity to translate the FLOPs savings into reduced cost of training. Remarkably, we found that even a naive block-sparse version of the DynSparse algorithm remains FLOP-Pareto efficient, which forms the first step towards more compute-efficient training of large-scale language models. However, further task performance improvements are necessary to fully translate the task performance advantages into time-to-train win on the Pareto curve. In particular, it will be important to shed further light on the conditions that enable the performance gains in unsupervised training, particularly the relationship between the number of available parameters and achievable task performance.
REFERENCES
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, arXiv Prepr. wxXiv001.132255 2020.
Brian R. Bartoldson, Ari S. Morcos, Adrian Barbu, and Gordon Erlebacher. The Generalization- Stability Tradcoff in Neural Network Pruning. ICLR, 2019.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep Rewiring: Training very sparse deep networks. 6th Int. Conf. Learn. Represent. ICLR Conf. Track Proc., 2017.
@msoofflow/dynaii.ccm

---

Under review as a conference paper at ICLR 2022
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The Lottery Ticket Hypothesis for Pre-traime to-itrinnd BERT Networks. NeurIPS, 2020.
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang. and Jingjing Liu. EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets. ACL-IJCNLP, 2021.
Tim Dettmers and Luke Zettlemoyer. Sparse Networks from Scratch: Faster Training without Losing Performance. arXiv Prepr. arXiv/900.008840, 2019.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT pp. 4171-4186, 2018.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the Lottery: Making All Tickets Winners. 37th Int. Conf. Mach. Learn. ICML., pp. 2923-2933, 2019.
Utku Evei, Yani A. Toannou, Cem Keskin, and Yann Dauphin. Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win. arXiv Prepr. arXiv2010.03533, 2020.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing Transformer Depth on Demand with Structured Dropout. ICLR, 2020.
William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv Prepr. arXiv2101.03961, 2021.
Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, ICLR, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear Mode Connectivity and the Lottery Ticket Hypothesis. ICML, 2020.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin, Pruning Neural Networks at Initialization: Why are We Missing the Mark? ICLR, 2021.
Trevor Gale, Erich Elsen, and Sara Hooker. The State of Sparsity in Deep Neural Networks. arXiv Prepr. 102.04902.009544 2019.
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng- Xu. Dynamic Channel Pruning: Feature Boosting and Suppression. 7th Int. Conf. Learn. Represent. ICLR, 2018
Graphcore. Graphcore Homepage, 2021. URL https: bestts::/www.qgee http://wwww.aapccc..aaaaaa
Scott Gray, Alec Radford, and Diederik P Kingma. GPU Kernels for Block Block-Sparse Weights, 2017.
URL
Amir Hadifar, Johannes Deleu, Chris Develder, and Thomas Demeester. Block-wise Dynamic Sparseness. arXiv Prepr: arXiv2001.04686 2020.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks. arXiv Prepr. arXiv2102.00554, 2021.
Siddhant M Jayakumar, Wojeiech M Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osidnero, Yee Whye Tch, Tim Harley, and Razvan Pascanu Deepmind. Multiplicative Interactions and Where Find Them. ICLR, 2019.
Siddhant M. Jayakumar, Razvan Pascanu, Jack W. Rae, Simon Osindero, and Erich Elsen. Top-KAST: Top-K K. Always Sparse Training. NeurIPS, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu. and Dario Amodei. Scaling Laws for Neural Language Models. arXiv Prepr. arXiv2001.08300 2020.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Tor. SNIP: Single-shot Network Pruning based Connection Sensitivity. ICLR, 2019.
10

---

Under review as a conference paper at ICLR 2022
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip H. S. Torr. A Signal Propagation Perspective for Pruning Neural Networks at Initialization. ICLR, 2020.
Dmitry Lepikhin, HyoukJoong Lee. Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, arXiv Prepr. arXiv2006. 16668, 2020.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. BASE Layers: Simplifying Training of Large, Sparse Models. arXiv Prepr. 1rXiv2103.16716. 2021.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the Intrinsic Dimension of Objective Landscapes. 6th Inl. Conf. Learn. Represent. ICLR - Conf. Track Proc., 2018.
Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, and Mykola Pechenizkiy. Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware. arXiv Prepr. urXiv/901.09/11 2019.
Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei, and Mykola Pechenizkiy. Selfish Sparse RNN Training. Proc. 38th Int. Conf. Mach. Learn., 2021a.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do We Actually Need In-Time
Dense arXiv2102.0288 2021b.
in Sparse Training, arXiv Prepr.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang, Learning Efficient Convolutional Networks through Network Slimming. In Proc. IEEE Int. Conf. Comput. Vis., volume 2017-Octob, pp. 2755-2763 2017.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. 7th Int. Conf. Learn. Represent. ICLR, 2017.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning Sparse Neural Networks through L_0 Regularization. ICLR, 2018
Ekdeep Singh Lubana and Robert P. Dick. A Gradient Flow Framework For Analyzing Network Pruning. ICLR, 2021.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nat. Commun., (1):23383, 2018.
Hesham Mostafa and Xin Wang. Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization. ICML, 2019.
Chaithanya Kumar Mummadi, Tim Genewein, Dan Zhang, Thomas Brox, and Volker Fischer. Group Pruning using a Bounded-I.p norm for Group Gating and Regularization. Ger. Conf. Pattern Recognit., 2019.
Sharan Narang, Eric Undersander, and Gregory Diamos. Block-Sparse Recurrent Neural Networks. arXiv Prepr. ww..vv711.007800 2017.
David Patterson, Joseph Gonzalez, Quoe Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon Emissions and Large Neural Network Training. arXiv Prepr. arXiv2104.103500 2021.
Alexandra Peste, Eugenia lofinova, Adrian Vladu, and Dan Alistarh. AC/DC: Alternating Com- pressed/DeComppressed Training of Deep Neural Networks. arXiv Prepr. arXiv2 12379, 2021.
Evani Radiya-Dixit and Xin Wang. How fine can fine-tuning be? Learning efficient language models. arXiv Prepr, arXiv 2004.14122 2020.
Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement Pruning: Adaptive Sparsity by Fine-Tuning. NeulPS, 2020.
11

---

Under review as a conference paper at ICLR 2022
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gard Gatte Gated Mixture-of offexes Layer. 5th Int. Conf. Learn. Represent. ICLR Conf. Track Proc., 2017.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and Policy Considerations for Deep Learning in NLP. ACL 2019 57th Annu. Meet. Assoc. Compul. Linguist. Proc. Conf., pp. 3645-3650, 2019.
Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, Liwei Wang, and Jason Lee. Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot. NeurIPS, 2020.
Hidenori Tanaka, Daniel Kunin, Daniel L.. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. NeurlPS, 2020.
Kale-ab Tessera, Sara Hooker, and Benjamin Rosman. Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization. arXiv Prepr. arXivv102.01611 2021.
Tulia Turc, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-Read Suudents Learn Better: On the Importance of Pre-training Compact Models. arXiv Prepr. arXiv 19tp:/90....0 089622 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Transformer: Attention is all you need. Adv. Neural Inf. Process. Syst. 30, pp. 5998 6008, 2017.
Dharma Teja Vooturi, Girish Varma, and Kishore Kothapalli. Ramanujan Bipartite Graph Products for Efficient Block Sparse Neural Networks. arXiv Prepr. arXiv2006. 13486, 2020.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv Prepr. arXiv2002.07376, 2020.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured Pruning of Large Language Models. Assoc. Comput. Linguist., pp. 6151-6162, 2019.
Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, and Hai Li. Learning Intrinsic Sparse Structures within Long Short- Term Memory. 6th Int. Conf. Learn. Represent. ICLR Conf. Track Proc.. 2017.
Huanrui Yang, Wei Wen, and Hai Li. DeepHoyer: Learning Sparser Neural Network with Differen- tiable Scale-Invariant Sparsity Measures. ICLR, 2019.
Matthew Shunshi Zhang and Bradly C Stadie. One Shot Pruning of Recurrent Neural Networks by Jacobian spectrum evaluation. ICLR, 2020.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch. ICLR, 2021.
12

---

Under review as a conference paper at ICLR 2022
A
TECHNICAL DETAILS
Optimizer:
Throughout this work we use element wise optimization based on Adam with and gradient
weight decay 0.01, 31 0.9, 82 0.999, 10-6 x loss-scaling- clipping, which is known work well with the sparse gradients found in NLP models.
Default learning rate schedule consists of 10000 linear warmup steps up to the maximum learning rate (0.0002 for BERT-Medium and 0.0001 for BERT-Base), followed by linear decay over the full training run.
Default dropout is 0.1 for all models larger then BERT-Small. To avoid artificial perfor- mance gains through an adjustment of the regularizer in the presence of sparsity induced regularization (Bartoldson et al. 2019), we keep dropout in the sparse models identical to the one used in the corresponding baseline.
Default floating-point precision: We use datatype FP16.16 16 (16 bit compute with 16 bit partials) throughout the model. The second-order moment in the Adam optimizer is com- puted and stored in FP32. Embedding is kept in FP16. The default loss-scaling factor for both BERT-Medium and BERT-Base 512.
The sparsity pattern is initialized randomly. The weights are initial- ized using a truncated normal initializer with initialization range of Ward 0.02. This choice was motivated by having compared different initializations for the sparse model and found that the dense default truncated normal gives the best task performance, shown in Figure 9 We found that preserving the variance of the activation statistics of the sparse model compared to the dense model (Evci al. 2020) does not lead to any performance gains.
Initialization scheme:
2.85
Init-type
gloret
truncated normal
Figure BERT-Medium with static unstruc- tured sparsity 0.9 imposed on all weights using Glorot (blue) or truncated normal (or- ange) initialization scheme. The marker shape indicates whether the standard deviation of the weight initializiation was increased.
2.80
rescale
MLM loss
False
2.75
True
2.70
2.65
learning rate (r)
Pre-training dataset: Phase pre-training is performed on Wikipedia and BookCorpus using Whole Word Masking with sequence length of 128.
Code for DynSparse library used in this work is available Graphcore's
Github
https:://qtthui com httpp//reereeeeeppess/e///////ee http://tteeeeeeee/////e
applicaatioosttesoffooooo/yaaarp http://ydydymmmmmssss.t.yom sparsity
A.1 HYPERPARAMETERS SPECIFIC TO DYNAMIC SPARSITY (DYNSPARSE)
Two very important hyper-parameten for DynSparse training are the sparsity pattern update frequency, ie. how often the network topology is modified, and the pruning ratio, which determines the fraction of the network topology modified at each update step. The sparsity ratio per layer is kept fixed throughout training.
Update frequency dependence: Comparing the task performance of 20, 40, 80, 160 and 320 updates at sparsity ratio 0.9, we have found that the task performance improves with the number of sparsity pattern updates (Table A.2 and A.4). We chose the optimal number of updates as n = 160 (n = 80) for sparsity ratio 0.9 and block size 1x (16x16).
Extreme sparsity regime: All hyperparameters have been optimized for sparsity 0.9. However, tested how well the pruning ratio and update frequency dependence optimized
13

---

Under review as a conference paper at ICLR 2022
Table A.2: Number of sparsity pattern up- dates dependence of unstructured 11) DynSparse BERT-Medium, 7 = 0.001397, sparsity s 0.9, 10 epochs, phase (pruning ratio = 0.5 with cosine decay and random reallocation).
Table A.3: Pruning ratio p, dependence of un- structured (1x1) DynSparse BERTT-Medium = 0.001397, sparsity & = 0.9, 10 epochs, phase I (number of updates n 160 with
cosine decay and random reallocation). Same
hyperparameters in Table A.2
MLM loss
NSP
Pr
MLM loss
NSP loss
40
2.468
0.645
0.25
2,439
0.655
80
2.430
0,656
0.50
2,413
0,684
160
2.409
0.626
0.75
2.411
0.668
320
2.419
0.649
1.00
2.459
0.698
Table A.4: Number of sparsity pattern up- dates n dependence of structured (16x 16) DynSparse BERT-Medium 7 0.001397, sparsity s = 0.9, 10 epochs, phase (pruning ratio Pr 0.5 with cosine decay and random reallocation).
Table A.5: Pruning ratio Pr dependence structured (16x 16) DynSparse BERT- Medium, 7 0.001397, sparsity epochs, phase I (number of updates n 160 with cosine decay and random reallocation).
0.9, 10
Same hyperparameters as in Table A.4
π
MLM loss
NSP loss
Pr
MLM loss
NSP loss
2.616
0.731
0.25
2.648
0.694
2.606
0.650
0.50
2.633
0.692
160
2.633
0.692
0.75
2.634
0.745
320
2.645
0.693
1.00
2.675
0.701
Table A.6: Pruning ratio pr and number of update n dependence of unstructured (1x 1) DynSparse BERT-Medium, 17 0.001837, sparsity 0.99, 10 epochs, phase (with cosine decay and random reallocation).
Pr
MLM loss
NSP loss
0.99
160
0.10
2.999
0.833
0.99
160
0.25
2.939
0.789
0.99
160
0.50
2.889
0.750
0.99
160
0.75
2.872
0.775
0.99
0.50
2.922
0.842
0.99
160
0.50
2,889
0.750
0.99
320
0.50
2,868
0.772
0.99
640
0.50
2.886
0.791
14

---

Under review as a conference paper at ICLR 2022
NLM lacs
nt 100
0.5
2.0
2.44
0.25
0.75
Figure 10: MLM loss vs pruning ratio Pr times number of sparsity pattern updates n for un- structured DynSparse training of BERT-Medium with sparsity ratio 0.9 for different values of (Top panel) pruning ratio pr (with 160) and (Bottom panel) sparsity pattern updates (with
2.42
40
60
80
100
120
140
160
MLM loss
P. 0.5
80
320
2.450
160
Pr 0.5). Same data as in Figure 5
2.425
Pr
Accumulated mask (rigl, sparsity=O.s, boom_up_2)
1000
1500
1.0
100
0.8
covered DDF
300
0.6
500
0.5 static
static
Accumulated mask (random, sparsity=0 9. boom, up_2)
0.4
0.5 rigl
500
1000
1500
2000
0.9 rigl
0.2
0.9 random
200
200000
400000
600000
800000
400
Iterations
Figure 11: (Leff panel) Fraction of explored degrees of freedom for static sparsity and unstructured DynSparse training using gradient based (RigL. (Evei al., 2019) vs random re-allocation (Dettmers & Zettlemoyer| 2019). (Right panel) Corresponding sparsity patterns for the first up-projection in the feedfoward component ("Boom-up") of the second transformer block, accumulated throughout training, for sparsity ratio 0.9 using gradient based (RigL) and random based reallocation. A black (white) dot corresponds to a parameter being non- zero (zero) at any point during training. The dark horizontal blocks in the Rigl. updates indicate collapse due outliers along the input dimension, which indicates that the effect arises from the activation part of the dense gradient update. This suggests that the collapse could be mitigated by reducing the influence of the activations during DpnSparse training update.
for sparsity 0.9 translates to sparsity 0.99, We have found that increasing the pruning ratio to pr 0.75 can lead to small performance gains, shown in Table A.6
Total number of pruned parameters The pruning ratio pr and the number of updates n jointly control the total number of pruned and re-allocated parameters. The total number of pruned and re-allocated parameters is proportional to their product. We obtain an optimal value of their product in terms of task performance as shown in Figure 10
Re-allocation criteria: We found that random re-allocation outperforms gradient based re- allocation. While the pruning criterion leads to a compression of the network topology, the growing criterion directs the evolution of the network topology and distinguishes DynSparse training as a form of neural architecture search during training from mere gradual pruning approaches. Understanding the requirements for efficient joint subspace exploration of parameter and network topology space using DynSparse training will be essential to scale towards larger language models. In Figure II] we show that for gradient-bart the dense gradient is dominated by outliers in the activation, c.g., along the input dimension of each layer, which imposes a strong bias on the available degrees of freedom during the update step. In agreement with this observation, a significantly larger fraction of the network parameters is explored during training, while for gradient-based re-allocation the training remains constrained into small subset of all network parameters (left panel of Figure
re- e-allocation,
find that for random-based re-allocation,
Block size metric The importance of blocks of parameters is assessed by evaluating the normnorm of the corresponding blocks (see Table A.7 and Figure 7).
15

---

Under review as a conference paper at ICLR 2022
Table A.8: Task performance of DynSparse BERT-Medium with sparsity 0.9 for various block sizes B compared dense BERT-Mini with similar number of FLOPs and linear in- terpolation
Table A.7: Task performance of DynSparse training of BERT-Mediimd with sparsity 0.9 for block size = 16 for various block size metrics.
baseline values ("Matched") with exaclty the same number of FLOPs. Hy- perparameters are not specifically tuned for different block sizes. See also BERT-Base
results in Table
Block metric
MLM loss
NSP loss
T,2 norm 2.611
0.684
Model
B
MLM
FLOPs
-,2-nom
2.623
0.684
-n-morm
2.627
0.664
Mini
2.614
2.617
10
Matched (dense)
2.603
2.738
109
m -norm
2.632
0.686
Medium 0.9)
16
2.621
2.738
109
L L -nom
2.635
0.720
- -nomm
2.637
0.670
Medium (s = 0.9)
2.591
2.738
109
Medium 0.9)
2.546
2.738
109
L'omor 2.603
0.665
Medium (s 0.9)
2.408
2.738
109
L'-m
2.606
0.650
L' L'-m
2.615
0.731
Table A.9: MLM validation loss of BERT-Small for results given in Figure
type
MLM loss NSP loss
0,25
zero
0.000343
2.390
0.653
0.50
zero
0.000589
2.485
0.687
0.75
0.001011
2.637
0.737
0.90
zero
0,001397
2.829
0.802
0.99
7ero
0.001697
3.244
0.907
0.25
untrained
0.000686
2.375
0.681
0.50
untrained
0.001178
2.491
0.675
0.75
untrained
0.002021
2.645
0.731
0.90
untrained
0.002795
2.850
0.829
0.99
untrained
0.003394
3.243
0.827
Block size dependence The block size dependence of BERT-Medium with sparsity 0.9 is given in Table A.8
Untrainable vs zero-valued parameters Numerical values for the results shown in the left panel of Figure are given in Table (A.9.
A.2 SPARSE FLOPS: FLOPS ESTIMATES FOR SPARSE MULTIPLICATION WITH DENSE INPUT
Throughout this report we assume the FLOPs for training dense layer with sparse weight elements to approximately scale as O(3 x 2 x B x O x f), where B the batch dimension, I is the input dimension, O the output dimension and f is the density of sparsity pattern imposed the corresponding dense layer, which is to the sparsity ratio as = 1 - s. The FLOPs estimate can be divided into the following components:
1. FLOPs estimate for sparse forward pass: Assuming sparse matrix M has sparsity ratio or density f = 8, the required matrix multiplication for given dense input and output is
Σ Miy
Ybi
.IM 40
where M has dimension [O, I] and dim(y) [B. O). dim(z) ) [B,
(a) Sparse Multiplication: performing the summation 261j Msy for i, iff M, * gives us a reduction of the total number of FLOPs by a fraction non-zero elements in M times B leading to B O X FLOPs.
16

---

Under review as a conference paper at ICLR 2022
(b) Sparse Addition: performing Σ, 2by requires us to calculate the exact number of non-zeros along the input dimension, giving B O x prob(out) x I prob(in) B X O X prob(out), where we defined some probability for non-zero values along the output dimension pro((out) and input dimension prob(in). Assuming uniform distribution, we estimate the FLOPs count to scale approximately linearly with the sparsity ratio O x I f B x O f/prob(in)
first order.
The total FLOPs of sparse multiplication used in the forward pass scales approximately linearly in the number of non-zeros, i.e. O(21 x B x O x f).
2. FLOPs estimate for recursive propagation of error through the network: Involves a multiplication of the dense error with the transposed sparse matrix leading O(2IxBxOx[) additional FLOPs.
3. FLOPs estimates for the outer product The weight update itself is formed by a sparse outer product, where only the sparse components need to be updated, which leads to further reduction in the number of FL.OPs that scales linearly with the density of the matrix.
A.3 RELATING IMPROVEMENTS IN FLOPs EFFICIENCY TO IMPLEMENTATION REQUIREMENTS
To relate the algorithmic improvement FLOPSs efficiency to implementation requirements in general hardware agnostic way, consider the sparse extra cost that we define as
At Apprsse
(5)
A Adnnse
where A (i sparse, dense) is the average time it takes to execute FLOP of type i for specific model size and sparsity ratio. For a given fixed number of training steps and the same task performance, DynSparse training with theoretical FLOPs /esaan (defined in Appendix (..endiii..))aaa is only faster than dense training with FLOPs pedems if the time to execute a sparse training step paprre is smaller than the time to execute a dense training step geesee Or formally:
teporse
<
tlense
Esparse
sparse
Fdense Adense
In other words, the utilization of fewer but "slower" FLOPs in the sparse model still translates to faster execution of the sparse model overall. Note that this comparison is performed at equal task performance and for the same number of training steps.
Using this formalism, we can view improvements in task performance in the context of throughput re- quirements for given algorithm in hardware agnostic way independent of the exact implementation. For tsparse- = fdense we can derive the maximum critical extra cost for DynSparse implementation before DynSparse training loses its advantages over dense computation in terms of time-to-train win. Specifically, for given fixed number of training step and same task performance, the critical cost factor is given by
Ccritical
Flense
pesparse
where sparre corresponds the DynSparse training and pdanse to the (interpolated) dense BERT family. We emphasize, that besides this requirement for a time-to-train win sparse training also allows run models with larger model dimensions for given number of parameters.
A.4 LEARNING RATE FOR SPARSE AND DENSE MODELS
The results of the learning rate sweep of BERT with various sparsities given in Table The corresponding learning rate sweep for the dense BERT-family is given in Table A.II] We confirmed that the optimal learning rates for static sparsity agree with the ones for DynSparse in Table We confirmed that the predicted learning rate dependence of the DynSparse model generalizes to block sparsity and across multiple model sizes given in Table A.12 for block sparsity 16x16.
A.4.1 LEARNING RATE FOR SPARSE MODELS
In Figure 12 we show the learning rate sweep of the BERT-Medium model with static sparsity, for various sparsity ratios. We estimate the optimal learning rate for sparse models through the minimum of cubie interpolation of the task performance vs learning rates for given sparsity ratio, indicated
17

---

Under review as a conference paper at ICLR 2022
Table A.10: Learning rate (17) sweep of static unstructured sparsity BERT-Medium, sparsity 8 = 0, 0.25, 0.5, 0.75, 0.9.
Table A.1I: Learning rate (17) sweep for dense BERT-family consisting of BERT-Tmy, Mini, Small, Medium and Base.
?
sparsity
MLM loss
NSP loss
0.0001
0.00
2.179
0.610
0.0002
0.00
2.115
0.598
model
MLM loss
NSP loss
0.0002
0.00
2.115
0.605
Mini 0.000050
3.062
0.839
0.0004
0.00
2.116
0.606
Mini
0.000100
2.833
0.811
0.0008
0.00
2.164
0.633
Mini
0.000400
2.625
0,742
0.0001
0.25
2.278
0.627
Mini
0.000800
2.606
0,775
0.0002
0.25
2,204
0.642
Mini
0.001600
2.628
0.779
0.0004
0,25
2.186
0.596
Mini
0.003200
2.665
0,783
0.0008
0.25
2.223
0.638
Small
0.000800
2.326
0.644
0.0001
0.50
2.412
0.679
Small
0.000400
2.310
0.621
0.0002
0.50
2.338
0.671
Small
0.000200
2.329
0,635
0,0004
0.50
2.283
0.631
Small
0.001600
2.418
0,768
0.0008
0.50
2.298
0.648
Medium
0.000200
2.115
0.605
0,0002
0.75
2.551
0.741
Medium
0,000400
2.116
0,606
0.0004
0.75
2,483
0.685
Medium
0.000800
2.164
0.633
0.0008
0.75
2.446
0.671
Medium
0.000100
2.179
0.610
0.0016
0.75
2.449
0.647
Base
0.000025
2.115
0,599
0.0032
0.75
2.547
0.707
Base
0.000100
1.878
0,542
0.0004
0.90
2.723
0.758
Base
0.000050
1.972
0.569
0,0008
0.90
2.677
0.711
Basc
0.000200
1.843
0.488
0.0016
0.90
2,648
0.706
0.0032
0.90
2.669
0.697
3.0
MLM loss
Figure 12: MLM validation loss for the Top panel dense BERT family Bottom panel static BERT of different sparsity ratios between 0 and 0.9 as function of learning rate. The solid line correspond cubic fit for all data with the same sparsity ratio.
miri
2.5
small
2.0
10
10-3
The minimum of the resulting fit corresponds to the optimal learning rate for given sparsity and is indicated by the black triangles connected by blue
NLM loss
2.8
0.25
0.5
lines.
5.9
10
10
learning rate (n)
18

---

Under review as a conference paper at ICLR 2022
Table A.13: Learning rate sweep BERT-Small alternating between dense and sparse training with either non-trainable parameter or zero-valued pa- rameter corresponding to sparsity
0.9, 10 epochs phase I, for various pruning methods. Op- timal values are given Table A.16 We switch = 160 times between sparse/non-traindae pa- rameters and the dense training.
Table A.12: Learning rate (1) sweep for DynSparse BERT-family BERT-Mini, Small, Medium and Base for sparsity 0.9 with block size 16x 16.
active
pruning
"
MLM
NSP
non-train
fixed
0.0002
2.366
0.671
non-train
fixed
0.0004
2.358
0.668
model
MLL
NSP
non-train
fixed
0.0008
7.242
0.693
non-train
magnitude
0.0002
2.379
0.658
Base 0.000160
2.520
0.692
non-train
magnitude
0.0004
2.354
0.675
Base
0.000320
2.429
0,693
Base
0.000640
2.340
0.647
non-train
magnitude
0.0008
11.160
0.766
Base
0.001
2.328
0.603
non-train
random
0.0001
2,431
0.733
Base
0.002560
2.369
0.656
non-train
random
0.0002
2.365
0.669
Medium
0.000125
2.878
0.892
non-train
random
0.0004
2.349
0.693
non-train
random
0.0008
7.272
0.693
Medium
0,000250
2.760
0.720
Medium
0.000500
2.670
0.730
fixed
2.5c-05
3.317
0.967
Medium
0.002000
2.640
0.715
fixed
5e-05
3.199
0.817
zero fixed
0.0001
3.277
0.819
Mini 0.001250
3.184
0.882
zero fixed
0.0002
3.329
0.884
Mini
0.002500
3.145
0.871
Mini
0.005000
3.147
0.869
zero
fixed
0.0004
3.358
0.964
fixed
0.0008
3.424
0.799
Mini
0,005120
3.195
0.907
Small
0.000313
2.927
0.865
7ero
magnitude
0.0002
2.746
0.756
magnitude 0.0004
2.685
0.711
Small
0.000625
2.841
0.773
Small
0.001250
2.788
0.861
zero
magnitude
0.0008
3.056
0.834
Small
0.005000
2.826
0.797
zero
magnitude
0.0016
6,538
1.217
zero
random
0.0001
6.232
1.142
zero
random
0.0002
6.132
1.273
zero
random
0.0004
6.094
1.185
random
0.0008
6.284
0.987
Table A.14: Learning rate sweep DynSparse BERT-Medium, sparsity s = 0.9, 10 epochs phase I, used to confirm that the optimal learning rates for static sparsity from Table A.10 A.1 translate into optimal learning rates for DynSparse.
MLM loss
NSP loss
0.00064
2,467
0.647
0.00128 28
2,410
0.670
0.0026
2.429
0,674
0.0051
2.521
0.654
19

---

Under review as a conference paper at ICLR 2022
dense sparse base
Mini
logfleaming rate)
Nedium
medium
small
-7.
MLM
0.0
2.6
0.5
-8.5
0.9
2.4
16.5
18.0
20-1 learning rate (n)
logtnumber parameters)
Figure 13: (Left panel) Dense fit the optimal learning rate estimated as the position of the black triangles from Figure 12 for BERT-Medium with various sparsities and the dense BERT-family as a function of the number trainable parameters N for various model sizes (indicated by symbol style and color) and sparsity ratios (colored crosses). The black lines indicate linear fits that are best approximated by log(n) = -0.8883 (o) [[]++ 6.13(+0.7) for the sparse models and log(n) -0, 44(50.05)) log(N) 0.47(=0.9) for the dense models. (Right panel) Testing the prediction of the optimal sparse learning rate from Eq. 3 (markerstyle "+") on BERT-family with sparsity 0.9 and block size 16x 16 (values given in Table A.12
Table A.16: MLM validation loss of BERT- Small trained by alternating between dense and training only a fraction of 0.1 of the non- embedding weights with the non-trainable parameter set to either zero or just untrain- able without modifications, We pick the best learning rate for each experiment using grid search over 2.5- 100.5, 2n with m 0, 1,2,... (Table A.13 (number of updates 160).
Table A.15: Learning rate sweep of DynSparse BERT-Smal unfreeze (freeze) ex- periment with initial (final) fraction of non- trainable parameters 0.9, 10 epochs phase I.
type
7
MLM loss
NSP loss
freeze
0,000087
2.467
0.715
non-train
selection
MLM
freeze
0.000175
2.407
0.703
freeze
0.000349
2.420
0.685
5c-05
zero
fixed
3.199
freeze
0.000699
2.540
0.695
0.0004
zero
magnitude
2.685
0.0004
zero random 6.094
unfreeze
0.000175
2.933
0.666
unfreeze
0.000349
2.598
0.676
0.0004
untrained untrained untrained
fixed 2.358
unfrecze
0.000699
2.440
0.703
0.0004
magnitude
2.354
unfreeze 0.001397
7,251
0.693
0.0004
random
2.349
unfreeze
0.002795
7.520
0.784
by the triangle markers in Figure 12 interpolation is best approximated by
We find that the optimal learning rate 1) calculated from the
log(n)(s)) L 0.220) 8.1.04)
(8)
as function of sparsity equivalently as (see Figure 12
log())(N)) = (0..338 + 6.13(+
(9)
for number of parameters N. Interestingly, a linear learning rate vs logarithmic memory fit as used in Kaplan al. (2020) (n) 0.003239 0.0001395 log(N) from Eq. (D1)) is leading to qualitatively worse agreement, which might be explained by our optimization for a fixed number of training steps.
A.5 ROLE OF SELECTION CRITERIA
To understand the role of magnitude pruning criteria in the DynSparse training dynamics, have disentangled the pruning step from the parameter re-allocation step by temporarily replacing the always sparse training algorithm with alteraation between dense and sparse training phases (Peste
20

---

Under review as a conference paper at ICLR 2022
et al. 2021). The dense training interval removes the influence from the regrowing selection all network parameters are periodically activated without preference. We have found that magnitude pruning ("magnitude") outperforms both pruning into fixed subspace chosen randomly at initializa- tion ("fixed") and changing random subspace re-drawn each time ("random") as shown in the top half of Table A6 The strong performance degradation of the random re-drawn sparsity patterns illustrates the importance of the large network parameter in preserving the task performance.
This picture changes if, instead of setting the parameter zero, make parameters non-trainable, which avoids the information loss associated with the pruning step. In fact, in this case, we find that randomly selecting the subset of trainable parameters outperforms both the selection of the parameters with the largest magnitude ("magnitude") as well as training fixed subset of parameters randomly chosen at initialization ("fixed") shown in the bottom part of Table A.16 Our results show that magnitude pruning gives performance advantages increased exploration coming from random parameter selection would, in the absence of information loss due to pruning, benefit the task performance were it not associated with information loss.
it preserves information. However, the
21


Reviews:
[{"review_id": "wxPBYGcj3tR", "reviewer": "Reviewer_idrP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Sparse training for neural networks, especially very large transformer-based architectures, is a very  interesting and timely topic when discussing model efficiency. However, the presented paper does not provide much novelty or method advancement. The authors simply apply an already existing method to one type of network. While parameter-studies in general can be of interest, yielding new insights into the training procedure, my main concern is that the studies presented in this paper do not follow a  clear research question, but seem rather random. More importantly, no generalized “lessons-learned” from these studies are drawn, and the translation towards other network architectures is not discussed. As such, the paper lacks significant scientific impact.\n\nSome further comments:\n- It is not explained, why the algorithm is adapted to a block-sparse structure. What is the benefit of this, why is it needed?\n- All results are reported in terms of reduced FLOPS, but actual training time (wall-clock time) is not considered. Especially on hardware specialized for dense/block structures (GPUs), accessing individual elements in a sparse pattern can, while requiring less operations, result in comparable compute times as their dense counterparts. Moreover, sub optimal usage of the device can thus negate the effects on efficiency, proclaimed by the sparse computation\n- In Figure 2, the caption does not match or explain the Figure. How does this Figure illustrate the pruning and re-allocation step? What do the masks and the colours stand for?\n- Equation 3: Where does this come from? This seems very empirical. What is the reasoning behind this?\n- No details on the utilized hardware and training setting are given", "summary_of_the_paper": "The paper studies the feasibility to use the dynamic sparsity (DynSparse) method of continuously pruning and re-allocating network weights during training for large unsupervised (self-supervised) language models, namely BERT. Towards this end, the authors present several parameter studies (w.r.t. hyperparameters) of the original BERT training procedure (Mask Language Modeling and Next Sentence Prediction) to assess the Pareto-curve of training BERT with sparse vs dense. ", "summary_of_the_review": "My main concern with this paper is that it is merely a parameter study of an existing method being applied to an existing architecture. While these kind of studies can still be of high scientific value, they need to provide new insight into the method or the architecture and yield some translation/generalization in terms of \"lessons learned\" for other applications in order to do so. I do not see this to be the case here. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mu4K90r0Nc", "reviewer": "Reviewer_5K4c", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths\n- The paper shows a simple algorithm (DynSparse) that could help find good sparse\nsubnetwork in BERT, reduce FLOPS and achieve comparable MLM validation performance.\n- The authors did intensive experiments on comparing several different design choices\nsuch as the metric to re-allocate weights, prune the weights, etc.\n- The sparse training tool developed is useful for the community.   \n\n\n\nWeakness\n\n- The novelty of this work is a bit limited. There is no new algorithm proposed to get better results for this sparse training of BERT. The discussions on what could be the reasons behind the design choice are not very convincing to me. For example, random re-allocation has more varieties of network topology explored than RigL, leading to better performance. If this understanding is correct, why would Figure 5 (a) have a convex curve?\n\n\n- Evaluation of downstream tasks. \nMaybe I haven't done a good review of this field, but I am skeptical that MLM validation loss is a good indicator of the quality of pre-trained BERT. Could the authors find references supporting this metric? What if we fine-tune the DynSparse BERT and BERT Large under the similar resource constraint and compare their performance?\n\n- Concern on the scale. \nI am not sure if the conclusions still hold when we \nscale to larger models. \n\"DynSparse training can achieve more efficient utilization of FLOPs or network parameters at any scale.\" It seems a overclaim to me as the paper didn't show the comparison with BERT large. \nIt would be helpful to label the point of BERT large and its DynSparse counterpart in Figure 3.\n\n- Missing comparison with baseline. \nThe conclusion from EarlyBert paper is that lottery ticket converges fairly fast and the subnetwork is identified during the early stage of training. This seems to conflict\nwith the findings in this work that network topology has to be updated sufficiently\nto get good results. Could the author elaborate more on this point?\nRegardless of always-sparse constraints, how does the quality of subnetwork found\nby DynSparse compared to EarlyBERT, in terms of performance in downstream tasks.\n", "summary_of_the_paper": "This paper did an extensive investigation of applying dynamic sparse training for BERT. \nThe main contributions of this work are several empirical observations:\n1. The authors empirically show that random re-allocation of weights is better than\ngradient-based approach (RigL) for unstructured DynSparse training of Bert. \n2. The authors investigated structured DynSparse training and found L1 norm is the\nbest metric to prune the weights.\n3. For a specific range of size of BERT, the DynSparse training shows Pareto improvement (MLM validation loss vs. Non-embedding FLOPS) over the dense BERT. ", "summary_of_the_review": "This work did a comprehensive empirical evaluation on applying DynSpare to BERT,\nin both structured and unstructured settings. The novelty of algorithm and theoretical understanding is limited. The empirical results could be more thoroughly conducted, e.g., the scale of BERT, downstream task evaluation. \nAs an initial step of exploring sparse training of BERT, it provides some promising results. The tool this work developed could benefit the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_yGVtkLaZ7X", "reviewer": "Reviewer_KM7E", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe motivation of using sparse training for BERT language modeling is clear and persuasive.\nExtensive experiments and technique details are provided to support claims and demonstrate the effectiveness of the proposed method. \n\nWeaknesses:\nIt will be better if the author provides more details on the difference between zero and untrained in computation or the time and space complexity analysis of these two methods.\nThe author mentioned in the part of the contribution that the structured DynSparse training of BERT without structured regularization gives performance gains compared to dense BERT baseline. Structured regularization is used in the block-sparse DynSparse algorithm. Are these two in contradict with each other?\nIt will be better if the author gives more description of the techniques in the contribution in the part of the methodology.\nThe author mentioned that the collapse could be mitigated by reducing the influence of activations during training updates. It will help if the author provides more details about how the activations are reduced and how many are reduced compared to the dense baseline.\n", "summary_of_the_paper": "Overall, this paper proposed a dynamic sparse pre-training method for BERT language modeling which showed better performance in the number of FLOPs when compared to statically sparse and dense models across a large scale of network sizes. \n\nTo be more specific, the proposed DynSparse has four contributions: 1) using random parameter re-allocation to alleviate network parameters collapse; 2) is scalable and FLOPs efficient; 3) showed effectiveness on structured training of BERT 4) structured regularization free while achieving Pareto improvements.", "summary_of_the_review": "Extensive experiments are conducted to demonstrate the effectiveness of the proposed method. However, I am not fully persuaded by the methodology. More details like complexity analysis and latency reduction experiments could be given.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YqygIYdFdsG", "reviewer": "Reviewer_vEBH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose to dynamically prune the transformer to improve the efficiency of BERT pre-training. To maintain the computational regularization, block sparsity is used in this work. The main technique of this paper is to add group LASSO regularization to induced sparsity and prune the network depending on the magnitude of the weights. The proposed method is simple yet effective. The resulting sparse pre-training method is nearly twice as efficient as the original BERT pre-training method. However, the novelty of this paper is limited because the proposed sparse pruning technique has been widely used to train convolutional neural networks. Moreover, it remains unclear whether the proposed method is tailored specifically for pre-training of sequence models. My additional concerns are listed below.\n\n1) Comparison with other dynamic approaches: As discussed in this paper, other lines of research exist to explore sparse transformer architectures [1, 2]. However, the proposed method is not compare with these prior works in the paper.\n\n2) Wall-clock evaluation: No wall-clock evaluation is provided. It is not clear whether the proposed method can provide real speedup on accelerator chips such as google TPU.\n\n3) Finetuning results: The authors only show the MLM loss after pre-training as the quality metric. However, it is also important to show the performance of models after fine-tuning on GLUE / SQuAD datasets.\n\n[1] Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping\n[2] Reducing Transformer Depth on Demand with Structured Dropout", "summary_of_the_paper": "The authors propose a dynamic sparse pretraining method to reduce the computational cost of BERT pretraining. ", "summary_of_the_review": "The novelty of the proposed method is limited and the empirical evidence is not strong enough to justify the method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "28G_j2M01Hz", "reviewer": "Reviewer_KoFV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Positives**\n\n* Reducing the training cost for language models like BERT is an important problem.\n\n* This work covers both *structured* and *unstructured* sparse training for BERT. The proposed block-sparse DynSparse can improve hardware utilization.\n\n* This work suggests minimum efficiency requirements for a time-to-train win for DynSparse over the conventional dense training.\n\n\n**Concerns/Questions**\n\n* This work in its current form lacks any quantitative comparison against other DynSparse work.  The authors argue for random re-allocation as a means to increase DOF. However, RigL [Evci et al. 2019] suggests that their gradient-based re-allocation outperforms SET, which also adopts random re-allocation, for both vision and NLP tasks (with RNNs, though). Is there any ground for random re-allocation to work better with BERT, or is it simply anecdotal? How would RigL perform with BERT?\n  \n* There is no evaluation of end-to-end time (\"time-to-train\"). Since the authors take into account the utilization in real accelerators by introducing block sparsity, I am wondering how much the proposed technique work reduce the end-to-end training time compared to the baseline dense training.\n  \n* Along the line of the previous one, the actual performance gains over dense training may be limited on commodity GPUs. According to Nvidia, cuSparse manages to deliver just about 50% of dense GEMM when using 32x32 blocks [R1]. Depending on the sparsity, the model may fall short of the minimum efficiency requirement of block sparse training to outperform dense training for BERT.\n\n* Evaluation seems a bit too narrow to convince the generality of the proposed technique due to the usage of just one model, fixed sequence length, and lack of ablation study and sensitivity analysis. Can the proposed technique be effective to other models than BERT?  The technique seems quite generic and I am wondering how effective it would be to other models. Is there anything specific to BERT?\n\n[R1] https://developer.nvidia.com/blog/accelerating-matrix-multiplication-with-block-sparse-format-and-nvidia-tensor-cores/\n", "summary_of_the_paper": "This paper advocates a straightforward, dynamic sparse (DynSparse) pre-training approach for BERT for more efficient FLOP utilization while maintaining comparable accuracy.  This paper utilizes random reallocation of weights (instead of gradient-based reallocation) to achieve higher total explored degrees of freedom (DOF). This work also proposes a block-sparse DynSparse scheme for higher effective utilization of FLOPS on real hardware accelerators (like GPUs and IPUs).", "summary_of_the_review": "This paper adapts DynSparse training to BERT to demonstrate its effectiveness. However, evaluation is somewhat narrow and it's not well justified why random re-allocation of weights works better than gradient-based re-allocation for BERT. Also, the performance gains in terms of end-to-end training time on the commodity accelerator like GPUs are still questionable.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All of the reviewers believe the paper should not be accepted, and I concur with their recommendation for the reasons they mention.

Four of the reviewers (vEBH, idrP, KoFV, 5k4c) believe the technique proposed in this paper is not particularly novel. Rather, the novelty is that it is being used on a BERT model rather than the computer vision models that are typically the starting point for pruning work. They also argue that the paper was not particularly thorough in its comparison to other pruning techniques (specifically dynamic sparsity techniques), which is essential for pruning work given how crowded and noisy the space is. Finally, they rightfully note that the paper does not look at the real-world speedups attainable on conventional hardware (GPUs and TPUs), the latter of which has no support for sparsity and the former of which (NVIDIA Ampere) has limited support for specific kinds of sparsity and especially limited support for sparse training.

The reviewers also raised several more specific methodological issues with evaluation (e.g., using the MLM loss rather than fine-tuning as a basis for evaluation), but the above concerns alone were enough to convince me that the paper does not merit acceptance at this time.
Decision: Reject