CrAM: A Compression-Aware Minimizer
Authors: Alexandra Peste
Adrian Vladu
Eldar Kurtic
Christoph H Lampert
Dan Alistarh
Venue: ICLR 2023 poster (2023)
Keywords: compression
sparsity
one shot pruning
optimization

Abstract: Deep neural networks (DNNs) often have to be compressed, via pruning and/or quantization, before they can be deployed in practical settings. In this work we propose a new compression-aware minimizer dubbed CrAM that modifies the optimization step in a principled way, in order to produce models whose local loss behavior is stable under compression operations such as pruning. Thus, dense models trained via CrAM should be compressible post-training, in a single step, without significant accuracy loss. Experimental results on standard benchmarks, such as residual networks for ImageNet classification and BERT models for language modelling, show that CrAM produces dense models that can be more accurate than the standard SGD/Adam-based baselines, but which are stable under weight pruning: specifically, we can prune models in one-shot to 70-80% sparsity with almost no accuracy loss, and to 90% with reasonable (∼ 1%) accuracy loss, which is competitive with gradual compression methods. Additionally, CrAM can produce sparse models which perform well for transfer learning, and it also works for semi-structured 2:4 pruning patterns supported by GPU hardware. The code for reproducing the results is available at: https://github.com/IST-DASLab/CrAM .


Published as a conference paper at ICLR 2023
CRAM: A COMPRESSION-AWARY MINIMIZER
Alexandra Peste' Adrian Vladu
Eldar Kurtic
Christoph H. Lampert' Dan Alistarh 1.3 CNRS & IRIF *Neural Magic, Inc.
Institute of Science and Technology Austria (ISTA)
ABSTRACT
Deep neural networks (DNNs) often have to be compressed, via pruning and/or quantization, before they can be deployed in practical settings. In this work we propose a new compression-aware minimizer dubbed CrAM that modifies the optimization step in a principled way, in order to produce models whose local loss behavior is stable under compression operations such as pruning. Thus, dense mod- els trained via CrAM should be compressible post-training, in a single step, without significant accuracy loss. Experimental results on standard benchmarks, such as residual networks for ImageNet classification and BERT models for language mod- elling, show that CrAM produces dense models that can be more accurate than the standard BGD/Adam-bas specifically, we can prune models in one-shot to 70-80% sparsity with almost no ac- curacy loss, and to 90% with reasonable (~ 1%) accuracy loss, which is competitive with gradual compression methods. Additionally, CrAM can produce sparse mod- els which perform well for transfer learning, and it also works for semi-structured 2:4 pruning patterns supported by GPU hardware. The code for reproducing the CrAM.
baselines, but which are stable under weight pruning:
results is available at: https: httpp://githhie //ghthut.....//.. com/
INTRODUCTION
The massive recent progress of deep learning models has been accompanied by an increase in compu- tational costs (Thompson et al., 2020). In turn, this has led to significant interest in model compression techniques order to reduce these costs. For many existing models, compression techniques such as distillation (Hinton et al., 2015), pruning (Hoefler et al., 2021) and quantization (Gholami et al., 2021) can usually reduce the number of parameters or FI.OPs of a given model by up to an order of magnitude with relatively little accuracy loss. However, performant compression still usually requires re- re-training or sie-ttning the model separately for each compression target, provided by the user as a target sparsity and/or quantization level. In turn, this compression process can be cumbersome and error-prone, as it requires additional computation and hyper-paramett tuning for each run.
In this work, we propose Compression-Awarr Minimization (CrAM), a method for training neural networks. which results in models that easily compressible one-shot, while still being highly- accurate. Specifically, CrAM enables training a single (dense) model, which can later be compressed to different target levels, with minimal or recalibration. Such flexibility is desirable, as models can be trained once, and then deployed on multiple devices, with different specifications. Having a single model that can be easily configured to meet the computational requirements of specific device can both reduce the overall computational cost, and also allow easier customization to individual devices.
CrAM is lonsely-inspired by the eccently-inrrodun 2021), which trains models that potentially converge to flatter minima, leading to better generalization compared to SGD-type baselines, by biasing the process towards minima of uniformly low loss. Multiple subsequent works have investigated and improved upon the original SAM algorithm, by either obtaining better generalization (K won et al., 2021), or by reducing the computational costs of SAM training (Liu et al., 2020; Du et al., 2022a). We the first carry over this idea the task of obtaining compressible models. Roughly speaking, CrAM works by optimizing not against the original "dense" model, but over compression projection applied the intermediate model iterate, at every optimization step. Thus, the CrAM update aims to bias optimization towards iterates that have both low loss and are robust under one-shot compression. Similarly to SAM, CrAM is simple to implement as part of a regular training loop and has a single scaling hyper- hyper-parameter, parameter, for which we
sharpness-
minimizer (SAM) (Foret et al.,
"Correspondence al 1eexanrra waannnra.pestee. pestetliai at..a..aa..

---

Published as a conference paper at ICLR 2023
provide a werl-performing default value. We detail the CrAM algorithm and provide a theoretical motivation leveraging fundamental results in robust optimization (Danskin, 2012) in Section 3.
To complement our algorithmic contribution, we perform an extensive experimental analysis of CrAM. We mainly focus on compression via weight pruning, but we also show that CrAM is compatible with weight quantization. Generally, CrAM models trained on large-scale image classification or language modelling tasks can improve over the dense baseline performance, while being very robust one-shot pruning, at different sparsity levels. For image classification, CrAM can train a highly-accurate dense ResNet50 model on ImageNet, that can be pruned in one-shot to 80% and 90% sparsity, and is competitive in terms of accuracy relative to state-of-the-an gradual pruning methods, following an inexpensive Batch Normalization re-tuning step on small calibration set.
Moreover, we show that full CrAM training is not necessary for good performance: specifically, a short CrAM finetuning period is sufficient to substantially improve mhe- pruning accuracy. For instance, using CrAM to transfer the standard BERT-base model (Devlin et al., 2019) on SQuADv 1.1 juestion-answering (Rajpurkar et al., 2016). obtain models that both more accurate and more compressible than with optimizers such as Adam (Kingma Ba, 2015) or SAM (Foret et al., 2021). In addition, a short (< 2 epochs) finetuning of the sparse model can provide substantial additional improvements: the 80%-sparse CrAM finetuned model reaches higher accuracy than the highly- competitive gradual pruning methods PLATON (Zhang et al., 2022) and Movement Pruning (Sanh et al., 2020), at fraction of the training budget.
CrAM lends itself to several extensions: it can be used with different layer-wise sparsity distributions, semi-structured N:M sparsity patterns, and one-shot pruning techniques. Sparse CrAM models can be successfully used for sparse transfer learning, where they can perform well on a wide range of "downstream" target tasks, even when compared to pruning methods that train a separate model for each sparsity level. We also provide evidence that the CrAM update can produce models that are robust to quantization.
Similar to SAM (Foret et al., 2021), one limitation is the added computational cost, as CrAM requires an additional backwards pass for the model perturbation. This can be addressed by only performing limited finetuning via CrAM instead of full retraining, or by only performing a regular optimization step for fraction of the time, both of which we show to have a limited impact on accuracy. Moreover, our approach is also compatible with efficient SAM-type updates (Liu et al., 2020; Du al., 2022a). variant of CrAM that uses sparse gradients, which could be
We also provide a leveraged by frameworks with support for sparse back- propapropagttion (Nikdan et al., 2023).
RELATED WORK
Sharpness-Aware Minimization (SAM). The recently introduced SAM optimizer (Foret et al., 2021) aims to minimize loss sharpness; this in turn should lead to flatter local minima, with better generalization. The authors show that SAM-trained models have higher validation accuracy compared to vanilla SGD-type baselines, that their performance continues to improve with prolonged training, and that they can also be successfully used for transfer learning. One important drawback of SAM is its computational overhead, as it requires twice as many forward-backwand network. Subsequent work has focused reducing computational cost by, for example, reducing the frequency of the extra gradient steps (Liu et al., 2022), computing the perturbations on a subset of the parameters (Du et al., 2022a), or by proposing new trajectory loss to replace the sharpness definition (Du et al., 2022b). We draw inspiration from properties of the initial SAM method proposed by Foret et al. (2021). Instead of attempting to minimize the maximum local increase loss (sharpness), our goal is to minimize the maximum local increase in loss due to compression.
passes through the
Training prunable networks. The increasing scale of deep neural networks have made their deploy- ment to edge devices dependent on compression techniques, such as quantization and/or pruning. While post- cost-training quantization can be efficient and successful without any retraining (Frantar & Alistarh, 2022), in the case of pruning the gold standard is still training a separate model for every target sparsity level (Zhu & Gupta, 2017; Singh & Alistarh, 2020; Evci et al., 2020; Peste et al., 2021), which can be expensive. A potential solution would be training a single dense model, which either contains multiple smaller ones that can be easily deployed, or which is itself prumable at multiple sparsity levels, without additional retraining. For example, "once-for-all" (OFA) (Cai et al., 2019) can train a large network that contains multiple specialized sub-nets, adapted to different resource constraint devices. However, obtaining the large OFA network is extremely expensive, and requires
2

---

Published as a conference paper at ICLR 2023
intensive finetuning to ensure good performance for the sub-nets. A similar idea has also been explored for automatic speech recognition (Wu et al., 2021). An orthogonal direction is to obtain "slimmable neural networks" (Yu et al., 2019; Yu & Huang, 2019b;a). by training a single model that can be executed at different widths; this usually achieved by performing multiple backpropagations using all the predefined widths, at each optimization step. Furthermore, BigNAS (Yu et al., 2020) inspired by these approaches, and achieves a similar goal to OFA, without additional finetuning. Compared to our method, these works focus on structured pruning, and require extensive finetuning, or more forward-backe backward steps each iteration. Related to one-shot pruning, Only Train Once (OTO) (Chen et al., 2021b) has been proposed for structured pruning, to train a large model that is easily slimmable one-shot. While we obtain better results than OTO for the same sparsity level, the two methods are not directly comparable, since we focus on unstructured sparsity.
Our work is more closely related to Miao et al. (2022); Zimmer et al. (2022), which propose leveraging Stochastic Frank- Wolfe (SFW) (Reddi et al., 2016) to encourage the weights to lie in convex hull spanned by sparse vectors; this would make the model prunable one-shot, without any finetuning. The methods proposed in Miao et al. (2022); Zimmer et al. (2022) result in highly- prunable models on relatively-sin small tasks. Under the same experimental setting, CrAM is able to match or outperform these methods; for instance, CrAM can prune VGG-16 trained on CIFAR-IO in one-shot to 95% sparsity without accuracy loss, outperforming SFW by more than 2% Top-1 accuracy. More importantly, we show that CrAM produces models compressible one-shot at both ImageNet scale and BERT language modeling scale. Remarkably, with one-shot pruning CrAM can offer competitive performance to gradual pruning methods, whether they are designed for CNNs (Kusupati et al., 2020; Lin et al., 2020) or for language modelling (Sanh et al., 2020; Zhang et al., 2022).
3 THE COMPRESSION-WAAR MINIMIZER (CRAM)
3.1 BACKGROUND
We now give an overview of our method, together with the corresponding algorithm and general- izations. CrAM aims to train models that are "compressible" in one-shot, following training, via sparsity or quantization. For this, we consider a compression operator C, such as for example Top-K. which keeps the highest K elements in absolute value of a tensor, and sets the rest to 0. We say that a model is easily compressible if small perturbations do not affect its performance after compression. To enforce this during training, we optimize against the dense model perturbation which has the largest impact the compressed model. We want minimize the "compression-awane (CrAM) loss, defined as:
L CGAM (0) = max L(C + 8)), S|
(I)
where 0 is the vector of parameters, I, the model loss and 5 is norm-hounded perturbation.
We approximate I ((C by taking gradient ascent step in the direction of the current update, followed by projection using the compression operator. This is inspired by the iterative hard thresholding (IHT) algorithm used for optimizing functions under sparse constraints (Blumensath & Davies, 2008; Foucart, 2011: 2012). To obtain the gradient with respect to the parameters, we employ straight-through estimator, and use the gradient evaluated at the perturbation. This gives us the following update for minimizing the CrAM loss at step
+ 1:
0, C(O, + P- (())))
o = O2 - L(,))
(2)
Alongside improving robustness to compression, we want to maintain the quality of the dense model, which cannot be guaranteed by only optimizing the CrAM loss. Therefore, we propose to explicitly optimize for the performance of the dense model. The main objective we use and which fulfills these criteria is the composite CrAM+ loss function, defined as:
I CrAM*
(0) = L()) + CrMM (0)
(3)
This requires simple modification to the CrAM update, at no extra cost, by simply adding the gradient ((,),) before the next update of the parameters 0.41. For O. = C(O. + (O.)).)) the CrAM+ update is the following: 0 = 0, - (VL VL(O)).
We can also add different regularization terms the objective in Equation 3, such as weight decay.
3

---

Published as a conference paper at ICLR 2023
Algorithm 1 Compression-Awan Minimization (CrAM/ CrAM*)
Require: Compression operators C {C1, C2,..., CM training data S, training iterations learning rate
!). perturbation step size p
1: Initialize the weights 8u
2: while S T do
3: Sample batch z E
4: Compute loss zz) and gradient 9, [[(0=;))
5: Uniformly choose compression operator C C
6: Get perturbed weights 0, = C(O, + pg.)
7: if C' = Top-K K then
8:
Let M, the linear projection operator onto the support of the largest coordinates of [0, + pgel= such that 0, (0, + pg.)
9:
0:
M. ((
10:
else
9: <0.:))) end if
12:
13:
if use CrAM then
14:
g. f 9. 9.
15:
end if
16:
Update the weights using gradient descent step: 0.+1 0: -1-9:
17:
end while
return Or
Furthermore, we can consider distribution over a set of compression operators C and approximate the expected CrAM loss under multiple types of compression, by sampling different operator each step. We refer to this version as C
3.2 THEORETICAL JUSTIFICATION OF THE CRAM UPDATE
To derive the CrAM update, and justify the choices made in designing our training method, we start from the optimization objective defined in Equation Although the loss includes an inner maximization problem, together with a potentially problematic compression operator, we show that under mild assumptions we can efficiently approximate a descent direction. We rely on a well-known theorem from robust optimization (Danskin, 2012), which allows one to obtain descent directions for min-max objectives under broad range of assumptions. Using Danskin's theorem (Theorem from Appendix A. we obtain that by computing the maximizer of the inner problem
6" arg max L(C( +))
(5)
[6] SP
and letting & 0 8". which compresses to the extrapolated iterate == C(Φ). we obtain descent direction -C)) Implementing this approach faces two difficulties - first, to compute the gradient we must proopprrpe through the composition of functions L(C(-)), which may cause trouble since C' is not necessarily differentiable; second, and more importantly, it is unclear how solve the inner maximization problem.
To address the first issue, we may choose to use a straight-through gradient estimator (Bengio et al., 2013), which permits us to only backpropagate through L, and use (() instead of the true gradient. To increase precision, in the case where compression performed via Top-K we interpret C as a "mask" operator M which zeroes out subset of coordinates dependent on Φ. Since except for articulation points, M, is constant and does not change the argument varies, we approximate VL((())) & MVL(M)) = MV)(
To address the second issue, rather than exactly maximizing the inner problem, we instead seek good enough maximizer using a standard iterative method. For this, we choose projected gradient ascent, which provides theoretical guarantees, even when the projection is performed onto non-convex domains (Peste al., 2021). For instance, if the compression operator is magnitude pruning, this becomes the iterative hard thresholding (IHT) method, frequently employed in the sparse recovery literature (Blumensath & Davies, 2008). Thus, to reach good iterate within this specific domain, in practice we perform a single step of (projected) gradient ascent, which matches the IHT iteration:
0, C (0, + VT((.))
4

---

Published as a conference paper at ICLR 2023
In Appendix A, provide full re-derivation of the CrAM update in Equation 2 under fairly reasonable assumptions on the objective function, along with a detailed discussion on the necessity of these assumptions; our derivations also hold for CrAM As side result, we also obtain simple re-derivation of the SAM update.
3.3 IMPLEMENTATION DETAILS AND EXTENSIONS
Multiple compression types. CrAM can be used to train models that are robust to multiple types of compression operators, by choosing between multiple compression projections each optimization step. Examples include pruning using different sparsity levels or quantizing at different precisions. We illustrate the general CrAM algorithm which handles multiple compression operators, and includes the explicit optimization of the dense model, in Algorithm I. We found that CrAM* with a different randomly chosen compression at each optimization step typically achieves the best trade-off between an accurate dense model and robustness to multiple one-shot compression schemes post-training; we call this variant CrAM+ -Multi, and will become the main method used in our experiments. When optimizing for robustness against sparse perturbations, we use the Top-K operator at each step, and choose the sparsity level uniformly at random among set of predefined values.
A Addressing the computational overhead. While CrAM requires twice as many forward-backward passes compared to regular training, we can reduce this overhead for TopK-CrA by making use of the sparsity in the intermediate updates. We found that using only the gradients from the support of O2 in V ((,) improves both the resulting dense model obtained with robustness to one-shot pruning. This observation is motivated by the fact that the straight-through (Bengio et al., 2013) gradient estimator with the identity function might be suboptimal (Yin et al., 2019), and better estimators can be defined using different functions. As seen in Section 3.2, we can assume, via Danskin's 's theorem (Danskin, 2012), that we can obtain descent directions for L LCAMM(0.) (0.) by evaluating (L()(,,)) where o, is the extrapolated point Φ = 0₂ + ((,) To evaluate the gradient, we may use trraighttthrouge estimator. For Top-K, is as an operator Mr which zeroes out a subset of coordinates dependent on Φ,. Provided that M, is constant and does not change as the argument varies, we can approximate V (I(((),)) ~ M, @ (L(M,,))) As both the iterate and gradient estimator are sparse, this implies a theoretical speed-up. We further show in Appendix C.6 results obtained from CrAM using infrequent mask updates.
as well as its
Alternative Updates. We note that alternative compression-awee updates can be derived. One such derivation, which we call Compressed-SAM (C-SAM), uses the gradient of the compressed model to perturb the dense model. While training with C-SAM can also result in models robust to one-shot pruning, we noticed that typically the accuracy of the resulting dense models is lower, compared to using CrAM. Moreover, CrAM can be easily modified to also optimize the dense model, while for C-SAM this modification would require an additional forward-backwer importance of the extra gradient step in CrAM, by comparing against simply applying the Top-K operator to the parameters. We provide an ablation study in Appendix C.I.
pass. We also examine the
Statistics Correction. It is well-known (Hubara al., 2021; Frantar & Alistarh, 2022) that pruning weights in a single step at high sparsity levels can have a large negative effect on normalization layers, due to mismatch between layer statistics, e.g. the running mean and variance of Batch Normalization (BN) layers, computed during training, and those of the pruned model. To correct for this, following prior work, we keep subset of randomly chosen 1000 training samples, to which apply standard training augmentations, and which are used post-pruning for resetting the BN statistics of the pruned model. We note that this procedure, which refer to Batch Norm Tuning (BNT) is very inexpensive, and does not finetune any model parameters. Furthermore, when training CrAM training for image classification, we only track the BN statistics on the dense model, before applying the compression perturbation. In the case of BERT models, we do not apply any statisties corrections.
4 EXPERIMENTS
Our experimental validation mainly focuses on sparsity, obtained by applying the Top-K operator, in the context of CrAM (i.e. TopK-CrAMI. The main method we propose is CrAM+ with multiple sparsity levels chosen uniformly random each step particular cases of this method, where only one sparsity level is used (e.g. CrAM+ -77). and also with the initial CrAM method with low sparsity (e.g. CrAM-k50). For image classification experiments,
We also experiment with
5

---

Published as a conference paper at ICLR 2023
all one-shot pruning results are presented after BNT on subset of 1000 training samples, i.e. 100 inference steps on batches of size 128, using standard random augmentations.
4.1 IMAGENET EXPERIMENTS
General Setup. We use a standard setup for training our ImageNet/ResNettS models, similar to Foret et al. (2021), which we describe in Appendix B. To match the number of backpropagation steps of CrAM, we additionally train the dense baseline for twice many epochs. We have found that 0.05 recommended by the authors of SAM (Foret et al., 2021) is a good value for CrAM, and we have kept it for all our ImageNet experiments. As stated, after one-shot pruning, we perform BNT on a subset of 1000 training samples (e.g. one per class), with standard augmentations. We show in the Appendix C.3 that the accuracy after BNT is extremely stable, w.r.t. the choice of calibration set.
(mngeNeetceeeett5 Validation
shot pruned models
Model
Sparsity
Valteetinn (M)
80%
90%
CrAM+
75.8
74.7
02
WoodFisher
76.7
75.3
AC/DC
76.2
75.2
CIRM-57 CotMe470 CAM
STR
76.1
74.3
DPF
75.1
74.6
Cence
Spanity (hh
Figure 1: One-shot pruning results, averaged over 10 BNT
Table 1: Accuracy of one-shot pruned
trials using randomly chosen calibration sets of 1000 samples.
CrAl'-Muli models vs. pruning methods.
Results for one-shot pruning. We validate that different versions of CrAM models are robust to post-training compression, by testing their accuracy after one-shot pruning, at different sparsity levels. We train models using CrAM-k50, CrAM+-k70 and CrAM+-Mull and global magnitude; for -Multi we choose the sparsity level randomly at each step from the set (50%, 90%). Using CrAM+ is crucial for preserving the dense model accuracy, at higher sparsities (2 70%) during training: however, when training with low sparsities, vanilla CrAM can result in dense models that are slightly better than the baseline (e.g. CrAM-k50), For both CrAM+-K7 and CrAM use sparse gradients for the Top-K K model perturbation, as described in Section 3.2. This improved substantially the accuracy after one-shot pruning, as well as the resulting dense model. Additionally, this could offer training-tine time speed-up compared to, for example, using dense gradients or training with SAM with the right framework support. We include an ablation study on the effects of sparse gradients in Appendix C.2.
The results from Figure show that CrAM models are substantially more robust to one-shot pruning, compared to standard SGD or SAM training. CrAM models do not lose accuracy at lower sparsity (e.g. at 50% for all or 70% for CrAM+ and CrAM+ and the resulting dense models tend to outperform the baseline (e.g. the ImageNet validation accuracy for CrAM+ -Mulii is 77.3% accuracy vs. 76.95% for the baseline). Overall, CrAM -Multi the most robust to one-shot pruning across different sparsity levels. For example, as shown in Table I, the results at 80% and 90% sparsity are competitive with state-of-the-ard pruning methods, such as WoodFisher (Singh & Alistarh, 2020) or AC/DC (Peste et al., 2021), and even surpass some pruning methods (STR (Kusupati et al., 2020), DPF (Lin et al., 2020)). We emphasize that CrAM requires single round of training (albeit with twice as many forward-backwart passes, compared to regular SGD), while standard pruning methods require training separately for each target sparsity, sometimes from a pretrained model (c.g. WoodFisher). In addition to global magnitude, CrAM can be used successfully with uniform magnitude pruning; show additional results in the Appendix C.4, as well as evidence that CrAM models are robust to sparse distributions different from those used during training.
Results for semi-structured sparsity. We show the robustness of CrAM on semi-structured N:M sparsity patterns, which can provide practical speed-ups (Mishra et al., 2021). CrAM+ trained using N:M sparsity preserves the dense model's 's accuracy (77.3%), and can be accurately pruned one-shot (+BNT) to 2:4 (77.0% ) or 4:8 (77.2%) patterns. This is competitive with -fate-of-the--a methods for N:M sparsity Zhou et al. (2021). We provide a full discussion in Appendix C.5.
Finetuning with CrAM. To reduce the training cost of CrAM, we investigate whether a model can be- come more robust to pruning after short finetuning with CrAM. We are inspired by Andriushchenko & Flammarion (2022), who showed that similar benefits to full SAM training can be obtained when SAM

---

Published as a conference paper at ICLR 2023
Average Relative increase In Error across Sparsities
70 50
60
30
Relative
20
10
90
Sparsity (16
Figure 2: Average relative increase in error, relative to dense, on 12 tasks, hetween models pruned one-shot, obtained from pruning methods, different sparsities. Lower is better. All models pretrained ImageNet/ResNet0.. For better visibility, error bars indicate 70% confidence intervals.
used only in the final training phase. We finetune pretrained ImageNet ResNet1 8 and ResNet50 models, from the Torchvision library, for 10 epochs, using BEAM**K70 and CrAM -Multi, both with sparse gradients. For CrAM+-Mult we randomly select each step sparsity level in the range 50%-90%. For comparison, we also finetuned using SGD with momentum or using SAM, under the same hyperparameters, We report in Tables 2 and the validation accuracy for the dense models, and after one-shot pruning at 50%-80% sparsity levels. Finetuning with CrAM*t preserves or outperforms the baseline accuracy, and results in good sparse models after one-shot pruning, at moderate sparsity levels (up to 70%). Results improve with longer finetuning: after 20 epochs, the CrAM+-Mull model can be pruned one-shot to 70% sparsity, with < 1% drop in accuracy, compared to the baseline.
Spariity
Sparsity
Model
60%
70%
80%
50%
70%
80%
SGD
70.4
68.9
67.0
62.3
50.1
SGD
76.8
75.4
73,6
69.0
53.1
70.5
67.4
75.8
57.8
CrAM+-k70
70.3
69.5
68.8
69.0
65.0
CrAM k70
76.8
75.9
75.5
75.4
72.0
Muuii
70.4
CrAM+-Muli-20
70.6
69.9
69.6
69.0
67.6
CrAM+-Multi-20
76.8
76.1
75.7
75.8
74.4
Table (ResNet18) Accuracy after finetuning for dense models, and after one shot pruning (+BNT)
Table (ResNet50) Accuracy after finetuning dense models, and after one shot pruning (+BNT)
Sparse Transfer Experiments. We additionally test how well the sparse models obtained through one-shot pruning after CrAM training on ImageNet transfer across different tasks. The setup is very similar to Salman et (2020); Kornblith al. (2019), where transfer performed across 12 bench- mark tasks. Following lofinova al. (2022), we use full finetuning of the non-zero weights, with fixed masks, and reinitialize the dense classification layer. We compare dense and one-shot pruned CrAM- k50 and CrAM -Muli models (before BNT), trained on ImageNet/ResNet50 to the corresponding ones obtained from SGD SAM. In addition, we compare with standard pruning methods used in the literature, such as lottery-tickets (LTH) (Chen et al., 2021a), AC/DC (Peste et al., 2021), STR (Kusu- pati et al., 2020) WoodFisher (Singh & Alistarh, 2020), for which use public finetuned models on the downstream tasks (lofinova et al., 2022). For each model and sparsity level, we aggregate the results over all tasks, measuring the relative increase in error of the sparse models, compared to the dense baseline, as was done in (lofinova et al., 2022). The results in Figure show that one-shot pruned CrAM models transfer well. In fact, both CrAM-K50 and CrAMt-Multi Muuli models transfer better than LTH or STR at 80% sparsity. Also, -Muli at 90% sparsity has a similar transfer performance AC/DC models, and gives better results compared to the other pruming methods used for comparison (LTH or STR), with the exception of the second-order WoodFisher method, which is the best performing method across both 80% and 90% sparsity. Moreover, the dense CrAM* -Mutti model has very similar transfer performance the baseline, with a less than 1% average relative increase in error, while dense CrAM-k50 or SAM, as well as the baseline trained for twice more iterations result in a minor improvement in transfer accuracy of around 2%, compared to the baseline. Compared to the standard pruning methods used for comparison, CrAM has the added advantage that it produces an accurate dense model, and both 80% and 90% models with a single ImageNet run.
Quantization. We additionally provide evidence that CrAM can be adapted to quantization. Namely, short finetuning using quantization version of CrAM pretrained ImageNet models can preserve
7

---

Published as a conference paper at ICLR 2023
their accuracy with respect to the baseline, after symmetric per-channel 4 bits quantization, while also boosting the accuracy of the dense model. More details can be found in Appendix C.7.
4.2 EXPERIMENTS ON LANGUAGE MODELLING
In addition to image classification, we also successfully apply CrAM for language models. We demon- strate that CrAM produces models that are more compressible and accurate than the ones obtained with standard optimizers like Adam (Kingma & Ba. 2015) and SAM (Foret et al., 2021). Also, we show that CrAM models are even competitive with gradual pruning methods, which usually require a higher computational budget produce accurate sparse models for each sparsity target independently.
General setup. We focus on the standard benchmark for compression methods: the BERT-base (De- vlin et al., 2019) model on the span-based question-answering task SQuADv1.1 (Rajpurkar et al., 2016). We consider the short fine-tuning setup (1-3 epochs) of the pretrained BERT-base model on a downstream task. Following the community standards (e.g. Sanh et al. (2020), Kurtic et al. (2022)) we sparsify the weights of the encoder part, and make use of the Top-K K operator at each step to impose uniform sparsity distribution over all layers.
Robustness to one-shot pruning. We fine-tune the model with Adam, SAM, and several variants of CrAM and test their robustness to one-shot pruning with the standard magnitude pruner. To identify we run a grid search (please see Appendix D for more details)
the optimal set of and pick the one with the best one-shot performance at 50% sparsity target. For a fair comparison, we allow Adam fine-tune for twice as many epochs as SAM and CrAM. The results presented in Table 4 suggest that CrAM models are more robust to one-shot pruning while still being able to match or even outperform the dense accuracy obtained with other optimizers.
Model
Dense
50%
60%
Sparsity
70%
50%
Model
Pruning
50%.
60%
Spansity
70%
80%
Adam
88.7
80.0
32.5
9.6
8.1
Fu regularization
gradual
84.6
K2.8
SAM
88.5
81.0
33.4
10.1
7.3
Magnitade
Movement
gradual
gradual
83.0
87.0
82.8
86.7
81.9
86.5
82.0
84.8
CrAM+-150
$8.9
$8.3
84.6
25.3
8.3
Soft Movement
gradual
85.8
N.A.
84.6
N.A.
CrAMf-160
88.7
88.1
87.8
75.7
10.2
PLATON
gradual
87.2
86.1
CrAM+-170
1180
88.4
88.8
86.9
87.8
87.0
85.5
84.9
86.9
84.7
33.9
CrAM--Muli
one-shot magninude
one one-shot aBERT
88.7
88.3
88.1
85.1
87.5
86.8
82.5
84.9
Mulii-Mulii
88.1
one-shot oRI-KT fine-tune
88.7
88.1 87.4
Table 4:
Valida-
Table 5: (SQuADv1. 1/BERT-base) Validation Fl score of the CrAMT model after one-shot pruning with magnitude and oBERT pruners. We additionally fine-tune the one-shot oBERT- pruned model and compare it with gradual pruning methods.
tion F1 score of models after fine-tuning with the corresponding optimizer and ap- plying one-shot magnitude pruning.
Comparison with gradual pruning methods. We investigate whether CrAM models can be competitive with those produced by gradual pruning methods, which progressively prune and fine- tune the model for many epochs. We adopt the CrAtt-Multi model from Table 4 and prune it in one-shot with the standard magnitude pruner, but also with the state-of-the-art BERT-pruning method called oBERT (Kurtic et al., 2022). Since one-shot pruning to high sparsity targets can severely impact the model's 's performance, we also investigate whether short fine-tuning (for at most 2 epochs) on top of it can bridge the gap towards full accuracy recovery. In Table we present the results and compare against the following gradual pruning methods: Eo regularization (Louizos et al., 2018), Magnitude (Zhu & Gupta, 2017). Movement (Sanh et al., 2020), Soft-Movement (Sanh et al., 2020) and PLATON (Zhang et al., 2022). For details regarding hyper-parameters, please see Appendix D. As can be seen from the results, one-shot pruned CrAM models are competitive with gradual pruning methods, which they outperform by huge margins when additionally fine-tuned for a few epochs. is worth emphasizing that competitive results obtained with two different one-shot pruners, magnitude and OBERT, suggest that CrAM models are indeed robust and compatible with different pruning techniques than the ones they have been trained with. We provide in Appendix D Tables 20 and 19 speed-ups for the sparse BERT models, and evidence that models become more robust to pruning even when CrAM is not used at every optimization step.
4.3 DETAILED COMPARISONS WITH OTHER METHODS
We perform a detailed comparison with other methods on CIFAR-10 (Krizhevsky et al., 2009), which common setup for all previous methods we consider. Specifically, we compare CrAM with
8

---

Published as a conference paper at ICLR 2023
state-of-the-arr gradual pruning methods (Lin et al., 2020) ResNet20 (He et al., 2016), with similar methods that train prunable networks (Miao et al., 2022) on VGG-16 (Simonyan & Zisserman, 2014) or R (He et al., 2016). We discuss all hyperparameters in Appendix B, and compare against one-shot pruning from dense baselines in Appendix C.8.
Comparison with Gradual Methods. We train CrAM+-Multi, with sparse intermediate gradients, and sparsity values chosen randomly among (50%, 70%, 80%, 90%, 95%]. The results in Table show that the dense CrAM+ -Muli model is highly accurate, and also very robust to one-shot pruning. even at high sparsities (e.g. 90% and 95%). Remarkably, our results for one-shot pruning (+BNT) are usually competitive with those obtained by other methods which train sparse models separately, for each sparsity target, for example through DPF (Lin et al., 2020). Notably, we substantially outperform DPF at 95% sparsity, after BNT.
Sparsity 80%
Architecture
Model
Dense
50%
70%
90%
95%
ResNer20
-rAMi -Multi
92.9 + O. N/A
92.8 + 0.1
92.7 + 0.2
92.6 + 0.2
91.2 + 0.1
89.2 = 0.1
DPF
N/A
92.4 0.1
92.2 2 0.2
90.9 - 0.1
88.0 = 0.3
CrAM++195
94.2 + 0.1
94.2 + 0.1
94.2 + 0.1
94.1 0.1
94.0 - 0.1
94.1 = 0.1
VGG-16
SFW
N/A
93.1
93.1
93.1
93.1
92.0
DPF
N/A
N/A
N/A
N/A
N/A
93.9 = 0.2
Table 6: (CIFAR-10) 10) Test accuracy (%) for CrAM after one shot pruning (+BNT). CrAM+ can outperform gradual pruning method DPF, up to 95% sparsity. DPF requires retraining for each target sparsity. CrAM outperforms similar method SFW (Miao et al., 2022).
Comparison with One-shot Methods. We compare against other methods for training prunable models, such as Miao et al. (2022); Zimmer et al. (2022). Both are based on Stochastic Frank- Wolfe (SFW) (Reddi et al., 2016), to encourage the parameters to lie in the convex hull spanned by sparse vectors, with directions given by the gradients. We compare CrAM against SFW for CIFAR- 10, on ResNet18 (He et al., 2016) and VGG-16 models, which is the same setup used in Miao et al. (2022) and Zimmer et al. (2022). We train CrAM+-K95 with sparse intermediate gradients, for 180 epochs (same as Miao et al. (2022)), using SGD with momentum and weight decay, and cosine learning rate schedule. We use BNT after pruning. On both ResNet18 and VGG-16, we obtain dense models that preserve the baseline accuracy: 94.2% (CrAM*+k95) vs. 93.9% (dense) on VGG-16 and 95.7% (CrAM+-k')
vs. 95.4% (dense) on ResNet18. Furthermore, on ResNet18 we maintain the model accuracy after pruning one-shot at 96% sparsity (95.4) 99..4%. after BNT) and have a 1.3% drop at 98% sparsity (94.4% Top-1); in comparison, Miao al. (2022) and Zimmer et al. (2022) obtain < 93% accuracy at 95% sparsity. We show a numerical comparison for VGG-16 in Table 6: Cr9M+-K95 preserves model accuracy even at 95% sparsity, which is competitive with the DPF gradual method, while SFW produces models that have lower accuracy even at higher density. CrAM has higher training costs than SFW, but requires much less hyper-parameter tuning, and leads to higher accuracies. Similar to Zimmer et al. (2022), we use BNT, but the additional cost of BNT is minimal; even without BNT, preserve accuracy at up to 80% sparsity (Appendix C.8), leading to better results than Miao et al. (2022); Zimmer et al. (2022).
5 CONCLUSIONS AND FUTURE WORK
We proposed a new method for training neural networks, CrAM, which results in models that are both highly accurate, and masily-compeessible.. Our extensive experimental analysis on large scale image classification (ImageNeuRessNets) and language modelling L.QQAADi..//ERTTTA.
shows that CrAM models can be pruned one-shot at a wide range of sparsity levels, while resulting in sparse models that are competitive with existing gradual pruning methods. Furthermore, we show that one-shot pruned CrAM models can transfer better to downstream tasks, compared to some of the existing pruning methods. While we focus on pruning as the main compression operator, we also give encouraging evidence that the CrAM update can be successfully adapted to other compression projections, such as quantization, and we plan to investigate this more closely in future work. We would like to explore whether prolonged CrAM-training would further enhance both the performance of the resulting dense model, as well as its robustness one-shot compression. Finally, we are interested in leveraging in the CrAM update different methods developed for reducing the computational complexity of SAM, in order to improve the efficiency of method.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
AP, EK, DA received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 805223 ScaleML). AV acknowledges the support of the French Agence Nationale de Recherche (ANR), under grant ANR-21-CE4S-000 -21-0164 (project COMCOPT), We further acknowledge the support from the Scientific Service Units (SSU) of ISTA through resources provided by Scientific Computing (SciComp)
ETHICS AND REPRODUCIBILITY STATEMENTS
Our work is technical in nature, considering existing models and datasets, and thus does not pose any direct ethical concerns. With regards to reproducibility, we release an implementation of our optimizer and an example experimental harness.
REFERENCES
Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware mini- mization. In International Conference on Machine Learning (ICML), 2022.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 3432, 2013.
Thomas Blumensath and Mike E Davies. Iterative thresholding for sparse approximations. Journal of Fourier analysis and Applications, 14(5-6):629-6544 2008.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network
and specialize for efficient deployment. International Conference on Learning Representations (ICLR), 2019.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021a.
Tianyi Chen, Bo Ji, Tianyu Ding, Biyi Fang, Guanyi Wang, Zhihui Zhu, Luming Liang, Yixin Shi, Sheng Yi, and Xiao Tu. Only train once: A one-shot neural network training and pruning framework. In Conference on Neural Information Processing Systems (NeurIPS), 2021b.
John M Danskin. The theory of max-min and its application to weapons allocation problems, volume 5. Springer Science & Business Media, 2012.
J. Devlin, Ming-Wei Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. North American Chapter of the Association for Computational Linguistics (NAACI.), 2019.
Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient sharpness-awand aware minimization for improved training of neural networks. International Conference Learning Representations (ICLR), 2022a.
Jiawei Du Daquan Zhou, Jiashi Feng, Vincent YF Tan, and Joey Tianyi Zhou. Sharpness- training for free arXiv preprint arXiv:220S.14088 2022b.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference Machine Learning (ICML), pp. 2943 2952. PMLR, 2020.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awan aware minimization for efficiently improving generalization. International Conference Learning Representations (ICLR), 2021.
Simon Foucart. Hard thresholding pursuit: an algorithm for compressive sensing SIAM Journal on Numerical Analysis, 49(6):2533-2563 2011.
10

---

Published as a conference paper at ICLR 2023
Simon Foucart. Sparse recovery algorithms: sufficient conditions in terms of restricted isometry constants. In Approximation Theory XIII: San Antonio 2010, pp. 65-77. Springer, 2012.
Elias Frantar and Dan Alistarh. Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning. Conference on Neural Information Processing Systems (NeurlPS), 2022.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103. 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint ar arXiv:1503.0255/1 2015.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research (JMLR), 2021.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accel- erated sparse neural training: A provable and efficient method to find N: M transposable masks. Conference on Neural Information Processing Systems (NeurIPS), 2021.
Eugenia Iofinova, Alexandra Peste, Mark Kurtz, and Dan Alistarh. How well do sparse ImageNet models transfer? In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better ImageNet models transfer better? In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal BERT surgeon: Scalable and accurate second-order pruning for large language models. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Bill Nell, Nir Shavit, and Dan Alistarh. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In International Conference on Machine Learning (ICML), 2020.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Leurning (ICML). 2020.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. ASAM: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In International Confer- ence on Machine Learning (ICML), 2021.
Robert Lang. A note on the measurability of convex sets. Archiv der Mathematik, 47(1):90-92, 1986.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario $aško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Majpe Major,, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac. Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics,
2021.
11

---

Published as a conference paper at ICLR 2023
Tao Lin, Sebastian U Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. International Conference on Learning Representations (ICLR), 2020.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Toward theory of optimization for over- over-paraeeterieed systems of non-linear equations: the lessons of deep learning. arXiv preprint arXiv:2003.000 003077 2020.
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scal- able sharpness-awwa minimization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Christos Louizos, Max Welling, and Diederik Kingma. Learning sparse neural networks through to regularization. International Conference on Learning Representations (ICLR). 2018.
Lu Miao, Xiaolong Luo, Tianlong Chen, Wuyang Chen, Dong Liu, and Zhangyang Wang. Learn- ing pruning-friest friendly networks via Frank Wolfe: One-shot, any-sparsity, and no retraining. In International Conference on Learning Representations (ICLR), 2022.
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08377 2021.
NeuralMagic. Deep Sparse: A fast CPU inference engine, 2021.
Mahdi Nikdan, Tommaso Pegolotti, Eugenia lofinova, Eldar Kurtic, and Dan Alistarh. Sparseprop: Efficient sparse backpropagation for faster training of neural networks, 2023. URL https:
/ www.iiioogg/aaaea..oaooaoooeoeae 2302. 04852.
Alexandra Peste, Eugenia Iofinova, Adrian Vladu, and Dan Alistarh. AC/DC: Alternating Com- pressed/DeCompressed Training of Deep Neural Networks. Conference Neural Information Processing Systems (NeurIPS), 2021.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2016.
Sashank Reddi, Suvrit Sra, Barnabás Póczos, and Alex Smola. Stochastic Frank-Wolfe methods for nonconvex optimization. In Annual Allerton Conference Communication, Control, and Computing (Allerton), pp. 1251. IEEE, 2016.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust ImageNet models transfer better? Conference Neural Information Processing Systems (NeurIPS), 2020.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. Conference on Neural Information Processing Systems (NeurIPS), 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Sidak Pal Singh and Dan Alistarh. WoodFisher: Efficient second-orde approximation for neural network compression. Conference on Neural Information Processing Systems (NeurIPS), 2020.
Kwong Meng Teo. Nonconvex robust optúmization. PhD thesis, Massachusetts Institute of Technology. 2007.
Neil C Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F Manso. The computational limits of deep learning, arXiv preprint arXiv:2007.05558, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac. Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45. Association for Computational Linguistics, 2020.
12

---

Published as a conference paper at ICLR 2023
Zhaofeng Wu, Ding Zhao, Qiao Liang, Jiahui Yu, Anmol Gulati, and Ruoming Pang. Dynamic sparsity neural networks for automatic speech recognition. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2021.
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Under- standing straight-through estimator training activation quantized neural nets. International Conference on Learning Representations (ICLR), 2019.
Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv: 1903. 11728, 2019a.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019b.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. International Conference on Learning Representations (ICLR), 2019.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoe Le. BigNAS: Scaling up neural architecture search with big single-stage models. European Conference on Computer Vision (ECCV), 2020.
Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance. In International Conference on Machine Learning (ICML), 2022.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning N: M fine-grained structured sparse neural networks from scratch. International Conference Learning Representations (ICIR), 2021.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint 171001770,,08888, 2017.
Max Zimmer, Christoph Spiegel, and Sebastian Pokutta. Compression-aware training of neural networks using Frank-Wolis arXiv preprint arXiv:2205.11922 2022.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
A THEORETICAL SUPPORT FOR THE CRAM UPDATE
In this section, we attempt to formally derive generic training method whose purpose provide accurate models, which perform well even after being compressed. To understand why we can hope achieve such guarantees, we first take brief detour the area of robust optimization.
A.1 ROBUST OPTIMIZATION
Generally, practical training methods are based on versions of stochastic gradient descent attempting to minimize a loss function T.(0). However, 0 might turn out to be bad solution the landscape of its neighborhood could contain large changes in value. To address this issue, one may attempt to flatten L such that it is less sensitive to sharp drops in value localized around very small region. To this extent, a standard robustification can be defined by
L(0) = max L - 8) S
which makes the value of T()) take that of the largest value of T. given by perturbation of within ball of radius p. While this robustified function may well suited generic training tasks, is priori unclear that it is amenable to optimization.
However, under certain conditions, we can efficiently optimize by using a classical theorem in robust optimization due to Danskin (Danskin, 2012).
Theorem 1. (Danskin) Let C C R be compact set, let a function & RR x C - R such that Φ y) is continuously differentiable for every fixed y E C and V Vab(2, x. y) continuous on I2" C. and let vb R + IR be defined as
v (x) max & (æ, y) yEC
Then vp is locally Lipschitz continuous, directionally differentiable, and its directional derivatives satisfy
dup (æ;; d) max
d V2 (x,
yEC-
where C* (x)) is the set of maximizers
(x(x)) y o (x, y²) max o (x, 8)
In particular, for some E R the set C*xx) { is singleton, then s is differentiable at and
VΦ((x) V yz)
This shows that, under certain assumptions, we can obtain directional derivatives for T. (.)) by simply maximizing T(O + 6) over ō E B2(p).
Corollary 2. Let be defined as in Equation (7), and define
C²
8 ||8|| < p, L(0 + 8) max L(O + 8*) |
and let 8 (0). Provided that L(0) is continuously differentiable, and 0 is not articulation point for T., T + 5) descent direction for T.(0) as long as is nonzero.
Proof. Let h VI.(0 + 5). We apply Danskin's theorem for ((0,6) = T.(0 + 5) and C B2(p). This shows that
dī h)
sup h TV + > h ( 5) h h 2 0. (E)
Provided that 0 is not an articulation point for T,, we also have that dī. -h) -dī((;h) 5. which concludes the proof.
14

---

Published as a conference paper at ICLR 2023
A.1.1 FROM ROBUST OPTIMIZATION TO SAM
Per Corollary 2, to obtain a descent direction it suffices maximize L(0 + 6) over the set of perturbations satifying ||o]] S p. In general, even when the underlying function L is convex, this may be difficult problem. Instead, one may simply attempt to obtain good local maximizer of L in bounded region around 0. The simplest possible way to do so is by performing a step of gradiens ascent, which can be regarded as a proxy for the maximization subproblem. Using this step, immediately obtain the iteration:
0. 0,-
VL VL(O.), 0,+1 - 0, LV))
P
which recovers the extrapolated SAM gradient step from Foret et al. (2021).
There is exhaustive research that has previously been done on robust optimization methods, For a comprehensive reference, we point the reader to Teo's PhD thesis (Teo, 2007).
A.2 ROBUST OPTIMIZATION FOR COMPRESSIBLE MODELS
With the robust optimization framework in mind, we are ready to attempt implementing similar scheme which exhibits robustness to compression.
To motivate the method, let us consider the post-training compression. After training the model to weights 0r we apply one-shot compression method C over some perturbation 0r + 8 of the weights. This captures several iterative methods for compression, such as iterative pruning, where changes in weights are alternated with one-shot pruning methods.
If our goal is to make the loss after compression robust within a small neighborhood of perturbations 6, we can establish formal objective minimize the robustified loss
L CrAM (0) max L (e++++))) 6:||6||5
for some magnitude p of allowed perturbations. In our case we will focus on the case where these are bounded in & norm, but this can be easily extended to other choices of the domain. Just as before, can now attempt minimize ICMA or find near-stationary point, by gradient descent. Using the robust optimization framework we may attempt to optimize it using Corollary 2 after replacing I(-) with L(C()).
Naturally, this poses some obstacles in our case. The main one is the fact that it is not true that L(C(O)) will generally be continuously differentiable, so the conditions required to obtain descent directions via an inner maximization loop are not satisfied. However, we can show that under certain conditions, continuous differentiability fails only at a set of points of measure O.
Definition 1. Let S be a countable set, let {Polies be a covering of R. with convex sets, and let S(x) denote the family of indices from S for which {Ti)ees. such that for any x the projections (T)(())) all map to the same point. We call a projective compression operator with respect to (1l) a mapping C IR + RR such that C(x) (x), for any E 5(x).
P.. Let a family of projection operators
For example, in the case of the Top-K compression operator, sets P such that all elements x E P share the indices A S [n]. |A| = l for the Top- elements in absolute value (with ties broken lexicographicalll), and furthenmore, all elements from P preserve the signs across A. is clear that any z E IR belongs in some such set P. and since there are finite number of subsets of size k and of possible signs for the components in [ri]. we have finite covering. Assume, without loss of generality, that set from the covering consists of elements for which the first & components are the highest in absolute value, and the signs across these components are shared across all x E P. Then, for any æ, y P, X E (0, 1) and i & k. we have that Aæ; + (1 \)ya1 = A|æ; + (1 A) y11 (since æ; and y, share the same sign). Using that A|a,| + (1- y 2 A| + (1- ,) for any < j < n, together with the triangle inequality, we obtain that |Ax; + (1 Auh, Aλæ, + (1 - for any i is k and j > k. Therefore, any P satisfying the conditions described above is a convex set. We can further define a projection for each subset A of coordinates of cardinality k. Then, it is clear that for any given vector æ, the set A E S(x)) iff the largest & coordinates of æ in absolute value (with ties broken lexicographically) are conclude that Top-K is projective compression operator.
define covering of R" with
supported in A. Therefore,
15

---

Published as a conference paper at ICLR 2023
Lemma 3 (Continuously differentiable functions induce few singularities after compression). Let L: "R" + be a continuously difserentiable function, and let C be a projective compression operator. Then the function g(x) == L(C(=)) is continuously disserentiable everywhere except at a set of points of measure O. Furthermore, so is the robustified function L [CAA(()) ) == maxiool L(C(x + +8))
Proof. First note that the boundary of any convex set has measure zero by standard arguments in convex analysis (Lang, 1986). Since a countable union of sets of measure zero has measure zero, it follows that the union of the boundaries of P.'s has measure zero. Now since L is continuously differentiable, within any set P. we have that g = (T), ), and hence remains continuously differentiable. Therefore the only region for which we can not argue about continuous differentiability
the complement of the union of interiors of P/'s, (U,intP;) )d S U,OP; which is a set of measure zero. Since g is well-behaved almost everywhere, all that remains to argue is that this is the same case with I. J.C.MM
For any fixed direction A0, we define the mapping (0) ~ (g(
and its robustification
O max M(O + 8) = max AO Vg + 6). [0
Hence the directional derivative w.r.t. AO of L. is discontinuous only when M0) is discon- tinuous. Finally, we note that this almost never happens, as M is continuous almost everywhere, and thus so must be M. Thus, directional derivatives are continuous except at set of measure which concludes the proof.
Finally, just like in the previous case, maximizing L(C(0 + 8)) over small perturbations is generally intractable. So we instead consider obtaining a good enough maximizer via a standard iterative method which has shown good performance in practice. More preciscly we consider the projected gradient ascent method, which provides strong theoretical guarantees, even when the projection is performed onto non-convex domains (Peste et al., 2021). In the case where the compression operator represents magnitude pruning, this corresponds to the iterative hard thresholding (IHT) method, frequently employed in the sparse recovery literature.
To reach a good iterate within this specific domain we instead perform single step of (projected) gradient ascent, which matches the IHT iteration:
0, C (0: + VL(O.))
(10)
We have therefore obtained re-derivation of the CrAM update in Equation 2. We note that in our experiments we use a fixed step size p for the interpolation step, instead of normalizing the gradients. Similar to previous work (Andriushchenko & Flammarion, 2022), we observed that normalizing the gradients did not result in significant difference the results, and for simplicity we have omitted the normalization step in our experiments.
Furthermore, we note that the analysis above holds also for the CrAM+. Namely, we can rewrite the CrAM+ loss as I CrAM* = m + 6)) + L()),, and use Lemma 3 to obtain that C is continuously differentiable almost everywhere, and are its directional derivatives.
B IMAGE CLASSIFICATION HYPERPARAMETERS
Hyperparameters for CIFAR-10 experiments We train ResNet20 models for 200 epochs, using
SGD with momentum and weight decay, and a cosine learning rate scheduler, with a learning rate warm-up of 5 epochs. Additionally, we trained the baseline model for twice many epochs, to match the number of backpropagation steps of SAM and CrAM. To determine the value of the hyperparameter 90% 10% train-validation split and found 0.1 and 0.15 to be the best values for SAM and CrAM + Multi, respectively (achieving highest validation accuracy). After finding the best value of p. we retrained using the entire training set, and starting from 3 different random seeds, and report the final accuracy after 200 epochs of training. We follow a very similar training recipe and hyperparameter search for ResNet18 and VGG experiments, but train instead of 180 epochs.
we performed grid search over values in the range 0.01 - 0.2, using a
16

---

Published as a conference paper at ICLR 2023
Hyperparameters for ImageNet experiments For our ImageNet experiments, we use standard data augmentation, and we train the models using SGD for 100 epochs, with batch size 512, momentum 0.9, and weight decay 0.0001. The learning rate is linearly increased for the first 5 epochs until it reaches a maximum value of 0.2, after which it is decreased at each epoch, using cosine scheduler. To determine the value of the hyperparameter P. we search over small grid, by training 90% of ImageNet using CrAM-K50, and using the remaining 10% of the dataset for validation. We have found p = 0.05 to give good results for CrAM-k50, and we have kept this value for all our other CrAM experiments. For SAM, we also use p = 0.05, which is the standard value recommended by the authors of Foret et al. (2021).
C ADDITIONAL IMAGE CLASSIFICATION EXPERIMENTS
C.I ABLATION STUDY FOR ALTERNATIVE UPDATES
Comparison between CrAM and C-SAM. We investigate the importance of individual components from the CrAM update by comparing against other similar updates, the CIFAR-10 dataset, using a ResNet20 model. One such update can be obtained by following closely the derivations for SAM (Foret et al., 2021). We assume |6] < p. define h(#)) ;= L(C(z)) (with C' the Top-K operator), and the loss max مل h(O + 8). We can assume that for a small enough p the function 1(0 + h)) is differentiable in 0 (for example, the perturbation is small enough such that the Top-K mask stays constant). By using a first-order Taylor approximation of h( + 5) around θ, together VL(C(e)) This enables us to define the with the quadratic constraint for 8, we obtain ô- P [VCCO compressed-See (C-SAM) update as:
Oz+1 = 0. - 1 C 00, (0, + (VIC(()) )
VLC(O)))
C-SAM:
(II)
We train C-SAM using sparsified gradients, for both the intermediate interpolation step, as well as in the final weight update. Namely, we always use the approximation: VL(C())) R Ms VL(M, o) for parameter o, where Ma is the Top-K mask of c. To determine the value of the interpolation step in C-SAM and CrAM, we perform a grid search over a 90-10% train/validation split of CIFAR-10; we re-run from 3 different seeds, using the best configurations and the entire training set, and report the test accuracy after 200 epochs of training. For fairness, we compare C-SAM with CrAM, and not choose
CrAM+, and for both we sparsified gradients. For C-SAM-Multi and CrAM-Multi, the sparsity levels during training uniformly at random among values in the interval 30% - 90%.
The results in Table 7 show that C-SAM can be more robust at higher sparsity levels, but with the cost of an accuracy drop for the dense models. Moreover, the dense model can be improved using CrAM + with no additional cost, whereas for C-SAM such modification would require a computational overhead. We additionally observed that the dense C-SAM-K70 model can be improved to almost match the accuracy of CrAM-k70, by not sparsifying the gradients in the interpolation; however, this also decreased the accuracy after one-shot pruning higher sparsity, to a level similar to CrAM.
Method
Dense
50% Sparse
60% Sparse
70% Sparse
80% Sparse
90% Spurse
CrAM-K50
92.8 = 0.1
92.8 0.1
92.7 = 0.0
92.0 + 0.1
89.7 + 0.2 90.6 0.2
73.5 + 2.4 $1.0 1.0
C-SAM-K50
92.6 = 0.3
92.6
92.5 : 0.3
92.0 1: 0.2
CrAM-k70
92.4 = 0.2
9.24
92.4 = 0.2
92.3 + 0,2
91.6 0.1 91.0
81.1 1.3
C-SAM-K70
91.4 = 0.0
9.04
91.4 0.0
91.3 0.1
85.3 i 0.5
CrAM-Multi
92.6 = 0.2
92.5
92.4
92.3 + 0.2
91.9 0..9 0.2
90.5 + 0.2
C-SAM-Multi
92.5 0.2
92.5
0.1
92.6 1 0.2
92.5 - 0.2
92.1
0.2
91.0
Table 7: (CIFAR-IO/RA)) Ne12 Comparison between CrAM and C-SAM. Test accuracy for the dense models and sparse models after one-shot pruning. We report the best value between the accuracy before and after BNT with 1000 training samples.
Importance of the extra- gradient step. Furthermore, we explore the importance of the extra- gradient step in the CrAM update. Notably, we investigate whether not using the extra-gradient achieves a similar effect to CrAM training. The removal of the extra gradient step would correspond to the following equation:
Top-K:
o+t1 0. (V.(.))
(12)
17

---

Published as a conference paper at ICLR 2023
Since the compression we use in our experiments is the Top-K sparsification, simply call this update "Top-K". This update has been previously studied in Lin et al. (2020), where the dense gradients, computed with respect to the sparse parameters, are used in the model updated. Generally, we have training instability using this update, partieularly at high sparsity. However, incorporating the optimization of the dense model, as well as sparsified gradients for the compressed parameters, greatly improved the stability and overall quality of the resulting models. These changes resulted in an update close to CrAM+ and of the same computational complexity, which will be referred to as Top-K*:
Top-K+:
0 = 0, (VV((,)) + Ma. VL((.)).
(13)
where 0, C(O,) and Ma. is its mask after applying Top-K.
Method
Dense
50% Sparse
60% Sparse
70% Spurse
80% Spurse
90% Spanse
CrAM -50
93.1 4: 0.1
93.1 0.1
93.0 0.1
92.3 +: 0.2
89.2 i: 0.2
71.1 + 1,5 56.7 = 1.0
To5-
92.7 0.1
92.6 0.0
9.1
91.6 0.1
86.5 = 0.3
CrAM*-k70
92.8 0.3
92.7
9.17
92.7 0.0
01.90.
80.8 + 1.4
Top-K+-k70
02.7+0.11
9.24
92.3
9.1
91.0 = 0.3
72.6 + 2.8
CrAM+-Multi
93.2
9.1
0.1
0.2
92.4 0.1
90.1 = 0.2
-oulk+-Multi
92.5
92.4 0.1
92.3
+.3+00 0.1
92.2
91.7
0.2
90.0 = 0.2
Table 8: (IFFAR-10/ResNe200 Comparison hetween CrAM+ and Top-K Test accuracy for the dense models and sparse models after one-shot pruning. For all sparse results we report best value between the accuracy before and after BNT with 1000 training samples.
The results of the comparison between CrAM and Top-K* + are presented in Table 8 and show that CrAM models tend to have higher accuracy for the dense models, as well as to be more robust to one-shot pruning at high sparsity (e.g. CrAM+ and Top-K trained with 50% or 70% sparsity, and one-shot pruned 80% and 90% sparsity).
The comparison between CrAM and C-SAM or Top-K shows that, although all methods can achieve good results with one-shot pruning, CrAM-trained models have the best trade-off between preserving (or improving) the dense model accuracy, while having good performance after one-shot pruning, at a range of sparsity levels.
CrAM vs. SAM on compressed models. We briefly highlight the differences between the CrAM update and simply applying SAM to compressed models. First, we note that the two objectives are different: while CrAM optimizes against the largest perturbation of the dense model, to minimize its effect after compression, training SAM on sparse models would be equivalent to optimizing against the largest perturbation on the sparse model. Namely, the SAM loss on compressed models could (L(C(O)) be written as max [8] Ep ((C(0)= + 5), with the intermediate update 0 = C(0) + P (VL((())) In practice, applying SAM to compressed models could only ensure generalization for the compressed models, whereas with CrAM (and, in particular, CrAM+) we can explicitly optimize the dense model, and ensure that it is robust to one-shot pruning.
C.2 IMPORTANCE OF SPARSE GRADIENTS
For all our image classification experiments, we observed an improvement in the robustness to post- training shot pruning, when using different trriihh-thhoouge estimator for the gradient (e.( where 0: = C(O, + p (L.).) Namely, instead of by- by-passn the Top-K K operator in the gradient, and estimating (O.) ~ Vol(0) e= we can assume that the masks M, of 0, change very little during training. This would allow us to use the approximation (oL zs M, VoL(O)) Please see Section 3.2 and Appendix Section A for more details. Training both CrAM and CrAM* with this new approximation for the CrAM loss gradient (which will be referred to as "sparse gradient") led to an improvement in the robustness to one-shot pruning, particularly at higher sparsity levels. Furthermore, our intuition that the masks are fairly constant during training is also confirmed experimentally: on CIFAR- 1 trained with frAM+-K70, the difference between consecutive O2 masks was lower than 0.6%. Interestingly, using sparse gradient under the same setup, encouraged more diversity in the masks, with the difference between them at later training stages increasing to around 2%. We speculate this could be potential reason for the improved robustness
18

---

Published as a conference paper at ICLR 2023
to pruning. Another aspect observed on CIFAR-I0 experiments is that using sparse gradients tends to decrease the dense model accuracy, when training CrAM at lower sparsity; for example, the dense model for CrAM-K50 reached 93.4% accuracy, which decreased to 92.8% when using sparse gradients. For this reason, on ImageNet we only experimented with the dense version of CrAM-k50, Nonetheless, using sparse gradients improved the robustness to pruning in all cases. For a better illustration of all these effects, we provide the results obtained with these different versions of CrAM on ImageNet, in Table 9 for one-shot unstructured global magnitude pruning and in Table 10 for semi-structured N:M pruning.
Spunity 70%
Model
50%
Spursity Pattern
CrAM-170
75.7
76.3
76.3
73.4
53,2
Dense
2:4
4:8
CrAM
77.3
77.3
76.8
73.9
51.9
CrAM-N:M
75.2
76.0
76.2
CrAM"-I70 (470 (SG)
77.2
77.2
77.2
76.3
62.1
CrAM -NM
77.1
76.2
76.7
CrAM-Multi
75.2
15.2
75.2
74.5
73.3
CrAM -N:M (SC)
77.3
77.2
CrAM Multi
76.4
76.1
SR SR-STE
77.0
77.4
CIAM" -Muli (SG)
77.3
77.2
77.0
75.8
74.7
Table 9: (ImageNe/RResNet50) Dense and one-shot pruning (+BNT) results. CrAM* with sparse gradi- ents (SG) improves the accuracy the dense model, and its robustness to one-shot pruning.
Table 10: (ImageNct/ResNet5) Dense and semi- structured one-shot pruning (+BNT) results. CrAM*. N:M with sparse gradients (SG) improves the accuracy of the dense model, and its robustness to N:M pruning.
C.3 VARIABILITY OF BATCH NORM TUNING RESULTS
We emphasize that CrAM relies a small calibration set of training samples to correct the Batch Norm statistics, namely running mean and variance, after pruning, particularly at high sparsity. We call this procedure Batch Norm Tuning (BNT). To ensure that the accuracies models are stable under the choice of the calibration set, we perform 10 independent trials of BNT, on 10 randomly chosen subsets of 1000 training samples, for each model and for different sparsity levels. The results of this experiment are presented Table 11, which also contains the raw numbers used in Figure 1. Notice that the accuracy after one-shot pruning and BNT is very stable, with respect to the choice of the calibration set. particular, for the CrAM+ model, the standard deviation S 0.1% across all sparsity levels considered. We also report the raw one-shot pruning accuracy, before BNT, for CrAM models Table 12.
report for sparse
Sparsity
Model
Dense
50%
60%
70%
80%
90%
Baseline
77.22
75.87
0.09
73.82 0.07
68.86 + 0.08
$1.96 0.27
8.57 + 0..1
SAM
77.35
76.47
0.04
75.11 +
71.87 +: 0.07
60.20
4: 0.13
18.25 - 0.18
CrAM-K50
77.48
77.3 + 0.07
76.61 + 0.05
74.77
+ 0.08
68.23
0.11
33.04 +: 0.16
CrAM+-K70
77.32
77.22
0.05
77.1 + 0.05
77.15
+ 0.05
76.3 + 0.08
61.92
+ 0.11
TrAM+-Mult
77.28
77.24 0.06
77.05 + 0.05
77.0
0.04
75.8
0.07
74.74
0.04
Table 11: (ImageNer/R<<Ne:500) Validation accuracy for the dense models, and after one-shot pruning using
global magnitude pruning, followed by BNT on 1000 samples. The results for one-shot pruning are the mean accuracies, and their standard deviations, when BNT is performed 10 different random calibration sets, of 1000 training samples each.
Sparsity
Model
Dense
50%
60%
70%
80%
90%
Baseline
77.22
74.35
68.9
46.36
2.0
0.1
SAM
77.35
75.02
70.4
52.8
3.66
0.11
CrAM-k50
77.48
75.91
73.54
63.05
13.59
0.16
CrAM -70
77.32
77.03
76.6
76.3
72.6
3.6
CrAM- Multi
77.28
76.23
74.87
75.69
72.32
52.25
Table 12: (ImageNe//Re<Net50) Validation accuracy for the dense models, and after one-shot pruning using
global magnitude, before BNT.
19

---

Published as a conference paper at ICLR 2023
C.4 RESULTS WITH UNIFORM SPARSITY
In this section we show that CrAM models can be trained to be robust to different sparsity distributions, such as uniform. We train CrAM+-Multi models for ImageNet/Resset50 under the same setup as in Section 4.1, but applying instead the Top-K operator at uniform sparsity across all prunable parameters (i.e. excluding BatchNorm and biases), while keeping the first and last layers dense. The resulting dense model achieves 77.1% accuracy, while one-shot uniform pruning at 80% and 90% sparsities gives, after BNT, 75.6% and 75.1% accuracy, respectively. Moreover, this model is also robust to one-shot pruning using global magmitude (c.g. 75.4% accuracy at 80% sparsity). Conversely, CrAM Mulli trained with global magnitude is robust one-shot pruning using uniform magnitude (e.g. 76.9% and 75.9% accuracy at 70% and 80% sparsity, respectively). This suggests that CrAM--rainee models can be robust to one-shot pruning using sparsity distributions different from the ones used during training.
C.5 CRAM FOR N:M SPARSITY PATTERNS
In this section, we show our full results regarding the robustness of CrAM models against semi- structured N:M sparsity patterns, where out of each block of M weights, N are sparse. In particular, the 2:4 pattern is supported on modem Ampere NVIDIA architectures, where has been shown to provide speed-ups (Mishra et al., 2021). We train CrAM+ models with the N:M pattern, by randomly choosing each optimization step between the 2:4 or 4:8 projections; this model will be referred to as "CrAM*-N:M'.. Similar to the previous experiments, we also use sparse gradients for the pruned model perturbation and have found this to have a positive impact on the one-shot pruned models. In Table 13 we show the one-shot pruning results (after BNT with 1000 samples). Note that CrAM+-N:t models do not lose accuracy when they pruned one-shot using the 2:4 and 4:8 patterns, which is competitve with tfatc-of-the-eee art methods for training N:M sparse models, such as SR-STE (Zhou et al., 2021); I); however, SR-STE requires different training run for each sparsity profile. CrAM -N:M model is also robust to one-shot pruning using unstructured patterns, at moderate sparsity levels; for example, the results in Table 14 show that models trained with CrAM+NN:M can be pruned one-shot to 70% sparsity with a minor accuracy loss, compared to the dense baseline.
Model
Dense
2:4
Dense
77.3
66.1
69.1
Top-K
50%
70%
SAM
77.4
69.4
71.8
Global
77.3
76.5
CrAM-K50
77.5
72.1
73.3
Uniform
77.3
76.5
CrAM+-N.M
77.3
77.0
77.2
SR-STE
77.0
77.4
Table 13:
Accuracy (%) after one-shot pruning
Table 14: [CrAM+ -N:M] Accuracy (%) after one-
(+BNT) using semi-structured 2:4 and 4:8 patterns.
shot pruning (+BNT), using unstructured sparsity
C.6 CRAM WITH INFREQUENT MASK UPDATES
As previously mentioned, when training image models with CrAM*, using sparse gradients of the intermediate model perturbation substantially improved both the accuracy of the dense final model, as well as after shot pruning. One hypothesis that would enable this approximation of the gradient would be that the masks of the compression perturbation change very little during training. We test the validity of this hypothesis by training version of CrAM* that only does infrequent mask updates. Namely, the masks obtained from the Top-K compression operator are kept fixed for a number of consecutive training iterations. In the case of CrAM-Multi, the masks for each sparsity level are changed each after 7 iterations with the corresponding sparsity target. We note that infrequent mask updates improve the practical training time of CrAM, as the Top-K operator can be skipped for most iterations. This brings the cost of an average CrAM* + iteration the same level as a SAM iteration, though we note that in theory, with specialized hardware, the computational cost of CrAM can be further decreased due the presence of sparse operators. We experiment using the same setup for training ImageNet on ResNer50 with CrAM+-Multi with global magnitude pruning at sparsity k (50%, 70%, 90%), and perform two separate runs, by varying the mask update frequency to T E (20, 100} iterations. As before, we also use BNT after pruning, on a subset of 1000 training samples, consisting of one example per class. The results presented in Table 15 show that using
20

---

Published as a conference paper at ICLR 2023
infrequent mask updates only has small impact the initial results; namely, observe that the results after one-shot pruning are slightly worse compared to the default ones, particularly at higher sparsity. In particular, the results for 80% sparsity have decreased the most, level which was not explicitly used during training. We also repeated the same experiment the CrAM-finetuning setup for ResNet50, with r = 100 iterations, and observed that results after one-shot pruning in fact improved slightly. This particularly encouraging, as using CrAM only for finetuning is a more attractive use-case, due to the reduced computational costs.
Frequency
Sparsity (%)
50
70
80
90
1
77.3
77.2
77.1
75.9
74.8
20
77.4
77.4
77.2
75.5
74.8
100
77.3
77.3
76.9
75.3
74.5
Table 15: (ImageNet/ResNet500) Validation accuracy (%) for the dense and sparse CrAM+-Mult models trained
with sparse perturbed gradients, using infrequent mask updates the global Top-K operator.
C.7 RESULTS ON QUANTIZATION
We have shown through previous experiments that CrAM can be successfully used with the Top- K operator to obtain models that preserve or improve the dense baseline's accuracy, while also being robust to post-training one-shot pruning, multiple sparsity levels. In this section we show encouraging evidence that CrAM can also be adapted to work with quantization. Specifically, we use CrAM where the compression operator C is the symmetric per-channel weight quantization bits (round to nearest integer). and finetune pretrained Torchivison ResNet18 and ResNet50 ImageNet models, for 10 epochs, using the same hyperparameters as in the previous experiments for sparsity.
ResNet18
ResNet50
Model
Dense
Bits
Dense
4 Bits
Bascline
69.8
66.2
76.1
74.1
SAM
70.5
67.1
76.9
74.8
CrAM+-4Biu
70.1
69.3
76.7
75.8
Table 16: [ImageNet] Validation acc. (%) the dense models, and after symmetric per-channel bits
quantization (+BNT).
C.8 COMPARISON WITH OTHER METHODS ON CIFAR-10
In this section we provide additional results accompanying those presented Section 4.3. Namely, we provide comparison between one-shot pruning CrAM+ vs. standard dense baselines (SGD, SAM), and we provide numbers before and after BNT for sparse models on VGG-16 and ResNet18.
In Table 17 we show the accuracy of the dense baseline, SAM and CrAM+-Multi, trained using sparsity levels sampled uniformly random in the range 30% 90% on ResNer20, before and after one-shot pruning at different sparsities. The results after one-shot pruning are presented after BNT random subset of 1000 train samples, over 100 batches. We note there small variations in the results after BNT, due to the choice of the random calibration set. These variations are small (+0..10.255) 1/0.2%) for CrAM* -Multi models, across all sparsity levels considered, but they larger for the one-shot pruned dense baselines at high sparsity (c.g. 80% and 90%). Moreover, the accuracy before BNT is still high for CrAM lower sparsity levels (e.g. 91.9% at 70% sparsity), but it degrades at high sparsity (e.g. 50.5% at 90% sparsity). We believe that this is only due to the NatchNorm statistics, which are adapted to the dense model during training, but they no longer reflect the distribution shift after weight pruning. This confirmed by the fact that 90% sparse models improve to over 90% test accuracy after only a few iterations of BNT, and are very robust to the choice of the calibration set.
Moreover, we show the extended results of CrAM+-K95 discussed in Section 4.3, before and after BNT, on ResNet18 and VGG-16. From Table 18 we can that one-shot pruning CrAM+-k95 without BNT preserves accuracy up to 80% sparsity, after which BNT is required to correct the NarchNorm statisties. Remarkably, the VGG-16 models at 97% and 98% sparsity have very low
21

---

Published as a conference paper at ICLR 2023
Sparsity
Model
Dense
50%
60%
70%
80%
90%
Baseline
93.0 # 0.1
92.2 = 0.0
91.0 = 0.3
88.0 0.2
78.0 = 1.1
45.8 3.0
SAM
93.5 + 0.1
92.8 0.2
92.4 0.0
90.7 0.3
85.2 -
54.6
CrAM -Multi
93.2 0.1
93.2
0.1
93.1 0.1
9.1
92.4
90.3 0.1
Table 17: (CIFAR- 10/ResNet20) Test acc. (%) the dense models, and after one-shot pruning (+BNT). The
baseline the model after SGD training. For all models we apply one-shot pruning at different sparsity (+BNT), but no additional retraining. Results are averaged across 3 runs from different seeds.
accuracy, which is improved greatly by BNT. Furthermore, also in this highly sparse regimes the accuracy very robust with respect the choice of the calibration set for BNT.
Sparsity 93%
Architecture
BNT
50%
80%
90%
95%
97%
98%
No 95.7+0.1
95.3=0.2
93.620.7
92.01144
89 89.8222.2
81.426.3
48.74:5.8
ResNet18
Yes
95.6+0.0
95.7+00.0
95.5+0.1
95.5+0.1
95.5++.1
95.2+0.0
94.5+003
No
94.240.1
93.910.2
86.721.6
48.5111.7
19.8
16.0=10.2 93,8+0.2
12.41:.11
VGG-16
Yes
94.2+0.1
94.2+0.1
94.0=0.1
94.0+0.2
94.1= 0.1
93.0=0.22
Table 18: (CIFAR-10 10) Test acc. (%) for the sparse models obtained with one-shot me--hoo--ruunng from CrAM -K95, before and after BNT. Results are averaged across runs from different seeds.
D LANGUAGE MODELS REPRODUCIBILITY AND HYPERPARAMETERR
To ease reproducibility of our results, we conduct all of our experiments with the popular open-source libraries: Transformers (Wolf al., 2020), and SparseML (Kurtz et al., 2020). We use the publicly available datasets via Lhoest et al. (2021), and focus on the BERT-base (Devlin et al., 2019) as is one of the most commonly used language models. It is composed of 12 identical transformer layers with 10M parameters. Following community standards, we prune all weights of the encoder part (85M) and report sparsities relative to this number.
General setup. Our SQuADv1.1 fine-tuning recipe with Adam, SAM and CrAM mostly follows the already established pretrained sert-basee bas.-aacaseeddddeeeee-e-- uncased (available for download at https://huggingffcc.oooooe uncased), sizh--siie-ee 16 6, max wew.---euueee-eenntt..
(Devlin et al., 2019; Wolf et al., 2020): start from the wtt.://eec.o//eebbbb.ee..eeeeeee
Adam, SAM, and CrAM optimization. For other hyper-parameteer each optimizer independently over the following values: learning-rate (3c-5, 5c 5, 5):5]: num-traii-eppo
conduct grid search for
E {2.3} for SAM and CrAM, and num-train-epochh. (2, 4, 6) for Adam (we allow 2x more epochs for fairness to SAM and CrAM); {0.0, 0.1, 0.2). We freeze the embedding layer in all experiments. To determine the value of the hyperparameter p, we performed grid search over values in the range le-4 to le-1. For each optimizer we pick the set of hyperparameters that produces the best results after one-shot magnitude pruning 50% sparsity, and they are as follows:
Adam:
label- Sabel-smonn wtt---sstthiin--atiic.... nun-train-epochh-22
Learning-rate--e-3
SAM: labell--moochhhng-rratt000...0.0
num-train-epoohhss
learning-rate=-e-b
CrAM (all CrAM rums use the same set of hyperparameterss: nun-traii-epochst
learning- 8,-5, label smoothing ratio=0, .2, 005.00
At each CrAM optimization step we apply Top-K (i.e. magnitude) sparsification over all layers uniformly.
One-shot pruning.
We apply one-shot pruning with two different pruners: magnitude and oBERT. For one-shot magnitude pruning impose uniform sparsity distribution over all layers. For one-shot oBERT pruning we adopt the suggested set of syyper-parameter by authors, which we briefly describe here for completeness: 1024 gradients, dampening 1e-7, block-size 50, recomputations, global
22

---

Published as a conference paper at ICLR 2023
sparsity distribution over all layers. For more details please refer to the oBERT paper (Kurtic et al., 2022).
Sparse fine-tuning of one-shot pruned models. We fine-tune one-shot oBERT-pruned models with the fixed sparsity mask and Adam optimizer. identify the best set of hyperparameters for fine-tuning of the sparse model, we conduct grid search over the following parameters: learning-Bn (3e-5, 5, 8,-5, 1e-4], num-train-epor -eochss [1,2], label smoothing E {0.0, 0.2]. warmup- ratio E (..0.0.13. We freeze the embedding layer and employ early-
stopping technique to prevent overfitting.
Sparsity
Model
4-cures, batchesiz==1 Throughput (itemsisce)
16-cures, batch-size=128 Throughput (items/scc)
Model
Dense
50%
60%
70%
80%
Speed-up
Speed-up
p(Adam) 0.0
88.7
88.3
88.1
86.8
82.5
Dense
4.0
1.0x
14.2
1.0x
P(Adam)
p(Adam) = 0.3 87.5
87.5
87.2
86.5
83.6
50% sparse
4.5
1.1x
18.0
1.3x
plAdam)
60%
5.2
1.3x
21.8
1.5x
p(Adam) = 0.8 87.0
87.1
86.8
85.2
79.1
80%
70% sparse
8.0
6.3
2.0%
1.6%
26.0
31.9
1.8x
2.3x
Table (SQuADv1.1/BBL))
Validation score models optimized with CrAM* where at each step with probability p(Adam) the standard Adam step applied instead of the FEAM*-Mulle step.
Speed-ups of prumed BERT-hase models relative to the dense model, benchmarked with the sparsity-awan inference engine DeepSparse (version 1.0.2) (Kurtz al., 2020; Neu- ralMagic, 2021) in two different scenarios AMD EPYC 7702 64-Core Processor.
Table 20:
Speed-ups of pruned BERT-base models. In Table 20 present speed-ups of pruned models in the sparsity- CPU inference engine DeepSparse (Kurtz et al., 2020; NeuralMagic, 2021) (version 1.0.2). We consider two different scenarios and report speed-ups relative to the dense model benchmarked in the same environment.
Robustness to one-shot pruning. In Table 19 we present results for runs where CrAM is not used at every optimization step, and demonstrate that even in this setup the obtained models are still more robust to pruning compared to the models fully fine-tuned either with Adam or SAM optimizers reported in Table
23


Reviews:
[{"review_id": "lWZwJ7goz9A", "reviewer": "Reviewer_VFwD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method to train a neural network, CrAM (and CrAM+), which results in a trained network that can be compressed (by pruning or quantization) while maintaining accuracy. The method is developed based on plugging compression in the SAM’s inner maximization step; the perturbed parameters are obtained by taking the ascent step (for maximization) and applying some compression on them. The authors attempt to provide theoretical justification for such an algorithm based on robust optimization literature. The authors present details on some extensions including batch norm tuning or SAM+multi, and further, experimental results on Imagenet as well as SQuAD for different comparisons.\n", "summary_of_the_review": "The paper explores an interesting combination of SAM and sparsity, but importantly lots of parts in the paper is not quite tight and needs further exploration.", "strength_and_weaknesses": "Strength\n* provide a range of experiments making comparisons on different scenarios, baselines, and domains\n* develop theory based on the mini max framework, by which compression interpreted as perturbation can be effectively performed as robust optimisation\n* N:M sparsity pattern is nice in practice adding it is definitely plus\n\nWeakness\n* CrAM, the basic version of the method, does not work well, whereas the provided theoretical justification is supposed to indicate CrAM should be enough, not CrAM+. The theoretical justification itself is only weakly justifying the method. For instance treating the mask operator and straight-through estimator the way the author does is very weakly justified.\n* overclaim for generality of the method. present as if it works for any compression methods, but mostly focuses on top-k based on magnitude of parameters. a quantization scheme is provided in the appendix but not enough to claim for compression in general.\n* far below the sparsity level compared to existing methods. present some effectiveness up to 90%, but considering the sparsity literature it is far below. for 95% sparsity it is a level at which the method does not work.\n* coated with various experiments, yet fails to concentrate on the central part. Although they conduct many interesting experiments, they are often quite partial point-by-point based and far from being comprehensive.\n* The only effective result is that CrAM/CrAM+ can give a network that is one-shot prunable. However, this is not considered enough given the fact that this method requires additional hyperparameter tuning such as the choice of sparsity levels during training, that there exist other one-shot training (and pruning afterwards) methods without requiring SAM framework or double the forward-backward computations.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "a_4Ly7-zTO9", "reviewer": "Reviewer_fNyD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new compression-aware minimizer dubbed CrAM that modifies the optimization step in a principled way, which results in dense models trained via CrAM can be compressible post-training, in a single step, without significant accuracy loss. Experimental analysis on image classification and language modelling shows that CrAM models can be pruned one-shot at a wide range of sparsity levels, while resulting in sparse models that are competitive with existing gradual pruning methods. Also, CrAM produces sparse models which perform well for transfer learning and works for semi-structured pruning patterns.", "summary_of_the_review": "The idea of CrAM is overall novel and supported by theoretical analysis. And it is the first carry over the sharpness-aware idea to the task of obtaining compressible models. Although the experimental results are not significant compared with the closed works, the paper is likely to have modest impact within the subfield. Hence, I recommend for acceptance.", "strength_and_weaknesses": "Strength.\n1) CrAM is simple to implement as part of a regular training loop and has a single scaling hyper-parameter.\n2) CrAM enables training a dense model, which can later be compressed to different target levels, with minimal or no recalibration.\n3) To derive the CrAM update, the paper provides the theoretical analysis to justify the choices made in designing the training method.\n4) Sufficient Experiments demonstrate the wide range of application, including one-shot pruning, quantization, semi-structured N:M sparsity patterns and transfer learning.\n\nWeaknesses.\n1) For most of the comparisons, the paper choose the SAM as the target, such as Table 2,3,4 etc, which is not suitable because it trains models that potentially converge to flatter minima for better generalization rather than compressible models. It is more appropriate to compare with compression-aware C-SAM.\n2) Compared with the gradual method DPF, CrAM performs worse at high sparsity, such as 90% and 95%. The paper explains that CrAM was not explicitly trained for such high sparsity. However, in the table 7, compared with C-SAM, CrAM still not perform very well at 80% and 90% sparsity with the same experimental configurations. It is better to clarify more about the phenomenon.\n3) In the paper, the computation cost is not specific, such as twice as many forward-backward passes or the cost of BNT is minimal. It is better to show the cost explicitly in the comparison tables, such as the training time for single iteration.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TDvIsTVivX", "reviewer": "Reviewer_6AwX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a specialized optimizer for optimizing the DNN to have good compression ability, i.e. the optimized model can be pruned/quantized to smaller models which have good accuracy.\n\nThe paper is based on a formulation related to the sharpness aware minimization, by adding a compression related constraint projection on the inner \"sharpness-aware\" gradient ascent step.\n\nExperiments result show that the proposed method can train a model which not only performs good on dense model but also on sparse models.", "summary_of_the_review": "The paper provides an intact derivation of extending SAM optimizer to the context of constrained optimization and especially model compression.\n\nThe experiments validate the proposed algorithm on different sparsity ratios and also dense model.\n\nFrom the updating rule, the proposed algorithm is the almost the same as using SAM on once-for-all network, so the novelty of this paper is not very significant.", "strength_and_weaknesses": "Strength\n- The proposed method can train a single set of model weights which performs well on multiple compression rate, i.e. sparsity ratios.\n- The formulation is a good extension of sharpness-aware optimization to the direction of optimization with constraints.\n- The authors also considered and discussed the computation complexity of the algorithm.\n- The experiments result show that the dense model results can be even better compared to training a standalone dense model.\n\nWeaknesses\n- Although the paper derives the method from the view of optimizer, the updating rule is almost the same as using SAM optimizer on supernet/one-shot NAS methods such as once-for-all network[1], bignas[2]. The novelty of this method is not very significant from this point of view.\n- It's better to have some discussion and comparison to the method which directly combines SAM with once-for-all network, e.g., is there any difference in the actual updating rule and how much difference in the results.\n\n[1] Cai, Han, et al. \"Once-for-all: Train one network and specialize it for efficient deployment.\" arXiv preprint arXiv:1908.09791 (2019).\n\n[2] Yu, Jiahui, et al. \"Bignas: Scaling up neural architecture search with big single-stage models.\" European Conference on Computer Vision. Springer, Cham, 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q_sEPXISI_", "reviewer": "Reviewer_FqjP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to have a compression-aware minimizer that can compress dense DNN into a sparse one, and most importantly, without much performance drop. Inspired by the sharpness-aware minimizer, the authors leverage the flat minima to find the weights' coordinates that are stable to the perturbation. With the price of doubled computational overhead, the proposed compression-aware minimizer can prune dense models in a one-shot to 70-80% sparsity with a small drop in accuracy. The authors also give theoretical proof to guarantee the convergence of the proposed CrAM. The experiments on the ImageNet dataset verify the effectiveness of the proposed CrAM. The paper is well-organized and easy to follow. \n", "summary_of_the_review": "\nTo conclude, this paper is a good research paper. It is well-written, and the proposed method is sufficient in derivation and effectiveness. It would be better if some more experiments could be added. \n", "strength_and_weaknesses": "Pros:\n1. The paper is well-written and well-organized. It is easy to follow and read. \n2. It is inspiring to leverage the flat minima for the model pruning task. This paper encourages the application of sharpness-aware training.\n3. The methodology and appendix E are good for readers. The derivation is sufficient; the theoretical proof makes the paper complete. \nCons:\n1. It would be better if the authors could have more experiments—for example, some parameter studies; some model pruning baselines. (One should be enough for the rebuttal period)  \n2. Appendix E is important; it would be better not to be placed the last part of the appendix. \n3. There are too many variants of CrAM in the experiments. It is better to give a universal parameters set of CrAM. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster