Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
Authors: Elias Frantar
Dan Alistarh
Venue: NeurIPS 2022 Accept (2022)
Keywords: compression
pruning
quantization
post-training
efficiency

Abstract: We consider the problem of model compression for deep neural networks (DNNs) in the challenging one-shot/post-training setting, in which we are given an accurate trained model, and must compress it without any retraining, based only on a small amount of calibration input data. This problem has become popular in view of the emerging software and hardware support for executing models compressed via pruning and/or quantization with speedup, and well-performing solutions have been proposed independently for both compression approaches.
In this paper, we introduce a new compression framework which covers both weight pruning and quantization in a unified setting, is time- and space-efficient, and considerably improves upon the practical performance of existing post-training methods. At the technical level, our approach is based on an exact and efficient realization of the classical Optimal Brain Surgeon (OBS) framework of [LeCun, Denker, and Solla, 1990] extended to also cover weight quantization at the scale of modern DNNs. From the practical perspective, our experimental results show that it can improve significantly upon the compression-accuracy trade-offs of existing post-training methods, and that it can enable the accurate compound application of both pruning and quantization in a post-training setting.

Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
Elias Frantar
Sidak Pal Singh ETH Zurich
Dan Alistarh IST Austria & Neural Magic dan. alistarh@ist.aa ac
IST Austria
elias. www.faaantrrss.aat. ac
sidak. singh@inf ethz. ch
Abstract
We consider the problem of model compression for deep neural networks (DNNs) in the challenging one-shho/postttaaining setting, in which we are given an accurate trained model, and must compress it without any retraining, based only small amount of calibration input data. This problem has become popular view the emerging software and hardware support for executing models compressed via pruning and/or quantization with speedup, and well well-performing solutions have been proposed independently for both compression approaches. In this paper, we introduce a new compression framework which covers both weight pruning and quantization unified setting, time- and space-efficient, and considerably improves upon the practical performance of existing post-training methods. At the technical level, our approach is based classical Optimal Brain Surgeon (OBS) framework of [LeCun, Denker, and Solla, 1990] extended also cover weight quantization at the scale of modern DNNs. From the practical perspective, our experimental results show that it can improve significantly upon the compression-accural trade-offs of existing post-training methods, and that it can enable the accurate compound application of both pruning and quantization a post-training setting.
exact and efficient realization of the
Introduction
The impressive recent progress of deep learning for solving challenging tasks across several domains has been accompanied by a significant increase parameter counts and computational costs for executing such models. A natural consequence has been a growing effort to reduce such costs via model compression, and the two most popular approaches for model compression are pruming- removing neural network weights by setting them to zero-and quantization, reducing the precision at which neural network weights and activations are stored and manipulated. Hundreds of such pruning and quantization approaches have been proposed and analyzed [17] [T], with the general goal of obtaining efficient variants of deep neural nets (DNNs) which would preserve accuracy while maximizing compression. Despite impressive progress, compression is still laborious process: the pruning and quantization stages are often done independently, and recovering model accuracy after compression often requires partial even full retraining of the compressed model.
An alternative but challenging scenario is the post-training compression setup 31 which we given a trained but uncompressed model. together with small amount of calibration data, and must produce an accurate compressed model one shot, i.c. a single compression step, without retraining, and with limited computational costs. This is motivated by practical scenarios such as the MLPerf Inference Benchmark [37], and the setting we focus on in this paper.
25 in
*Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Compression via weight pruning started with seminal work by LeCun et al. [23], complemented by Hassibi and Stork [T3. who proposed a mathematical framework called the Optimal Brain Surgeon (OBS), for choosing the set of weights to remove from a trained neural network, by leveraging second-
order information. (We describe their approach in Section 3 Recent advances, e.g. [6
10]
showed that OBS can lead to state-of-the-art compression at DNN scale, by introducing numerical methods which can approximate the second-order information needed by OBS at the massive parameter counts of modern models. However, these approaches do not apply to the post-training setting, as they require gradual pruning, as well as significant retraining, to recover good accuracy.
An alternative approach, which is standard in the context of post-training compression, has been to break the compression task into layer layer-wise sub-pre problems, identifying a compressed weight approxi- mation for each layer, given sub-sample of the layer's inputs and outputs based on calibration data. This line of work, e-g. [4 31 19, introduced elegant solvers for the resulting layer-wise weight quantization problem, which achieve state-of-the-art results for post-training quantization. Recently, AdaPrune [8] showed that this approach can also be effective for post-training weight pruning.
In this context, a natural question is whether existing approaches for pruning and quantization can be unified in order cover both types of compression in the post-training setting, thus making DNN compression simpler and, hopefully, more accurate. This question is also of practical importance, since both GPU and CPU platforms now jointly support sparse and quantized formats [30 35 and, as we illustrate experimentally, the resulting models could be executed with compound speedups.
Contribution, In this paper, we provide a mathematical framework for compression via pruning or quantization, which leads to state-of-the-art accuracy-versee sennraayyveeruu-compressssoooo trade-offs in the challenging post-training compression setup. Our framework starts from the layer-wise compression problem described above, by which the global compression task, defined either for pruning or quantization, is first split into layer-wise sub-problems, based on the layer behavior on the calibration data. Specifically, given a layer E defined by weights We, and layer inputs X₁, we aim to find a compressed version of the weights We which minimizes the output difference relative to the uncompressed layer, measured via the squared error between the original and compressed layer, acting on the sample input |[WXe - W.X under a fixed compression constraint on We.
Although solving this problem optimally for spansity or quantization constraints is NP-hard 2 31, is a key step in all state-of the-hc post-training compression methods, both for pruning 18 9 and
for quantization [31 18 24 Once this is solved per layer, solution the global problem can be
obtained by combining layer-wise solutions, which is handy especially for non-uniform compression, e.g. [15 Thus, several approximations for this problem have been proposed [1 19 18]
We show that there still is significant room for improvement when solving the layer-wise compression problem. Roughly, our approach is to specialize the OBS framework to the squared error formulation above: in this case, the framework can in theory produce an exact greedy solution, but a direct implementation would have infeasible ((d*) computational cost, where d is the layer dimension. Our main technical contribution is a series of algorithms which reduce this computational cost, without any approximations, to Old dew)) where deal is the column dimension of the weight matrix. In practice, these improvements are significant enough to allow us to implement the exact OBS greedy solution, which prunes one weight at time, and updates all remaining weights after each step, the scale of modern DNNs with tens of millions of parameters, within reasonable time, on a single GPU. We provide efficient implementations of our methods at witttt::////t.. http://ggiuhuuub.ommi.oaaeee Dww..////S/S/////////........ /BCC
In turn, this algorithmic development allows us to apply the OBS approach to quantization. The resulting algorithm, called the Optimal Brain Quantizer (OBQ), quantizes weights iteratively one-at- a-tim a-time, depending on their impact on the loss increase, after which it applics a dlosed-form update to the remaining unquantized weights, further reducing the loss. This solves the two problems efficiently, and in a unified manner-we we call the unified framework the Optimal Brain Compressor (OBC).
We apply OBC to standard tasks and models covering image classification, object detection, and language modelling applications. We first show that our framework yields significantly better solutions for the layer-wise compression problem, which leads to higher-accuracy end-to-end compressed models for both pruning and quantization, relative to the corresponding state- of-the-art techniques, often by significant margins. Second, we show that our pruning and quantization approaches can be compounded, with surprisingly strong results: we obtain a 12x reduction in theoretical operations with 2% accuracy drop for GPU-supported compound compression [30].
Experimental Results.
2

---

and a 4x speedup in actual runtime with only 1% accuracy drop for a CPU-based sparsity-award runtime [35] Together, these results suggest for the first time that post-training compression can be competitive with full retraining.
2 Related Work
Optimal Brain Surgeon (OBS). The classic OBS framework 23 13 was originally applied to networks with hundreds of weights; more recently, methods such as WoodFisher [39] rendered the approach computationally feasible for DNNs by using a block-diagonal Fisher approximation of the Hessian, while follow-up methods introduced more efficient and general algorithms for handling the inverse Fisher matrix, or customize this approximation to specific model families (21). Earlier work
called Layer-wise OBS (L-OBS) [6] was inspired by the K-FAC approximation (29 12 L-OBS
approximates the OBS framework not for the global objective, but for a quadratic per-layer loss, while also pruning all weights based on single Hessian computation. At a high level, our approach is similar, in that we apply OBS layer-wise; however, we apply OBS exactly, that is, pruning one weight at a time, and exactly recomputing the Hessian after every pruning step. This is made computationally tractable by several new algorithmic ideas, and yields significantly improved results relative to L-OBS. This prior work on pruning considered settings with extensive finetuning. By contrast, we will focus on the post-training setting, where only a small amount of calibration data is available.
Post- Wust-Trainin Quantization. This setting has been primarily considered for quantization, and most state-of-the-arr methods work by performing layer-wise compression. Specifically, BitSplit O optimizes the quantized weights bit by bit, while AdaRound [31] finds a weight rounding policy through gradient based optimization with an annealed penalty term that encourages weights to move towards points on the quantization grid. AdaQuant [19] relaxes the AdaRound constraint, allowing weights to change during quantization-ave -wwre optimization, via straight-through estimation [33]. BRECQ (24) suggested that accuracy can be improved further by integrating second-order information into the layer-wise losses and by jointly optimizing hand-crafted blocks of related layers.
A key step of AdaRound, AdaQuant and BRECQ is to quantize layers incrementally. in sequential order, so that errors accumulated in earlier layers can be compensated by weight adjustments in later ones. This sigmificantly improves performance, but reduces flexibility, as the entire process may need to be re-done whenever we wish change compression parameters of one layer. We instead target independent compression of each layer, allowing the end model to be simply "stitched" together from layer-wise results. Despite operating independently on each layer, we find that, after correcting basic statistics such as batchnorm, our method performs on par to sequential ones for uniform quantization.
Post-Training Sparsification. The layer-wise approach was shown to also be effective for post- training pruning by AdaPrune [8), which pruned weights to the GPU-supported N:M pattern [45] AdaPrune first drops parameters according to their magnitude [46] and then reoptimizes the remaining weights to reconstruct the pre-compression calibration set output. This is similar to [16] g which also perform layer-wise reoptimization of the remaining weights. Follow-up work [10] noted that the results of AdaPrune can be improved further by performing more frequent pruning/optimizann Our algorithm pushes this idea to the limit, performing full reoptimization after every single pruned weight, while remaining computationally tractable. We further use a more sophisticated weight selection metric which incorporates second-orde information. Finally, [00 also introduces global AdaPrane, a more expensive global optimization step applied on top of the layer-wise AdaPrune results, which can bring additional accuracy gains. This can also be applied to our pruned models.
steps.
Non-Uniform Compression. An orthogonal practical question is how to compress different layers to maximize accuracy under a given resource constraint, such as latency or energy consumption. Existing methods can be roughly categorized into search-based and boseer-based approaches. The former, e.g. AMC 15 or HAQ AT, search for a layer-wise compression policy directly via, for example, reinforcement learning or genetic programming 45 whereas the latter, e.g. HAWQv3 44] or AdaQuant [9, construct a relaxed version of the overall problem that is then solved exactly. We focus here on solver-based approaches, as they can rapidly adapt to different scenarios when combined with accurate independent layer-wise compression schemes; however, our techniques could be of interest for search-based methods as well. Concretely, we use the problem formulation of AdaQuant [19 to which we apply the DP algorithm of SPDY [0 to achieve fast solving times even with large number of possible choices per layer.
3

---

3 Problem Definition and Background
The Layerwise Compression Problem. Following prior work on post-training compression, c.g. [3] 19I we define the problem as follows. Mathematically, we model a layer i as a function fe(X,, W2) acting on inputs Xt, parametrized by weights We. The goal of layer-wise compression is to find a "compressed" version of We that performs as similarly as possible to the original weights. More formally, the compressed weights We should minimize the expected layer output change as measured by some loss L while at the same time satisfying a generic compression constraint, which we denote by c(W) > C, which will be customized depending on the compression type:
argming, Ex, (X,, We), feee,, We)) subject C(We)>
(1)
The expectation over the layer inputs X₃ usually approximated by taking the mean over a small set of N input samples. This low-data setting is one of the primary applications of layer-wise compression. Further, most works [42 [3 19] focus on compressing linear and convolutional layers, which can be unfolded into linear ones, as these are prevalent in practice, and use the squared loss to measure the approximation error. This definition of the loss can be motivated, via a sequence of approximations, from second-order information: please see R for precise derivation. Furthermore, this approximation approach has been shown to work well in many applications [31 19 9.
We follow these conventions as well, and work with the specific layer-wise compression problem stated formally below, where the weights We are dow deal matrix (for conv-layers deal corresponds to the total number of weights in a single filter), and the input X has dimensions deal X N.
argmin w. W,, 3 S.L. C(W)) C.
(2)
The Optimal Brain Surgeon (OBS) Framework. The OBS framework 23 13 considers the problem of accurately pruning a trained dense neural network. It starts from the Taylor approximation at the given point (assumed to have negligible gradient), and provides explicit formulas for the optimal single weight to remove, as well as the optimal update of the remaining weights which would compensate for the removal. More precisely, let H denote the Hessian matrix of the loss at the given (dense) model. Then the weight to prune wpp which incurs the minimal increase in loss and the corresponding update of the remaining weights 6, can be calculated follows:
w H
Wp
Upp = argmin my
6p
H
(3)
H
pp
pp
where [H- denotes the pth diagonal entry of the inverse Hessian, and H is its pth column.
OBS for Layer-Wise Pruning. We will now instantiate this framework for the layer-wist pruning problem, defined above. First, the loss in equation 2 is quadratic and since our starting point is given by the dense weights achieving the minimal loss of O, the assumptions of the OBS framework fully met, meaning that its formulas are exact for this specific problem formulation. Thus, iterating the OBS framework to remove one weight at time would yield an exact greedy solution for the layer-wise pruning problem, as it takes the (locally) optimal decision at each step. While this greedy approach does not guarantee convergence to global optimum, such approaches can be very effective for dealing with problem instances that are too large to be handled by exact methods.
4 An Optimal Greedy Solver for Sparsity
The obvious challenge is that applying the OBS framework in its true form, i.e. pruning single weight at a time using the exact formulas in (3). is computationally very demanding. The Hessian H is d x d matrix where d drow deut= which is already expensive to store and compute with. Additionally, this matrix needs to be updated and inverted at each of the (dd) steps with a computational complexity of (dd²). Clearly, an O(d*) total runtime too inefficient for pruning most layers of modern neural networks, as dl is usually 2 10t or even 2 10 for several layers. However, as we will now show, it is actually possible to reduce the overall costs of this process to O(dro d&a) time and (ddaa) memory, making it efficient enough to prune e.g. all layers of a medium-sized model such as ResNet50 in bit more than one hour on single NVIDIA RTX 3090 GPU. We emphasize that the techniques we we do not rely on any approximations.
introduce are exact; unlike prior work 6 39

---

The ExactOBS Algorithm. In the following, we introduce our efficient instantiation of the OBS framework, for the layer-wise compression problem. which we call ExactOBS, in step-by-step fashion. We start by rewriting the matrix squared error in (2) as the sum of the squared errors for each row in the weight matrix. As we are always dealing with fixed layer f, we drop the subscript to simplify notation. The objective is then equívalent to Eis1 | X
This way of writing the error makes it clear that removing a single weight [Wi; only affects the error of the corresponding output row YI:- = Wi.X. Hence, there is no Hessian interaction between different rows and so it suffices to work only with the individual deol deol Hessians corresponding to each of the drow rows. Further, as the dense layer output Y = WX is fixed, the objective for each row has standard least squares form and its Hessian is given by H 2XX
Although this observation already reduces computational complexity, two key challenges remain: (a) applying OBS to each row still costs Oddeal &&al time, which is too slow for large layers, and (b) we need fast access to the Hessian inverses of all drw rows, since we want to prune the minimum score weight of the whole matrix rather than just per row in each step. In particular, (b) requires (d (al)) GPU memory, which likely to infeasible.
Step 1: Handling a Single Row. We first describe how to efficiently prune weights from a single row with deal parameters. For simplicity, we denote such row by with corresponding Hessian H. The full algorithm for this procedure is given in Algorithm [ in the following, we provide a detailed description. The key idea is to avoid having to do the full ( (NNd calculation and (d inversion of H in each step. The former is easy, as the weights themselves do not enter the calculation of H 2XXT, and the Hessian for the weights with pruning mask M denoted by HAs is thus simply comprised of the corresponding rows and columns in the fully dense version H. Hence, we only have compute (which is actually the same for all rows) once, from which we can then extract the rows and columns corresponding to M as needed.
Critically, this trick is not applicable to the inverse, as (HM) - # M.. However, using the fact that the removal of one parameter p simply drops the corresponding row and column from H, we can actually update the inverse remove parameter p directly using single step of Gaussian elimination, with cost (dP..)) The following result, whose proof is in the Appendix, formalizes this.
Lemma (Row& & Column Removal). Given an invertible deat X deal matrix H and its inverse H-1 we want to efficiently compute the inverse of H with row and column p removed, which we denote by H p- This can be accomplished through the following formula:
H
H
(4)
which corresponds to performing Gaussian elimination of row and column in H followed by dropping them completely. This has (d time complexity.
The resulting pseudocode is shown in Algorithm I where we avoid constantly resizing H (and correspondingly changing indices) by utilizing the fact that row and column have no effect on any future calculations after they have been eliminated by Lemma [ as they are 0 (and the non-zero diagonal element is never accessed again). One check that this algorithm applies OBS single row W with per-step cost O (d and thus O d&a)) overall time for pruning weights.
Algorithm Prune S deal weights from row with inverse Hessian H (2XXT) )- according to OBS Oik da time.
Less compute:
elements
Pruning Traces
Weight Rows
Loss Changes
Global Mask
M {1, dcol
Weight
Weight
for i = L,..., k do
p argmin pe
PEM H H- 1pp W
w < w H
Wpp
by
H-1 1-1_ <- H
H
p.
memory: resolve not pruned weights
M + M {p}
Figure/ Efficient global OBS using the row- wise results.
end for
5

---

Step 2: Jointly Considering All Rows. Applying the OBS framework to the full weight matrix W rather than just to each row independently requires fast access to all dnw row-wise inverse Hessians, in order to select the weight with the smallest overall pruning score in each step. However, storing drw matrices of size deat x deal each in GPU memory can too expensive; while it would be possible to offlood some Hessians to main memory, this could result in large number of expensive memory transfers. However, since there is no Hessian interaction between rows, the final compressed weights of each row only depend on the total number of parameters that were pruned in it. Similarly, the change in loss incurred by pruning some weight only depends on the previously pruned weights in the same row, which also means that the order in which weights are pruned in each row is fixed.
The consequence of these insights is that we can process each row independently, pruning all weights in order while always recording the corresponding change in loss òC, = w / 1 -pp- At the end, we know SCp for all d weights and can then simply determine the global mask that would be chosen by OBS on the full matrix by selecting the weights with the lowest values in order, requiring only ((d) extra memory. We note that once the per-row masks M; are known, we can directly solve for the optimal update of the remaining weights via the corresponding group OBS formula [21] 8M.- = H.. (H "=M))"' W.. This will be considerably faster in practice than simply rerunning the iterative pruning process in Algorithm [ Alternatively, if enough CPU memory is available, one can keep the full pruming trace of each row, that is, the full weight vector after every individual pruning step, in CPU memory and ultimately simply reload the entries corresponding to the global mask. This requires (d deal) extra CPU memory but avoids second computation pass to reconstruct the not pruned weights and will therefore be faster. Figure | visualizes both options just discussed.
Implementation Details. In practice, the matrix H might not always be invertible for reasons such as using too few data samples or dead/ linearly dependent inputs. The former can usually be addressed by extending the calibration dataset with augmentations (additional augmented samples only need to be accumulated into the Hessian once and are thus very cheap to include) and the latter can be prevented by adding a small diagonal dampening term to the Hessian before inverting it. Second, direct GPU implementation of Algorithm |
will perform a large number of small CUDA calls, which can be expensive. This overhead can be removed by using batch operations to process multiple matrix rows simultaneously-foo more details please see our sample implementation. Finally, when applied to an already sparse weight matrix, the complexity of our algorithm can scale cubicly with the row-density by working with a dense version of the weights / Hessians consisting only of the non-zero elements and mapping the pruning result back at the end.
N:M Sparsity. Our method can be easily extended to various forms of semi-strucfured sparsity. This includes, for example, the N:M sparsity pattern 45). which enforces exactly N non-zero values in each block of M consecutive weights, and is becoming popular due support on newer NVIDIA hardware [30] Adapting our algorithm to this pattern requires only one simple change: instead of selecting the weight with the smallest change in loss, we select the weight with the smallest change in loss that is in a block with < N pruned weights. We note that all rows have exactly the same sparsity 1 --N/M in the N:M pattern and so we can terminate per-row pruning as soon as this target sparsity value is reached. For the same reason, there is no need for the global mask selection step described earlier. Thus, our method will be even more efficient in this case.
Block-Sparsity. Another practically relevant pruning pattern, particularly in the context of CPU acceleration 7 (2)) is block-pruning, where zeros appear only in consecutive blocks of size c, which is typically a small number like 4 or 8. We follow recent work (21 that extends the OBS framework to pruning small groups of connected weights in order to account for the correlation between them, using the following formulas for the target block and weight update, respectively:
WP argmin w ((((H))))) 8p=-H ( 'WP.
(5)
where denotes the set of indices corresponding to one block. Algorithm I can easily be adapted
to operate on blocks using the above equations and applying the update of H via Lemma successively for all p E P. Although there are now only c per row, each update of H also takes O(c- dea) time and so the overall asymptotic runtime stays the same. Additional practical overhead only comes from the extra Oc22.d, dda.) terms that are the result of computing and multiplying with the x matrices ((H-))p)

---

5 The Optimal Brain Quantizer (OBQ)
Although the classical OBS framework 23 13 has inspired long line of work pruning methods for DNNs 33 To 27]. so far it has not been used for quantization. We now show that our results from the previous section can in fact be extended to quantization in an effective and accurate way, via a method which we call the Optimal Brain Quantizer (OBQ), in the spirit of 22 13
The Quantization Order and Update Derivations. Under the standard assumption that the gradient at the current point w is negligible, the OBS formulas for the optimal weight to be pruned up and the corresponding update 6, can be derived by writing the locally quadratic problem under the constraint that element p of 6p is equal to pp which means that upp is zero after applying the update to w. This problem has the following Lagrangian:
L(6p. A) Ho + Xe - (-wp))
(6)
where H denotes the Hessian at w and Op is the pth canonical basis vector. The optimal solution is then derived by first finding the optimal solution to 6p via setting the derivative aL/06, to zero and then substituting this solution back into L and solving for X: please see e.g. [I3 39 for examples.
Assume setting in which we are looking to quantize the weights in layer on a fixed grid of width while minimizing the loss. To map OBS to quantized projection, we can set the target of the Lagrangian constraint in O6) to (quant(w) wp) - wp). where (uant(u:p) is the weight rounding given by quantization; then wp quant( (pp)) after the update.
Assuming we wish to quantize weights iteratively, me-at---timm,, can derive formulas for the "opti- mal" weight to quantize at a step, in terms of minimizing the loss increase, and for the corresponding optimal update to the unquantized weights, in similar fashion as discussed above:
(quant( (pp)) wp) )
Wp-
quantt wp) H.
Wp- = argmin upp
6p
(7)
ppp
In fact, since wp is a constant during all derivations, we can canjust substitute it with (quant(w;, wpp) in the final result. We note that the resulting formulas are a generalization of standard OBS for pruning, if quant(-) always "quantizes" weight to O, then we recover the original form.
Quantizing Full Layers. At first glance, OBQ might appear curious since one usually quantizes all weights in a layer, leaving no more weights to update. At the same time, the weight selection metric influences only the quantization order, but not the quantization value. However, this view changes when considering OBQ in the context of our efficient one weight-at-a-time pruning algorithm described in the previous section. Specifically, using OBQ, we can greedily quantize the currently "easiest" weight by the above metric, and then adjust all the remaining unquantized weights to compensate for this loss of precision, thus changing their value. We then choose the next weight to quantize, and so on. This can result in quantization assignments that are different from the ones that would have been chosen by rounding initially, and in better overall quantization results. Concretely, to realize this, we can plug into Algorithm I in order to iteratively quantize weights for given layer, leading to the similar Algorithm in the Appendix, thus essentially unifying pruning and quantization.
Quantization Outliers. One practical issue with this greedy scheme can occur especially when applied to quantization grids that permit some outliers in order to achieve lower error on the majority of weights, which are currently standard 4 34]. Since these outliers can have high quantization error, they will usually be quantized last, when there are only few other unquantized weights available that may be adjusted to compensate for the large error incurred by quantizing the outliers. This effect can become worse when some weights are pushed even further outside the grid by intermediate updates. We prevent this with simple but effective heuristic: we quantize outliers, e.g. weights with a quantization error > A/2 where A is the distance between quantized values, as soon as they appear (which typically happens only few times per layer). With this heuristic, OBQ yields a highly effective layer-wise quantization scheme, as our experiments in the next scction demonstrate. Finally, we note that the OBQ version of the techniques discussed in Section Thas all the same runtime and memory characteristics (barring the global step in Figure I] which is unnecessary for quantization).
Experiments
Objectives, Models & Datasets. To demonstrate the effectiveness and flexibility of our method, we
7

---

consider several different standard post-training compression scenarios [1 19 18] We begin with settings where only a single type of compression applied: concretely, we consider unstructured pruning for given FL.OP targets, global 2:4 and 4:8 pruning, as well as uniform weight quantization. Additionally, we also study two practical tasks that feature joint pruning and quantization: a GPU scenario where quantization and N:M pruning are combined, as well as CPU scenario combining
quantization and block pruning. We work with variants of the following models and tasks: ResNet [14]
for image classification on Imagenet (38). YOLOv5 (20] for object detection on COCO (26) and
BERT [5] for question answering on SQuAD 36. Our smaller BERT models denoted by BERT3 and
BERT6 correspond to the smaller 3 and 6 layer variants of BERT-base respectively. trained by (21.
The Appendix contains additional experiments as well as runtime information of our algorithms.
Experimental Setup. All of our calibration datasets consist of 1024 random training samples. For ImageNet, where we use roughly 0.1% of the training data, we additionally apply standard flipping and cropping augmentations to artificially increase the size of this dataset by 10x; other tasks do not use any augmentations. While the effect of augmentations is typically minor, they are very cheap to include for our method. For ResNet models, batchnorm statistics are reset using 100 batches of 128 samples from the calibration set with standard augmentations. For other models, we apply mean and variance correction 32 | after all normalization layers (so that the correction parameters can be easily merged and incur no extra cost) on a single batch of samples of size 128 (for YOLO) and 512 (for BERT). We found this to be more effective than batchnorm tuning for YOLO, and the BERT models have no batchnorm layers.
When compressing to given FLOPP or timing constraint, we need to solve the problem of identifying per-layer compression targets, which match the constraint, while maximizing accuracy. To identify these non-uniform targets, we follow the approach of [10]: we first collect a "model database" contain- ing for each compression level (e.g. bit-width or sparsity setting) the corresponding (independently) compressed version of each layer. For building a joint sparse and quantized database we simply sparsify layers first and then apply quantization to the remaining weights. Next, similarly to [19], we compute the layer-wise calibration losses (without augmentations) for all compression levels, corresponding to the models with exactly one layer compressed to a certain level. Then, given layer-wise FLOP timing information, we set up a constrained layer-wise compression problem of the form described in AdaQuant [99 and solve it with the dynamic programming algorithm of SPDY 10] This returns optimal per-layer assignment of compression levels, for which we can then easily produce the corresponding model, via a two-step process: we first stitch together layers at the corresponding compression levels from the database, and then perform the discussed statistics correction to recover some extra accuracy
Unstructured Sparsity. We begin our experiments with unstructured sparsity, comparing against global magnitude pruning (GMP) [46), the approximate layer-wise OBS method L-OBS (6. and the post-training pruning state-of-the-art method AdaPrune [18). As samity check, examine in Figure 2 whether our method provides better results in terms of laier-wise squared error, pruning the first layer of ResNet18 8 (RN18) model to several sparsities. In this metric, ExactOBS performs best by a wide margin ahead of AdaPrune, which significantly outperforms the other two methods.
Next, in Table D we turn our attention to the practical problem of pruning various models to achieve a given FLOP reduction of 2 applying the per-layer target sparsity optimization technique described above. Our ExactOBS generally performs best (except for YOLOv51 2 where all methods perform similarly in terms of mAP@0.5) and at 4x FLOP reduction even with a > 1% gap to the next best method. Interestingly, on the hard-to-prume BERT model, ExactOBS appears to be the only method which still produces reasonable results at higher reduction targets. For BERT 3x and 4x, where the performance drop of all methods is > 2%. we additionally assess the compatibility of our results with the more powerful (but also more expensive) post processing method global AdaPrune [10] While this global optimization technique is able to recover lost accuracy, the ExactOBS models still maintain a > 0.5% and > 2% Fl advantage, respectively (see Table 5).
N:M Sparsity. Next, we study the performance of our method for semi- remi-structured sparsity via the N:M pattern. Specifically, we compare against the 4:8 results of AdaPrune with batchnorm tuning [8 on ResNet models (see Table 2) in addition to a 2:4 comparison on BERT models (see Table 3) We highlight that ExactOBS matches or even slightly exceeds the 4:8 results of AdaPrune with the considerably more stringent 2:4 pattern, which is already well supported on NVIDIA hardware. Furthermore, in a 2:4 comparison on BERT models, ExactOBS achieves 1-2% higher Fl scores.
8

---

ResNet18 first layer
5030
Exacicos Adutrune
ResNet50
76.13
YOLOvvI 66.97
BERT 88.53
Method
4030
L-O83 CMP
2x
3x
2x
3x
3x
GMP
74.86
71.44
64.84
65.83
62.30
55.09
65.64
12.52
09.23
L-OBS
75.48
73.73
71.24
66.21
64.47
61.15
77.67
3.62
6.63
2030
AdaPrune
75.53
74.47
72.39
66.00
64.88
62.71
87.12
70.32
18.75
1030
ExacOOBS
75.64
75.01
74.05
66.14
65.35
64.05
87.81
85.87
82.10
5.00 5.59 0.35 0.47 555 0.56 0.72 6.73
Sparsty mattv staps
Table I: Unstructured pruning for different FLOP reduction targets.
Figure 2: RNI8 squared error.
AdaPrune 4:8
ExactOBS 2:4
Model
Model
Dense
AdaPrune
ExacIOBS
4:8
BERT3
84.66
82.75
83.54
ResNet18
69.76
68.63
68.81
69.18
ResNet34
73.31
72.36
72.66
72.95
BERT6
88.33
85.02
86.97
BERT
88.53
85.24
86.77
ResNet50 76.13
74.75
74.71
75.20
Table 3: Semi-structured 2:4 pruning of all lay- ers except the embeddings.
Table 2: Semi-structured N:M pruning (+ batchnorm tun- ing) of all layers except the first and the last.
Quantization. Additionally, we compare OBQ's independent performance (after batchnorm tun- ing) with the state-of-the-art sequential post-training methods AdaQuant [19], AdaRound [3] and BRECQ (24). We perform standard asymmetric per-channel quantization of all weights, using the authors' implementations. We rerun all methods on Torchvision (28] ResNets to ensure uniform baseline. The quantization grids for OBQ as well as AdaRound are determined with the same LAPQ (34 procedure that is used by BRECQ. Surprisingly, we find that, despite optimizing layers independently, OBQ achieves very similar (sometimes even slightly better) accuracies as existing non-independent methods for 4 and bits. This suggests that it should be well-suited for mixed precision applications where one needs to quickly generate many non non-uniform models optimized for different constraints. (However, we note that ExactOBS can also be applied sequentially; see Appendix.)
ResNNet88
69.76
ResNer50
13
BERT
Method
I.w. Ind.
Methods
4bit
3bit
2bit
4bit
3bit
2bit
3x
4x
AdaRound
yes no
69.34
68.37
63.37
75.84
75.14
71.58
gAP AdaPrune
86.99
84.10
AdaQuant
yes
no
68.12
59.21
00.10
74.68
64.98
00.10
gAP ExaciOBS
87.57
86.42
BRECQ
no
no 69.37
68.47
64.70
75.88
75.32
72.41
Table 5: Further improving re-
OBQ (ours)
yes yes
69.56
68.69
64.04
75.72
75.24
70.71
sults in Table [ with > 3% per- formance drops through more ex- pensive post-processing via global AdaPrune (gAP).
Table 4: Comparison with ofate-of-the-art post-training methods for asymmetric per-channel weight quantization of all layers. We mark whether methods are Layer-wise (Lw.) or Independent (Ind.).
BOP-Constrained Mixed GPU Compression. We now consider a practical setting where we are given trained model together with some calibration data and want to compress this model for efficient inference on an NVIDIA GPU which supports 8-bit and 4-bit arithmetic, also in combination with 2:4 sparsity. Thus, there are 4 possible compression choices per layer: 8bit weights 8bit activations (8w8a). 4w4a, 8w8a+ + 2:4 and 4w4a + 2:4. Unlike in the previous section, we do symmetric per-channel quantization of the weights as it has better hardware support; activations are quantized asymmetrically per-tensor. We then generate mixed precision configurations for various BOP (number of bits times FLOPPs) reduction targets and visualize the resulting compression-acoon accurracy trade-off curves in Figure 3 In summary, at the cost of a ~ 2.5% relative performance drop, we can achieve a 12 - 14x BOP reduction for ResNets and a 7 - 8x reduction for the more challenging YOLO and BERT models (relative to the compute in compressible layers). To the best of our knowledge, we are the first to consider joint N:M pruning and quantization in a post-training setting. Recent work 3 also studies joint 4w4a + 2:4 compression for ResNet18 8 but with 90 epochs of (sparse) Quantization-Aware Training (QAT) on the full dataset and report 67.33% accuracy. Although not perfectly comparable (we keep the first layer dense and their dense baseline has 0.94% higher accuracy and uses 4:8 sparse activations), we achieve similar 67 20% accuracy for 4w4a + 2:4 post training, which emphasizes the effectiveness of our methods for joint sparsification and quantization.
Sime-Constrainee CPU Compression, Lastly, we explore similar scenario, but targeting actual CPU inference speedup a 12-core Intel Xeon Silver 4214 CPU using the DeepSparse inference

---

Mixed quantization + 2:4 pruning
Mixed quantization 2:4 pruning
76
67
74
65
Accuracy
63
70
PAP @
68
61
ResNet50
ResNet34
59
YOLOv5I
18
YOLOv5m
57
Dense
62
55
5x
7x
9x
11x
13x
15x
5x
6x
7x 8x
9x
10x
BOP reduction factor
BOP reduction factor
(a) ResNet variants.
(b) YOLO variants.
Mixed quantization + 2:4 pruning
Blonk-pruning + quantization
88
76
86
Accuracy
84
BERT
BERT6
80
OBC (ours)
78 Dense
Dense
71
5x
9x 10x
3.5x 4.0x 4.5x Inference time speedup
BOP reduction factor
(c) BERT variants.
(d) ResNet50 CPU.
Figure 3: (a) to (c) Mixed quantization and 2:4 pruning for various BOP reduction targets. (d) Joint block-pruning and quantization for CPU inference time speedups.
which provides acceleration for joint 8-bit quantization and block-sparsity with blocksize 4. In this case, we work with real layer-wise timing data (for batchsize 64), as in [9]. There are 30 available block-sparsity targets per-layer. in steps of pruming 10% of the remaining weights, all of which are further quantized to 8 bits. The base acceleration of the dense 8 bit model is z 2.7x on top of which sparsity speedup acts roughly multiplicatively. Figure 2d shows results for ResNet50 and several (real-time) speedup targets-ww we achieve 4x and 5x (actual) speedup with 1% and 2% accuracy loss, respectively. These are the first full post-training results in this setting (the authors of [10] only performed 4-block pruning post-training, followed by 5 epochs of QAT the entire ImageNet dataset), and they show very encouraging accuracy-speedug trade-offs.
engine
35 22
7 Conclusions & Future Work
We have presented a new efficient and accurate approach for solving the layer-wise compression problem, and built it to obtain state-of-the-art post-training compression solutions for both pruning and quantization. Our framework should be naturally extensible to structured pruning, which in fact should allow for further optimizations, and should also be compatible with further compression via unstructured pruning and quantization. Our results suggest that post-training compression may be able reach comparable accuracies much more expensive retraining methods. We plan to investigate this in future work, in particular in the context of more resource-intensive models, such as very large-scale language models.
Acknowledgements
We gratefully acknowledge funding from the European Research Council (ERC) under European Union's 's Horizon 2020 programme (grant agreement No 805223 ScaleML),, as well as computational support from AWS EC2. We thank Eldar Kurtie providing us BERT code and pretrained models, and the Neural Magic Team, notably Michael Goin and Mark Kurtz, for support with their software.
10

---

References
[1] Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. In Conference on Neural Information Processing Systems (NeurIPS), 2019.
[2] Thomas Blumensath and Mike E Davies. Iterative thresholding for sparse approximations. Journal of Fourier Analysis and Applications, 14(5-6):629-6333 2008
[3] Brian Chmiel, Itay Hubara, Ron Banner, and Daniel Soudry. Optimal fine-grained N:M sparsity for activations and neural gradients. arXiv preprint arXiv:2203, 10991, 2022.
[4] Yoni Choukroun, Eli Kravehik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In International Conference on Computer Vision Workshop (ICCVW), 2019.
[5] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
[6] Xin Dong, Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon, In Conference on Neural Information Processing Systems (NeurIPS), 2017.
[7] Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast sparse convnets. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[8] Utku Evei, Nicolas Le Roux, Pablo Castro, and Leon Bottou. Mean replacement pruning. 2018.
[9] Elias Frantar and Dan Alistarh. SPDY: Accurate pruning with speedup guarantees. arXiv preprint arXiv:2201.13096 2022.
[10] Elias Frantar, Eldar Kurtic, and Dan Alistarh. M-FAC: Efficient matrix-free approximations of second-order information. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
[11] Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630 2021.
[12] Roger Grosse and James Martens. Kronecker-factored approximate Fisher matrix for convo- lution layers. In International Conference on Machine Learning (ICML), 2016.
[13] Babak Hassibi, David Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks, 1993.
[14] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[15] Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. AMC: AutoML for model compression and acceleration on mobile devices. In European Conference on Computer Vision (ECCV), 2018.
[16] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In International Conference Computer Vision (ICCV), 2017.
[17] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. arXiv preprint arX1v:2102.00554 2021.
[18] Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Seffi Naor, and Daniel Soudry. Acceler- ated sparse neural training: provable and efficient method to find N:M transposable masks. In Conference Neural Information Processing Systems (NeurIPS), 2021.

---

[19] Itay Hubara, Yury Nabshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In International Conference on Machine Learning (ICML), 2021.
[20] Glenn Jocher. YOLOv5. https://sittuu..mmmmmtrrrssy///s
2022.
[21] Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The Optimal BERT Surgeon: Scalable and accurate second- order pruning for large language models. arXiv preprint urXiv:2203.07259, 2022.
[22] Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Bill Nell, Nir Shavit, and Dan Alistarh. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In International Conference on Machine Learning (ICML), 2020.
[23] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Conference on Neural Information Processing Systems (NeurIPS), 1990.
[24] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang. Peng Hu, Qi Zhang. Fengwei Yu, Wei Wang. and Shi Gu. BRECQ: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations (ICLR), 2021.
[25] Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370-403, 2021.
[26] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference Computer Vision (ECCV), 2014.
[27] Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang. Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning (ICML), 2021.
[28] Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In ACM International Conference on Multimedia, 2010.
[29] James Martens and Roger Grosse. Optimizing neural networks with kronccker-factored approx- imate curvature. In International Conference on Machine Learning (ICML), 2015.
[30] Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08:75 2021.
[31] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? Adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML), 2020.
[32] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling, Data-free quantization through weight equalization and bias correction. In International Conference Computer Vision (ICCV), 2019.
[33] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yclysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. white paper on neural network quantization. arXiv preprint arXiv:2/06.08295, 2021.
[34] Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M Bronstein, and Avi Mendelson. Loss aware post-training quantization. Machine Learning, 1 110(11):3245335 2021.
[35] MeuralMagic. The DeepSparse Inference Engine. hutp://gthhhccommeeenaaaggggggge 2022.
12

---

[36] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Conference on Empirical Methods Natural Language Processing (EMNL.P, 2016.
[37] Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, et al. MIperf inference benchmark. In 2020 ACM/IEEE 47th Annual International Symposium Computer Architecture (ISCA), pages 446-459. IEEE, 2020.
[38] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-255)) 2015.
[39] Sidak Pal Singh and Dan Alistarh. WoodFisher: Efficient second-order approximation for neural network compression. In Conference on Neural Information Processing Systems (NeurIPS). 2020.
[40] Chaoqi Wang. Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the Kronecker-factored eigenbasis. In International Conference on Machine Learning (ICML), 2019.
[41] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: Hardware-aware automated quantization with mixed precision. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[42] Peisong Wang. Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via bit-split and stitching. In International Conference on Machine Learning (ICML), 2020.
[43] Haichuan Yang, Shupeng Gui, Yuhao 7hu, and Liu. Automatic neural network compression by mparsty-quantizatiin joint learning: A constrained optimization-banl based approach. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[44] Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael Mahoney, et al. HAWQ-v3: Dyadic neural network quantization. In International Conference on Machine Learning (ICML), 2021.
[45] Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning N:M fine-grained structured sparse neural networks from scratch. In International Conference on Learning Representations (ICLR), 2021.
[46] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint
2017.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
Assets open-source with standard permissive licenses.
(d) Did you discuss whether and how consent was obtained from people whose data you're you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "ewe2chbZVad", "reviewer": "Reviewer_7fQD", "summary": "This paper tackles the important problem of post-training compression in deep nets, namely pruning and quantization. The paper adopts the Optimal Brian Surgeon framework for the per-layer squared loss objective, and presents efficient and exact solvers for the greedy and sequential Hessian-based pruning algorithm, allowing it to scale to modern deep nets. The authors also propose a quantization variant, Optimal Brain Quantizer, which leverages the same exact and efficient solver and enables accurate post-training quantization. The authors validate the efficacy of their methods across various tasks, networks, and compression strategies, showcasing real hardware benefits. ", "questions": "Please see Weaknesses above\n", "limitations": "The authors should mention the limitations of their method, ideally in the last section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written and very easy to follow. Everything was well organized\n- The proposed method is simple and scalable, allowing the authors to implement the exact OBS solver at the scale of modern deep nets\n- The idea of extending OBS for quantization seems novel to me, and is very interesting\n- The experimental results demonstrate that the proposed method is better/competitive with existing post-training methods\n\nSlight Weaknesses:\n- The overall novelty of the method is hard to assess. Please correct me if I'm wrong, but my current understanding is that the main contribution is having an efficient and exact OBS solver, but the implementation itself is built on existing/simple ideas. For instance, the theoretical result (Lemma 1) seems to be a direct application of the Woodbury-Sherman-Morrison formula [1] (if so, it should be cited in the text). I like how the paper is written, but I feel some improvements can be made to better highlight the technical contributions, and distinguish it from existing methods/ideas.\n\n- A comprehensive comparison that includes both the setup cost (time to compress), runtime cost (inference time), and task performance (accuracy) is missing in the main text. Such comparisons are vital to improving the quality of the manuscript, and help the reader better asses the performance of the proposed method compared to prior art.\n\n[1] Woodbury, Max A. Inverting modified matrices. Statistical Research Group, 1950.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aQ7H3GQYvnc", "reviewer": "Reviewer_Xxed", "summary": "This paper proposes a method for post-training quantization and pruning by primarily building on the Optimal Brain Surgeon (OBS) framework. It achieves this by creating an efficient version of the Hessian-based approach in OBS and applies it to quantization and pruning separately. The quantization approach is called Optimal Brain Quantizer (OBQ) and their pruning approach is ExactOBS. They evaluate their methods in multiple modes, including unstructured sparsity, block sparsity, quantization, and mixed compression. They find their method performs better or equal in most of these modes, with the added benefit in some cases of being layer-wise and more parallelizable. ", "questions": "As correctly predicted by the authors, it is not  intuitive that one-by-one quantization of weights would give a strong benefit. How much are the (not-yet-quantized) weights being adjusted after the quantization of other weights?\n\nThere is a significant body of work on post-training quantization methods that use various amounts of calibration data, including those that use no data at all. These include HAQ, OCS, ZeroQ, QAT, etc. Are these not directly compared against because they are not solely weight quantization? Or do the baselines included claim to outperform them?\n\nThe quantized experiment mentioned using a consistent pre-trained model for comparison. Is this true with all the experiments?", "limitations": "No significant limitations than otherwise mentioned.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=== Strengths ===\n\nThe authors include a strong motivation for their work especially from the perspective that modern hardware supports quantization and block sparsity.\n\nIt seems that in general although exact methods are usually unnecessary in machine learning, this may not be true in the context of post-training quantization and limited calibration data. This provides again a high-level motivation for pursuing an exact OBS solution.\n\nThe authors include a variety of evaluation tasks, including classification, detection, and question answering.\n\nPaper written in an organized and clear way.\n\n=== Weaknesses ===\n\nIn the end, what matters with compression is the accuracy vs. real-world performance curves. It would be useful to show these comparing against other methods to the extent that they are supported.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Re5IAKgh04I", "reviewer": "Reviewer_CZ4H", "summary": "This paper introduces a compression framework for post-training quantization and pruning. The paper takes a layer-by-layer compression method to quantize and sparsify model weights and shows that this can be done under a unified setup without retraining a network from scratch. The evaluation on ResNet and BERT models show that the proposed method lead to 2 points drop in accuracy when composing quantization and pruning while obtaining 12x model size reduction. ", "questions": "Can you provide a more direct comparison of the proposed unified method with the existing method that also combines sparsification and quantization without layer-by-layer compression? For example, the sparsification can be weight magnitude-based pruning or movement pruning [1], and quantization can be quantize-aware training [2]. \n\n\n[1] Sanh et al. \"Movement Pruning: Adaptive Sparsity by Fine-Tuning\"\n[2] Jacob et al. \"Quantization and training of neural networks for efficient integer-arithmetic-only inference.\"", "limitations": "Yes. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper explores the combination of sparsification and quantization from an interesting angle: the optimal brain surgeon framework, and introduces two techniques to induce sparsity and quantization in a layer-by-layer manner. \n- Promising results on combining quantization and sparsification using layer-wise compression techniques without extensive retraining. \n \nWeaknesses:\n\n- The problem definition of the work is a bit confusing. Usually, the problem of compression has been defined as an optimization problem where the optimization objective is to reduce some metrics such as model size or latency while satisfying certain accuracy constraints (e.g., little or no accuracy loss). However, the paper defines the problem as a layer-wise compression problem, which raises a few concerns. First, it includes the solution space of compression as part of its problem definition, e.g., saying that the compression has to be layer-wise, but layer-by-layer compression is only one out of many ways to perform compression, and there is no clear advantage on why one has to use layer-by-layer compression. Therefore, including it in the problem definition seems to artificially reduce the solution space (e.g., in the evaluation, the paper does not compare its method with many existing compression methods that also combine sparsification and quantization). Second, the goal of the problem is to find a \"compressed version of a layer weight that performs similarly to the original weights\", but this appears to be an indirect optimization target where the paper does not show whether it has a direct correspondence to the metrics people care in practice (e.g., model size, latency, compression cost). The paper might need to provide more justification on the rationale behind defining the problem this way. \n \n- The paper introduces the challenge of compression from a rather indirect angle. There have been numerous work on both pruning and quantization in recent years that target compressing a model. However, instead of talking about the challenges of state-of-the-art approaches, the paper starts with the challenges in the OBS (optimal brain surgeon) framework, which was published almost 30 years ago, and its variations, but none of those appear to have been established as the state-of-the-art for model compression. In contrast, there are actually quite a few more recent work that also performs quantization and pruning together in a unified manner, such as https://arxiv.org/abs/2101.09671 and https://arxiv.org/abs/1811.01907. Therefore, it would be better to more directly compare and position itself with respect to those existing works. Btw, the paper does not cite nor compare those unified compression frameworks. \n \n- The difference between the paper's proposed post-training compression and existing compression methods is not very clear. Notably, popular and state-of-the-art compression methods do not require full retraining, such as quantize-aware training.  Instead, only a small portion of the training data (e.g., <10%) is used to calibrate the quantization errors, as in https://arxiv.org/pdf/1702.03044.pdf. Similarly, for pruning, it is rarely the case to train a sparse model from scratch and again only a small number of iterations are performed to calibrate the model to recover the accuracy, as in https://arxiv.org/pdf/1709.05027.pdf. Since the proposed method still need training data (a subset of training samples) and refinement/calibration training, it is unclear how fundamentally this method has an advantage over existing compression techniques.  \n\n\n---------------\nPost-rebuttal comments\n\nThe authors addressed my concerns. I raised my score from 4 to 6 to reflect my post-rebuttal perspective. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept