Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The idea of this work is to first represent DNNs with add-shift-add format instead of mult-add. By doing so, they observe that the first accumulation step contribute a lot to the energy consumptions. As such, they propose the bit-pruning for the first add in terms of the shiftadd perspective.

The key is not representing weights in add-shift-add. Rather, the key is to regularize and prune the number of adds.

I like the idea and am willing to take a further look at the implementation later on.
strength_and_weaknesses: Strength:

* The idea sounds interesting to me.
* The proposed bit-pruning method beats weight pruning across several experiments.
* This paper clearly describe the most related background work, ShiftAddNet, and explain why it is not direclty comparable.

Weakness:

* Some math formulation could be more concise and not necessarily complex. Or you can give more high-level insights before diving into the math. Also, the figure quality is not enough, I strongly encourage the authors to revise their figure to the best level you can draw.
* Apart form the energy cost, latency comparisons or number of operations are expected and should be more direct.
* More quantization works and bit pruning works should also be compared with, I believe there are a lot bit-pruning works, either compare in a table with checks or compare in terms of experiments or both.
clarity,_quality,_novelty_and_reproducibility: It is written clearly, and well organized.
summary_of_the_review: In a nutshell, I recommend this paper due to their novel insights for bit pruning in terms of shiftadd representation of DNNs.

BTW, I wonder how this would be efficient and fast on GPUs? I understand that may need customized CUDA implementations but am curious which stage are you currently in.

Also, more quantization works and bit pruning works should also be compared with. More baslines and better figures would further strenthen this paper.

As for the difference with ShiftAddNet, I buy the explaination that they are not direclty comparable. ShiftAddNet reparameterizes DNNs with bit-shifts and adds. This work "translate" multiplications with shift and adds and prune the first accumulation process.

They can even be merged as a hybrid model, like ShiftAddNAS [1] did.

[1] ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks, ICML 2022
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Bit-pruning proposes pruning at the bit level in the context of unstructured pruning methods. At its core, formulation presented by the Authors aims to replace the scalar-scalar multiplication with its equivalent scalar-Nbit multiplication followed by N-shifts and finally a sum over N shifted values. When extending this to a dot-product (the basic OP in matrix-matrix and matrix-vector multiplications), it can be seen as a add-shift-add OP. In this formulation to compute is dominated by the number of bit-level additions before the shift and therefore this is the metric to minimise. By means of an additional loss term that penalises the number of bits in the weights representation as well as the distortion this introduces, the proposed bit-pruning mechanism achieves large energy reductions for a fixed accuracy value. This method is evaluate on image classification with ResNets. 
strength_and_weaknesses: ### Strengths
*    The proposed method is simple and seems to deliver very good results, specially at higher accuracy ratios, where weight pruning understandably struggles to zero-out weights without impacting on accuracy.
*    ImageNet results shows around 50% energy reduction, that's good.


### Weaknesses
*    Unstructured sparsity benefits from models being over-parameterised for the task (i.e. the task at hand is easy so a fair amount of weights can be discarded without impacting on accuracy). It would be interesting to see the performance of Bit-pruning when the simpler tasks (specially CIFAR-10) is done with a light-weight model, for example a small mobilenet.
*    What approach was used to perform weight-pruning? using traditional magnitude-based pruning? or something else?
*    Some key implementation details aren't clear from the text: are all convolutional layers using Bit-pruning? both 3x3 and 1x1 convolutions? Also the input layer?
* Why all layers use the same $\lambda$? Probably earlier layers would had benefit for less sparse weights retaining in that way some of the degradation that would otherwise propagate to the rest of the network. Another motivation for this is that early layers have comparatively much fewer OPs than deeper layers so the potential speedup might not be worth it. Could the Authors comment upon this? 
clarity,_quality,_novelty_and_reproducibility: The paper is well written, with sufficient background for those that aren't familiar with lower-level compute and HW considerations. I'm unsure about the novelty here (unstructured sparsity is, as the authors state, not so easy to accelerate unless custom HW is available -- I believe limited work has been done in this front as a result but I'm not super familiar with the Literature these days) but the method proposed is simple enough, well motivated, and seems to deliver very good results. The Authors state the code will be open-sourced, that's always a plus. 

The above being said, the worthiness of Bit-purning is subject to the fact that an accelerator for unstructured sparsity can actually be implemented and yield the estimated accuracy-energy trade-off. 
summary_of_the_review: This is a good paper that shows how pruning at the bit level can yield much better energy-accuracy ratios than weight-pruning that prunes weights directly. Even when at lower bit-widths, bit-pruning (which uses 8-bit weights) outperforms the weight-pruning counterpart using 4-bit weights. I would had loved to see this method applied to smaller models (even if CNNs) or to Transformers (even if only a single ViT).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors present an energy saving approach that replaces dot product operations with an equivalent add-shift-add operation. In the context of neural networks with integer weights, the authors demonstrate how such a technique can save significant amounts of energy while controlling for the accuracy degradation.

The paper introduces a theoretical foundation based on the binary decomposition of the dot product operation. They then replace the costly multiplication operations by unrolling additions up to the length of the binary encoding (e.g. 8 bits), and then use shifts and additions to reconstruct the dot product exactly. 

The savings come from pruning at the binary level, where the authors introduce sparsity in a way that reduces the number of computations to be carried on.

The authors then present an empirical evaluation on a simulator, as they did not have access to real hardware that could realize this kind of operations.
strength_and_weaknesses: The major weakness is that the experiments were not carried on real hardware, which could demonstrate the benefits of the approach.

The introduction of the loss function that helps with the discretization is well defined, however, there are no more details on the impact it has on the optimization. The proximal weight function defines a discretization that is not smooth, and when we take into account the optimization step size, we could see oscillatory behaviors during training.

The empirical evaluation does a good job comparing the simulated energy consumption, however, more details on the intended training use-case would be beneficial (see next section).
clarity,_quality,_novelty_and_reproducibility: The paper is well written and technically sound. The authors do a good job explaining how the dot product can be implemented as an add-shift-add transformation, and the impact this can have considering sparsification at the bit level.

One aspect of the training that is not clear to me, is whether you intend to train the whole network from scratch using the sparsification, or simply refine the resulting network? 
summary_of_the_review: The authors propose an interesting paper that can be viewed as a further discretization not in terms of bit counts as it is common in the DL literature, but they go one step further and take energy consumption into account by means of replacing a core operation such as the dot product, with a simplified version implemented by additions and shift operations.

The paper raises some further questions: Are you training the whole network using your approach? If so, as you are training with the weights in full precision and simply adding a penalization in the form of a proximal weight, it is not clear to me what the benefits are of penalizing during the whole training history. Or do you train at high precision and then simply refine with your penalty?

Furthermore, a regularization term in the loss, does not imply that the regularization term will have a cost of zero, except potentially when this is a significant part of the loss function. Do you employ a dynamic regime to rescale your regularization? i.e., do you increase your $\eta$ parameter during training? or is it kept constant?.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the potential energy saving of implementing multiplication as a add-shift-add operation, and propose to utilize the bit-level sparsity of the weight to further reduce energy cost. The paper proposes a bit-sparsity regularization that promote weight to be sparse in binary format, and claim to achieve a better energy-accuracy tradeoff.
strength_and_weaknesses: ## Strength
1. The paper provides interesting observation on the potential energy consumption improvement brought by add-shift-add operation
2. The paper advocates the exploration of utilizing bit-level sparsity, which may inspire future work

## Weakness
1. From the novelty perspective, the paper ignores previous attempts on inducing bit sparsity, and didn't perform comparison on closly related work. See the novelty section in the next review question for details
2. The paper does not clearly explain why the proposed regularization is effective. See clearity section of the next review question
3. The paper does not perform fair comparison in the experiments, and miss discussion on important baseline and limitation. See quality section in the next review question for details
clarity,_quality,_novelty_and_reproducibility: ## Novelty
This paper is novel in the aspect that it motivates the use of random bit-level sparsity, and propose a regualrization method to induce such sparsity. However, this novelty is incremental since bit-level pruning has been explored in previous work (e.g. [1] [2]), as well as utilizing shift add operations (e.g. [3]). Also the paper does not make direct comparison with these previous explorations, which further doubts the significance of the proposed approach.

[1] https://arxiv.org/pdf/2102.10462.pdf

[2] https://arxiv.org/pdf/1909.08496.pdf

[3] https://proceedings.neurips.cc/paper/2020/file/1cf44d7975e6c86cffa70cae95b5fbb2-Paper.pdf

## Clarity
The paper clearly explains why bit sparsity and add-shift-add operation can bring additional energy savings. However, the paper does not give a clear explaination on why the proposed regularization in Eq. (6) is effective. For instance:
1. Why is $X^{(l)}$ involved in the weight regularization formulation?
2. Why is square root needed in the formulation of $C_{mv}$?
3. Why can the proposed regularization guarantees to induce bit sparsity? For example the weight value of 15 and 17 would have the same proximal $\hat{W} =16$, and have the same regularization value, but obviously 15 more 1-bits than 17. How is such situation avoided by the proposed regularization?

## Quality
The paper provides multiple experiments comparing the energy-accuracy tradeoff of BP vs. WP. However, the experiments are not well designed to ensure a fair comparison. For example:
1. Even under WP, nonzero weight also contains zero bits, which can also lead to energy saving if add-shift-add scheme is used. The paper ignores this in the computation and assumed no bit sparsity in the WP model
2. Quantizing the model to a lower precision naturally leads to less bits, thus less 1-bits. So its inadequate to only compare the proposed method with WP, but also to quantization to a lower precision.
3. Given the random access pattern of 1-bits in the model after bit prune, it would require specialized data pipeline to fulfill zero skipping in the add-shift-add operation, and would lead to additional overhead for data loading and storing. These potential overheads are not clearly discussed in the paper
summary_of_the_review: In summary, the paper makes progress on the exploration of bit sparsity, and how to utlize it to improve energy efficiency. However, the missing discussion on closely related work, the lack of clearity in the proposed method, and the unfair experiment setting make it hard to evaluate the significance and true effectiveness of the proposed method. Also, the potential overhead of the add-shift-add scheme may also hinders the real application of the proposed random bit sparsity. Therefore I would recommend rejection for now.

## Post rebuttal

I would like to thank the author for the extensive responses. Overall I like the idea of exploring potential energy savings of the random bit-level sparsity via the add-shift-add scheme. Also, the additional comparison provided by the author on using add-shift-add on weight pruned model indicates improving efficiency with add-shift-add is not naive for any model, and the explicit exploration of bit-level sparsity is required. This enhances the significance of the proposed regularization method.

As for the proposed regularization-based training method, the response from the author largely resolves my doubt on the effectiveness and the design choices. I think this paper should be worth acceptance given all the additional results and explorations provided by the author. To this end I'm increasing my score to weak accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The submission proposes a low-energy model compression strategy based on an analysis of the basic operation of a dot product.  The strategy takes the perspective of a basic "add-shift-add" operation, and then looks at pruning the number of operations by sparsification.  The rebuttal/discussion process clarified the applicability and hardware assumptions of this model, which in certain circumstances was demonstrated to increase efficiency / decrease energy usage favorably to other methods.  The application requires (i) the appropriate kind of model compression, (ii) appropriately optimized hardware, and (iii) an appropriately optimized linear algebra library.  As such, the immediate application may be limited to specialized settings.  It is a valuable exercise to explore interactions between statistical accuracy, and computational and energy efficiency, and this submission does so in a concrete and interesting way.  The reviewers were generally appreciative of the contributions of the submission, with a consensus that it is at least above the threshold for acceptance.
justification_for_why_not_higher_score: The specific strategy for network compression is not broadly ready for prime-time without some additional linear algebra library development.  It's an interesting idea, but the reviewers were not overwhelmingly positive that it is a priority for a spotlight.
justification_for_why_not_lower_score: The topic of computational and energy efficiency in NN deployment is an important one.  This paper makes a strong argument for a specific view on basic compute operations that can lead to improved efficiency/accuracy tradeoff.
Decision: Accept: poster