Bit-Pruning: A Sparse Multiplication-Less Dot-Product
Authors: Yusuke Sekikawa
Shingo Yashima
Venue: ICLR 2023 poster (2023)
Keywords: pruning
non-uniform quantization
power of two
2bit

Abstract: Dot-product is a central building block in neural networks.
However, multiplication ($\texttt{mult}$) in dot-product consumes intensive energy and space costs that challenge deployment on resource-constrained edge devices.
In this study, we realize energy-efficient neural networks by exploiting a $\texttt{mult}$-less, sparse dot-product. We first reformulate a dot-product between an integer weight and activation into an equivalent operation comprised of additions followed by bit-shifts ($\texttt{add-shift-add}$).
In this formulation, the number of $\texttt{add}$  operations equals the number of bits of the integer weight in binary format. 
Leveraging this observation, we propose Bit-Pruning, which removes unnecessary bits in each weight value during training to reduce the energy consumption of $\texttt{add-shift-add}$. Bit-Pruning can be seen as soft Weight-Pruning as it prunes bits, not the whole weight element.
In extensive experiments, we demonstrate that sparse $\texttt{mult}$-less networks trained with Bit-Pruning show a better accuracy-energy trade-off than sparse $\texttt{mult}$ networks trained with Weight-Pruning. 

Published as a conference paper at ICLR 2023
BIT-PRUNING
A SPARSE MULTIPLICATION-LES DOT-PRODUCT
Yusuke Sekikawa, Shingo Yashima
DENSO LABORATORY, INC., Tokyo, Japan
(sek kawa. yusuke, yashima. shi (ig)) écore
ABSTRACT
Dot-product is a central building block in neural networks. However, multipli- cation (mult) in dot-product consumes intensive energy and space costs that challenge deployment on resource-constrained edge devices. In this study, we realize energy-efficient neural networks by exploiting a mult-less, sparse dot- product. We first reformulate a dot-product between an integer weight and acti- vation into equivalent operation comprised of additions followed by bit-shifts (add-shia ft ft-add). In this formulation, the number of add operations equals the number of bits of the integer weight in binary format. Leveraging this obser- vation, we propose Bit-Pruning, which removes unnecessary bits in each weight value during training to reduce the energy consumption of add-shi ft -add. Bit-Pruning can be seen as soft Weight-Pruning as it prunes bits, not the whole weight element. In extensive experiments, less networks trained with Bit-Pruning show a better accuracy-energ: trade-off than sparse mult networks trained with Weight-Pruning. (Code available
demonstrate that sparse mult-
INTRODUCTION
Modern deep neural networks (DNNs) contain numerous dot-products between input features and weight matrices. However, it is well known that multiplication (mult) in dot-product consumes intensive energy and space costs, challenging DNNs' deployment on resource-constrained edge de- vices. This drives several attempts for efficient DNNs by reducing the energy of mu.t.
Quantization (Yin et al.] 2019 Esser et al. 2020 Li Baillieul] 2004) discretizes the weight and/or activation into a low-bit repre- sentation; low-precision mult requires less en- ergy than the high-precise preciisrciiionn counterpart. Power of two networks (Li et al.)
Table 1: Energy [aJ] and area [um²]] cost on ASIC (45mm technology) and FPGA (ZYNQ-7ZC706), Data adapted from You al. 2020 Horowitz 2014
Gholami et al.
ASIC
FPGA
2019b Miyashita
Operation
Format
Energy
Area
Energy
al. 2016; Elhoushi al. 2021) restricts the mult operation to the power of two (PoT); PoT mult can be realized by bit-shift (shift), which consumes orders of magnitude less en- ergy than mult. Although these approaches have successfully reduced energy consumption, they define the eefegy-efficient model structure before training. That is, they limit model capacity and impose training with precision that is difficutt to use gradient-based optimization, e.g., approximate gradient with the straight-through estimator (STE) (Yin et al. 2019), making challenging to achieve good accuracy in the low-bit regime (e.g., 4bit, 2bit, or binary).
mult
FIX32
3.1
3495
19.6
FIX8
0.2
282
0.2
FIX32
137
FIX8
0.03
shift
FIX32
0.13
0.1
FIX8
0.024
0.025
Aside from low-bit approaches, Weight-Pruning (Frankle & Carbin 2018 Chen al. 2022 Yang
et al. 2020; Wortsman et al. 2019) learns efficient model structure during training by removing unimportant weight from high capacity models. Owing to its data-driven nature, it could remove the unnecessary mult without sacrificing accuracy (c.g.. 95% reduction in unstructured pruming). Empowered by the recent progress of in/near-memory computing architectures (Gholami et al.),

---

Published as a conference paper at ICLR 2023
Tmnt
Rodk
Tacid
Zarh
W
B, EEF
Ug
W
X X
Dot-product realized using sparse muit-add (left) and equivalent representation using add-shitt saniiiit-add(righl aad(righh) In the mult-aae computation is reduced by pruning the entire weight (muit)). In contrast, in the add-shi- st -tdaddd computation is reduced by pruning the hit (add) in binary format.
Figure 1:
several DNN accelerators designed for the unstructured sparsity have emerged (Han 2016
Bamberg et al. 2023 Dictrich et al. [021] Zhang et al.] 2020). realizing the efficient inference of these sparse networks. Furthermore, optimized implementation for general- pprrpoee CPUs also proved to be a good candidate for unstructured sparsity (Kurtz et al. 2020).
Motivated by the significant energy reduction of sparse mult operations by Weight-Pruning and recent advances in the frameworks supporting unstructured sparsity, we envision new frontiers in accuracy/energy trade-off by realizing spurse and mu t-lexs dot-product for hardware support- ing the unstructured sparsity. Noting that the dot-product between integer weights and activation can be decomposed into a PoT basis and binary vector, we first reformulate a dot-product between integer weight and activation mmult-adoe into equivalent operation comprised of additions fol- lowed by bit-shifts and additions (add-shi ft-add, Figure []. this formulation, the number of the first add operations equals the bitcount of the weight elements in binary format. From this observation, we propose Bit-Pruning, which removes unnecessary add (therefore unnecessary bits in weight) during training in data-driven manner reduce the energy consumption in the add-shi faaifittadd operation. Because of the difficulty of optimization in binary format, opti- mize parameters on a weihh-ighcciee precision differentiable mult moult-add network with bit-sparsity regulariza- tion, which promotes weights to be sparse in binary format. After training, the obtained bit-sparse mu t-add network is converted an equivalent add-sh ft fadd network comprised of sparse add for efficient inference on DNN accelerators supporting unstructured sparsity.
As depicted in Figure I Bit-Pruning can be seen soft and fine- mneggraneed Weight-Pruning it does not necessarily remove the whole weight elements but removes only some unnecessary bits of the weight values. This interpretation raises the following question:
Which pruning offers better accuracy/energy trade-off? mult in mult-add or add in add-shife -add network (Bit-Proe Phii-Prnnnng?
network
We conducted extensive evaluation answer the question (Section 4D: the results suggest that pruning the add is several times more energy efficient than pruning the mult.
We do not argue that the add-shi sadadifffdfddi representation itself is efficient. In fact, the estimated energy consumption of (dense) mult addd and shi-t-iift-e without pruming are almost the same, grained pruning by removing bits rather than weights can find a more efficient sparse structure; the add- shift add representation permits this investigation.
Remark:
in Section 3 Our research interest is investigate whether the fine-
2 PRELIMINARIES
2.1 DOT-PRODUCT WITH MULT-ADD
The dot-product is a fundamental building block of neural networks. In this study, we mainly focus on the dot- dot-product that appears in convolution Let the weights and activation be quantized to M
'The same discussion be applied for dot-product in another computation block such as multi-layer perceptron self/cross attention.
2

---

Published as a conference paper at ICLR 2023
and N bits, respectively, where M, N are sufficiently fine quantization levels such M, N = 8, 32. Convolution with the input E ZCxW* and weight W ZC.xCxkkk is equivalent to matrix multiplication of unfolded input X &CCkkWW and the reshaped weight W e 2C,xCkE
= == WW
For brevity, consider single imput pixel and output channel j. Then, the convolution which corresponds to the dot-product of the u-th column vector of the unfolded input X., and j-th row vector of the weight W, is written as follows:
K OjH = W Σ XR.
Wj
(2)
where K- = C,kk. Computing using naive dense mult-ada requires K mult followed by
(K 1) addd which consumes lot of energy contains many t operations (Table 1
2.2 SPARSE MULT-ADD BY WEIGHT-PRUNING
When DNN processor supports unstructured weight and dynamic activation sparsity (Han et al., Bamberg al. 2023 mult t can be skipped either when the activation or weight is zero. In this case, the number of mul: operation become mult K where muit is the ratio of the nonzero element defined as bit count (W, # 0 X., * 0)/K. K. Therefore, the dot-product using sparse mult-add consumes less energy than dense mult -add.
2016
The sparse mult add dot- dot-product is typically realized by Weight-Prunitee which trains a network with the spight-sparsity regularization Lug Liu et al. 2015 Han et al. 2015):
L(W) Lrack (W)) + 17 Lwg)
Cwg Σ 1 19 w
(3)
where Lrask a loss specific to the given task (e.g., cross-entropy for classification), W (W())/ |= is a of weight matrices from all L layers, (!) is an input of I-th layer, and 17 is hyper-parameter to balance the accuracy and computational energy. In practice, 4 -nom or Hoyer(- Square) loss (Yang al. 2020) applied to relax lo-norm for efficient training. Note that when the target hardware only supports unstructured weight sparsity (no activation sparsity), the same argument can be applied by replacing X ( with numcl(X")
3 BIT-PRUNING
For more efficient DNN inference, we propose a sparse and mult-less dot-product by softening the Weight-Pruning Our intuition is that better energy-accuracy tradeoff could be achieved by bit-level fine nnee-gaaiedd pruning instead of the weight-level course pruning. We first reformulate a dro-pur between integer weight and activation into equivalent operation comprised of add-shift- which does not contain mu 1 (Section 3.1 which consumes lot of energy, Figure Dp. In this formu- lation, can see that the number of the first add (which directly translates energy consumption) equals the bit bount of the weight element in binary format. Then, we propose Bit- Bit-Pruning, which removes unnecessary bits of weight elements during training to achieve efficient inference (Section 3.2
3.1 REFORMULATE DOT-PRODUCT WITH ADD-SHIFT-ADD
M-bit weight matrix W E ZCOK is represented in binary format with tensor B
{0, 1 I.CMMx, where Bj,,& (0, M is the binary representation of the weight value W E Z:
M 2m
W
Σ
Bj.mkk
(4)
FFor & E z!x! and E J O E z | is defined as (ar @ y)i.JA = 3i Vi.k- For simplicity, we assume the weight element W positive in the following explamation.
3

---

Published as a conference paper at ICLR 2023
Cark
Omutt @add
2siil
X
2
3add
X
Figure 2: Matrix view the mu -add (left) add-shitt -ad (right) dot-product.
Then, rewrite the mu 1 a add dot-product
into the following add-shi ft-add formulation:
K
M
X
k,u
2m
B Bymm,k
m=1
M
2
2m
(X (XTT
(5)
772
This is computed first by accumulating the input X., along the binary vector Bj.m using add, then multiplying 2m using sh ft, and finally aggregating results of all M bits by add. When the operation mapped into the hardware supporting the unstructured sparsity, the first add using the binary vector B,. requires bitcount J.m.) times add. The shi ft and last add are executed for M times (Table 2
3.2 SPARSE ADD-SHIFTI SHD BY BIT-PRUNIE PRU-PIG
In the shi-- add formulation of ( the number of the first add operations equals the bi bitcount of the binary tensor B, which directly translates to energy consumption. Therefore, goal is to learn the sparse binary tensor of the add-shift-ade network that also achieves good performance by minimizing the following loss:
C(B) Crask (B) + n bitcount(B))
(6)
where = {B is a set of binary tensors from all L layers. However, binary parameters prohibit gradient computation, making it difficult train with standard backpropagation.
Instead of directly optimizing the binary tensor B, as done by Yang et al. (202T). we deal with the original weight matrix W while promoting it be sparse in the binary format B. Notice that any dot-product with the integer weight can be precisely expressed as the add-sta shift ft-add dot product, and the number of add counts for B depends on the corresponding integer weight W. For example, when W j.A is zero, then B j. = 00, 0, 0, 0, 0, 0,
and there is no add. When W is PoT (e.g., 23), then B A [0. 0, 0, 1, 0, 0, OJT, and there is only one add. In such way, number of add in aadddhhttt ft-fftt-dcc dot oot-prduuct depends on the value of the corresponding weight j.A (Figure Bah. and there is favorable value in terms of energy consumption.
We leverage this to propose framework called Bit- Bit-Pruning, which optimizes the network's 's energy consumption in data-driven manner. The binary tensor B of the idd-shi ft-add network is trained the high-precision mult-add network having weight W (Figure and sparsification of B is realized by imposing the bit-sparsity regularization Lbit weight W:
L(W) Lrask (W) + (W))
Lea Σ
x (W W())
where Luit measures a deviation of the current weight W from bit-sparse proximal weight W. explained as follows. Similar to Weight-Pruning 3.. we relax the lo-norm in 7 with In-nomm or Hoyer(-Square) loss in practice.
Proximal weight Given the current weight W. the proximal weight W is computed (element-wise) as minimizer of the sum of weight proximity cost Cmo and add count cost Cadd:
W j, arg min (m. Ws.)) +
(00 Cadd(w))
(8)
4

---

Published as a conference paper at ICLR 2023
Number of add
300
50
100
Weight (a) Number of addition (8hit)
128
Prox. weight
64-
T
200
Weight (b) Proximal weight (A 0.1)
welght
64-
32
100
-50
150
Weight (c) Proximal weight (A = 1.0)
Figure 3: (a) Number adi each weight (add count cost C.da). (b)-(c) Proximal weight W (arrow), which is a solution of ( (number of add for rdd-shiit-adee dot dot-product is encoded as color). Refer to Appendix for the visualization of corresponding loss landscape (Cmu(w, (,, w)) + Cada(w)).
where Cada(w) the bitcout of w in binary format (Figure 3al and Cmy (w,, w²) mea- sures the proximity of two weight values w and w'. In this study, adopted Cmv (w, (w,w') == |sgn(w)]w[ (agp/-ssnw(w sgn(w") [''l), We choose p = 1/2 to discount the change in Crav for larger value (assuming accuracy is less affected by the change when it is large). Intuitively, Codd encourages Wj, be sparse in the binary format while C... keeps it close the current weight W,,s and (1) controls its balance (larger X induce more sparsity). Owing to the discrete structure of the weight space (e.g.. bit), the solution of g can be analytically computed for given À (Figure 3b Bcd.
Efficient sparse inference. Once the training has been completed, the high-precision mult addd network is converted into a mathematically equivalent add shissiiii ft add network for efficient in- ference. It is expected to be comprised of sparse add which consumes less energy.
3.3 WIICH IS EFFICIENT? PRGGIIN-RUNING OR BIT-PRUNING
An important question in this study is whether pruning the add (Bit-Pruning) is more effi- cient than pruning the mult (Weight-Pruning). When considering the dot-product between two K --dimensional,, M-bit vectors, the en- ergy consumption of sparse add-shift-and (Eadd-shift-add over that of sparse mult -add shift
Table 2: Number of operations for dot-produce be- tween K K-dimensional M-bit vectors.
mult Ladd
add-shi sii-sshit--ae -td sparse/ dense
dense
sparse
mult add
Omalt K
K -
dmait K 0
Massa K-M-1 M -1
(Emiitadds) is calculated from Table 2
Eada shiff--if--dd (Mosas K) K)E + (M - Esieee + (M E)Eadd
MEade Oadd
Emolt-ada
(amuit K)Emuit + (Amult (K 1))
Emul: Juilt
where Omult and aadd the ratio of nonzero elements in weight (W,) and the ratio of nonzero bits in its binary format (B,), respectively. Emolt: Eada- and Eshift are energy consumption of corresponding operations and we assume K >> M in the last approximation. In the case of ASIC,
5

---

Published as a conference paper at ICLR 2023
Table 3: Experimental setup
CIFAR-10
CIFAR-100
ImageNet ConvNeXt-B
Network
ResNet18
ResNet18
Batch size
512
Training epochs Optimizer
200
100
AdamW Loshchilov & Hutter
2019)
Scheduler
OneCycle Smith & Topin 2019
Cosine decay
Loshchilov & Hutter 2017
Weight quantization M
Activation quantization N
4/8/32
Weight initialization
Kaiming-uniform He et al.
2015
Pretrained
(0) for Cmove (Bit-Pruning only)
1.0 for all layers (Figure 3c
roughly get Emuit M Eada from Table ! therefore, the computational efficiency of Bit-Pruning over Weight-Pruning is almost determined by the ratio of obtained connection density aada/ Omolt.
We expect that the add in Bit-Pruned network is much smaller than the m in Weight- Pruned network when both networks achieve comparable accuracy. This because Bit-Pruning removes connections more finely than Weiight-Pruning in other words, given the same nonzero tio, i.c. add = mull. Bit-Pruned weight can have more nonzero element (at most times) than Weight-Pruned weight when it learns a weight that is sparse in binary format. In the extreme case where all the nonzero weights in a network are represented as PoT, the computational cost in add-sth srd-shifee ft add representation is M times smaller than that in mult add representation. The evaluation on actual DNN training is presented in Section
3.4 BIT-PRUNING AS A UNIFIED LOW-ENERGY MODEL
(Non-uniform) quantization. Bit-Pruning effectively applies different non-uniform mixed- precision quantization for each weight element. As result of the Bit-Pruning, we will get sparse B; this has a similar effect to the non-uniform quantization of weight. Existing quantization tech- niques learn (non-uniform) step size for group of weights; contrary, Bit-Pruning realizes pseudo- non-uniform quantization by selecting a bit pattern.
PoT dot- doooprr Bit-Pruning can be interpreted the soft version of the (additive) PoT quan- tization. The bit-sparsity regularization prefers sparse binary representation, which corresponds to the PoT; however, it could learn choose individual non-PoT values for each weight element when necessary (i.e., when it achieves less total energy for the same accuracy). The learned weight by Bit-Pruning dominated by PoT, while some have multiple nonzero bits (See Figure D)
Unstructured Weight-Pruning. Bit-Pruning can be seen as a soft, fine-grained version of Weight- Pruning. Both bit-sparsity and weight-sparsity regularization loss are minimized when W 0: this corresponds to zero add in add-shi ft-add and zero mult in mult-add. However, bit- sparsity regularization does not necessarily remove the whole weight elements but only some unnec- essary bits in the weight element.
4 EXPERIMENT
4.1
EVALUATION OF ACCURACY/ENERGY TRADE-OFF
Which energy efficient; pruning add (Bit-Pruning) or pruning mult PWeight- We eval- uated the accuracy/energy trade-off of both methods on CIFAR-10 (Krizhevsky et al. a). CIFAR-100 Krizhevsky et al.) bb. and ImageNet (Deng et al. 2009p For both bit-sparsity regularization Cait which is used for all the conv layer, approximate
7 and weight-sparsity regularization Lungt lo-norm using Hoyer-Squan loss (Yang et al. 2020). We use the same network architecture, quan- tization method (we use LSQ Esser al. (2000)) and training strategy except for the loss function for both frameworks. The primary experimental setup is summarized in Table 3 To compare the trade-off, train networks across various sparsity coefficients 7 of Luut and Leat (fixed across the entire training sequence).
PPytorc official model zoo /pyt:://pyttrch..... wwtp:/tooocooor.gvii..oo http://isiio//mii/oooooooo ht///eeseext.t
"See appendix for more detailed experimental setup (Appendix and additional results (Appendix

---

Published as a conference paper at ICLR 2023
WP
WP
WPPadd) 8Pimult)
WP/addi BP(mutt)
WiFedd) (SP/mur))
60-
Energy Lell
20-
86
88
90
92
86
94
86
88
90
92
94
Accuracy 196
Accuracy 181
Acturacy (16]
CIFAR- 10/4-bit
(b) CIFAR- 10/8-bit
(c) CIFAR 10/32-bit
80
WP
WF
BP
WP(addd)
WPLadd)
WPtaod]
BP(malt)
BPimmutt
60-
BP(multi
Energy 641
Energy 64)
20
55
60
70
Accuracy [6]
Accuracy (5))
Accuraccy [%]
(d) LUFAR-100/4-5
(c) CIFAR-100/8-bi
(f) DLFAAR-100/2-bb
Figure 4: Accuracy/energy trade-off of ResNet18 trained on CIFAR-10 and CIFAR-100 for difference acti- vation quantization level. Bit-Pruning aodd-shift add) (BP) and Weight-Pruning (mul (mult-add) (WP) compared. Bit-Pruning always uses an 8-bit width binary tensor B. and Weight-Pruning quantizes weight using the same quantization level for activation. The result of Weight-Pruning in add-shife -add representation (WP(add)) and Sit-Pruning in mu tadd representation (BP(mult) is provided for reference. The multiply- accumulate (MAC) of the original dense mu molltdad add network is about 0.55 x 10", and the estimated energy consumption (from Table 26μJ, 107pJ, and 1721 pJ for 4-bit, 8-bit, and 32-bit network, respectively.
CIFAR-1OCIEAR-100. We ResNet18 (He al., 60] as one of the most popular architectures and train them from scratch. Figure summarizes the results on CIFAR-I0 (Figure 4a 4h 4c
WP
SP WPladd) BPimuti
and CIFAR-100 (Figure 4d 4e 41
ImageNet. We conducted an experiment using pre-trained Con- vNeXt (Liu et al. 2022) on ImageNet (Deng et al.) 2009). The ConvNeXt designed to be energy efficient by its network archi- tecture; c.f., it heavily utilizes depthwise convolutions reduce the number of mu mt. Results are summarized in Figure
Summary. In all ranges of accuracy, add-shi ft-add net- work trained with Bit-Pruning consumes less energy than that of mult -add network trained with Weight-Pruning. In particular, huge energy saving is observed in high accuracy regime, demon- strating the superior data-adaptivity Bit-Pruning over Weight-Pruning. We a similar trend for the lazge-sized, already efficient networks (ConvNeXt) on a large-scale dataset (ImageNet). The number of add (in add-shift-aa
Figure 5: Result of ConvNeXt-B on ImageNet. 8bit activation. See Fig- ure 4 for legend.
network) and mult (in mult-ade network) can be com- pared by multiplying M (number of bits for weight) to the add count in the plots (Because we have Fmul= M Fa from Table D Section 3.3) The number of add (from Bit-Pruning) larger than mult (from Weight-Pruning. however; their ratio is significantly less than M: therefore, we'll get the better trade-off.
Notice that the Bit-Pruned network does not show efficiency in mult-ade representation (BP(mult)), and the Weight-Pruned network does not show efficiency add-shift-ade rep- resentation (WP(add)). This is because the Bit-Pruned network prefers sparse weights in binary format (Figure od]. which is not necessarily zero, contrary, the Weight-pruned network prefer small weight, which not necessarily sparse in binary format (Figure

---

Published as a conference paper at ICLR 2023
10a
10
3
105
Frequency (log)
Frequency (log)
201
201
107
20
10R
-100
Weight Initial (Kaming uniform)
Weight
(b) Naive
107
10F
Frequency (log)
201
10a
Frequency
201
20
10
10-
-100
50
0
50
100
-100
-50
50
100
Weight
(c) Weight-Prone Pruning
(d) Bit- Bit-Pruning (Ours)
Figure 6: Weight histogram 10/ 13th convolution layer). Naive a model trained using only task loss (test accuracy: 93.1 16). We choose a model performing around 93.0% test accuracy for Weight- Pruning and Bit-Pruning. The color indicates the number of addition for sod-shif--ade
4.2 ABLATION BASE BIT WIDTH
Can obtain a better accuracy-energy trade-off by pruning bits from wider bit tensor? We conducted the accuracy/energy trade-off comparison between Bit-Pruning with different base bit widths (4-bit/8-bit) (Figure 7 The result shows the network with wide tensor (8-bit) shows better trade-off than one using a narrow tensor (4bit). We consider it is because a network with wide bit has more choices for selecting good PoT values, suggesting that better bit-sparse network is obtained from the larger design space, just like the case Weight-Pruning (Frankle Carbin
2018
4.3 LEARNED WEIGHT DISTRIBUTION
Figure 6 visualizes histograms of learned weight. Starting from the initial uniform distribution, both Bit-Pruning and Weight-Pruning drive most of the weight to zero; this corresponds to removing mult in the mult-add network and add in the add-shift-ache network, respectively. In Bit-Pruning, most of the remaining nonzero weights are concentrated on sparse values binary format, such POT. Whereas,
did not show such particular preference.
5 RELATED WORKS
ShiftAddNet (You et al. 2020 is mult-less DNNs utilizing shift ft and add for efficient train- ing and inference. It realizes a better accuracy/energy trade-off than the former study, AdderNet (Chen al. 2020), utilizing only add. Ours and ShiftAddNet have similarities in that both utilize shift and add to realize mult-less DNNs. However, their computational model is fundamen- tally different from ours; ShiftAddNet computes the interaction between weight and input by first computing the pro-put with PoT weight followed by the l norm with add weight. It does not have an equivalent mult-add dot-product. Its limited expressiveness and the non-differentiabiiity of the PoT operation make training harder (requiring specifically designed gradient computation) and prohibit it from reaching the accuracy the corresponding mult-add network. Ours can be learned as ordinal high-precision differentiable mult -add network; therefore, it is possible to reach the performance of the original network. Besides, motivation and target hardware different; AhiftAddNet aims to realize efficient training and inference dense vector-type processors, while
8

---

Published as a conference paper at ICLR 2023
target is hardware supporting unstructured sparsity solely for inference
F Instead, using the
cent extension, ShiftAdNNAS You et al. (2022), ours and ShiftAddNet can be combined, e.g,, the best combination of shift Stift- add and add-shift-adoe are automatically searched.
Lottery ticket hypothesis states that "dense, hundomly-nnitialzzenn feed-forward networks contain sub-networks that have the equivalent accuracy the original network (winning tickets): unstruc-
tured weight pruning naturally uncovers the winning tickets" (Frankle & Carbin 2018 Ramanujan
al. 2020p. The comparison of different bit widths Bit-Pruning (Section 4.2 might indicate the
lottery ficket hypothesis in bils; a network having more connections in terms of add-shi sshift-ad
representation (i.e., the large binary tensor B) has more chance of containing the winning ticket.
Bit-level sparsity have been considered in several areas. BSQ Yang et al. 2021 proposed novel technique for learning the layer-wise bit-width for mixed-precision networks by induc- ing grouped bit-label sparsity. The novel notion of bit-slice sparsity (sparsity within sliced subdivision of binary weight) introduced by Zhang et al. (2019) to realize efficient infer- ence emerging ReRAM-based DNN accelerators. Because their motivation for considering the bit-level sparsity differs from ours, their proposed method for inducing is also totally dif- ferent. Therefore, it cannot efficiently sparsify the add in the add-sin shi ff--tdddd network
15
4bit
CONCLUSION
abit
We propose Bit-Pruning, a novel framework for learning efficient mult-less sparse DNNs. The intensive evaluation shows prun- ing add (bit) of the add-shi ftdadd network is more energy efficient than pruning mult (weight) of the mult-add network.
6.1 LIMITATIONS FUTURE WORK
86
88
90
92
94
Learning weight moving cost. In Bit-Pruning, sparse add for efficient inference is achieved by guiding the weights closer to the proximal weight (sparse in binary format and close to the current weight) defined in ( The distance to current weight needs to be defined as weight moving cost Cme- In this study, we predefined it (Section 3.2). We consider this to be sub-optimal, and it might be possible to determine the cost (e.g.. by learning from data to achieve a better tradeoff, e.g., some weight can move a lot without affecting the task loss and vice versa.
15
4bit
Evaluation on quantization susceptible module. In this study, evaluate Bit- Bit-Pruning on simple classification tasks using sim- ple network modules such as conv. Some DNN modules are known to be more susceptible the object detection head (Li et al. 2019a), transformer module (Bamberg al. 2023). and recurrent module (He et al.] 20166). making it challenging quan- tize them in lower bit regime. Bit-Pruning might be an ideal choice for realizing energy-efficient inference on these modules because could virtually utilize sish-precision weight to achieve the accuracy of the original high-precisio network.
50
55
60
65
70
Accuracy 196)
quantization noise, such as
Figure 7: Ablation of bit width. CIFAR-10/100 (top/hottom).
Benchmark on real hardware We show the estimate of computational gain by Bit- Bit-Pruning using the basic statistic known in the literature (Table we have not evaluated the actual energy consump- tion, wall-clock time, or area cost in the actual hardware. The proposed add-shi thiff--ddd or exist- ing sparse mult-ada add models require specialized hardware that supports unstructured/dynamiie sparsity or optimized CPU implementation Kurtz et al. (020)) to exploit the sparsity to the full ex- computing architecture (Gholami
tent. The sparsity- -ware accelerators often utilize al.). We believe the Bit-Pruning is more suitable than Weight-Pruning for this architecture because add logic is cheaper in size and energy place in or near memory than mult logic.
For this reason, it is difficult to compare with Shift Shift in terms of computational efficiency. "Refer to Appendix E H for discussion and experiments for comparing the ability induce sparsity.

---

Published as a conference paper at ICLR 2023
REFERENCES
Lennart Bamberg, Arash Pourtaherian, Luc Waeijen, Anupam Chahar, and Orlando Moreira. Synapse compression for event-based aunvolutionall-nuraaaneeewvwee accelerators. IEEE Trans- actions on Parallel and Distributed Systems, 34(4): 2023.
Hanting Chen, Yunhe Wang. Chunjing Xu, Boxin Shi, Chao Xu. Qi Tian, and Chang Xu. Addernet: Do we really need multiplications in deep learning? In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1468-1477, 2020.
Tianlong Chen, Xuxi Chen, Xiaolong Ma, Yanzhi Wang, and Zhangyang Wang. Coarsening the granularity: Towards structurally sparse lottery tickets. In International Conference Machine Learning, pp. 3025-3039, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2009.
Anastasia Dietrich, Frithjof Gressmann, Douglas Orr, Ivan Chelombiev, Daniel Justus, and Carlo Luschi. Towards structured dynamic sparse pre-training of BERT. arXiv preprint arXiv:2108.00 06277, 2021.
Mostafa Elhoushi, Zihao Chen, Farhan Shafiq, Ye Henry Tian, and Joey Yiwei Li. DeepShift: Towards multiplication-less neural networks. In /EEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2359-2368, 2021.
Steven K Esser, Jeffrey McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S Modha. Learned step size quantization. In International Conference on Learning Repre- sentations, 2020.
William Falcon and Kyunghyun Cho. A framework for contrastive self-supervised learning and designing new approach. arXiv preprint wrXiv:2099.01144, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03633 2018.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pp. 291-326.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems, volume 28, 2015.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. Eie: Efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3)244--254, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance imagenet classification. In IEEE International Conference Computer Vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016a.
Qinyao He, He Wen, Shuchang Zhou, Yuxin Wu, Cong Yao, Xinyu Zhou, and Yuheng Zou. Effective quantization methods for recurrent neural networks. arXiv preprint
2016b.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.084155 2016.
Mark Horowitz. 1.1 computing's energy problem (and what we can do about it). In IEEE Interna-
fional Solid-State Circuits Conference Digest of Technical Papers, pp. 10-14, 2014.
10

---

Published as a conference paper at ICLR 2023
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chiell Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for MobileNetV3. In IEEE/CVF International Conference Computer Vision. pp. 1314-1324, 2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: arXiv: 1704.0466, 2017.
Yerlan Idelbayev and Miguel A Carreira-Perpinánn Low-rank compression of neural nets: Learning the rank of each layer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8049-8059, 2020.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 (canadian institute for advanced research). Jttt://///www. cs toroott..eaa//Kki2//cc3.3. html
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-100 (canadian institute for advanced research). URL http:://ww// cs. toronto.......... wwt/kkdiz/ciiiaaeeae. wtt..oorttt...abb/krir////eeeeee
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting activa- tion sparsity for fast inference on deep neural networks. In International Conference on Machine Learning. pp. 5533-5543, 2020.
Keyong Li and John Baillieul. Robust quantization for digital finite communication bandwidth (dfcb) control. IEEE Transactions on Automatic Control, 49(9):1573-1584, 2004.
Rundong Li, Yan Wang, Feng Liang, Hongwei Qin, Junjie Yan, and Rui Fan. Fully quantized net- work for object detection. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2810-2819, 19, 2019a.
Yuhang Li, Xin Dong, and Wei Wang. Additive Powers-of-Two quantization: An efficient non- uniform discretization for neural networks. arXiv preprint urXiv:/909) 1314,, 2019b.
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convo- lutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 806-814, 2015.
Zhuang Liu, Hanzi Mao, Chao-Yuse Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 11976-11986, 2022.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Inter- national Conference on Learning Representations, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Daisuke Miyashita, Edward H Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. arXiv preprint arXiv:1603.01033 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems, volume 32, pp- 8024-8035. 2019.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Raste- gari. What's hidden in a randomly weighted neural network? IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11893-11902, 2020.
Leslie N Smith and Nicholay Topin. Super-convergencee Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi- Domain Operations Applications, volume 1 11006, pp. 369-386, 2019.

---

Published as a conference paper at ICLR 2023
Ross Wightman. Pytorch image models.
http:://ggttaa..cmm///tgtttttt
-etoorch-imagemmooee modell 2019.
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. In Ad- vances in Neural Information Processing Systems, volume 32, 2019.
Huanrui Yang, Wei Wen, and Hai Li. DeepHoyer: Learning sparser neural network with differen- tiable scale-invariant sparsity measures. In International Conference on Learning Representa- tions, 2020.
Huanrui Yang, Lin Duan, Yiran Chen, and Hai Li. BSQ: Exploring bit-level sparsity for mixed- precision neural network quantization. In International Conference Learning Representations, 2021.
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Under- standing straight-through estimator training activation quantized neural nets. In International Conference on Learning Representations, 2019.
Haoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li, Sicheng Li, Zihao Liu, Zhangyang Wang, and Yingyan Lin. ShiftAddNet: A hardware-inspired deep network. Advances in Neural Information Processing Systems, volume 33, pp. 2771-2787 2020.
Haoran You, Baopu Li, Shi Huihong, Yonggan Fu, and Yingyan Lin. Shiftaddnas: Hardware- inspired search for more accurate and efficient neural networks. In International Conference on Machine Learning, pp. 35566-255801 2022.
Fin-Fang Zhang. Ching-Ein En Lee, Chester Liu, Yakun Sophia Shao, Stephen W Keckler, and Zhengya Zhang. SNAP: An efficient sparse neural acceleration processor for unstructured sparse deep neural network inference. IEEE Journal of Solid-State Circuits, 56(2):636-647, 2020.
Jingyang Zhang, Huanrui Yang, Fan Chen, Yitu Wang, and Hai Li. Exploring bit-slice sparsity in deep neural networks for efficient ceram-based deployment. In Workshop on Energy Efficient Machine Learning and Cognitive Computing, pp. 1-5, 2019.
12


Reviews:
[{"review_id": "t-fxeEU9jCC", "reviewer": "Reviewer_MKH6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors present an energy saving approach that replaces dot product operations with an equivalent add-shift-add operation. In the context of neural networks with integer weights, the authors demonstrate how such a technique can save significant amounts of energy while controlling for the accuracy degradation.\n\nThe paper introduces a theoretical foundation based on the binary decomposition of the dot product operation. They then replace the costly multiplication operations by unrolling additions up to the length of the binary encoding (e.g. 8 bits), and then use shifts and additions to reconstruct the dot product exactly. \n\nThe savings come from pruning at the binary level, where the authors introduce sparsity in a way that reduces the number of computations to be carried on.\n\nThe authors then present an empirical evaluation on a simulator, as they did not have access to real hardware that could realize this kind of operations.", "summary_of_the_review": "The authors propose an interesting paper that can be viewed as a further discretization not in terms of bit counts as it is common in the DL literature, but they go one step further and take energy consumption into account by means of replacing a core operation such as the dot product, with a simplified version implemented by additions and shift operations.\n\nThe paper raises some further questions: Are you training the whole network using your approach? If so, as you are training with the weights in full precision and simply adding a penalization in the form of a proximal weight, it is not clear to me what the benefits are of penalizing during the whole training history. Or do you train at high precision and then simply refine with your penalty?\n\nFurthermore, a regularization term in the loss, does not imply that the regularization term will have a cost of zero, except potentially when this is a significant part of the loss function. Do you employ a dynamic regime to rescale your regularization? i.e., do you increase your $\\eta$ parameter during training? or is it kept constant?.\n", "strength_and_weaknesses": "The major weakness is that the experiments were not carried on real hardware, which could demonstrate the benefits of the approach.\n\nThe introduction of the loss function that helps with the discretization is well defined, however, there are no more details on the impact it has on the optimization. The proximal weight function defines a discretization that is not smooth, and when we take into account the optimization step size, we could see oscillatory behaviors during training.\n\nThe empirical evaluation does a good job comparing the simulated energy consumption, however, more details on the intended training use-case would be beneficial (see next section).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kodsW5xd9rM", "reviewer": "Reviewer_79uo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the potential energy saving of implementing multiplication as a add-shift-add operation, and propose to utilize the bit-level sparsity of the weight to further reduce energy cost. The paper proposes a bit-sparsity regularization that promote weight to be sparse in binary format, and claim to achieve a better energy-accuracy tradeoff.", "summary_of_the_review": "In summary, the paper makes progress on the exploration of bit sparsity, and how to utlize it to improve energy efficiency. However, the missing discussion on closely related work, the lack of clearity in the proposed method, and the unfair experiment setting make it hard to evaluate the significance and true effectiveness of the proposed method. Also, the potential overhead of the add-shift-add scheme may also hinders the real application of the proposed random bit sparsity. Therefore I would recommend rejection for now.\n\n## Post rebuttal\n\nI would like to thank the author for the extensive responses. Overall I like the idea of exploring potential energy savings of the random bit-level sparsity via the add-shift-add scheme. Also, the additional comparison provided by the author on using add-shift-add on weight pruned model indicates improving efficiency with add-shift-add is not naive for any model, and the explicit exploration of bit-level sparsity is required. This enhances the significance of the proposed regularization method.\n\nAs for the proposed regularization-based training method, the response from the author largely resolves my doubt on the effectiveness and the design choices. I think this paper should be worth acceptance given all the additional results and explorations provided by the author. To this end I'm increasing my score to weak accept. ", "strength_and_weaknesses": "## Strength\n1. The paper provides interesting observation on the potential energy consumption improvement brought by add-shift-add operation\n2. The paper advocates the exploration of utilizing bit-level sparsity, which may inspire future work\n\n## Weakness\n1. From the novelty perspective, the paper ignores previous attempts on inducing bit sparsity, and didn't perform comparison on closly related work. See the novelty section in the next review question for details\n2. The paper does not clearly explain why the proposed regularization is effective. See clearity section of the next review question\n3. The paper does not perform fair comparison in the experiments, and miss discussion on important baseline and limitation. See quality section in the next review question for details", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XFPRwWYpS6O", "reviewer": "Reviewer_bvLy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The idea of this work is to first represent DNNs with add-shift-add format instead of mult-add. By doing so, they observe that the first accumulation step contribute a lot to the energy consumptions. As such, they propose the bit-pruning for the first add in terms of the shiftadd perspective.\n\nThe key is not representing weights in add-shift-add. Rather, the key is to regularize and prune the number of adds.\n\nI like the idea and am willing to take a further look at the implementation later on.", "summary_of_the_review": "In a nutshell, I recommend this paper due to their novel insights for bit pruning in terms of shiftadd representation of DNNs.\n\nBTW, I wonder how this would be efficient and fast on GPUs? I understand that may need customized CUDA implementations but am curious which stage are you currently in.\n\nAlso, more quantization works and bit pruning works should also be compared with. More baslines and better figures would further strenthen this paper.\n\nAs for the difference with ShiftAddNet, I buy the explaination that they are not direclty comparable. ShiftAddNet reparameterizes DNNs with bit-shifts and adds. This work \"translate\" multiplications with shift and adds and prune the first accumulation process.\n\nThey can even be merged as a hybrid model, like ShiftAddNAS [1] did.\n\n[1] ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks, ICML 2022", "strength_and_weaknesses": "Strength:\n\n* The idea sounds interesting to me.\n* The proposed bit-pruning method beats weight pruning across several experiments.\n* This paper clearly describe the most related background work, ShiftAddNet, and explain why it is not direclty comparable.\n\nWeakness:\n\n* Some math formulation could be more concise and not necessarily complex. Or you can give more high-level insights before diving into the math. Also, the figure quality is not enough, I strongly encourage the authors to revise their figure to the best level you can draw.\n* Apart form the energy cost, latency comparisons or number of operations are expected and should be more direct.\n* More quantization works and bit pruning works should also be compared with, I believe there are a lot bit-pruning works, either compare in a table with checks or compare in terms of experiments or both.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5woFJI555r", "reviewer": "Reviewer_Rw8U", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Bit-pruning proposes pruning at the bit level in the context of unstructured pruning methods. At its core, formulation presented by the Authors aims to replace the scalar-scalar multiplication with its equivalent scalar-Nbit multiplication followed by N-shifts and finally a sum over N shifted values. When extending this to a dot-product (the basic OP in matrix-matrix and matrix-vector multiplications), it can be seen as a add-shift-add OP. In this formulation to compute is dominated by the number of bit-level additions before the shift and therefore this is the metric to minimise. By means of an additional loss term that penalises the number of bits in the weights representation as well as the distortion this introduces, the proposed bit-pruning mechanism achieves large energy reductions for a fixed accuracy value. This method is evaluate on image classification with ResNets. ", "summary_of_the_review": "This is a good paper that shows how pruning at the bit level can yield much better energy-accuracy ratios than weight-pruning that prunes weights directly. Even when at lower bit-widths, bit-pruning (which uses 8-bit weights) outperforms the weight-pruning counterpart using 4-bit weights. I would had loved to see this method applied to smaller models (even if CNNs) or to Transformers (even if only a single ViT). ", "strength_and_weaknesses": "### Strengths\n*    The proposed method is simple and seems to deliver very good results, specially at higher accuracy ratios, where weight pruning understandably struggles to zero-out weights without impacting on accuracy.\n*    ImageNet results shows around 50% energy reduction, that's good.\n\n\n### Weaknesses\n*    Unstructured sparsity benefits from models being over-parameterised for the task (i.e. the task at hand is easy so a fair amount of weights can be discarded without impacting on accuracy). It would be interesting to see the performance of Bit-pruning when the simpler tasks (specially CIFAR-10) is done with a light-weight model, for example a small mobilenet.\n*    What approach was used to perform weight-pruning? using traditional magnitude-based pruning? or something else?\n*    Some key implementation details aren't clear from the text: are all convolutional layers using Bit-pruning? both 3x3 and 1x1 convolutions? Also the input layer?\n* Why all layers use the same $\\lambda$? Probably earlier layers would had benefit for less sparse weights retaining in that way some of the degradation that would otherwise propagate to the rest of the network. Another motivation for this is that early layers have comparatively much fewer OPs than deeper layers so the potential speedup might not be worth it. Could the Authors comment upon this? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster