S$^3$: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks
Authors: Xinlin Li
Bang Liu
Yaoliang Yu
Wulong Liu
Chunjing Xu
Vahid Partovi Nia
Venue: NeurIPS 2021 Poster (2021)
Keywords: Convolution Neural Network
CNN
Edge Computing
Edge
Quantization
Low-bit
Multiplication-free
Bit-shift

Abstract: Shift neural networks reduce computation complexity by removing expensive multiplication operations and quantizing continuous weights into low-bit discrete values, which are fast and energy-efficient compared to conventional neural networks. However, existing shift networks are sensitive to the weight initialization and yield a degraded performance caused by vanishing gradient and weight sign freezing problem. To address these issues, we propose S$^3$ re-parameterization, a novel technique for training low-bit shift networks. Our method decomposes a discrete parameter in a sign-sparse-shift 3-fold manner. This way, it efficiently learns a low-bit network with weight dynamics similar to full-precision networks and insensitive to weight initialization. Our proposed training method pushes the boundaries of shift neural networks and shows 3-bit shift networks compete with their full-precision counterparts in terms of top-1 accuracy on ImageNet. 

S Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks
Xinlin Li', Bang Liu?, Yaoliang Yu', Wulong Liu', Chunjing Xu', and Vahid Partovi Nia'
Noah's Ark Lab, Huawei Technologies. *Department of Computer Science and Operations Research (DIRO), University of Montreal. *Cheriton School of Computer Science, University of Waterloo.
Abstract
Shift neural networks reduce computation complexity by removing expensive multiplication operations and quantizing continuous weights into low-bit discrete values, which are fast and energy-efficient compared conventional neural net- works. However, existing shift networks are sensitive to the weight initialization and yield degraded performance caused by vanishing gradient and weight sign freezing problem. To address these issues, we propose S re-parameterization, a novel technique for training low-bit shift networks. Our method decomposes a discrete parameter in sign-sparse-shift 3-fold manner. This way. it efficiently learns low-bit network with weight dynamics similar full-precision networks and insensitive to weight initialization. Our proposed training method pushes the boundaries of shift neural networks and shows 3-bit shift networks compete with their full-precision counterparts in terms of top-1 accuracy on ImageNet.
1 Introduction
While deep neural networks (DNNs) have achieved widely success in various tasks, the training and inference of DNNs usually cost prohibitive resources due to the fact that they are often over- parametrized and composed of computational costly multiplication operations for both forward and backward propagation [28). To enable the application feasibility of DNNs in resource-constraine scenarios, substantial efforts have been made to reduce the computation complexity of neural networks while preserving their accuracy. Among different proposed approaches for this purpose, low-bit neural networks with binary weights (6 [22] or ternary weights [5] are recently designed to replace the expensive operations like multiplication with cheaper ones, e.g., replacing multiplication with sign flip operation during inference.
We focus on low-bit shift neural networks 11 30 that replace multiplication with the bit-shift operation. Following the conventional linear algebra notations, we denote scalars by x, vectors in bold small letters x, and matrices by bold capital letters X. Traditional neural networks require feature
computing the inner product w T x = E. Wirr, where w is the weight vector and vector. In comparison, shift neural networks limit the values of the weight vectors to set of discrete values Wshin E {0} U (+22). }. In this case, multiplication can be replaced with bit-shift operations, multiplying w; = 2 is equavilent to shifting places to the left (if is positive) or to the right (if p is negative). The bit-shift operation alone only covers discrete neural networks with positive weights, therefore, most shift neural networks further add sign flip operation the shift to allow w; = {- Inference with bit-shift operations is highly hardware friendly and can save up to 196x energy cost FPGA and 24x ASIC compare their multiplication counterparts 28
Several works aim to further improve the memory efficiency of shift neural networks by reducing the bit-width of the weights. Zhou et al. 30 propose to fine-tune pre-trained full-precision weights
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

with a progressive power-of-two weight quantization strategy. They suggest to keep portion of weights unquantized during fine-tuning to improve model performance. DeepShift (9) is the state-of- the-art technique for training low-bit shift networks from random initialization. Initialization with pre-trained full-precision weights provides significant performance gain for DeepShift. All latest techniques heavily rely on the initialization of full-precision pre-trained weights, which implies some optimization difficulties exist in the current training diagram of low-bit shift networks. However, although extremely memory efficient and hardware friendly, low-bit shift networks suffer from significant accuracy drop on large datasets compared to their full-precision counterpart, and the performance of them is sensitive weight initialization.
To address these problems, in this work, we analyze the current training diagram of low-bit shift neural networks, and find that the accuracy degradation and the weight initialization sensitivity of the low-bit shift networks are caused by the design flaw of the weight quantizer during training. Specifically, training shift neural networks is different from training full-precision networks. The gradient-base based optimization algorithms are designed for optimizing continuous variables, but the weights of shift neural networks are discrete values. To learn the discrete weights with a gradient-based approach, it is necessary to exploit weight e-paaaaeterizationn One of the most widely adopted discrete weight re- p--pammtterrzation technique is using a quantizer function to map a continuous parameter w E IR to a discrete weight w E {01, 02 ... Ui } with possible values. As the weight quantizers are generally non-differentiable functions, a gradient approximator, such straight-through estimator [3. is utilized to approximate their derivatives during training. Our analysis shows the quantizer leads to severe gradient vanishing and weight sign freezing.
In this paper, we design an effective training strategy to combat the gradient vanishing and weight sign freezing problem in low-bit shift networks. We first impose dense weight priori during training. which equals maximizing the to norm of the discrete weights. However, optimizing to norm during training is a combinatorial problem, instead we propose S re-parameterizzttin and regularize the sparsity parameter. 5 is a new technique which re-parameterizes the discrete weights of shift networks in sign-sparse-shift, 3-dimensional augmented parameter space to disentangle the roles of quantization values, and hopefully train more effectively thanks to more orthogonal axes in the augmented space. Although method introduces extra parameters during training, its memory occupation under low-bit is slightly higher than the full-precision network since the additional memory overhead depends the weight size. In contrast, the activation size dominates the training memory occupation, especially using a large batch size.
We evaluate our approach the ImageNet dataset. While all previous methods require at least 5-bit weight representation to achieve the same performance as the full-precision neural networks on large datasets such as ImageNet, our experimental results show that our proposed method surpasses all previous methods, pushes this boundary further to 3-bits. Besides, our approach requires no complicated weight initialization or training strategy. Moreover, we define two indices of weight dynamics, named weight sign variation rate (WSVR) and weight low-value rate (WLVR), and compare the trend of these two indices during the training process of different methods. Experiments show that the weight dynamics of the ternary weights trained with our proposed technique better align with the weight dynamics of full-precision weights than trained with a traditional quantizers. This desirable dynamics is caused by our re-pat parametrization, and is the key to efficient training. We hope that this opens the possibility for methods similar S to be used in training low-bit networks.
2 Related Works
Different approaches have been proposed to reduce the computational complexity of neural networks. A type of the popular multiplication-fre free neural networks is low-bit neural networks with binary weights 6 |2] or ternary weights [15]. However, they suffer from under-fitting on large datasets, leading to an accuracy gap compared with their full-precision counterparts. Other approaches include replacing multiplication operations with addition operations
5 25 26 or bit-shift operations []] 9 30]. Although these approaches achieve a low accuracy drop on large datasets, they require higher weight representation bit-width. Other follow-up works try to improve the accuracy of multiplication- free neural networks by replacing multiplication with both addition and bit-shift operation (28). sum of binary bases [7 29], or sum of shift kernels [6 However, their computational cost is high as they use multiple operations per kernel.
2

---

Training shift neural networks with discrete weight values is challenging task. Many weight re- parameterization approaches have been proposed to solve the challenge. One way is using stochastic weight 6 18 23 but these methods suffer from the slow computation of sampling. Another way is map or threshold continuous weights to discrete values. Since the thresholding function is non-differentiable, the backward gradients across the quantizer are approximated by a gradient approximator. The choice for quantizer gradient approximation.
utilizing a quantizer function 66 22 [.]
(STE)
is a popular 3
There are also works study the vanishing gradient problem (VGP) that appears frequently in deep neural network training. [ [0] point out that the zero values in full- full-precision weights lead to VGP when training highly pruned neural networks, which eventually jeopardizes the accuracy of the model. 8 (0) suggest to maintain a portion of the full cell-precision weights during quantized network training improving the performance of the model. 30) shows that applying progressive quantization on low and high value weights differently can further improve the performance of the trained model. Quantizing high-value weights first and keep more low-value weights remains full-precision during training helps to achieve better shift neural network compared with randomly applying progressive quantization strategy on weights regardless their values. This observation coincide with our analysis, and we argue that the performance gain of progressive quantization methods come from the gradient flow preserved by maintaining a portion of the negative impacts caused by weight quantizers still missing. To the best of our knowledge, we are the first to study the VGP caused by weight quantizer design.
full pellipreiiionn weight during training. However, the study
Training Low-bit Networks
We first introduce the general form of the weight quantizers for discrete weight training, and then explain how quantization function leads to training difficulties. In low-bit networks, including shift networks, the weights are discrete values while usually the gradient-based optimization algorithms are designed for optimizing continuous variables. Specifically, quantizer function one of the most widely-uset approach for training discrete weight neural networks. It maps continuous weight to one of the & discrete weight values Q(w) E {1), U2.... }.
Several quantizers of low-bit networks 30 are in the following form of staircase function:
15 31 27
and shift networks
01
122
h
v w
< l2
Q(w)
(1)
Uk
tk S
However, training quantized neural network with a quantizer function the form Equation | is challenging due two issues, vanishing gradient (VGP) and weight sign freezing, We discuss the vanishing gradient issue section 3.1 and the weight sign freezing issue in section 3.2
3.1 Gradient vanishing
To clarify further why quantization and VGP are entangled, suppose ReLU-activated fully-coneccceeeeee layer can be generally formulated as:
W Wio
w 01
= ReLU
(2)
w
The gradient update toward oy can be calculated as:
@Lss ØLoss
0r
@oss Oz
W + 0zlt1 2 + Wha
W
(3)
0x Ox
a(W =0 + W 21
10
Derivative of ReL.U
And the derivative of ReL.U is,
Or 0x1+1
[1 Wo What > W What 50
(4)
a(W %6 + WT Who
3

---

Waig gelxoout
Weer (-1,0,1)"
Wigaase
0.73 0.10
STE
W spaars E Rinxout
sparse)
Whe
4.20 0.21
1
Wsign
0.01 0.91 0.51
STE
Figure 1: Left panel: decomposing Wter in Equation (6). Blue vectors demonstrate the original ternary space, while the black vectors show the decomposed augmented space. The augmented parameters are projected back to the ternary space after multiplication. Right panel: efficient back propagation on Wher in the 5 re-parameterized space.
025
6:45
3.5
415
(c) 7-th layer. Figure 2: Density polygon of the weights in the multiple layers of ResNet20 trained on CIFAR- 10, full-precision (blue), versus ternary before passing through the quantization function (orange). Full- precision and quantized training have different dynamics. Ternary weights are kept away from zero, preventing weight sign change.
(a) 3-th layer
(b) 3-th layer, ternary
(d) 7-th layer, ternary
By observing Equation B). we can see that the gradient information Boos propagating from x²+1 0z
to a vanishes in two distinct situations: i) the corresponding derivative ReLU is zero. (i.e. the neuron not activated), and ii) the corresponding weight W is zero. Both scenarios lead gradient information loss, and ultimately cause inferior performance even after training convergence. The situation i) is widely known as the Dying ReLU issue caused by the sparse derivative of ReLU. with non-sparse derivatives are developed remedies.
Several ReLU-variants 20 12
The VGP caused by the quantizer similar to the dying ReLU issue. As the first step, we motivate S technique on a ternary network and then generalize it further for higher bit shift networks. The ternary quantizer function ( contributes to situation ii) by mapping portion of non-zero full-precision weights w to the exact zero value Wter = (w). This leads to smaller to norm for the discrete weight term Wher compared its full-precision counterpart w, [IQ(w)]l llo S ||w||o- Shift networks exchange the costly multiplication operation with cheap shift operation, so inherently it is non- uniform quantized network and faces the same problem. The VGP caused by the quantizer the main optimization difficulty in training shift neural networks as well. This issue appears more severely in low-bit networks which include fewer quantized range with higher sparsity rate.
3.2 Weight sign freezing
Another issue of the low-bit quantizer is the weight sign freezing effect. This effect prevents the weight sign variation and leads to different weight dynamics between the full-precision weights and the low-bit discrete weights.
The weight dynamics of full-precision weights are as follows. In general practice, the weights are initialized with a zero-centred distribution in small variance and trained with & regularization, so they stay close to the origin point during training. When the full-precision neural network train with noisy gradient updates, such as an SGD optimizer and large learning rate at the beginning of training, weights oscillate around the origin point. With the learning rate decreasing, more and more weights converge to specific sign representing the corresponding input feature, which correlates to the prediction output. The remaining weights whose input feature is neither positively correlated nor negatively correlated are concentrated in the region close to the origin point to oscillate.

---

However, we notice that the weight dynamics of low-bit discrete weights trained with quantizer are dramatically different. We train ResNet-20 models with full precision weights and ternary weights using quantizer and compare the histogram variation of the same layer in Fig. 2 Take ternary quantizer [15] as an example. The histogram variation of a ternary weight trained with the quantizer has two peaks. They are symmetric to zero; this is because the positive and negative thresholds of the ternary quantizer attract a large portion of weights oscillating around them and prevent the weights across zero and switching their sign. The intuition of full full-precision weight dynamics and the observation from weight histogram comparisons inspire us to design the experiments in section 5.2 Our experiments show that the ternary networks trained with a quantizer have very low WSVR during the whole training process. We call it the weight sign freezing issue.
S Re-paraaetterizttion
We suggest to initialize the networks with dense weight prior, which is equivalent to maximizing the 4n norm of the weight tensors at initialization. We initialize ternary weights away from zero so that VGP is partially solved at early training epochs. Dense weight initialization provide additional benefits for low-capacity models, and encourages training around non-zero weight values. This characteristic empirically increases the model capacity and alleviate under-fitiing problem commonly observed in low-bit models.
The remedy proposed initialization only solves training difficulty at early training. Efficient training over discrete weights still remain slow. As the follow up remedy we propose to re-parameterize the discrete weight of shift networks into sign-sparse-shiff 3-fold manner, parameterization allows us to regard the low bit network as projection of an augmented space, in which single parameter takes care sparsity, single parameter takes care of of switching signs, and another parameters takes care of the weight magnitude. This decomposition has two benefits. First, it promote the weight sign variation by allowing the discrete weight switch between positive and negative values without approaching to zero. Second, it allows to control the amount of sparsity by controlling the sparsity parameter during longer epochs.
call $ This re-
4.1 Ternary network
Almost all ternary networks such as 15 are trained with the following quantizer
A < w
Wher Qa(w)=
A < A w < -A
(5)
We propose to replace the quantizer the form of with the following $ re-parameterized counterpart, see Figure
Wher = ((Waparee) (21 (wiign)))) - -1}.
(6)
in which 1(-) is the Heaviside function, i.e. (xx if x and equals zero otherwise. Note that w, Wspaase, Usign are full precision parameters, and ternary weights are recovered after passing through the Heavidside function.
The 3 ce-parameterization approach for ternary weight training decomposes the discrete ternary weight into two binary parameters, one representing sign, another representing sparsity. Motivated from the ReLU activation, we propose to impose dense weights by penalizing the loss function with the magnitude of negative weights. This is simple regularization that adds little to training computation and is effective to enforce dense weights.
Resaarse Wsparse) maxx - Wsparse; 0)]l2
4.2 Shift networks
We further generalize our approach toward shift neural networks. Following 9 the discrete weight of shift networks can be decomposed into two parts: a ternary term Wher and a scaling term composed
of power of numbers 2 that shifts the network S bits Wshin Wer 25 shifting to left or right depends on the sign of S e E.
5

---

W [Rlxxout
1(W
Previous Layer
2
x R
W
I(W{2)
0.03 0.50 0.34
W
E (0, + 1, 4)
x
W
shin
W
xouut
21( sign)
lign
055-2224
240
Den
x mut
W
R Inxout
1(W sppasse
sparse
0.41
STR
RRW
|mmax(
W
Sparse
0)[|,
Loss
Figure 3: Illustrating the overview of 3bit shift network trained with S re- parameterization and dense weight regularizer (forward computation dark green, backward in light red)
We apply the decomposed Wher Wsign Usperse described in the session 4.1 and re- re-write shift networks in terms of the decomposed ternary weights. Following [9] we focus on positive S values. The re- parameterization for the negative S values can easily be achieved by adding constant bias S. We propose to re-parameterize this shift weights as a combination of binary variables recursively.
Sg = 0, S, = 1(w,)(S,-1+ + 1).
(8)
For instance, 8 reparameterizes a 3-bit shift networks whose discrete weight values are Wshif (0, +1, +2, +44 to:
Wshiit = 25 (Wsparse) (21 (Wsign) - 1}.
(9)
while S2 = (1w + 1}, so it reduces to ternary network for t = 0. Note that in this reparametrization all weights (Wainn Wsparse W1, W2) are trained in full precision. demoon- strates forward pass and back propagation using this reparametrization.
Ultimately
[(w) = Loss(w) w) + XR(w) + a'Raparse (w)
optimized, which R the & norm and a > 0 is a regularization constant controlling the network sparsity during training.
5 Experiments
In this section, we describe our experiment setup and benchmark S pe-aaamtttrriztiion over SOTA low-bit DNNs. Then we compare the dynamics of weights with different training paradigms to show our method could better align with the training of full-precision models. Finally, we present ablation studies of the dense weight regularizer and the number of training epochs.
5.1 Benchmark 3 re-parameterization over SOTA low-bit DNNs
Models and datasets. We evaluate our proposed method on ILSVRC2012 7 dataset with different bit-widths to demonstrate the effectiveness and robustness of our method. We use ResNet-18 and ResNer-50 as our backbone with the same data augmentation and pre-processing strategy proposed in 13.. Following common practice in most previous methods 22 19 29), all convolution layers are quantized except for the first one.

---

Table I: Comparison of SOTA methods using ResNet- 18 trained ImageNet
Kernel operation
Methods
Bit- Width
Initialization
Top-1 Acc. (%)
Top-5 Acc. (%)
FP32
32
Random
69.6
89.2
Multiplication
TTQ
2
Pre-trained
66.6
87.2
Lq-Net
29
2
Random
68.0
88.0
Sum of Sign Flips
Lq-Net
29
3
Random
69.3
88.8
Lq-Net
29
4
Random
70.0
89.1
BWN [22]
Random
60.8
83.0
HWGQ 4
Random
61.3
83.2
BWHN [4
Random
64.3
85.9
RR-net 21
Random
66.5
86.8
Sign Flip
TWN
Random
61.8
84.2
LR-net
23
2
Random
63.5
84.8
SQ-TWN 8
2
Random
63.8
85.7
INQ E0
2
Pre-trained
66.02
87.13
Ours
Random
66.37
87.18
INQ
30
3
Pre-trained
68.08
88.36
INQ
80
4
Pre-trained
68.89
89.01
INQ
60
5
Pre-trained
68.98
89.10
Shift + Sign Flip
DeepShift
Random
65.63
86.33
DcepShift
Pre-trained 68.32
88.41
Ours
3
Random
69.82
89.23
Ours
4
Random
70.47
89.93
Table 2: Comparison of SOTA methods using ResNet-50 trained on ImageNet
Keree operation
Methods
Bit Width
Initialization
Top-1 1 Acc. (%)
Top-1 Acc. (%)
Multiplication
FP32
32
Random
76.00
93.00
INQ [30]
Pre-trained
74.81
92.45
Shift Sign Flip
DeepShift
9
6
Pre-trained
75.29
92.55
Ours
Random
75.75
92.80
Training settings. We train the networks with 200 epochs utilizing the cosine learning rate, and the initial learning rate le-3. The networks are optimized with SGD optimizer, and the momentum and weight decay are set to 0.9 and le-4 respectively. The hyper-paramett parameter a of dense weight regularizer set to le-5 without using decay scheduler.
Baselines. We evaluate the proposed approach over two SOTA methods for training low-bit shift networks, including DeepShift (9] and Incremental Network Quantization (INQ) (30]. We also compare the results of shift network to other neural networks with higher computational cost, including multiplication-base ConvNet in full-precision
[33 and LqNet
(29
Experiment results. The results summarized in Table Table 2 show that our method signifi- cantly improves the accuracies of low-bit bit multiplication-fre networks. Our proposed method on the 3-bit shift networks has the most exciting result: it achieves state-of-the-an accuracy of low-bit shift networks with Top-1 accuracy of 69.82% ResNet- 18 and 75.75% on ResNet-50 without extra technique at initialization or during training.
5.2 Compare and analyze the dynamics of weights with different training paradigms
The dynamics of weights during training reveals the learning process of neural network. Here define two indices reflecting the weight variation behaviour during training, namely weight sign variation rate (WSVR) and weight low-value rate (WLVR). Then compare the variation trend of them among different models during training.
Weight Sign Variation Rate (WSVR) indicates the frequency of weight sign variation during training. To measure WSVR, we take weight snapshots every 10 epochs during training and measure the percentage of weights with different sign between two neighbouring snapshots as the WSVR.

---

Full-Precision Network
0,30
Full-Precision Network
Waight Sign Variantion Rate
025
Temary Network (5*)
Ternary Network (5?)
Terarr Network (Quantizer)
Ternary Network (Quantizer)
0.20
Weight Low-Value
0.15
0.15
0.10
0.35
0.00
0.00
80
202
Epochs
Epochs
(a) Weight Sign Variation Rate
(b) Weight Low- Value Rate
Figure 4: Weight dynamics comparison using ResNet-20 trained on CIFAR-10. 10. Each curve represents
one layer in the model. The colour of the layer gradually becomes opaque from front to end.
Weight Low-Value Rate (WLVR) indicates the percentage of weight values relatively close or equal unified index of continuous and discrete weights reflect their generalized
to zero. We sparseness during training. For discrete weight networks, we consider weight sparsity as WLVR. For model with continuous weight trained without an regularizer, the low-value weights oscillate in neighbouring region of zero. To measure WLVR, we normalize the weight tensors of each layer to the range between -1 and 1, then count the percentage of weights between -0.03 and 0.03 WLVR. Our conclusion relies on the variation trend of the curves, which is not significantly affected by this pyper-parammter.
Experiments setup. We compare the two indicators mentioned above on three models: a full- precision model, a ternary weight model including quantizer [15], and an S re-parameterized ternary weight model described in section 4.1 This analysis chooses the ResNet-20 model as the backbone and trains all three models the CIFAR-10 dataset for 150 epochs with cosine annealing learning rate and the initial learning rate set to 0.1. All models are converged and reach a reasonable validation accuracy on CIFAR1O (> 91%). We measure WSVRs and WLVRs for each layer in all three models during training and summarize the results in Figure 4
Result analysis. We can observe from Figure 4a that the full-precision network has high WSVRs at the beginning training and then slowly decreases to zero in the end, indicating that the weight signs oscillate rapidly initially and gradually stabilize in the end. On the contrary, the ternary weight model with a traditional quantizer remains low WSVRs during the whole training process, indicating that the weight sign variation is not frequent as its full- precision counterpart. Although our proposed approach has a different trend of WSVRs with the full-precision model initially, it is in good alignment in the rest of the training stage.
Figure [b) shows that the full full-precision network has low WLVRs in the early stage of training and then slowly increases, indicating that the portion of low-value weights increases during the training of the full precision model. However, the WLVRs variation trend the ternary weight network with quantizer is toward the opposite direction-the weight sparsity of the ternary weight network trained with a quantizer decrease during training. On the trend of this indicator, our method better aligns with the full-precision model.
We argue that the design of 3 re-parameterization the key to better alignment with the weight dynamics. By decomposing into Wsign and Waparse. the discrete weight can oscillate between -1 to I values directly, and the dense weight regularizer keeps the w Wsporse stay one as long as possible until both +1 and -1 status increasing the loss value and the gradient signal from the optimizer push the discrete weight values zero status. This decomposition design leads to weight dynamics very close to the full-precision weight dynamics described in section 3.2 i.e. oscillation and slowly converge to a specific weight sign, otherwise push to zero.
5.3 Ablation studies
In this section, we verify the efficiency and robustness of our method via extensive experiments.
8

---

o=0
1e-3
Testing Accuracy (%)
c= 1e-4
Training
2m 1e-6
a-0
20
g 1e-3
1e-5
19
202
250
Epochs
Epochs
(a) Training Loss
(b) Testing Accuracy
Figure 5: Sensitivity analysis for dense weight regularizers without using decay scheduler
Table 3: Dense weight regularizer hyper-parameter comparisons using ResNet- 18 ImageNet
Decay
ResNet- 18
Scheduler
ImageNet
le-2
le-3
lc-4
le-5
le-6
Top-1
69.78
69.85
69.84
69.82
69.83
None
Top-5
89.21
89.24
89.23
89.23
89.23
Top-1
69.91
69.68
69.74
69.62
66.70
Linear
Top-5
89.23
89.06
89.13
89.10
86.93
Top-1
69.91
69.85
69.85
69.57
63.82
Cosine
Top-5
89.15
89.20
89.38
89.04
86.50
Dense weight regularizer. We verify wide range of hyper-parameters of the dense weight regular- izer and two decay schedulers: linear decay and cosine decay, to prove our method's efficiency and robustness. We choose the 3-bit S re-parameterized shift network and ResNet- backbone on the ImageNet dataset as the test case. All hyper-parameters are the same as described in section 5.1] The ablation study results are summarized to Table
The results show that the performance of S re- parameterized networks insensitive to the choice of value of the dense weight regularizer. The decay schedulers for have limited impact the performance of the trained model as well, indicating that the main benefit dense weight priori comes from the early stage of training. This result implies the performance gain from initialization with a pre-trained full-precision model may come from the good initial estimation of weight sign from the pre-trained weights.
Although the experiment results are consistent across wide range of values and two different decay schedulers, Figure 5 shows that the dense weight regularizer is indispensable part to train an S re-parameterized network-otherwis the training suffers from a convergence problem.
Training epochs. Due to their frequent sign variation instability, binary parameters' training requires Since the number of training epochs is critical for binary parameter training, we verify the neural network performance affected by We choose the 3-bit $ re-parameterized shift network and ResNet- 18/50 backbone on ImageNet dataset the test case. All hyper-parameters are the same as described in section [.1 except for the number of epochs. Table shows that the network performance can significantly improve with more epochs. With the training of 200 epochs, we close the accuracy gap between 3bit shift networks and full-precision CNN on BosNeet8//50 ImageNet experiments.
more epochs compared to their full-precision counterparts 6
Conclusion
Although low-bit shift networks are memory efficient and hardware friendly during inference, they are not able to achieve competing performance with their full-precision counterparts due to the gradient vanishing and weight sign freezing problems when trained by existing methods. Our proposed $ reparametrization technique efficiently addresses these issues and bridges the accuracy gap between

---

Table 4: Training epochs comparisons using ResNets trained on ImageNet
R Repaarso
Epochs
Experiments
Accuracy
90
120
150
180
200
ResNet-18
Top-1
68.
69.24 24
69.28
69.63
69.83
lc-5
ImageNet
Top-5
88.35
88.77
88,93
89.16
89.23
ResNet-50
Top-1
74.67
75.47
75.76
75.75
le-5
ImageNet
Top-5
92.27
92.60
92.72
/
92.80
low-bit shift networks and their full-precision counterparts by carefully design decomposed space for optimization so the discrete weight dynamics can match the continuous weight dynamics. Extensive experiments and ablation studies demonstrate the superior effectiveness and robustness of our method for training low-bit shift networks. Moreover, alignment between the dynamics of weights in training low-bit shift network and its full-precision counterpart. Our future work is to further explore the theoretical ground of our reparametrization technique and weight dynamics.
show that S reparametrization enables better
Acknowledgement
We would like to thank the area chair and the anonymous reviewers for their thoughtful discussions that led to significant improvement of the paper. We want to thank our colleagues Yunhe Wang and Chao Xing for providing us with helpful suggestions during the initial development phase of the paper. We also enjoyed technical discussions with Mostafa Elhousie about DeepShift networks. We appreciate the initial hardware cost analysis by our colleague Seyed Alireza Ghaffari.
References
[1] Simon Alford, Ryan Robinett, Lauren Milechin, and Jeremy Kepner. Training behavior of sparse neural network topologies. In 2019 IEEE High Performance Extreme Computing Conference (HPEC), pages 1-6. IEEE, 2019.
[2] Milad Alizadeh, Javier Fernández-Marqués. Nicholas D Lane, and Yarin Gal. An empirical study of binary neural networks' optimisation. In International Conference Learning Representations, 2018.
[3] Yoshua Bengio. Estimating or propagating gradients through stochastic neurons. CoRR, abs/1305.2982,, 2013.
[4] Zhaowei Cai, Xiaodong He. Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by hale-wave gaussian quantization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5918-5926, 2017.
[5] Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. Addernet: Do we really need multiplications in deep learning? In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 1468-1477, 2020.
[6] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pages 3123-3131, 2015.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-258 leee, 2009.
[8] Yinpeng Dong, Renkun Ni, Jianguo Li, Yurong Chen, Jun Zhu, and Hang Su. Learning accurate low-bit deep neural networks with stochastic quantization. arXiv preprint 1rXiv: 2017.
[9] Mostafa Elhoushi, Zihao Chen, Farhan Shafiq, Ye Henry Tian, and Joey Yiwei Li. Deepshift: Towards multiplication-less neural networks. arXiv preprint
2019, 2019.
10

---

[10] Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint vrXiv:1906.10735 2019.
[11] Denis A Gudovskiy and Luca Rigazio. Shiftenn: Generalized low-precision architecture for inference of convolutional neural networks. arXiv preprint arXiv:/706.0220 02393, 2017.
[12] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, In Proceedings of the IEEE international conference on computer vision, pages 1026-1033 2015.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[14] Qinghao Hu, Peisong Wang, and Jian Cheng. From hashing to cnns: Training binary weight networks via hashing. In Thirty-Second AAAI conference on artificial intelligence, 2018.
[15] Fengfu Li, Bo Zhang, and Bin Liu, Temary weight networks. arXiv preprint 1cXiv:1605.047111 2016.
[16) Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks. In International Conference on Learning Representations, 2019.
[17] Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. Advances in Neural Information Processing Systems, 30, 2017.
[18] Zhouhan Lin, Matthieu Courbariaux, Roland Memisevic, and Yoshua Bengio. Neural networks with few multiplications. arXiv preprint arXiv: arxiv:/510.03009, 2015.
[19] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang- Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European conference on computer vision (ECCV), pages 722-737, 2018
[20] Andrew Maas, Awni Y Hanmun, and Andrew Ng. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3. Citeseer, 2013.
[21] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2250-2259, 2020.
[22] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pages 225-542.60 Springer, 2016.
[23] Oran Shayer, Dan Levi, and Ethan Fetaya, Learning discrete weights using the local reparame- terization trick. arXiv preprint arXiv: 2017.
[24] Wei Tang, Gang Hua, and Liang Wang. How to train a compact binary neural network with high accuracy? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[25] Yunhe Wang, Mingqiang Huang, Kai Han, Hanting Chen, Wei Zhang, Chunjing Xu, and Dacheng Tao. Addernet and its minimalist hardware design for energy-efficient artificial intelligence. arXiv preprint arXiv:2101.1001 2021.
[26] Yixing Xu, Chang Xu, Xinghao Chen, Wei Zhang, Chunjing Xu, and Yunhe Wang. Kernel based progressive distillation for adder neural networks. In NeurIPS, 2020.
[27] Zhaohui Yang, Yunhe Wang, Kai Han, Chunjing Xu, Chao Xu, Dacheng Tao, and Chang Xu. Searching for low weights in quantized neural networks. In NewrIPS, 2020.
[28] Haoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li, Sicheng Li, Zihao Liu, Zhangyang Wang, and Yingyan Lin. Shiftaddnet: A hardware-inspired deep network. In NewrIPS, 2020.
11

---

[29] Dongging Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantiza- tion for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pages 365-382, 2018.
[30] Aojun Zhou, Anbang Yao. Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quanti- zation: Towards lossless cnns with low-precision weights. arXiv preprint arXiv: (edi::7702030444 2017.
[31] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint arXiv: 2016.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] see subsection 4.2 it emphasize
all weights are training in full-precision. see seessuseection 5.3) We deseribe the binary parameters require a higher number of epochs to train.
(c) Did you discuss any potential negative societal impacts of your work? [No], We don't tt see there any.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] We will provide
the code upon publication. We only ran experiments on publicly available datasets.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No]. . It was too costly to train multiple times, we just ran once.
(d) Did you include the total amount of compute and the type of resources used (c.g.. type of GPUs, internal cluster, or cloud provider)? [No], but we gave standard training details in in|section 5]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "fGZAVeo8SzQ", "reviewer": "Reviewer_geT6", "summary": "The paper suggests a new training technique for neural networks, where each parameter is broken down into multiple parameters, representing its sign, sparsity, and magnitude (or shift). They explain why this new technique is useful for overcoming vanishing gradients and sign-freezing. They show that this training can produce a network with 3 bits weights, with the same quality as the standard fp32 network.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: \n\nThe method suggested in the paper is novel. Related work is adequately cited.\n\n**Quality**\n\nThe submission is technically sound.\n\nThe paper starts by discussing the problems of training low-bit networks. The paper claims sign-freezing and vanishing gradients are at fault, and provide some evidence that this is the case.\n\nThe next part discusses the proposed technique, of S3 reparameterization. I found the details of the implementation of sign-sparse-shift training to be lacking. (See limitation 3,4)\n\nThe experimental results in the paper cover the necessary benchmarks, showing benefits for both resnet18 and resnet50 with Imagenet. The baselines give a relatively good representation of competing methods. I would suggest adding several more baselines for a clearer picture: I was using this survey [1] for comparison, and some of the lines seem to also be relevant for this paper as well.\n\nWhile the competition on reducing bit width is stiff (And may also include activation width, where this approach may be less appealing), I consider the ability to train an efficient network from scratch to be this paper's great advantage, as most papers that used low bits implementations rely heavily on a pre-trained model.\n\nThe ablation study was very helpful in showing the robustness of the new technique. However, all the results in the study were lower than the published result by ~0.5%, including the marked result ($\\alpha = 1e-5$, No scheduler). This can be explained by the reduced number of epochs, but its means that the ablation study covers an area that is somewhat different from the one we are interested in. \n\nOf course, the paper can still benefit from the inclusion of more benchmarks to show that the results generalize well to other domains. The sensitivity analysis is useful at verifying that the S3 training indeed solves the problem it was designed to.\n\nOne big limitation of the paper is the reparameterization of the weights during training is expected to be costly (in memory, at least). I don't think this limitation has been properly addressed, and I would have missed it in my initial read if it wasn't noted in lines 199-200.\n\n**Clarity:**\n\nOverall, the paper is clearly written.\nAs mentioned above, some details that can help a reader reproduce the work are missing.\n\n**Significance**\nThe results are very important, and pending further research, have the potential to be translated to more efficient hardware for inference. As a successful training technique for low bits width, this work may also inspire theoretical works that can further advance our understanding of the theory of learning.\n\n[1] https://arxiv.org/abs/2101.09671", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. My main concern with the paper is reproducibility. Without additional details and/or code, the experimental part can not be properly evaluated. \n\n2.  Please highlight that the benefit to network efficiency is for inference only, and specify the cost for the technique during training. The limitations should be clearly mentioned in either the introduction or the conclusion.\n\n3. For the ablation study to be more effective, it should scan the same area as the final result- please increase the number of epochs to 200. It will also be helpful to measure the standard deviation of the main results.\n\n4. Regardless of reproducibility, section 4 should be expanded. Since reparameterization of every value utilizes Heaviside functions, the values of $w_{sign}, w_{sparse}, w_{n}$ can not be optimized with standard backpropagation. According to Figure 3, STEs are used, but this is not explained in the text as far as I can tell. The issue of initializing these parameters is also not mentioned-- initialization might have important consequences since sign freeze and (sparse-freeze) can still occur if the values were not initialized properly. \n\n(Minor)\n\n5. The data in Table 4 would be clearer if visualized in a plot with the familiar format (Top1 accuracy vs Epochs). If space is a concern, I think it would be preferable to have a figure visualizing it in the appendix than a table within the main text.\n\n6. The front page claim of out-performing full-precision networks is exaggerated. For it to be made, I would expect clear error bars for both results, proving that the result is significant. The ablation study implies the standard deviation is on par with the current accuracy gap.\n\n7. Expanding table 1 with more baselines will give the reader a better picture of the tradeoffs between different approaches.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "MvofoYyC3ht", "reviewer": "Reviewer_FA7K", "summary": "This paper proposes a new weight re-parametrization technique $S^3$ for training low-bit shift network. It claims that this new re-parametrization technique is weight-initialization insensitive and accuracy persistent compared with previous methods. In ImageNet, it achieves comparable top-1 accuracy with 3-bits, less than previous methods requires at 5-bit weight representation. Additionally, they also design two indices of weight dynamics to illustrate the mechanism of how $S^3$ works.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This paper proposes a novel weight re-parametrization method. More Related work on BNN or other sophisticated weight re-parametrization methods on low-bits networks should be considered.\n\nQuality: There are serval doubts on the discussion on the \"Gradient vanishing\". In most cases on studies on low-bits networks, the backbone is mainly ResNet. With residual connection and batch norm, it might be impossible to cause VGP. It is hard to acknowledge the correspondence or relationship between VGP and the proposed $S^3$, especially in Binarized quantizer function since the weight representation is {0,1}.\n\nFor Equation 4, what exactly are these $\\mathbf{W}$s representing? The decomposition mentioned in paper is unclear. What is the motivation for that? and any theoretical support for $w_{sparse}$ and $w_{sign}$. This can be really important. If the authors can illustrate somewhere in the main text, that would be appreciated.\n\nOne minor issue, it seems that the authors only use ImageNet dataset, therefore, it might be appropriate to remove descriptions such as, we evaluate on multiple datasets in Line 65. This can be misleading for readers.\n\nClarity: This paper is well-written, and the layout is good to read.\n\nSignificance: $S^3$ achieves lower bits than previous low-bits shift networks. However, the state-of-the-art is not sufficient. There are also Binary Neural Network which has even lower than the number reported in the paper, and in the same time, they achieve comparable results under the same setting [1]. More comparisons with BNN are encouraged here. It would be better if the author can demonstrate the applicability of training BNN.\n\n[1] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In CVPR, 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "MAE1K85z2ri", "reviewer": "Reviewer_KeAd", "summary": "The paper proposes an idea of a different reparametrization for ternary networks, decomposing any values into learning a 'shadow weight' for both the sign and the value itself.\n\n", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To me, this paper is a strong reject as it currently stands. The paper is written in a haphazard fashion, is imprecise in it's claims and methodology, and unconvincing in it's justifications of the method and the results. I will go over the parts of the paper that are unclear first from a methodological point of view.\n\nSection 3.1 on the vanishing gradient problem is very imprecise. W^l being zero, and sparse, are not the same thing. I don’t see how a regularizer helps weights to be non-zero to counteract this. Also, sparse weights don’t necessarily lead to vanishing gradients; you could 0-out 50% of your weights, and double the remaining ones, and you’d still have a similar magnitude in your gradients. (1) refers to a quantiser, but not necessarily to a ternary one, this should be made more precise. The paper claims to be the first to analyze the vanishing gradient problem for quantization... but I don't see any further analysis of this that leaves me convinced this is a real issue. A plot to show vanishing gradients happening due to quantization would help. \n\nSection 3.2 I don't see it proven that the sign is the actual issue here. If the same analysis was done for 4 bit quantization, wouldn't we see that the same problem occurs around +2 and +3? There doesn't seem to be any evidence for the sign itself being very specific to this problem. And if the sign isn't specifically the problem, I don't see how the suggested parametrization helps. \n\nSection 4 - The underlying assumption that a weight that is 0, stays 0 and somehow ensures the network has a lower capacity doesn’t hold. Even when initialised at 0, weights can update to be higher or lower, unless full output neurons are initialised to be entirely 0 or you have a dying ReLU-like problem.\n\nDiscussing the reparametrization (4) in ternary, it seems to me that for w_ter = {w_sparse, w_sign}, that {0, 0} ->-1, {1,0}-> -1, {0, 1} -> 0, {1,1}->1. It is insightful to plot what happens with the values w_sparse and w_sign (e.g. wolfram alpha z = x * (2*y-1) for x=-0.1...1.1 and y=-0.1...1.1 ). In this curve, it's clear that if you initialize anywhere and want to move your value of z up (with a positive gradient), both w_sparse and w_sign would be increased. However, if the gradient of z has to go down, w_sign goes down but w_sparse goes up too! This means that when optimizing, w_sparse will always go up, and only w_sign is left as a factor that is actually optimized over meaningfully. This means that with this refactorization, you end up with one significant variable you are optimizing anyway. It would be insightful if the authors plotted the behavior of these gates for a toy example, and showed e.g. that jumps from -1 directly to 1 could happen with the shadow weights (I contest this is not possible in a smooth manner with the current formulation). \n\nOne very important point is that to me, the arguments for the weight initialization and the WSVR issue are at odds with each other. For weight initialization, it's mentioned that weights should be initialized far from 0, and the paper introduces the dense weight regularizer to actively encourage weights to be either -1 or 1. However, when discussing the weight sign freezing issue, the argument is that weights don't switch enough from -1 to 1. Weights that would be close to 0 would normally have an ability to switch to -1 or 1 when doing a normal optimization. We see that in graph 4.b, the ternary network (quantizer) remains on a very low sparsity, while the other networks gain higher sparsity over time. So despite the dense weight regularizer, the ternary network (S3) gets higher sparsity than the ternay network (quantizer)? That's the opposite of what the logical argument of the authors would suggest.\n\nThe results section is unclear to me. the 'sign flip' and 'shift+sign flip' methods are not introduced as is. It's also unclear where the difference in performance between methods is coming from. In the ResNet18 table, it seems the shift+sign flip method is better than the baseline. It might just be that the author's training pipeline is better? It should be clear how the other methods are ran for comparison, to see if this is fair. It's also unclear where the gains are coming from. Which part of the method is working? Is it the initialization? The quantization scheme? An ablation study that shows the difference between everything that is added would help tremendously. Perhaps INQ + better initialization could already give a better result?\n\nThe WSVR doesn’t seem like a sensible measure to me. I could have a procedure where I just randomly flip all weights in the network every epoch, which wouldn’t help with optimisation at all, but would give me a perfect WSVR score. If anything, 4.a. just shows to me that the learning rate is low for the ternary network, as weights not flipping means essentially nothing is being learned (as there’s just 3 possible values of the weights!). \nThere’s a more sensible analysis to show if your ternary optimisation method works, and it’s to actually see how your gradient estimate predicts a real decrease in the loss function. And the authors fail to show that in this paper. \n\nFrom a structure point of view: \nPlease introduce shift networks, and your final method earlier in the text; all of section 3 talks about shift networks, but you haven’t introduced in your paper what you are talking about. The reader now has to refer to another work to figure out what shift networks are, and hope it’s the same notion as you mention. Please define what you mean by this yourself in the text. \nSection 3.2 WSVR is named as an acronym for the first time since the introduction. Please properly name and define this before using it as an acronym. \n“Different weight dynamics” What do you mean with this exactly, please be more precise. \nSection 4.2 leaves me confused. First we’re looking at ternary networks, and S indicates we shift left or right. Then S is claimed to be positive, so I guess we only shift right? Then S2 is being introduced as ternary networks, but it seems the rest of the paper uses the S^3 definition as a ternary network (see fig 4.3)? It would really help if the authors worked on the flow of the article, clarifying what their actual method is and building up the understanding, and make sure there is consistency in the paper.\nThe order of figure 1 and 2 is reversed", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "8_KjxEy7lw", "reviewer": "Reviewer_CUjS", "summary": "The paper presents the sign-sparsity-shift low-bit (i.e. 3-bit) reparametrization of weights for the multiplication-free networks, whose learning dynamics (as well as ImageNet results) closely resembles their full-precision counterparts.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n+ The algorithm is efficient and easy to implement.\n+ The empirical results are relatively impressive.\n+ The writing is clear and easy to follow.\n\nCons\n- The number of baselines being compared against in Table 1 & 2 [8, 12, 25] is simply too small, considering the heated competition in this area. As stated in Sec 2, there exist many papers about improving multiplication-free networks [23, 14, 24, 13] and discrete weights [5, 15, 18], which also should be compared against as much as possible (e.g. on a speed-accuracy chart) whether the authors believe they're too slow or not. Otherwise, the authors should provide a more detailed explanation regarding why they cannot be fairly compared against. In addition, [27] seems highly relevant but is missing in the paper.\n- It's unclear why the shift network is parameterized recursively as St in Eq 6, which seems very inefficient in bit usage (when t is larger than 2, which wasn't but should also be tested in the paper) and strongly biases St towards 0 at initialization (if w's are all initialized symmetrically around 0, which is commonly the case). The alternative of using the staircase function (Eq 1) for the shift network should also be considered and tested in the paper.\n\n[27] Ultra-Low Precision 4-bit Training of Deep Neural Networks, NeurIPS, 2020\n\nPost rebuttal\n- I'm satisfied with the authors' detailed feedback including comparison of their design of the shift network (Sec 4.2, Eq 6) against the more commonly used staircase function, additional ImageNet results of the 4-bit extension of their algorithm (which reasonably addressed my concern about its scalability) and additional, favorable comparison to a much wider range of prior work. I've upgraded my score to 8.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Following the authors' response, this paper had two very positive reviewers (8), one slightly positive reviewer (6), and one very negative reviewer (2). 

The main contribution of this paper, that excited the positive reviewers, was it showed for the first time it is possible to train from scratch with shift based quantization on ImageNet, and at a very low precision (3bit). This surprising observation that the suggested re-parameterization + "densifying regularization" are needed is insightful and should be helpful for developing other methods.

The very negative reviewer was mainly concerned with the re-parameterization, and suggested many weights should get "stuck" during optimization. Though the reviewer argument initially had some errors in the phase space diagram, we had a discussion whether the correct phase space diagram could should have such issues. However, during the discussion, I became convinced this is not a critical issue, because of the toy examples below (found in the discussion) by the most positive reviewer, in which we actually converge to the correct solution. Interestingly, this convergence seems to work be because of the "densifying regularization", and I encourage the authors to verify and discuss this. The results added by the authors during the review process are useful for this.

I would ask also the authors to polish the writing to help readability and impact of this paper. I tend to agree with the negative reviewer that some parts of this work are imprecise and confusing (and so did even the most positive reviewer).

**** Example ****

Suppose the training loss is $\frac12 (w_{ter} - 0)^2$ then $\frac{dL}{dw_{ter}} = g = w_{ter}= H(x)(2 H(y) - 1)$

Then, the STE gradient of the regularized loss L is

$\frac{dL}{dx} = ((2 H(y) - 1)  H(x) )  (2 H(y) - 1)    - \alpha I_{x<0}  = H(x) - \alpha H(-x) $ (i.e., 1 for x>0, -$\alpha$ for x<0)

, where $H$ is the heaviside function. Therefore the step $-\frac{dL}{dx}$ will always drive $x$ toward 0, and if $\alpha$ is small, we will have $x<0$ for most iterations.

Another one example:
$L = \frac12 (w_{ter} - 1)^2$, $g = w_{ter} = H(x)(2 H(y) - 1) - 1  $

$\frac{dL}{dx} = ((2 H(y) - 1)  H(x) - 1)  (2 H(y) - 1)     - \alpha I_{x<0}  = H(x) - (2 H(y) - 1)  - \alpha H(-x) $

$\frac{dL}{dy} = 2 H(x) g = 2 H(x) (( H(x) (2 H(y) - 1)) -1)  = 2 H(x) ((2 H(y) - 1)  - 1) = 4 H(x) (H(y) - 1) $

so, the step $-\frac{dL}{dy} = -4H(x)(H(y) - 1) $ will be non zero only if $x>0$ and $y < 0$. If $x>0$ it will eventually become positive as required.

If, on the other hand, $y<0$, the step $-\frac{dL}{dx} =   -H(x) - 1 + \alpha H(-x)  $  so it will move down while x<0, and, indeed, stay negative. Therefore, (-1,-1) won't shift to (1,1) in this setup. 
Decision: Accept (Poster)