Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper presents the sign-sparsity-shift low-bit (i.e. 3-bit) reparametrization of weights for the multiplication-free networks, whose learning dynamics (as well as ImageNet results) closely resembles their full-precision counterparts.
main_review: Pros
+ The algorithm is efficient and easy to implement.
+ The empirical results are relatively impressive.
+ The writing is clear and easy to follow.

Cons
- The number of baselines being compared against in Table 1 & 2 [8, 12, 25] is simply too small, considering the heated competition in this area. As stated in Sec 2, there exist many papers about improving multiplication-free networks [23, 14, 24, 13] and discrete weights [5, 15, 18], which also should be compared against as much as possible (e.g. on a speed-accuracy chart) whether the authors believe they're too slow or not. Otherwise, the authors should provide a more detailed explanation regarding why they cannot be fairly compared against. In addition, [27] seems highly relevant but is missing in the paper.
- It's unclear why the shift network is parameterized recursively as St in Eq 6, which seems very inefficient in bit usage (when t is larger than 2, which wasn't but should also be tested in the paper) and strongly biases St towards 0 at initialization (if w's are all initialized symmetrically around 0, which is commonly the case). The alternative of using the staircase function (Eq 1) for the shift network should also be considered and tested in the paper.

[27] Ultra-Low Precision 4-bit Training of Deep Neural Networks, NeurIPS, 2020

Post rebuttal
- I'm satisfied with the authors' detailed feedback including comparison of their design of the shift network (Sec 4.2, Eq 6) against the more commonly used staircase function, additional ImageNet results of the 4-bit extension of their algorithm (which reasonably addressed my concern about its scalability) and additional, favorable comparison to a much wider range of prior work. I've upgraded my score to 8.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new weight re-parametrization technique $S^3$ for training low-bit shift network. It claims that this new re-parametrization technique is weight-initialization insensitive and accuracy persistent compared with previous methods. In ImageNet, it achieves comparable top-1 accuracy with 3-bits, less than previous methods requires at 5-bit weight representation. Additionally, they also design two indices of weight dynamics to illustrate the mechanism of how $S^3$ works.
main_review: Originality: This paper proposes a novel weight re-parametrization method. More Related work on BNN or other sophisticated weight re-parametrization methods on low-bits networks should be considered.

Quality: There are serval doubts on the discussion on the "Gradient vanishing". In most cases on studies on low-bits networks, the backbone is mainly ResNet. With residual connection and batch norm, it might be impossible to cause VGP. It is hard to acknowledge the correspondence or relationship between VGP and the proposed $S^3$, especially in Binarized quantizer function since the weight representation is {0,1}.

For Equation 4, what exactly are these $\mathbf{W}$s representing? The decomposition mentioned in paper is unclear. What is the motivation for that? and any theoretical support for $w_{sparse}$ and $w_{sign}$. This can be really important. If the authors can illustrate somewhere in the main text, that would be appreciated.

One minor issue, it seems that the authors only use ImageNet dataset, therefore, it might be appropriate to remove descriptions such as, we evaluate on multiple datasets in Line 65. This can be misleading for readers.

Clarity: This paper is well-written, and the layout is good to read.

Significance: $S^3$ achieves lower bits than previous low-bits shift networks. However, the state-of-the-art is not sufficient. There are also Binary Neural Network which has even lower than the number reported in the paper, and in the same time, they achieve comparable results under the same setting [1]. More comparisons with BNN are encouraged here. It would be better if the author can demonstrate the applicability of training BNN.

[1] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In CVPR, 2020.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper suggests a new training technique for neural networks, where each parameter is broken down into multiple parameters, representing its sign, sparsity, and magnitude (or shift). They explain why this new technique is useful for overcoming vanishing gradients and sign-freezing. They show that this training can produce a network with 3 bits weights, with the same quality as the standard fp32 network.
main_review: **Originality**: 

The method suggested in the paper is novel. Related work is adequately cited.

**Quality**

The submission is technically sound.

The paper starts by discussing the problems of training low-bit networks. The paper claims sign-freezing and vanishing gradients are at fault, and provide some evidence that this is the case.

The next part discusses the proposed technique, of S3 reparameterization. I found the details of the implementation of sign-sparse-shift training to be lacking. (See limitation 3,4)

The experimental results in the paper cover the necessary benchmarks, showing benefits for both resnet18 and resnet50 with Imagenet. The baselines give a relatively good representation of competing methods. I would suggest adding several more baselines for a clearer picture: I was using this survey [1] for comparison, and some of the lines seem to also be relevant for this paper as well.

While the competition on reducing bit width is stiff (And may also include activation width, where this approach may be less appealing), I consider the ability to train an efficient network from scratch to be this paper's great advantage, as most papers that used low bits implementations rely heavily on a pre-trained model.

The ablation study was very helpful in showing the robustness of the new technique. However, all the results in the study were lower than the published result by ~0.5%, including the marked result ($\alpha = 1e-5$, No scheduler). This can be explained by the reduced number of epochs, but its means that the ablation study covers an area that is somewhat different from the one we are interested in. 

Of course, the paper can still benefit from the inclusion of more benchmarks to show that the results generalize well to other domains. The sensitivity analysis is useful at verifying that the S3 training indeed solves the problem it was designed to.

One big limitation of the paper is the reparameterization of the weights during training is expected to be costly (in memory, at least). I don't think this limitation has been properly addressed, and I would have missed it in my initial read if it wasn't noted in lines 199-200.

**Clarity:**

Overall, the paper is clearly written.
As mentioned above, some details that can help a reader reproduce the work are missing.

**Significance**
The results are very important, and pending further research, have the potential to be translated to more efficient hardware for inference. As a successful training technique for low bits width, this work may also inspire theoretical works that can further advance our understanding of the theory of learning.

[1] https://arxiv.org/abs/2101.09671
limitations_and_societal_impact: 1. My main concern with the paper is reproducibility. Without additional details and/or code, the experimental part can not be properly evaluated. 

2.  Please highlight that the benefit to network efficiency is for inference only, and specify the cost for the technique during training. The limitations should be clearly mentioned in either the introduction or the conclusion.

3. For the ablation study to be more effective, it should scan the same area as the final result- please increase the number of epochs to 200. It will also be helpful to measure the standard deviation of the main results.

4. Regardless of reproducibility, section 4 should be expanded. Since reparameterization of every value utilizes Heaviside functions, the values of $w_{sign}, w_{sparse}, w_{n}$ can not be optimized with standard backpropagation. According to Figure 3, STEs are used, but this is not explained in the text as far as I can tell. The issue of initializing these parameters is also not mentioned-- initialization might have important consequences since sign freeze and (sparse-freeze) can still occur if the values were not initialized properly. 

(Minor)

5. The data in Table 4 would be clearer if visualized in a plot with the familiar format (Top1 accuracy vs Epochs). If space is a concern, I think it would be preferable to have a figure visualizing it in the appendix than a table within the main text.

6. The front page claim of out-performing full-precision networks is exaggerated. For it to be made, I would expect clear error bars for both results, proving that the result is significant. The ablation study implies the standard deviation is on par with the current accuracy gap.

7. Expanding table 1 with more baselines will give the reader a better picture of the tradeoffs between different approaches.
review_title: null

final_score: rating: 2: Strong rejection
review_content: summary: The paper proposes an idea of a different reparametrization for ternary networks, decomposing any values into learning a 'shadow weight' for both the sign and the value itself.


main_review: To me, this paper is a strong reject as it currently stands. The paper is written in a haphazard fashion, is imprecise in it's claims and methodology, and unconvincing in it's justifications of the method and the results. I will go over the parts of the paper that are unclear first from a methodological point of view.

Section 3.1 on the vanishing gradient problem is very imprecise. W^l being zero, and sparse, are not the same thing. I don’t see how a regularizer helps weights to be non-zero to counteract this. Also, sparse weights don’t necessarily lead to vanishing gradients; you could 0-out 50% of your weights, and double the remaining ones, and you’d still have a similar magnitude in your gradients. (1) refers to a quantiser, but not necessarily to a ternary one, this should be made more precise. The paper claims to be the first to analyze the vanishing gradient problem for quantization... but I don't see any further analysis of this that leaves me convinced this is a real issue. A plot to show vanishing gradients happening due to quantization would help. 

Section 3.2 I don't see it proven that the sign is the actual issue here. If the same analysis was done for 4 bit quantization, wouldn't we see that the same problem occurs around +2 and +3? There doesn't seem to be any evidence for the sign itself being very specific to this problem. And if the sign isn't specifically the problem, I don't see how the suggested parametrization helps. 

Section 4 - The underlying assumption that a weight that is 0, stays 0 and somehow ensures the network has a lower capacity doesn’t hold. Even when initialised at 0, weights can update to be higher or lower, unless full output neurons are initialised to be entirely 0 or you have a dying ReLU-like problem.

Discussing the reparametrization (4) in ternary, it seems to me that for w_ter = {w_sparse, w_sign}, that {0, 0} ->-1, {1,0}-> -1, {0, 1} -> 0, {1,1}->1. It is insightful to plot what happens with the values w_sparse and w_sign (e.g. wolfram alpha z = x * (2*y-1) for x=-0.1...1.1 and y=-0.1...1.1 ). In this curve, it's clear that if you initialize anywhere and want to move your value of z up (with a positive gradient), both w_sparse and w_sign would be increased. However, if the gradient of z has to go down, w_sign goes down but w_sparse goes up too! This means that when optimizing, w_sparse will always go up, and only w_sign is left as a factor that is actually optimized over meaningfully. This means that with this refactorization, you end up with one significant variable you are optimizing anyway. It would be insightful if the authors plotted the behavior of these gates for a toy example, and showed e.g. that jumps from -1 directly to 1 could happen with the shadow weights (I contest this is not possible in a smooth manner with the current formulation). 

One very important point is that to me, the arguments for the weight initialization and the WSVR issue are at odds with each other. For weight initialization, it's mentioned that weights should be initialized far from 0, and the paper introduces the dense weight regularizer to actively encourage weights to be either -1 or 1. However, when discussing the weight sign freezing issue, the argument is that weights don't switch enough from -1 to 1. Weights that would be close to 0 would normally have an ability to switch to -1 or 1 when doing a normal optimization. We see that in graph 4.b, the ternary network (quantizer) remains on a very low sparsity, while the other networks gain higher sparsity over time. So despite the dense weight regularizer, the ternary network (S3) gets higher sparsity than the ternay network (quantizer)? That's the opposite of what the logical argument of the authors would suggest.

The results section is unclear to me. the 'sign flip' and 'shift+sign flip' methods are not introduced as is. It's also unclear where the difference in performance between methods is coming from. In the ResNet18 table, it seems the shift+sign flip method is better than the baseline. It might just be that the author's training pipeline is better? It should be clear how the other methods are ran for comparison, to see if this is fair. It's also unclear where the gains are coming from. Which part of the method is working? Is it the initialization? The quantization scheme? An ablation study that shows the difference between everything that is added would help tremendously. Perhaps INQ + better initialization could already give a better result?

The WSVR doesn’t seem like a sensible measure to me. I could have a procedure where I just randomly flip all weights in the network every epoch, which wouldn’t help with optimisation at all, but would give me a perfect WSVR score. If anything, 4.a. just shows to me that the learning rate is low for the ternary network, as weights not flipping means essentially nothing is being learned (as there’s just 3 possible values of the weights!). 
There’s a more sensible analysis to show if your ternary optimisation method works, and it’s to actually see how your gradient estimate predicts a real decrease in the loss function. And the authors fail to show that in this paper. 

From a structure point of view: 
Please introduce shift networks, and your final method earlier in the text; all of section 3 talks about shift networks, but you haven’t introduced in your paper what you are talking about. The reader now has to refer to another work to figure out what shift networks are, and hope it’s the same notion as you mention. Please define what you mean by this yourself in the text. 
Section 3.2 WSVR is named as an acronym for the first time since the introduction. Please properly name and define this before using it as an acronym. 
“Different weight dynamics” What do you mean with this exactly, please be more precise. 
Section 4.2 leaves me confused. First we’re looking at ternary networks, and S indicates we shift left or right. Then S is claimed to be positive, so I guess we only shift right? Then S2 is being introduced as ternary networks, but it seems the rest of the paper uses the S^3 definition as a ternary network (see fig 4.3)? It would really help if the authors worked on the flow of the article, clarifying what their actual method is and building up the understanding, and make sure there is consistency in the paper.
The order of figure 1 and 2 is reversed
limitations_and_societal_impact: NA
review_title: null

Meta-review: Following the authors' response, this paper had two very positive reviewers (8), one slightly positive reviewer (6), and one very negative reviewer (2). 

The main contribution of this paper, that excited the positive reviewers, was it showed for the first time it is possible to train from scratch with shift based quantization on ImageNet, and at a very low precision (3bit). This surprising observation that the suggested re-parameterization + "densifying regularization" are needed is insightful and should be helpful for developing other methods.

The very negative reviewer was mainly concerned with the re-parameterization, and suggested many weights should get "stuck" during optimization. Though the reviewer argument initially had some errors in the phase space diagram, we had a discussion whether the correct phase space diagram could should have such issues. However, during the discussion, I became convinced this is not a critical issue, because of the toy examples below (found in the discussion) by the most positive reviewer, in which we actually converge to the correct solution. Interestingly, this convergence seems to work be because of the "densifying regularization", and I encourage the authors to verify and discuss this. The results added by the authors during the review process are useful for this.

I would ask also the authors to polish the writing to help readability and impact of this paper. I tend to agree with the negative reviewer that some parts of this work are imprecise and confusing (and so did even the most positive reviewer).

**** Example ****

Suppose the training loss is $\frac12 (w_{ter} - 0)^2$ then $\frac{dL}{dw_{ter}} = g = w_{ter}= H(x)(2 H(y) - 1)$

Then, the STE gradient of the regularized loss L is

$\frac{dL}{dx} = ((2 H(y) - 1)  H(x) )  (2 H(y) - 1)    - \alpha I_{x<0}  = H(x) - \alpha H(-x) $ (i.e., 1 for x>0, -$\alpha$ for x<0)

, where $H$ is the heaviside function. Therefore the step $-\frac{dL}{dx}$ will always drive $x$ toward 0, and if $\alpha$ is small, we will have $x<0$ for most iterations.

Another one example:
$L = \frac12 (w_{ter} - 1)^2$, $g = w_{ter} = H(x)(2 H(y) - 1) - 1  $

$\frac{dL}{dx} = ((2 H(y) - 1)  H(x) - 1)  (2 H(y) - 1)     - \alpha I_{x<0}  = H(x) - (2 H(y) - 1)  - \alpha H(-x) $

$\frac{dL}{dy} = 2 H(x) g = 2 H(x) (( H(x) (2 H(y) - 1)) -1)  = 2 H(x) ((2 H(y) - 1)  - 1) = 4 H(x) (H(y) - 1) $

so, the step $-\frac{dL}{dy} = -4H(x)(H(y) - 1) $ will be non zero only if $x>0$ and $y < 0$. If $x>0$ it will eventually become positive as required.

If, on the other hand, $y<0$, the step $-\frac{dL}{dx} =   -H(x) - 1 + \alpha H(-x)  $  so it will move down while x<0, and, indeed, stay negative. Therefore, (-1,-1) won't shift to (1,1) in this setup. 
Decision: Accept (Poster)