Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: To be honest, I only read several papers on the neural network quantization. I am not familiar with this research topic, so I  provide my judgement based on my own limited knowledge rather than thorough comparison with other related works.

1. The motivation is clear. The 1-bit activation networks usually deteriorates the performance greatly.
2. The gradient mismatch for discrete variable did bring difficult for optimization. Do you mean 1-bit activation has larger gradient mismatch than other bits, at least in the defined cosine similarity by this paper?
3. As to Eq(3), Appendix C.1 describes the way to choose step size. I understand the logic, but for the detailed method, is it cross-validation with grid search or some other tricks?
4. Is there any relation between the decoupling method in Section 5 and the proposed estimated gradient mismatch in Section 4.2?
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new measure of gradient mismatch for training binary networks, and additionally proposes a method for getting better performance out of binary networks by initializing them to behave like a ternary network.

I found the new measure of gradient deviation fairly underdeveloped, and I suspect the method of converting ternary activations into binary activations works for a different reason than that proposed by the authors.

There were English language issues that somewhat reduced clarity, though the intended meaning was always understandable.

Detailed comments:

"Binary Neural Network (BNN) has been gaining interest thanks to its computing cost reduction and memory saving." --> "Binary Neural Networks (BNNs) have been garnering interest thanks to their compute cost reduction and memory savings." (will stop making English language corrections from here on)

"Therefore, we argue that the sharp accuracy drop for the binary activation stems from the inefficient training method, not the capacity of the model."
This could also be due to poor initialization in the binary case. e.g., it might make sense to initialize the binary network with bias=-0.5, so that the nonlinearity has a kink at pre-activation=0, rather than pre-activation=0.5.

"Unfortunately, it is not possible to measure the amount of gradient mismatch directly because the
true gradient of a quantized activation function is zero almost everywhere. " It *is* possible to measure the mismatch to the true gradient exactly. One could even train using the true gradient. It's just that the true gradient is useless.

Fig 1b -- this is a nice baseline.

"the steepest descent direction, which is the direction toward the point with the smallest loss at given distance"
This is not the usual definition of steepest descent direction. If you're going to redefine this, should do so mathematically and precisely (for instance, you are going to run into trouble with the word "distance", since your coordinate discrete gradient more closely resembles an L\infty-ball perturbation, rather than an L2-ball perturbation.

eq. 3:
Note that this equation is equivalent to taking the true gradient of a function which has been boxcar-smoothed along each parameter. This may more closely resemble existing measures of deviation than you like.

You should also consider the relationship to an evolutionary strategies style gradient estimate, which similarly provides an unbiased gradient estimate for a smoothed function, and which allows that estimate to be computed with fewer samples (at the cost of higher error).

Sec. 4.2 / Figure 3:
The results in this section will be *highly* sensitive to the choice of epsilon. You should discuss this, specify the epsilon used, and experimentally explore the dependence on epsilon.

"The results indicate that the cosine similarity between coarse gradient and CDG can explain the relationship between gradient mismatch and performance of model better than previous approaches. "
Don't know that I followed this. Gradient mismatch is never formally defined, so it's hard to know what this says about its relationship. Additionally, CDG sounds more like something which is correlated with, rather than an explanation for, performance.

" cosine similarity between coarse gradient and CDG can explain the relationship between gradient mismatch and performance of model better " --> " cosine similarity between coarse gradient and CDG can explain the relationship between gradient mismatch and performance of model better "

"we shift the bias of BN layer which comes right before the activation function layer. "
Did you try using these bias values without pre-training as a ternary network? I suspect it would work just as well!

"Please note that BN layers followed by binary activation layer can be merged to the threshold of the binary activation layer, incurring no overhead at inference stage."
Did not understand this.

"it is expected that the fine-tuning increases the accuracy even further"
Does it improve the accuracy further? Should state this as result, not prediction, and should have an ablation experiment showing this.

"Table 2 shows the validation accuracy of BNN in various schemes."
Why not test accuracy?

Figure 6:
What are the filled circles?
What was the sampling grid for the HP search? The images have high spatial frequency structure that I suspect is an artifact of the interpolation function, rather than in the data.

----

Update post-rebuttal:

The authors have addressed the majority of my concerns, through both text changes and significant additional experiments. I am therefore increasing my score. Thank you for your hard work!
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper studies activation quantization in deep networks. The authors first compare the coordinate discrete gradient and those obtained by various kinds of straight-through estimators, and found 1-bit activation networks have much poorer gradient estimation than 2-bit ones. Thus they speculate that this explains the poorer performance of 1-bit activation networks than 2-bit ones. To utilize higher precision of activation, the authors then propose to decouple a ternary activation into two binary ones, and achieve competitive results on typical image classification data sets CIFAR-10 and ImageNet.

The paper is overall well-written and easy to follow. The decoupling method is simple and straightforward. The experiments are also well conducted. One main concern is that since the computation of the decoupled binary model and the coupled ternary model are the same, why does the decoupled binary model can finally to tuned to perform better than the original ternary model? Is there any intuition or theoretical explanation? Yet another concern is that ternary activation basically can be viewed as binary+sparse activations, can it be even more computationally cheaper than the decoupled binary activation?

Question:
1. One line below eq (2), does STE mean the estimated gradient? How can the difference be calculated based on different things (i.e., activations and gradients)?
review_title: Official Blind Review #1

Meta-review: Three reviewers suggest acceptance. Reviewers were impressed by the thoroughness of the author response. Please take reviewer comments into account in the camera ready. Congratulations!
Decision: Accept (Poster)