Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper does a theoretical analysis of various gradient approximations for the non differentiable sign function. They investigate 3 gradient estimators based on the STE assumption, namely (identity) STE, ReLU STE, and clipped STE (cReLU). They show theoretically that a gradient estimator which breaks the scale symmetry (clipped STE) in the network is more likely to achieve a local minimum than one that keeps the symmetry.
main_review: Strong points:
* A novel approach on how to analyze various gradient estimators.
* Interesting new theoretical insights into various flavors of the STE gradient approximation.
* Detailed derivations in the appendix.

Weak points:
* No experimental results, even not on MNIST or a toy example to show that the theoretical findings correlate with empirical observations.
* The straight through estimator (STE, identity function) is know to be a biased estimator. In the paper this discussion is not coming back, but I think it would be interesting to discuss whether the other estimators (ReLU, cReLU) are biased and whether one is more biased than the other. Is there a trade-off to be made between being more biased vs having a higher chance to converge to a local minimum?
* The optimization problem defined in 2.1 (eq 1 & 2) is non-convex. For a non-convex optimization problem it is unclear to me based on the manuscript how the gradients in stationary points are important and increase the chance to get to the true global minimum (page 5, below eq 23). Some further explanation on the reasoning could be helpful to understand for the reader and to verify that these claims are true.
* A discussion on whether or not this analysis also holds for 1) weight quantization and 2) multi bit quantization would be interesting. Especially for the latter case it is common practice that the gradient is clipped due to the clipping in quantization (STE is applied to the rounding operation, thus the gradient of the full quantization operation is clipped due to the gradient of the clipping, see Esser et al. 2019).
* The clarity of the paper could improve and there are some misconceptions:
    * The straight through estimator is clearly defined in literature (see Bengio et al. 2013) as back propagation through a function as it would have been the identity function (thus gradient of 1 in the backwards path). However, in the manuscript any approximate gradient is defined as a STE gradient. This misuse of terminology makes it extra difficult and confusing to understand the work, especially since it is of theoretical nature. (“In this paper, we refer to the proxy of the gradient as the STE gradient.”)
    * It is a bit confusing that sigma is defined as step function. But based on the text (binary activations) and later equations it seems that it is a sign function. Would be good to be clear and maybe mathematically describe the exact function.


summary_of_the_review: Interesting and novel perspective on various gradient estimators for the sign function used in binary activations. The main drawbacks are that there is no empirical data to support any of their claims, several aspects are unclear based on the manuscript and the paper could improve in its overall clarity. In the current version and my understanding of it the drawbacks outweighs the strong points.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Training quantized neural network by the biased Straight-Through estimator (STE) is a common practice in the NN quantization literature. However, due to the biased nature of STE, there is no principled guidance in making algorithmic choices for such training methods. This work focuses on the choice of surrogate differentiable functions in STE, and looked at three common choices: identity, ReLU, and clipped ReLU. Through analysis of the scale invariance of gradients in STE, the paper suggests clipped ReLU might pick up local minima degenerated in scales, due to its property of breaking scale symmetry among the three. The authors further analyzed the stationary points of a simple misspecified model with Gaussian inputs to confirm their observations.
main_review: 
I have read the rebuttal. I'd like to thank the authors for the revision, although it does not resolve my concerns.

==========================

## Summary

Training quantized neural network by the biased Straight-Through estimator (STE) is a common practice in the NN quantization literature. However, due to the biased nature of STE, there is no principled guidance in making algorithmic choices for such training methods. This work focuses on the choice of surrogate differentiable functions in STE, and looked at three common choices: identity, ReLU, and clipped ReLU. Through analysis of the scale invariance of gradients in STE, the paper suggests clipped ReLU might pick up local minima degenerated in scales, due to its property of breaking scale symmetry among the three. The authors further analyzed the stationary points of a simple misspecified model with Gaussian inputs to confirm their observations.

## Comments

* The clarity of the paper is very poor. Currently the paper is more like a technical note than a polished conference paper. There are deep stacks of equations everywhere and many of them lack clear explanation. To support my claim, here are a few examples:

Equation (1) is confusing as a convolutional neural network because the hidden layer output is a simple matrix vector product, I figured it out after some thought that you restructured the input by replicating its features in a window to form each row of Z. However, you mentioned this nowhere and it is even not stated what $m$ and $n$ is.

* The main claim is not rigorous, which makes it very difficult to understand the implications of the results and the potential impact of the paper.  Many arguments are hand-wavy and when approximation is used, there is no characterization of the error, to list a few:

"the cRelu STE gradient for |w| << 1 is approximately identical with the Relu STE gradient"

"this relation implies that the Relu gradient is approximately embedded in the cRelu STE gradient"

"may not be zero" 

"there is still a chance to be zero"

"using cRelu STE is the most likely to achieve the true minimum in the three STEs"

All these make the conclusion less convincing and the lack of empirical evidence makes it worse.

## Overall suggestions

I vote for rejection mainly due to the questionable analysis (lack of rigor) and the difficulty to interpret the relevance of the result. Also the work does not go beyond one-hidden layer network as it claimed in the abstract.

## Questions and minor points

* Is my understanding correct that the hard $g_j$ has no scale invariance (it only has scale invariance in the loss)? 
* I'm always curious, have the quantized neural network community ever looked into other unbiased gradient estimators developed by the Monte-Carlo gradient estimation people? Or is it true that STE empirically outperformed all of them so it is still the default practice?

summary_of_the_review: I vote for rejection mainly due to the questionable analysis (lack of rigor) and the difficulty to interpret the relevance of the result. Also the work does not go beyond one-hidden layer network as it claimed in the abstract.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper considers a network with one hidden layer with binary valued neurons sharing the same weight vector but having non-overlapping receptive fields. The authors analyse the stationary points of a particular MSE loss when a straight through estimator (STE) is used for the loss gradient. They analyse three variants for the STE: (1) derivative of the identity function, (2) derivative of ReLU and (3) derivative of a piecewise linear squashing function. Assuming a simple independent distribution over the inputs and a particular loss, the authors show that the first two variants are biased and only the third STE variant can be unbiased. The authors conjecture that the reason for this is that the third STE variant breaks the scaling symmetry w.r.t. the weights of the first layer.
main_review: **Strengths:**
The authors propose a clear, concise and closed form analysis of the true gradient (i.e. the gradient of the expected loss) and its stationary points. This is possible due to the simplicity of the model and the assumed data distribution. The same is done for each of the three STE variants. This proves that the first two STE variants are biased in general. Moreover, the authors prove in particular that none of the global optimisers of the expected loss is a stationary point for these STE variants. (Notice that the globally optimal weights of the hidden layer form a ray due to scale invariance). On the other hand, they show that at least one of these global optimisers is also a stationary point for the third STE variant.

**Weaknesses**
The same model has been studied in the previous work Yin et al. (2019) for a different loss. It has been shown there that the first STE variant is biased, while the second and third where found to be unbiased in this case. The loss studied there was the MSE loss w.r.t. a ground truth model with the same architecture. The loss considered in the submitted manuscript is the MSE loss w.r.t. a ground truth model that uses ReLU activations in the hidden layer. The submitted manuscript thus also rules out the second STE variant as biased. 

The authors cite the previous work Shekhovtsov & Yanush (2020) which derives STEs as a linearisation of an unbiased gradient estimator with low variance. In particular, it is shown there that the properly derived ST estimator is unbiased for models studied in the submitted manuscript, albeit for a different type of loss. It is clearly seen that their derived unbiased ST estimator breaks the scale invariance. I am pretty sure that the derivation given there can be generalised for the model & loss combination studied in the submitted manuscript.
summary_of_the_review: The paper provides a convincing in-depth analysis of STE properties for a particular simple combination of model and data distribution. However, the novelty is limited in view of the previous work Yin et al. (2019). Its significance is restricted due to the very specific assumptions on the model, data distribution and loss.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper investigates analytically the scale invariance properties of 
different straight-through estimators used in training neural networks with binary activation and weights. 

The main claim is that the cReLU STE, among the 3 considered, is the one that is more likely to give zero gradients in the minima of the population loss. This is (weakly) supported by theoretical analysis on a little variation of the shallow setting analyzed in Yin et al. (2019), where the teacher now contains ReLU activations.   

main_review: The paper contains little novelty compared to Yin et al. (2019)
Also, it is not well written. The claims are weak and it is not clear if they are specific to the specific setting chosen.
There is no numerical validation.

- Claims such as
"""
Considering the scale invariance of Relu STE gradient, this relation implies that the
Relu gradient is approximately embedded in the cRelu STE gradient, so that if the back-
propagation using Relu STE converges to a point, the back-propagation using cRelu STE
can also converge to that point.
"""
are not entirely clear nor sufficiently motivated.


- The main claim reads
"""
In other words, because of the breaking of the scale symmetry, the cRelu STE may pick up one of local minima,
which degenerate in all scales, while because of keeping the scale symmetry, identity STE and Relu
STE may not. Therefore, we conclude that the back-propagation using cRelu STE is the most likely
to achieve the true minimum in the three STEs.
"""
In other words, the authors claim that some minima of the population loss of the original model *may* also be minima of the surrogate model with cReLU STE if the weights are appropriately rescaled. This claim has only the weak analytical support
given by the analysis of the Gaussian input case of the next Section 3. No numerical support is provided for
convergence of the training dynamics in this toy model, let alone deeper networks and structured datasets.

- In Section 3, the authors show that the cReLU STE has an additional stationary point (the global minimum) compared to the
ReLU STE. This is an interesting result, and also somehow natural given that a non-invariant gradient has more degree-of-freedoms
for satisfying the stationary point condition. The single setting explored, the one with the ReLU teacher, doesn't seem enough to support general claims though. Can the author check analytically a few other cases, e.g. tanh activation. Or at least experimentally (possibly also
in deeper networks)? 

- Reviewing the paper was made more complicated by the poor writing, 
starting from the very first sentence of the abstract  "In the quantized network, its gradient shows either vanishing or diverging."

summary_of_the_review: While there is some novelty to the paper, I don't think the importance of the results and the novelty level
are high enough for the ICLR standards.
Moreover, there is a complete lack of experimental validation and some poor writing. 
I cannot recommend the manuscript for publication on ICLR.
review_title: null

Meta-review: I agree with the reviewers that this work is not well-presented, and it seriously lacks rigor and experimental support. The writing of this work also needs significant improvement. The authors made many claims without offering rigorous proofs, and hand-waved their argument throughout without strong empirical support. In the end, the authors' response did not address the reviewers' concerns satisfactorily and no one is excited enough to defend the current draft. Please consider revising your draft according to the reviewers' comments.
Decision: Reject