Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper provides an empirical study of binary weight networks (BWNs), where they find that 1 the commonly adopted scaling factor is not critical 2 there exists a subnetwork that stabiles early in training 3 the 3x3 filters in VGG and ResNets demonstrate a sparse distribution. They combine all the observations and propose a novel quantization algorithm that achieves more aggressive compression than standard BWNs.

pros:
+ I appreciate the careful examination of design and training details of standard BWNs. The identification of a persistent subnetwork and the analysis on the sparse distribution of kernels are particularly interesting.

+ The proposed quantization algorithm is interesting, which has a potential of squeezing more redundancy out of standard BWNs

cons:
- If I understand correctly, in the proposed algorithm the kernel distribution is only drawn from the last conv layer of the full precision network, which is then shared across all layers when retraining the BWN. This seems a strong assumption and needs to be justified. What's the reason to believe that the selected frequent kernels are shared across different layers?

-In Algorithm 1, W = where(abs(W ) > ∆E, sign(W ), W ) is not motivated and explained well. What's the reasoning of using the threshold when computing the distance to the frequency binary kernels? 

-The experimental results seem to be really hard to interpret for me, and this is perhaps the weakest point of the this paper. In particular, Table 1 needs to have proper baselines. This includes the full precision, standard BWN accuracies, as well as controls which allow one to draw comparisons between the proposed algorithm and basic binarization by equating certain quantities. 

I suggest the authors work on the suggested improvements which will make this a much stronger contribution.

*****post rebuttal updates*****
I want to thank the authors for responding to my questions. The additional explanations are indeed helpful for clarifying my first two questions (selection of the binary kernel and the use of ∆E). However, I still have concerns about Table 1 (and Table 2). For example, I have a really hard time interpreting the significance of achieving a 3.2x CR with a loss of 3% (92.3 - 89.2 from VGG-7) in acc with the proposed method (although the paper argues that it's a "bearable" loss). Considering that this is the main experiment supporting the efficacy of the proposed quantization algorithm, I think the paper needs more controlled experiments to demonstrate the practical usefulness of the proposed algorithm. As a result I'm keeping my original score and hope the authors can work on the improvements for the next version. 
review_title: Interesting findings but needs more clarity and stronger results

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
The authors show empirically that weight signs are more important than weight magnitudes. They also analyze the optimization process and the structure of optimized binary networks to note that 
i) there are clusters of weights whose weight remain almost unchanged during the optimization 
ii) optimized convolutional networks have simple sub-structures  
They exploit the latter to propose a new quantization method that increases the compression of binary networks.
 
strengths:
The idea of studying how binary weights update during the optimization is interesting and may help understand the optimization process of binary but also real-valued networks. The experiments include numerical results for a wide range of data sets and networks. 

weaknesses:
The meaning of the given simple proof is not clear. What are gamma and epsilon in equations 1 and 2? The paper would have much more impact if, after showing that BN can absorb the redundant factors, all real-valued parameters were dropped. In some sense, the proof seems to show that the obtained result about the little influence of scaling is somehow expected. 

Regrading the sign flipping, observing that "flipping weights with large full precision magnitude will cause a significant performance drop compared to those weights close to zero" seems also something that one can expect.

Finally, the presence of clusters is not really explained and shown in a very accessible way. Are there other results except from Figure 4 (whose caption sounds a bit hermetic: "The X-axis indicates the index of a 3 x 3 binary weight kernel while Y-axis indicates the frequency that the certain appears in one certain Conv layer")

questions: 
- what happens if all scaling factors are kept fixed?
- is alpha usually shared by all weights in the network or is layer-specific?
- the weights distributions in figure 1 are obtained by adding a regularization term? 
- what happens if also the first full-precision Conv layer is quantized? 
review_title: Extensive empirical results about BWN

final_score: rating: 3: Clear rejection
review_content: **Summary:**
This paper proposes some interesting observations for training BWNs. 1: The scaling factors can be removed with batch normalization used. 2: The signs of the weights with large norms are determined and fixed at the early training stage. 3:  The binary weight networks can be further compressed. Moreover, the authors provide some empirical visualizations and results to demonstrate its analysis. However, the paper seems to be incomplete and needs to be further improved. 

**Pros:**  
The observation that the signs of weights with large norms are determined and fixed at the early training stage is interesting. The idea that weights with large norms are stable and sensitive on sign changes can be utilized for improving the training of BWNs, maybe BNNs as well.

**Cons:**   
1: Extensive descriptions are very confusing. I can only list some of them. 

1): The contributions of the paper include the observation that binary kernels in the learned convolutional layers tend to be centered on a limited number of fixed structural patterns. However, it is still not clear to me what the “fixed structural patterns” are. 

2): In Sec. 3, the paper claims to “quantize the less frequent kernels to those high frequent kernels to save space” and “we sort these binary kernels according to their appearance frequency…” . However, the paper fails to explain the motivation for exploring the frequency of kernels. Some theoretical explanations are needed. 

3):  In Figure 4’s caption, what does “the certain appears in one certain Conv layer” mean?
  
4):  In Sec. 4.1, the authors propose to apply QBN on model compression.  However, it is not clear to me what the points the paper intends to express. Specifically, the authors claim that “we can reduce the number of parameters to represent a binary-kernel by changing the small magnitude weights’ signs”. Then the question comes. How can the number of parameters be reduced via changing signs? Also, the sentence “we can compress their parameters to an extremely small number by replacing the whole 3×3 binary-kernel with fewer”  is incomplete. 

2: Extensive symbols are undefined, which makes me hard to understand the paper properly, especially in Algorithm 1, the main algorithm of the paper. For example, definition of $n$ in calculating $E$;  definition of  “where()”;  definition of $K_m$, are not explained.

3:  Extensive technical details are missing, which makes the algorithm difficult to understand. 
1):   The authors should at least explain how to generate the power-of-two number of binary kernels in details in Sec. 3.1.  
2):   During training, the paper generates $k$-bit (i.e., k is the bitwidth) number of binary kernels. However, during testing, the learnt binary weights should be fixed ($k=1$) and why the “Quant Bit” in Table 1 can be larger than 1? This makes me very confusing.

4:  This paper only binarizes weights while I am wondering whether simultaneously binarizing both weights and activations can have the similar observations. 

5:  There is no comparisons with current state-of-the-arts.  Also, in Sec 4.1, the paper claims that “the compressed ratio can achieve to 1.5$\times$ with a bearable accuracy drop (less than 3% on Cifar-10)” does not stand. The 3% loss on Cifar-10 is significant. 

6:  There is no conclusions and future works discussions. 

7:  It is widely known that the scaling factors can be absorbed into BN layers. In terms of this, it should not be a contribution of this paper.

8: The paper introduces extensive heuristic hyper-parameters. Specifically, it does not give any strategy to automatically determine the optimal quantized bitwidth and threshold $\Delta$ for each layer. 

9:  There are lots of grammar and typo issues. For example, the words in equations should use straight format (use \rm in latex).
review_title: Official Blind Review #2

final_score: rating: 6: Marginally above acceptance threshold
review_content: Overview:
The Authors show that scaling factors with hand-crafted or learnable methods are not so important when training Binary Weight Networks (BWNs), while the change of weight signs is crucial. They make two observations: The weight signs of the primary binary sub-networks are determined and fixed at the early training stage. Binary kernels in the convolutional layers of final models tend to be centered on a limited number of fixed structural patterns. Based on these observations, they propose a new method called binary kernel quantization to further compress BWNs. 

Strength bullets:
1. They propose a binary kernel quantization to quantize the binary kernel, which effectively reduces the number of all possible kernels, and can further compress BWNs to 2-5 times.
2. The Authors observe that the weight with the large norm, fixing their signs in the early training stage. And they compared this phenomenon with the lottery ticket hypothesis and propose the primary binary sub-networks. I think it's an interesting idea.
3. The paper is well written and well-motivated. It contains extensive and systematic related work-study. I like it. 

Weakness bullets:
1. The binary kernel quantization method is only applied to 3x3 kernels in this paper. When the size of the convolution kernel is 5x5 or 7x7，how does it work? I hope this method can extend to another cases and it also can work well. 
2. As shown in table1, why authors report the best accuracy among different seed? I think authors should report the averaged results with different random seeds instead of the best result.

-----Post Rebuttal-----

I have read all feedback and especially thanks to the authors' efforts on the extra experiments. I think the author has addressed my first concern. For the second one, I would prefer to see the errorbar.

I tend to keep my scores unchanged. I think the findings are interesting [share similar thoughts as R4's], while the experiments part need to be improved. Although I like the ideas and observations, I don't feel especially strongly in favor of it and cannot champion it.
review_title: This idea is novel and experiment results are supervising. I tend to accept this paper.

Meta-review: ## Description
The paper discovers interesting phenomena in training neural networks with binary weights:
- Connection between latent weight magnitude and how important its binarized version for the network performance
-training dynamics, indicating that large latent weights are identified and stabilize early on
- Observation that amongst learned binary kernel, several specific patterns prevail, up to the bits who's reversal has very little effect. This is so regardless of the architecture, the layer considered or the dataset. 
The paper further demonstrates how these observations may be used to compress binary neural networks below 1 bit per weight.

## Review Process and Decision
The reviewers welcomed the experimental investigation of new phenomena, but commented the overall technical quality of the work as somewhat substandard. The redundancy of consecutive affine transforms is known and not connected to binary weights investigation. The investigation itself lacks a more in-depth analysis. The proposed compression results appeared not convincing to reviewers since a significant drop of accuracy occurs. The AC shares these concerns and supports rejection.

## General Comments
From my perspective, the study undertaken is methodologically „wrong“. An ad-hoc training method is investigated, which is not even clearly defined in the paper (there are many „STE“ variants) and for which it is not known what it is doing, what are the real-valued weights for and whether they are needed at all (as empirically argued by Helwegen et al. (2019)). As such, the investigation makes impression of poking a black box (the training method in this case). At the same time, there are more clear learning formulations, applicable in the setting of the paper (binary weights), in particular considering the stochastic relaxation:
* Shayer et al. (2017): Learning Discrete Weights Using the Local Reparameterization Trick
* Roth et al. (2019): Training Discrete-Valued Neural Networks with Sign Activations Using Weight Distributions
* Peters et al. (2018): Probabilistic binary neural networks

These methods are approximate, but at least the optimization is well posed and it is known what do the real-valued weights represent (e.g. logits of binary weight probabilities).
From this perspective, it can be seen that latent weights close to 0 correspond to Bernoulli weights that are almost fully random (and thus only contribute noise) and are fragile to gradient steps. Therefore the model can only perform well if it learns to be robust to their state or their state becomes more deterministic (corresponding to large latent weight). So one would actually expect to see in these models phenomena similar to the observed in the paper and not bee too much surprised or astonished by them. Furthermore, there are recent works explaining STE and its latent weights as optimizing the stochastic relaxation:
* Meng et al. (2020): Training Binary Neural Networks using the Bayesian Learning Rule
* Yanush et al. (2020): Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networks. 

The authors are encouraged to make the observed phenomena more explainable by connecting to the mentioned works.

## Further Details

*  „We show that in the context of using batch normalization after convolutional layers, adapting scaling factors with either hand-crafted or learnable methods brings marginal or no accuracy gain to final model.“

From theoretical perspective, this is obvious and known to me. Practically, there could be in principle some difference due to the learning dynamics, and verifying that there is none is a useful but a weak contribution. The section devoted to this issue can be given in the appendix but is not justified in the main paper.

* „change of weight signs is crucial in the training of BWNs“

The sign determines the binary weights, so this is by definition.

* „ Firstly, the training of BWNs demonstrates the process of seeking primary binary sub-networks whose weight signs are determined and fixed at the early training stage, which is akin to recent findings on the lottery ticket hypothesis for efficient learning of sparse neural networks“

In the lottery ticket hypothesis paper it is shown explicitly that the identified sparse subnetwork changes during the learning the most rather than retains its initialization state or the state in the beginning of the training. It is therefore could be of a different nature.

Decision: Reject