Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the analysis of hyperparameters in binary neural networks (BNN). The hyperparameters could explain the BNNs with latent real-valued weights during training. However, the magnitude of binary weights is not interpretable with these hyperparameters yet. This paper provides an interpretation of these magnitude-based hyperparameters based on higher-order gradient filtering during network optimization.
strength_and_weaknesses: Pros:
- This paper has good motivation for interpreting the magnitude-based hyperparameters on BNN. 
- It provides some interesting insights such as: 
  - A gradient filtering perspective on latent weight hyperparameters.
  - A justification of which magnitude-based hyperparameters to use and offer a setting with fewer hyperparameters to tune.

Cons:
- It would be better to add a summary paragraph in section 3 to state their concise conclusion of magnitude-based hyperparameters.
- As mentioned, the performance improvement is marginal in Table 2 based on ReActNet-A. 
- The paper claimed that it provided a “simplified” setting with fewer hyperparameters to tune. However, I did not see the big advantages of the proposed optimizer compared to latent-weight optimization in terms of accuracy, and training/optimization speed.
- The experiment section is relatively weak as the experiment is given on one architecture based on a few baseline methods, “ReActNet-A” on Imagenet and “Bi-Real” on CIFAR. It is expected to show how the proposed optimizer performs on several recent BNN optimization methods with various types of architecture and different types of datasets (modality such as visual and text, distribution such as natural, digital, fine-grained) and task (classification, detection, and segmentation.)
- The metrics in Table 2 haven't been explained well. More details should be given.

clarity,_quality,_novelty_and_reproducibility: The paper well clarifies the interpretability issue of binary BNN weights and mag-based hyperparameters. The writing is good and easy to follow; yet the experiment is expected to be strengthened. 

The paper does not provide code and pseudo code/algorithm flow. 

summary_of_the_review: Overall, this paper provides some interesting insights for interpreting binary weights of BNN to magnitude-based hyperparameters. However, the major concern arises in the lack of experiments to support their statements. Also, the proposed optimizer does not offer many advantages compared with the latent-weight-based optimization.

**Post after rebuttal**

I appreciate the additional experimental results and detailed responses provided by the authors. After checking the latest revision in the manuscript, I decided to raise my score to encourage further exploration of hyperparameters for binary neural networks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Binary Neural Networks use values of -1 and +1 for the network's weights instead of floating point number to save on memory. Successful training of these networks, and their ultimate performance, depends on several hyperparameters. This paper redefines or removes many of these hyperparameters. This means that there are fewer hyperparameters to tune and removes the need for latent weights. Experiments support that there is little loss in performance compared to methods with latent weights.
strength_and_weaknesses: **Strengths**
- The main idea of the paper is simple.
- Removing hyperparameters and reducing the need for tuning is important---especially in architectures that are large or expensive to train.
- The paper is easy to follow.

**Weaknesses**
- Experiments are limited to two models and datasets, both of which are image classification.
clarity,_quality,_novelty_and_reproducibility: - Figure 1 seems pretty unnecessary. I imagine most readers will be familiar with with binary vs real valued numbers.
- The analysis is quite simple. This is a good thing, but I don't know this area well enough to comment on how novel it is.
summary_of_the_review: The paper is clear and presents a nice idea. The experiments support the main claim of the paper but are limited in scope. I do not know the literature on Binary Neural Networks well enough to comment on the paper's novelty.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides an analysis of standard optimization hyperparameters like weight initialization, learning rate, weight decay,  and momentum in the context of the latent-weight free interpretation of binary neural networks.

In a binary neural network, all weights and activations are +1 or -1. Although the weights are binary, they are typically optimized with real-valued gradients. In the latent-weight free interpretation, optimization can be viewed as a weight-flipping algorithm. When the gradients have accumulated past a certain threshold, the optimizer flips the weights.

But what should our understanding of traditional optimizer hyperparameters like weight initialization and learning rate be if we are to view optimization as weight-flipping? This paper argues that we should interpret them as impulse response filters on the gradients. This understanding reduces the number of hyperparameters that need to be tuned.
strength_and_weaknesses: Strengths:
- This paper builds upon the latent-weight free interpretation of binary neural networks, and provides a new understanding of traditional optimization hyperparameters in such a setting.

Weaknesses:
- This paper fails to demonstrate a significant empirical benefit from their new interpretation of binary neural network optimization beyond a reduction in the number of hyperparameters in some cases.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and original. While the authors did not include code with their submission, they have stated that they intend to release their code to make their experiments reproducible.
summary_of_the_review: This paper provides an interesting new understanding of optimization in the binary neural network setting. However, it also fails to show significant empirical benefits from such an understanding.

I am on the fence about recommending this paper for acceptance, because it's not clear to me how the community can use this newfound understanding to significantly advance our theoretical understanding of binary neural networks or their empirical application. However, I also have low confidence in my assessment. I think the authors can improve the paper by adding new empirical or theoretical results, or at least sketching out how their contribution is significant to the community.

Edit: Thanks for the authors' response. The quotes from relevant papers in the literature has increased my confidence in the significance of the work. Even though the empirical contributions are still weak, I see the great value this piece of work brings in filling this gap in the literature. I have updated my score to reflect that. I highly encourage the authors to re-write the Introduction section to emphasize the significance of this work by highlight the gaps in the literature and how this work fills them.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: For binary neural networks, some hyperparameters and optimization tricks such as weight decay lose their meaning. In this paper, the authors reinterpret these methods through a perspective of gradient filtering. Specifically, if the binary weights are taken to be the sign of latent weights the authors show that SGD with momentum and weight decay is equal to smoothing the gradient twice with an EMA filter. 



strength_and_weaknesses: Strengths:

* The authors provide a clear and intuitive explanation for how hyperparameters interact when training binary networks with SGD.
* The authors clearly verify their mathematical arguments experimentally.

Weaknesses:

* The mathematical arguments are rather simple and not surprising. 
* It is not clear how the analysis extends to adaptive optimizers such as Adam. Note that Adam is used for the imagenet experiments.
* The authors refer to “our optimizer”, but it seems to me like this is just a way to parametrize well-known methods. 
* Only results matching SOTA are reached.



clarity,_quality,_novelty_and_reproducibility: Regarding the mathematical arguments in the paper, the clarity could be improved by using more formal language with theorems, assumptions, and so on explicitly stated. The current mathematical exposition is rather hand-wavy. One weakness is that the analysis might not extend to adaptive optimizers, could the authors comment on this?

The paper also repeatedly refers to “our optimizer”. Could you clarify what this optimizer is?

Regarding the experiments, the figures and experimental setup are rather clear. However, I find the hyperparameters to be rather strange. E.g. a batch size of 510 is used. Could the authors motivate their choice of hyperparameters? Furthermore, could the authors discuss how the choice of hyperparameters for their method and their competitors be discussed?

The empirical results are also broken down by one or two-stage training. Is there any reason why two-stage training might not be preferred? It seems to give better results, but maybe it consumes more resources? Furthermore, could your method be applied to two-stage training and then improve the final results?




summary_of_the_review: The paper visits an interesting topic and brings some relevant and thoughtful, albeit maybe simple, mathematical arguments which illuminate how hyperparameters interact when using SGD for binary networks. It is not clear how the analysis extends to adaptive optimizers, and what exactly the proposed method is is also not clear. While the authors clearly verify the theoretical results, the method does not seem to bring any performance benefits over two-stage methods.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: ### Description

The paper proposes an analysis of heuristic straight-through SGD methods for training binary neural networks. The current heuristics contain many hyperparameters such as latent weight initialization, latent weight learning rate, latent weight decay. Exploiting the invariances of the learning algorithm due to the binarization operation, the paper infers a simplified equivalent learning algorithm with effectively fewer hyperparameters and a new interpretation as second order gradient filtering.

### Summary of Evaluation
The weaknesses pointed out by reviewers were:
* The mathematical arguments are rather simple and not surprising (H9io).
* It is not clear how the analysis extends to adaptive optimizers such as Adam. Note that Adam is used for the imagenet experiments (H9io).
* Several reviewers remarked they do not see an advantage in terms of accuracy.
* Several reviewers and AC pointed out minor issues (clarity, literature, technical)

I believe all these issues have been clarified. In particular the Adam is used for optimizing real-valued parameters along the considered simplified optimizer for binary weights. And in particular, the method is not supposed to improve accuracy. The paper was overall positively evaluated by reviewers and no issues requiring a major revision were identified. Despite the technical contribution is rather simple compared to many other papers at ICLR, it is useful and rather solid compared to many papers in the binary NNs field and seems to offer a new view allowing to build new connections. Therefore we give it a green light.

### Recommendations for the final version

Authors are encouraged to make improvements for the final version:
* Discuss related work on interpreting latent weights (as already discussed). 
* Show that the gradient filter is really estimating the gradient: assuming that $\nabla L_i$ are i.i.d. for all $i$ (ie. it is a stochastic gradient evaluated at the same point in the parameters space), show that the filtered $g_i$ (18) is unbiased: $E[g_i] = E[\nabla L_i]$. Adjust scaling to satisfy this, if necessary. Visualize filter weights in comparison with EWA (see below)
* Look up and connect to the related work on applying other filters than EWA in deep learning training, 
e.g. Yang "Stochastic Gradient Variance Reduction by Solving a Filtering Problem"
* Improve the paper clarity, conciseness and soundness of arguments

### Follow-up on the discussion

I thank the authors for the explanations.

>  The Xavier / He initializations

This was concerning the paragraph "Latent weight initialization" where the paper says " no longer makes sense to use real-valued weight initialization techniques". I am quite convinced that the gradient filtering view allows to simplify the existing heuristic SGD algorithm from which the paper starts. However I am not so convinced that this allows to fully change the interpretation. In particular more argumentation of the claim "no longer makes sense to use real-valued weight initialization techniques" is neede. Perhaps saying that the variance analysis underlying these techniques does not apply would be verifiable and more understandable. The random initialization at zero is still used in the proposed scheme, somewhat artificially. Also note that in the Adam they perform a bias correction of the EWA averages for the first and second moment, which assumes zero initialization. The correction is such that the filtered mini-batch gradient is an unbiased estimate of the full gradient (assuming we do not make optimization steps, ie. the learning rate is infinitesimally small).

> "Expand using past two gradients" -- As far as we know, this is not possible without changing the working of the filter.

Yes, I agree.
Another, perhaps a better option to understand it would be to plot the filter weights. I.e. to expand (18) as a linear combination of all past gradients only and show the weights of this linear combination as plot. E.g. with EWA the weights can be written explicitly $w_0 = (1-q)^n$, $w_i = (1-q)^{n-i}q$. It should be possible to compute such weights for the proposed filter at least numerically and visualize for different choices of a parameters. This would be very complementary to Fig 2. Consider improving the paper. Then, one could attempt to make a rational choice of the filter. I would argue that a filter that makes a flat average of $k$ past gradients would be useful -- it has a good variance reduction and low lag, however implementing it requires remembering $k$ past gradients. If the second order could move the weight mass more towards such a rectangular filter compared to EWA that would be convincing for me and possibly useful in a wider context. Please check and discuss in the paper the related work on gradient filtering. It would strengthen the paper to show such connections.






justification_for_why_not_higher_score: The contribution of the paper was evaluated as not very strong and the paper is not as comprehensive in terms of analysis, arguments and connections to the literature. The practical gain was recognized by reviewers as useful but not exciting.
justification_for_why_not_lower_score: The paper contributes a useful knowledge, clearly demonstrated theoretically and experimentally. The average score is positive. No major issues identified.
Decision: Accept: poster