Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces Fourier frequency domain to solve the gradient of sign function’s estimation problem in BNN, and uses noise adaptation module to compensate the approximation error. The experiments show promising improvement. It is a good work in both theoretical and experimental perspectives.
main_review: The method based on Fourier Series to approximate the gradient of sign function is novel. Extensive experiments on image classification task show the effective of this paper’s methods. In addition, this paper proposed a noise adaptation module into the training phase which can refine the gradient effectively.

Introducing the noise adaptation module to estimate the gradient more accurately  sounds okay. The module is only used in training stage and discarded during inference, which is achieved by controlling the hyper-parameter \alpha in Eq.14 and gradually decrease it to zero, which is simple and clever.

The experiments on typical BNNs including VGG, AlexNet and ResNet have demonstrate the effectiveness of the proposed method.

limitations_and_societal_impact: STE based on Taylor expansion, this paper’s method based on Fourier Series, what is the necessity to use FS to estimate the gradient of sign function?

In line 168, you mentioned SGD, however in line 284 you said you use Adam optimizer. I s there any problem in learning rate adaptation or others?

In Equation 15, you use an MLP with two fully-connected layers with relu as the noise adaptation module. Will the performance be better if you use a more complicate DNN module?

A recently approach [Lin2020] proposed another approximation function for BNN, is that a comparable approximation compared to sine series?

Is the BN applied to the noise module?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an interesting training strategy to train binary neural networks. The gradient of the non-differentiable sign function is approximated by the sine series. Besides, in order to prevent using infinite sine series to accurately approximate sign function, a noise adaptation module is further introduced. Extensive experiments on CIFAR-10 and ImageNet are conducted to validate the effectiveness of the proposed method.
main_review: Strength:

+ Approximating a function by series decomposition is reasonable and has strong theoretical guarantee. It is interesting to apply series decomposition to train the binary neural networks.

+ To reduce the gap between the real gradient and the approximate gradient, a modification module is further proposed. This module can improve the approximation precision to make the training robust. Ablation study also verifies the effectiveness of this module.

+Extensive experiments on multiple datasets validate that the proposed method can improve the performance of the trained binary neural networks.

Weaknesses:

- I am a little confused about Fig.1, it seems that all of the three approximation functions can approach sign function when the corresponding parameters go to infinity. How do you judge which approximation function is better?
- In Eq.7, the sign function is equal to the square wave only when |t|<T. Then how do you deal with those input that is larger than T?

- The number of series seems to be a critical hyper-parameter. How to determine it?

- How do you decrease \alpha in Eq.14 during training?

- In STE estimation function the input is clipped to [-1,1], is the same clip used in this method?

-------------------------------------after post-rebuttal------------------------------------
My concerns are well addressed in the authors' rebuttal. So I will keep my original score of 7.  
limitations_and_societal_impact: Dealing with input that is out of the range and the way of determining the hyper-parameters are my major concerns.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose an interesting strategy to train binary neural networks. First, the gradient of sign function is approximated by its series decomposition, i.e., the sign function is decomposed to a series of sine functions. A noise adaptation module is designed to modify the approximated gradient. The experiments conducted on CIFAR-10 and ImageNet datasets with different models show an obvious improvement compared to the baseline methods.
main_review: Basically I think the paper bears some merits as follows:
1. This paper uses the gradient of sine series as the proxy to update the binary neural networks, which bypass the problem of calculating gradients accurately. I appreciate this exploration, as in my mind how to train binary neural networks is still an open challenge as the gradient of sign function cannot be calculated directly.

2. The devised modification module avoids stacking too many sine series and makes the training more stable.  

3. Experimental results are strong. Ablation studies also validate the effectiveness of the proposed method.


Concerns:
1. The noise adaptation module is designed with two FC layers. Is it the optimal formulation? It is interesting to further discuss how to design effective noise adaptation module.

2. The performance in Fig. 3 and Fig. 4 converges very fast with a small number $n$. Is this an ablation study with or without noise module? If without the noise module, why does an inaccurate approximation still results in a good performance? If the ablation study is conducted with the noise module, what will happen to n when removing the noise module? Will you need a larger n to make the performance saturate?

3. I admit that the improvement over the baseline is large. However, a relatively weak baseline is used in the experiments. I would like to see what the improvement is over the state-of-the-art stronger baseline such as ReactNet [1].

[1] ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions. ECCV, 2020.

Overall the paper is clear and well motivated, I would suggest a clear accept if my above concerns can be effectively addressed and clarified by the authors.
limitations_and_societal_impact: None.
review_title: null

Meta-review: The paper proposes a new method to approximate the gradient of the sign function in binary neural networks. Specifically, the gradient of the sign function is estimated in the frequency domain using Fourier series with a fixed number of sine functions with different periods. In addition, to further compensate for the approximation error, this paper proposes a noise adaptation module to refine the gradient. The paper is well motivated and well organized. The proposed solution is novel and technically sound and shows promising results on image classification tasks. Reviewers all agreed that the paper has novel ideas and is clearly written.Thus, I am recommending acceptance.

Nevertheless, there are some detailed technical concerns raised by reviewers. The authors have addressed most of them in the rebuttal. I suggest the authors carefully revise the manuscript according to the reviewers’ comments. In particular, the following modifications need to be made. 1) add comparison to Lin2020 and REactNet [1]. 2) further justification of the advantage of the Fourier series approximation over the others.

Decision: Accept (Oral)