Learning Frequency Domain Approximation for Binary Neural Networks
Authors: Yixing Xu
Kai Han
Chang Xu
Yehui Tang
Chunjing Xu
Yunhe Wang
Venue: NeurIPS 2021 Oral (2021)
Keywords: binary neural network
frequency domain approximation
fourier series
noise

Abstract: Binary neural networks (BNNs) represent original full-precision weights and activations into 1-bit with sign function. Since the gradient of the conventional sign function is almost zero everywhere which cannot be used for back-propagation, several attempts have been proposed to alleviate the optimization difficulty by using approximate gradient. However, those approximations corrupt the main direction of factual gradient. To this end, we propose to estimate the gradient of sign function in the Fourier frequency domain using the combination of sine functions for training BNNs, namely frequency domain approximation (FDA). The proposed approach does not affect the low-frequency information of the original sign function which occupies most of the overall energy, and high-frequency coefficients will be ignored to avoid the huge computational overhead. In addition, we embed a noise adaptation module into the training phase to compensate the approximation error. The experiments on several benchmark datasets and neural architectures illustrate that the binary network learned using our method achieves the state-of-the-art accuracy. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/FDA-BNN.

Learning Frequency Domain Approximation for Binary Neural Networks
Yixing Xu', Kai Han', Chang Xu², Yehui Tang 4.3, Chunjing Xu', Yunhe Wang"!
Huawei Noah's Ark Lab
The University of Sydney Speking University
(yixing.xu) xu, kai. hai, tangyehui, xuchunjing, yumhe.vang/huussei.com c. xu@dssyney.. edu. au
Abstract
Binary neural networks (BNNs) represent original full-precision weights and acti- vations into 1-bit with sign function. Since the gradient of the conventional sign function is almost zero everywhere which cannot be used for back- paopapropagttion, several attempts have been proposed to alleviate the optimization difficulty by using approximate gradient. However, those approximations corrupt the main direction of factual gradient. To this end, we propose to estimate the gradi- ent of sign function in the Fourier frequency domain using the combination of sine functions for training BNNs, namely frequency domain approximation (FDA). The proposed approach does not affect the low-frequency information of the original sign function which occupies most of the overall energy, and high-frequency coefficients will be ignored to avoid the huge computational over- head. In addition, embed noise adaptation module into the training phase to compensate the approximation error. The experiments on several benchmark datasets and neural architectures illustrate that the binary network learned us- ing our method achieves the state-of-the-art accuracy. Code will be available at http:///wwecoooooooomommm///eete
Introduction
The success of deep convolutional neural networks (CNNs) has been well demonstrated in several real-world applications, e.g., image classification [24 35 59 I6], object detection [42], semantic segmentation [34], and low-level computer vision [46]. Massive parameters and huge computation- complexity are usually required for achieving the desired high performance, which limits the application of these models to portable devices such mobile phones and smart cameras.
In order to reduce the computational costs of deep neural networks, number of works have been proposed to compress and accelerate the original cumbersome model into portable one. For example, filter pruning methods 33 60
aim to sort the filters based on their importance scores in which unimportant filters will be treated as redundancy and removed to derive
lightweight architectures. Knowledge distillation methods [19 36 52 51] are explored to transfer the knowledge inside a teacher model to student model. Tensor decomposition method [40] decomposes the original large weight tensor into several small ones to simplify the overall inference quantize the widely used 32-bit floating point weights and activations into low-bit ones to save the memory consumption and computational cost.
process. Model quantization methods 33 22 54 32 31
"Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Sign(s)
Sing(s)
Tamby/8)
(S,(s)
(a)
(b)
(0)
FFT Sign(s)
FFT Siny(X)
FFT Tanby/X)
FFT S8y(x)
(e)
(n)
(g)
(h)
JFFT Sig Sign(x)
Sign(x)-FFT Sing/oi
JFFT Sign(X)FFT Tanby(X)
JFFT Sign(x)-FFT S8./8|
(0
Figure 1: From top to bottom: original functions in spatial domain, corresponding functions in frequency domain and the difference between the current function and sign function in frequency domain. From left to right: sign function, combination of sine functions, tanh function in [10] and SignSwish function in 6 (short as SS).
Among the aforementioned algorithms, binary neural network (BNN) is an extreme case of model quantization method in which the weights and activations in the original neural network will be quantized to 1-bit (-1 and +1) values. Obviously, the memory usage for weights and activations binary convolutional layers are 32x lower than that in the full- polliprecision network with the same model architecture. BNN was firstly proposed by Hubara et.al. (22) by utilizing stright-througg estimator (STE) M for handling gradient problem sign function. Then, XNOR-Net 4I] introduced channel- wise scaling factor for weights to reduce the quantization error. Dorefa-Net [ used single scaling factor for all the channels and achieved competitive performance. In addition, there are number of methods for optimizing the original structure to improve the performance of learned
BNN. ABCNet (26 used multiple parallel binary convolutional layers. Bi-Real Net (30 inserted skip-connections to enhance the feature representation ability of deep BNNs.
The optimization difficulty caused by the sign function BNNs is a very important issue. In practice, the original full-precision weights and activations will be quantized into binary ones with sign function during the feed-forward procedure, but the gradient of sign funccion cannot be directly used for back- prook-ropagation. Therefore, most of the approaches mentioned above adopt the straight- through estimator (STE) I for gradient back pack-propagation to train BNNs. The inconsistency between the forward and backward pass caused by the STE method prevents BNN from achieving better performance. Additionally, several attempts were proposed estimate the gradient of sign function for replacing STE. For example, DSQ [10] introduced tanh-alike differentiable asymptotic function to estimate the forward and backward procedures of the conventional sign function. BNN+ (6) used SignSwish wish activation function to modify the back- pack-propagation of the original sign function and further introduced a regularization that encourages the weights around binary values. RBNN [25) proposed taining-awarr approximation function replace the sign function when computing the gradient. Although these methods have made tremendous effort for refining the optimization of binary weights and activations, the approximation function they used may corrupt the main direction of factual gradient for optimizing BNNs.
To better maintain the main direction of factual gradient in BNNs, we propose frequency domain approximation approach by utilizing the Fourier Series (FS) to estimate the original sign function in the frequency domain. The FS estimation is a lossless representation of the sign function when using infinite terms. practice, the high-frequency coefficients with relatively lower energy will be ignored to avoid the huge computational overhead, and the sign function will be represented as the

---

combination of fixed number of sine functions with different periods. Compared with the existing approximation methods, the proposed frequency domain approximation approach does not affect the low-frequency domain information of the original sign function, i.e., the part that occupies most energy of sign function [50] as shown Fig. [ Thus, the main direction of the corresponding gradient w.e..hh original sign function can be more accurately maintained. In addition, to further compensate the subtle approximation error, we explore a noise adaptation module in the training phase to refine the gradient. To verify the effectiveness of the proposed method, we conduct extensive experiments on several benchmark datasets and neural architectures. The results show that our method can surpass methods for training binary neural networks with higher performance.
other
2 Approach
In this section, we first provide the preliminaries of BNNs and STE briefly. Then, we propose to estimate sign function with the combination of sine functions and analyze the inaccurate estimation problem. Finally, we introduce noise adaptation module to solve the problem and derive our FDA-BNN framework.
2.1 Preliminaries
The conventional BNNs quantize the weight matrix W E gexxexxkx and activation matrix A E R xhxw to 1-bit, and obtain the binary matrices Ws gexexxkkk and As E [xxxxxxxxe for the intermediate convolution layers using sign function as follows:
wh {
-1,
1.
Wf > 0,
Wy 50, ob -
{-
-1, af 50
1. of >0 0,
(1)
where wy and a are the elements in the original 32-bit floating point weight matrix W and activation matrix A, respectively, and wh and dg are the elements the binarized weight matrix W,, and activation matrix As. respectively. Given the 1-bit weights and activations, the original convolution operation can be implemented with bitwise XNOR operation and the bit-count operation [30]. During training, both the full-precision weight matrix W and the 1-bit weight matrix W, are used, while only W, is kept for inference.
Note that it is impossible to apply back- propagation scheme on the sign function. The gradient of sign function an impulse function that almost zero everywhere and unable to be utilized in training. Thus, the STE method [ is introduced to compute its gradient as:
ac Clip(
OC
1,,
Ow
Owb
which C the corresponding loss function for the current task (i.e., cross-entropy loss for image classification) and
1, if 11,
Clip(æ, 1, v, if -
(3)
1, otherwise.
Since STE is an inaccurate approximation of the gradient of sign function, many works have focused on replacing STE during back-propagation. BNN+ [6] introduced a SignSwish function, while DSQ [10] proposed tanh-alike function and minimized the gap between the sign function with learnable parameter. Such methods focus on approximating sign function in spatial domain, and corrupt the main direction of the gradient. Thus, we propose new way estimate sign function to maintain the information in frequency domain.
2.2 Decomposing Sign with Fourier Series
In the following sections, use f(-) and f-) to denote original function and its corresponding gradient function.
Recall that the gradient of sign function is an impulse function which is unable to be back-propagated. Thus, zero-order (derivative-free) algorithms such as evolutionary algorithms need be applied to reach the optimal solution, which is quite inefficient. Thus, propose to find surrogate function
3

---

that can be empirically solved by first-order optimization algorithms such as SGD, while theoretically has the same optimal solution as sign function.
has been proved that any periodical signal with period 7 can he decomposed into the combination of Fourier Series (FS) 45
ao f(t)
2 a cos(twt) + b, sin (iwt)),
(4)
in which w = 2x/ T is the radian frequency, an is the direct component and {bx),-2 (fa, ((a))==) are the coefficients of the sine (cosine) components. Specifically, when the periodical signal the square wave, we have
4 a, 0 for all i; be
if i is odd,
in 0, otherwise.
(5)
and derive the FS for the square wave st):
4 sin ((2) + 1)wt)
st)
(6)
π 1=0
2i
Note that when the signal is restricted into single period, the sign function is exactly the same the square wave:
stgn(() = s(t), | < T.
Thus, the sign function can also be decomposed into the combination of sine (cosine) functions, and its derivative is computed as:
4w
(())
((2i 1)wl).
In this way, propose to replace the derivative used in STE (Eq. 22) with Eq. (8) during back- propagation for better approximating the sign function.
The FS decomposition is a lossless representation of the sign function when using infinite terms by transforming the signal from spatial domain into frequency domain, as shown in Theorem First, we can rewrite Eq. (6) as
4
n sin ((2i + 1)wt)
(1)
π
+
i=U
where n is the number FS terms. The corresponding derivative is
4w
(t)
cos ((2i + 1)wt).
(10)
T=0
In the following, we show that increases, the mean squared error between the estimator s, (t) and the ground-truth s(t) decreases, and converges to 0 when n + co.
Theorem 1. (Mean Square Convergence (43]) Given s(-) the square wave function which is integrable in a period, and 8n(-) is the nhth partial sum of the Fourier Series of s(.), then (:) converges to s(-) in the mean square sense, i.e.
1 T / (s(() (t)lladt + as n 00.
(11)
The above theorem shows that the estimator (t) converges to the ground-truth s(t) when using an infinite number of FS terms, thus theoretically has the same optimal solution. In practice, we use fixed number of FS terms for two reasons: 1) We can avoid the huge computational overhead by ignoring tigh-frequency coefficients. 2) We can still accurately maintain the main direction of the corresponding gradient the./.he original sign function. Compared to other estimation methods that directly estimate sign function the spatial domain, our method benefits to the approximation process since the proposed approximation approach does not influence the low-frequency domain information of the original sign function, which occupies most of the energy.
2.3 Solving the Inaccurate Estimation Problem

---

Although using infinite FS terms makes the esti- mator S, (t) converges the ground truth s(t), an error
Sign
01
STE
8()
4C3
=()==(()) = - (1)
(12)
(6)
occurs in reality when n is finite. The error comes from the infinite high-order FS terms and has little energy. Basically, a simple way to solve the inaccurate estimation problem is to in- crease the number of FS terms n to make 8, (t) approaches st). However, the gradient 8 (t) will approach the impulse function and desta- bilize the back pack-propagttton process. Besides, increasing the number of FS terms n will in-
6,()
8t 9,(0)
3z
8()
6(-)
(e)
(4)
Figure 2: (a) The forward pass of the combination of sign function and noise adaptation module. (b) Corresponding backward pass of (a). (c) The for- ward pass of the combination of the sine module and the noise adaptation module. (d) Correspond- ing backward pass of (c). The actual forward and backward pass are bolded in the figure.
crease the training time. Thus, we propose to use another way to solve the aforementioned problem.
Specifically, we treat the error r(t) between the sign function and the estimation function as noise. In the previous works, there are several ways to deal with noise. The first is change the training framework to derive a robust network for noise [15 14 55 9. The second is to improve the training loss function noise robust loss [56].
We propose to use a learnable neural network called noise adaptation module to deal with noise. Compared with increasing the number of FS terms, neural network has learnable parameters which make the training more flexible. Besides, the neural network has been proved to be Lipschitz continuous [44] 61 which means the gradient is continuous and avoid the problem from impulse function. We train the noise adaptation module through an end-to-end manner which is reasonable when initialized the weight of the module in correct way. is easy to know that the distribution of r(t) in Eq. (12) has zero mean and finite variance regardless of the number of FS terms n. Assuming that the input of noise adaptation module is Gaussian distribution, then the initialization of the weights of noise adaptation module need to have zero mean in order to make the distribution of output fit that of r(t). As long as the initial output is unbiased, the proceeding learning process will automatically approach the final goal through an end-to-end training manner with gradient descent based optimization methods.
Specifically, given the input vector t E RIxd the process of sign function can be replaced by the combination of a sine module (-(-) and noise adaptation module ē(-) which estimates the error r(-) using two fully-connected layers with ReLU activation function and shortcut:
ē(t) otW
17()),
(13)
in which (t)) max(t, 0), W1 dx & and W₂ E R xd are the parameters of the two FC layers in which le is used to reduce the number of parameters and set to 64 in the following experiments.
m()) is the shortcut connection.
There are two benefits from adding the shorteut connection. The first is that it helps the flow of gradient and alleviates the gradient vanishing problem 28 38, 20). The second is that given n(t)= f as example, we have (t) a which equals to adding bias to the gradient. This benefit the training process since given limited training resources and time, only finite number of FS terms can be used compute 5, (t) in Eq. ( and raise the problem that the tail of the estimated gradient &, (t) oscillates around at a very high frequency. This phenomenon does harm to the optimization process, since a very little disturbance on the input will results in two different gradients with opposite direction, and adding bias term on the gradient helps to alleviate this problem.
Based on the analysis above, given Eq. p and Eq. 133, with the combination of sine module and noise adaptation module:
able to approximate sign function
S, (t) + aé(t),
(14)
in which is the hyper-parameter to control the influence of noise adaptation module. In order to keep the weights and activations as 1-bit in inference, the sine module is used only in the backward pass and replaced as sign function in the forward pass, as shown in Fig. 2 Besides, a is gradually
5

---

decreased during training and reaches O at the end of training to avoid the inconsistency between training phase and inference phase. The weights of both BNN and noise adaptation module are updated in an end-to-end manner by minimizing the cross-entropy loss Lee.
14 and the loss function lees the gradients in Fig. 2 (d) can
Specifically, given Eq.
Algorithm Freed-Forward and Back-Propase Pack-Propagation Pro- cess of convolutional layer in FDA-BNN.
be computed as:
Oloe Olce wa @ ((tWi) 2 W, Ot
Require: A convolutional layer with weights W and input features (activations) A, the noise adapta- tion module with parameters Wa1 (Ww1) and Wa2 (Ww2) for activations (weights), the loss function loe. the learning rate y, and training iterations T.
Oz
olce (())
dz
Olee
dz о 4w n Σ
cos ((2i (2i + 1)wt), (15)
1: f = to 7 do
2:
Feed-Forward:
in which dhse is the gradient back- propagated from the upper layers, @ rep- resents element-wiss multiplication, and Ofse
3: Quantize the weights and activations: W.
Sign(W). = =Agnn(A);
4:
Apply the convolutional operation the bina-
is the partial gradient on t that back- propagate to the former layer.
rized weights and activations: W, ( As;
öt
5:
Back-Propagee Propagttoo:
6:
Compute the gradient of W, and Ag using stan- if.s;
dhee
Oz Hbee WT @ ((tWi) 2 0): (16)
af.e dard back propaaappg,,,on, and get aW, and OA,
0W,
=t
7:
ah. Compute the gradient of A, and derive DA
Olce
=t(W))
Ulee
(17)
using Eq. (15);
OW2
Oz
8:
Compute the gradients on parameters of noise adaptation module for activations: Ot.e. and Olsz using Eq. (6) and Eq. [7
in which Uha and the gradients used aw; aW, update the parametens in the noise adap- tation module.
BW.1
aWaz
9:
Compute the gradients of W and derive 86... aw and also He and 8b.x with the same manner ow BW
The formulations from Eq. 9 to Eq. 17 are applied on both weights and activa- tions for replacing the back-propagation sign function. The feed-forward and back- pockppropggttion process of convolution- al layer in FDA-BNN is summarized in Alg. 1
step 7-~8.
10:
Parameter Update:
11:
W < UpdateParameters/ W, atc
ww 7)
12:
Wa1 +- UpdateParameters (Wa1. 7W at
Dige.
7);
13:
Waz + UpdateParameters: Waz, oW,2
Otce
7);
14:
W "1 UpdateParammeters W wl, JW..
7)
3 Experiments
15:
Ww2 + UpdateParametees// W Ww2, oW .7).
In this section, we evaluate the proposed method on several benchmark datasets such as CIFAR-10 23 and ImageNet 7 on NVIDIA-VI00 GPUs to show the superiority of FDA-BNN. All of the proposed FDA-BNN models follow the rule in previous methods that all the convolutional layers except the first and last layers are binarized [77 6. All the models are implemented with PyTorch (37 and MindSpore 21).
16:
end for
3.1 Experiments on CIFAR-10
The CIFAR-10 23
dataset contains 50,000 training images and 10,000 test images from different categories. Each image is of size 32 x 32 with RGB color channels. Data augmentation methods such as random crop and random flip are used during training. During the experiments, we train the models for 400 epochs with a batch size of 128 and set the initial learning rate as 0. The SGD optimizer is used with momentum set of 0.9 and weight decay of Ic-4.
The widely-used ResNet-20 and VGG-Small architectures in BNN literature 30 10 are used to demonstrate the effectiveness of the proposed method. Dorefa-Net [57] is used as our baseline quantization method, and the sine module and noise adaptation module are used during training.
Other competitors include: 1) methods using STE for back-propagation, egg., BNN (2], Dorefa- Net [57] XNOR-Net 41). TBN 49 ; 2) methods using approximated gradient rather than STE during back- hack-propagation, egg., DSQ 10. BNN+ (.. -R-Net [39]: 3) fine-tuning methods, egg., LNS [1]
The experimental results are shown in Tab.

---

Table 1: Experimental results CIFAR-10 using different 1-bit quantized models. Bit-width (W/A)
denotes the bit length of weights and activations, and BackProp denotes the way of computing gradients and updating parameters.
Network
Method
Bit-width (W/A)
BackProp
Acc (%)
FP32
32/32
92.10
Dorefa-Net
17
1/1
STE
84.44
XNOR-N -et 41
1/1
STE
85.23
LNS [7
1/1
STE
85.78
ResNet-20
TBN 49
1/2
STE
84.34
DSQ 10
1/1
Tanh-alike
84.11
IR-Net [39
1/1
Tanh-allik
85.40
FDA-BNN
1/1
Fourier Series
86.20
FP32
32/32
94.10
Dorefa-NC Net
57
I/I
STE
90.20
Binary 22
1/1
STE
89.90
XNOR-Ne -et
1/1
STE 89.80
VGG-small
BNN+ [6
171
SignSwish
91.31
DSQ 10
1/1
Tanh-alike
91.72
IR-Net [39
1/1
Tanh-alike
90.40
EDA-BNN
1/1
Fourier Series
92.54
The results show that the proposed FDA-BNN method outperforms all the competitors and achieves state-of-the-arr accuracy. For ResNet-20 architecture, the proposed FDA-BNN achieves an accuracy of 86.20% which improves the baseline model by 1.76%, and is 0.42% higher than the previous SOTA method LNS [17 which is fine-tuning based method and costs 120 more epochs than our method. When applying the proposed method to VGG-small model, the proposed method achieves an accuracy of 92.54% which is also the highest and improves the baseline model by 2.34%. FDA-BNN method outperforms other gradient estimation methods which approximate sign function in spatial domain such as DSQ [TO,, BNN+ (6] and IR-Net (39].
3.2 Ablation Study
In this section, conduct several ablation studies further verify the effectiveness of each com- ponent in the proposed method, the impact of shortcut in noise adaptation module and the effect of hyper-parameters.
Firstly, we conduct experiments to understand the effect sine module and noise adaptation module. The models are verified on CIFAR-10 using ResNet-20 architecture. As shown Tab. - the first line represent the standard Dorefa-Net baseline. We can see that only use sine module can benefit the training process and improve the accuracy from 84.44% 85.83%. When combining the sine module and noise adaptation module together, we reach the best performance, i.e., 86.20% accuracy, which shows the priority of using both modules in FDA-BNT method during training. Table 2: Ablation study sine module and noise adaptation module.
Sine Module
Noise Adaptation Module
Acc (%)
84.44
85.83
86.20
In order to further verify the usefulness of the noise adaptation module, we plug the module into other gradient approximation methods such as DSQ [10] and BNN+ [6]. The experiments are conducted on CIFAR-10 dataset with ResNet-20 model, and the results Tab. show that fitting the error between the surrogate function and the original sign function benefits to different gradient approximation methods.
Then we evaluate the influence of using different 77-) on the branch of shortcut in noise adaptation module. shown Tab. using shortcut results a better performance than not using it, and the shortcut function 17(x) asin(x) performs the best during the experiment. Note that this is consistent

---

Table 3: Plugging noise adaptation module into
Table 4: Ablation study n7-).
different methods.
w/o noise module
w/
Functions of shorteut branch
Ace (%)
noise module
7( =0 (w/o shortcut)
85.27
DSQ BNN+
84.11
84.46
n =0
85.93
84.59
84.87
n(x) =a sin
86.20
FDA-BNN
85.83
86.20
the analysis in Sec.2.3. Given n(x)) sin(),, the gradient flow can still be strengthened. Besides, the gradient r/ (a: = a cos(æ) equals to adding a positive bias the given range, and thus can still alleviate the problem that the tail of the estimated gradient & (a:) oscillates around 0 with very high frequency. a is set to 0.1 during the experiment.
Using different number of FS terms during training will influence the final performance of BNN. Intuitively, larger n will reduce the difference between s.,() and sign(-), but at the same time lead to more time training since combination of gradients will be computed (Eq.
(10).. We show the training time spent and the final accuracy achieved by using different n in Fig. 3 We can see that the number of FS terms n increases, the training time continues to increase while the accuracy saturates at about n. = 10, which means a moderate n. is enough for training an accurate BNN.
As shown in Fig. large n leads to a waste resources and time. Hence, we explore three different settings of using hyper-parameete during training.
small number of FS terms do harm to the performance of BNN while
Setting 1: the number of FS terms fixed pre-defined number np during training.
Setting 2: the number of FS terms is gradually increased from 1 to pre-defined number np.
Setting 3: the number of FS terms is gradually increased from (best result in setting 1) to a pre-defined number np-
Results of different settings shown in Fig. Setting performs well at wider range than setting 2 and they reach comparable performance when Mp is large enough. Setting 3 performs the best, since it start from moderate and avoid the large gap between s.,()) and sign(-) the beginning training, and is able to approach the sign function as the training proceed. Thus, in the following experiments start from moderate number n, (e.g., n, 9) and gradually increase it to np 2n...
0.9
80
0.
Trannin Tine (en Japochi
Accureynn)
800
0.3
04
0.2
74
Setting Setting Setting
0.1
20
Figure 3: Training time spent and the final accu- racy achieved when using different number FS terms. The blue line denotes the accuracy and the red bar denotes the training time.
Figure 4: Accuracies under different settings of hyper-parameter n during training.
3.3 Experiments ImageNet
In this section, we conduct experiments on the large-scale ImageNet ILSVRC 2012 [7] dataset. ImageNet dataset contains over 1.2M training images with 224x224 resolutions from 1,000 different

---

Table 5: Experimental results ImageNet using different 1-bit quantization methods. 'W/A' denotes the bit-width of weights and activations, and BackProp denotes the way computing gradients and
updating parameters. **' indicates that ReActNet (29 is used as the baseline method and only the
way of computing gradient of sign function is replaced with the proposed method. Network
Method
W/A
BackProp
Top-1
Top-5
FP32
32/32
69.6
89.2
TBN [49
I/2
STE
55.6
79.0
Dorefa-Net [57
1/1
STE
52.5
67.7
Binary-Net
22
1/1
STE
42.2
67.1
XNOR-Net 41
1/1
STE
51.2
73.2
ABCNet 26
1/1
STE
42.7
67.6
Bireal-Net 33
1/1
STE
56.4
79.5
Bireal-Net +PRcLU
1/1
STE
59.0
81.3
BOP [18]
1/1
STE
54.2
77.2
ResNet-18
LNS 17
1/1
STE
59.4
81.7
ReActNet 129
1/1
STE
65.5
86.1
HWGQ
172
Piece-wiss wise function
59.6
82.2
PCNN (J 1)
1/1
DBPP
57.3
80.0
Quantization Networks 53
1/1
Sigmoid
53.6
75.3
ResNetF
2
1/1
ApproxSign
58.1
80.6
BONN
1/1
Bayesian learning
59.3
81.6
IR-Net
39
1/1
Tanh-alike
58.1
80.0
RBCN
27
1/1
Project-based
59.5
81.6
RBNN 225
1/1
training-aware
59.9
81.9
FDA-BNN
I/1
Fourier Series
60.2
82.3
FDA-BNN*
1/1
Fourier Series
66.0
86.4
FP32
32/32
56.6
80.2
Dorefa- Net [57
1/1
STE
43.6
BinaryNer 22
1/1
STE
41.2
65.6
AlexNet
XNOR-NC Net [1
1/1
STE
44.2
69.2
LNS 17
1/1
STE
44.4
FDA-BNN
1/1
Fourier Series
46.2
69.7
categories, and 50k validation images. The commonly used data augmentation method is applied during training. The training images are resized, random cropped, random flipped and normalized before training, while the test images are resized, center cropped and normalized.
In the following, we conduct experiments on widely-used ResNet- 18 and AlexNet. We use the Adam optimizer with momentum of 0.9 and set the weight decay to O. The learning rate starts from 1e-3. The experimental results are shown in Tab. [5] Following the acknowledged setting 22 (7), the first and last layers of the network are not binarized for classification, and all methods shown in the table except for ABCNet (26) and Binary-Net 20 do not quantize the down-sample layers in the ResNet shortcut branch.
We can see that on ResNet-18, the proposed FDA-BNN achieves top-1 accuracy 60.2% and top-5 accuracy of 82.3%, which improves the baseline method (BirellNeet + PReLU activation function) by 1.2% and 1.0%, and surpass all other competitors including TBN 49 and HWGQ 4 which use 2-bit activations for quantization. When using ReActNet (2] as the baseline method and replace the way of computing gradient of sign function with our method, the proposed FDA-BNN achieves a top-1 accuracy of 66.0% and top-5 accuracy of 86,4%, which outperforms the baseline method by 0.5% and 0.3%. For AlexNet, we use the quantization method in Dorefa-Net (57] as the baseline method, and achieve top-1 accuracy of 46. 2% and top-5 accuracy of 69.7 7%, which also outperforms other state-of-the-ar methods such as LNS [7 by 1.8% on top-1 accuracy, which shows the superiority of the proposed FDA-BN BNN method.

---

4 Conclusion
In this paper, we propose a new frequency domain algorithm (i.e, FDA-BNN) to approximate the gradient of the original sign function in the Fourier frequency domain using the combination of series of sine functions, which does not affect the low-frequency part of the original sign function that occupies most of the energy. Besides, since only finite FS terms are used in reality, noise adaptation module is applied to fit the error from the infinite high-order FS terms. In order to gradually approach the sign function, we increase the number of FS terms during the training. The experimental results on CIFAR-10 and ImageNet datasets using various network architectures demonstrate the effectiveness of the proposed method. Binary networks trained using method achieve the state-of-the performance.
Funding Disclosure
We thank anonymous area chair and reviewers for their helpful comments. This work was supported in part by the Australian Research Council under Projects DE1 RE180101438 and DP210101859.
References
[1] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432. 2013.
[2] Joseph Bethge, Haojin Yang, Marvin Bornstein, and Christoph Mcinel. Back to simplicity: How to train accurate bnns from scratch? arXiv preprint wxXv:1906.00837, 2019.
[3] Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++4: Improved binary neural networks. arXiv preprint arXiv: 1ttp://999.33333 2019.
[4] Zhaowei Cai. Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. In Proceedings of the IEEE conference computer vision and pattern recognition, pages
2017.
[5] Xinghao Chen, Yiman Zhang, Yunhe Wang, Han Shu, Chunjing Xu, and Chang Xu. Optical flow distillation: Towards efficient and stable video style transfer. In European Conference on Computer Vision, pages 614-630. Springer, 2020.
[6] Sajad Darabi, Mouloud Belbahri, Matthieu Courbariaux, and Vahid Partovi Nia. Bau+: Improved binary network training. arXiv preprint 18Xiv:1812.11880 2018
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenct: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009.
[8] Minjing Dong, Hanting Chen, Yunhe Wang, and Chang Xu. Crafting efficient neural graph of large entropy. In Sarit Kraus, editor, Proceedings of the Twenty- Eighth International Joint Conference Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 16, 2019, pages 2244 2250. ijeai.org. 2019.
[9] Jacob Goldberger and Ehud Ben-Reuven. Training deep netwolss using a noise adaptation layer. 2016.
[10] Ruihao Gong, Xianglong Liu, Shenghu Jiang. Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 4852-4861, 2019.
[11] Jiaxin Gu, Ce Li, Baochang Zhang, Jungong Han, Xianbin Cao, Jianzhuang Liu, and David Doermann. Projection convolutional neural networks for 1-bit cnns via discrete back propagation. In Proceddings of the AAAI Conference Artificial Intelligence, volume 33, pages 8344-8384 2019.
[12] Jiaxin Gu, Junhe Zhao, Xiaolong Jiang, Baochang Zhang, Jianzhuang Liu, Guodong Guo, and Rongrong Ji. Bayesian optimized 1-bit enns. In Proceedings of the IEEE International Conference on Computer Vision, pages 4909-4917, 2019.
[13] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, pages 1737-1746, 2015.
10

---

[14] Bo Han, Jiangchao Yao, Gang Niu. Mingyuan Zhou, Ivor Tsang. Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. In Advances in Neural Information Processing Systems, pages 5836-5846, 2018
[15] Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Rohust training of deep neural networks with extremely noisy labels. In Advances in neural information proessing systems, pages 8527-8537, 2018.
[16] Kai Han, Yunhe Wang. Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[17] Kai Han, Yunhe Wang, Yixing Xu, Chunjing Xu, Enhua Wu, and Chang Xu. Training binary neural networks through learning with noisy supervision. In International Conference on Machine Learning, pages 4017-4026. PMLR, 2020.
[18] Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang- -ing Cheng, and Roeland Nusselder. Latent weights do not exist: Rethinking binarized neural network optimization. In Advances in neural information processing systems, pages 7533-7544, 2019.
[19] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 503.0530, 2015.
[20] Furong Huang, Jordan Ash, John Langford, and Robert Schapire. Learning deep resnet blocks sequentially using boosting theory. In International Conference on Machine Learning, pages 2058-2067. PMLR, 2018.
[21] Huawei. Mindspore.
http://wwww..iindee
cn/ 2020.
[22] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks. Advances in neural information processing systems, 29:4107-4115, 2016.
[23] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[24] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, pages 1097-1105, 2012.
[25] Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Yan Wang, Yongjian Wu, Feiyue Huang, and Chia- Wen Lin. Rotated binary neural network. arXiv preprint arXiv:2009.1300 2020.
[26] Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances in neural information processing systems, pages 345- 353,5,33222 2017
[27] Chunlei Liu, Wenrui Ding, Yuan Hu, Baochang Zhang, Jianzhuang Liu, Guodong Guo, and David Docrmann. Rectified binary convolutional networks with generative adversarial learning. International Journal of Computer Vision, pages 1-15, 2021.
[28] Tianyi Liu, Minshuo Chen, Mo Zhou, Simon S Du, Enlu Zhou, and Tuo Zhao. Towards understanding the importance of shortcut connections in residual networks. In Advances in neural information processing systems, pages 7892-7902, 2019.
[29] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In European Conference on Computer Vision, pages 143 143-159. Springer, 2020.
[30] Zechun Liu, Baoyuan WU, Wenhan Luo, Xin Yang, Wei Liu, and Kwang- Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cans with improved representational capability and advanced training algorithm. In Proceedings of the European conference on computer vision (ECCV), pages 722-737. 2018.
[31] Zhenhua Liu, Yunhe Wang, Kai Han, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. arXiv preprint arXiv:2106. 14156, 2021.
[32] Zhenhua Liu, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao. Evolutionary quantization of neural networks with Speech and Signal Processing (ICASSP), pages 2785-2789. IEEE, 2021.
In ICASSP 2021-2021 IEEE International Conference on Acoustics,
[33] Jian han-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pages 2017.

---

[34] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic segmentation. In Proceedings of the IEEE international conference on computer vision, pages 1520-1528. 2015.
[35] Ini Oguntola, Subby Olubeko, and Christopher Sweeney. Slimnets: An exploration of deep model compression and acceleration. In 2018 IEEE High Performance extreme Computing Conference (HPEC). pages 1-6, IEEE, 2018.
[36] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 3967 -976, 2019.
[37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, 2019.
[38] George Philipp. Dawn Song, and Jaime G Carbonell. Gradients explode-deep networks are shallow resnet explained. 2018.
[39] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2250 2259, 2020.
[40] Stephan Rabanser, Oleksandr Shehur, and Stephan Günnemann. Introduction to tensor decompositions and 2017.
their applications in machine learning. arXiv preprint
[41] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classifi- cation using binary convolutional neural networks. In European conference on computer vision, pages 525 545.552.666 Springer. 2016.
[42] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, pages 91-99, 2015.
[43] BRYAN RUST. Convergence of fourier series. 2013.
[44] Kevin Scaman and Aladin Virmaux. Lipschitz regularity of deep neural networks: analysis and efficient estimation. arXiv preprint arXiv: 1005, 2018
[45] William D Stanley, Gary R Dougherty, Ray Dougherty, and H Saunders. Digital signal processing. 1988.
[46] Ying Tai, Jian Yang, and Xiaoming Liu. Image resolution via deep recursive residual network. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3147-3155, 2017.
[47] Ychui Tang, Yunhe Wang, Yixing Xu, Yiping Deng, Chao Xu, Dacheng Tao, and Chang Xu. Manifold regularized dynamic network pruning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5018-5028, 2021.
[48] Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. arXiv preprint arIiv:2010. 10732, 2020.
[49] Diwen Wan, Fumin Shen, Li Liu, Fan Zhu, Jic Qin, Ling Shao, and Heng Tao Shen. Tbn: Convolutional neural network with ternary inputs and binary weights. In Proceedings of the European Conference on Computer Vision (ECCV), pages 315-332, 2018
[50] Zifan Wang, Yilin Yang, Ankit Shrivastava, Varun Rawal, and Zihao Ding. Towards frequeney- explanation for robust cnn. arXiv preprint wrXiv:2005.03144 2020.
[51] Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, and Chang Xu. Positive- unlabeled compression on the cloud. Advances in Neural Information Processing Systems, 32:2565-2577 2019.
[52] Yixing Xu, Chang Xu, Xinghao Chen, Wei Zhang, Chunjing Xu, and Yunhe Wang. Keree based progressive distillation for adder neural networks. arXiv preprint MxXiv:2009.13444 2020.
[53] Jiwei Yang. Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng. Jianqiang Huang, and Xian-sheng Hua. Quantization networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7308-7316, 2019.
[54] Zhaohui Yang, Yunhe Wang, Kai Han, Chunjing Xu, Chao Xu, Dacheng Tao, and Chang Xu. Searching for low-bit weights in quantized neural networks. arXiv preprint arXiv:2009.06699 2020.
12

---

[55] Xingrui Yu, Bo Han, Jiangchao Yao. Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? arXiv preprint 1rX1::1901.0221,, 2019.
[56] Zhilu Zhang and Mert Sabuneu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing sysiems, 31:8778 8788, 2018.
[57] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1rXiv:1606.060 2016.
[58] Mingjian Zhu, Kai Han, Yehui Tang, and Yunhe Wang. Visual transformer pruning. arXiv preprint arXiv:2104.085 085000 2021.
[59] Mingjian Zhu, Kai Han, Enhua Wu, Qiulin Zhang, Ying Nie, Zhenzhong Lan, and Yunhe Wang. Dynamic resolution network. arXiv preprint arXiv:2106. 2021.
[60] Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-awarc channel pruning deep neural networks. Advances Neural Information Processing Systems, pages 875-886, 2018.
[61] Dongmian Zou, Radu Balan, and Maneesh Singh. On lipschitz bounds general convolutional neural networks. IEEE Transactions on Information Theory, 6663))6738-17777766 2019.
13


Reviews:
[{"review_id": "tCWNpm7nslU", "reviewer": "Reviewer_oG66", "summary": "BNN is an important network when we cannot support huge computational costs. However, its optimization is still an open question and cannot be addressed well. This paper proposed a novel gradient computation method for sign operation in BNN. The author used the combination of sine functions to approximate sign operation, and a noise adaptation module is applied to fit the high order error between sine functions and sign operations.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. Basically, this is a simple, novel, and effective method that approximates sign operation in the frequency domain aspect, which is different from other gradient approximation methods. As all known, previous studies only consider the optimization in the spatial space. This paper revisits this problem in the frequency domain, which is very interesting to me.\n\n2. The paper is technically sound and adding a noise adaptation module is a clever way to fit the high order error. The experimental results demonstrate the effectiveness of the proposed training method.\n\n3. In experiments, both CIFAR10 (10 classes) and ImageNet (1000 classes) are used to validate the proposed method. The results show that the proposed method is accurate and efficient.\n\nCons:\n\n1. Figure 1 is important to help readers understand why your motivation is working. More explanation should be added in the caption.\n\n2. Notations are unclear. In Section 2.1, there are some notations that are not explained, e.g., c_i, h…\n\n3. Some parts are not self-contained. In Theorem 1, what is the convergence rate regarding n?\n\n--------------------------------\nPOST REBUTTAL\n--------------------------------\n\nAfter reading comments from other reviewers, I am confident that this paper should be accepted by NeurIPS 2021. Thus, I would like to increase my score to 8.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Relatively weak baselines such as Dorefa-Net and Bireal-Net+PReLU are used. I understand the rebuttal time is limited, thus it is optional but better that the author conduct experiments based on a stronger baseline such as ReactNet [1].\n\n[1] ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6hrs"}, "extra_text": {}}, {"review_id": "boD4v8ehnYa", "reviewer": "Reviewer_UYYe", "summary": "This is a novel gradient estimation method for binary networks based on the Fourier series estimation of sign function. By doing this, the energy in low-frequency domain is preserved, and that of high frequency is estimated by a noise adaptation module.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Basically, the idea of using Fourier series to estimate sign function is straight-forward and easy to understand, but treating it as the frequency domain approximation is interesting. It is a novel gradient estimation method that is orthogonal to a series of binary network algorithms, which means it can fit well into them to further improve their performance.\n\nIt is also interesting to solve the inaccurate estimation problem through a learnable neural network, which is the noise adaptation module. While the Fourier series is a surrogate optimizer of sign function, enlarging the FS terms to reduce the estimation error will make it hard to train the binary network. Thus, the author treats the high-order estimation error as noise and use a learnable network to fit the error which makes it easier to train. During training, the weights of the noise module continue to decrease so that the network is finally consistent to that used for inference.\n\nExperimental results on CIFAR-10 and ImageNet show an obvious improved performance compared to the baseline methods.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The noise adaptation module is trained through an end-to-end manner, and there is no specific target used to guide the training, the author should further explain this in the paper.\n\nAlthough there are obvious improved performances compared to Dorefa net and Bireal net, the author should conduct more experiments on stronger baselines to further demonstrate the usefulness of the proposed method.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5hours."}, "extra_text": {}}, {"review_id": "Qp7GxsLqlDz", "reviewer": "Reviewer_f6JJ", "summary": "This work focuses on training binary neural networks. Instead of using the traditional STE method to estimate the gradient of sign function whose gradient is almost everywhere which cannot be used for back-propagation, the paper proposes to estimate it with Fourier series in the Fourier frequency domain. To avoid using infinite terms of Fourier series, an additional noise adaptation module is further applied during training.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ Compared to the previous gradient estimation methods such as using tanh function, signswish function, etc, the proposed method does not contaminate the low frequency part of the sign function in the frequency domain perspective which occupies most of the energy, thus keep the main direction of the factual gradient.\n\n+ Estimating sign with Fourier series is reasonable, and a theorem is provided to show the rationality.\n\n+ The high frequency part of Fourier series is ignored to avoid the huge computational cost. Instead, a noise adaptation module is used to solve the inaccurate estimation problem. The weight of the noise adaptation module is gradually decreased during training. This is a good compromise between finding the correct direction of the gradient and the consistency between training and inference process. \n\n+ There are a bunch of ablation studies to verify the effectiveness of sign module and noise adaptation module, and the hyper-parameter used during the experiments.\n\n+ The experiments on CIFAR10 and Imagenet show that the proposed method can benefit from the Fouries series estimation.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-\tIn table 2, it seems that the ablation study is not complete. I also want to know what is the final accuracy when only use the noise adaptation module during training. I understand that the performance may not be good when using noise adaptation module along, but the result should still be given for integration.\n\n-\tIn table 4, it seems that the shortcut is important during training. The author should further explain this.\n\n-\tWhat is the number of FS terms used when conducting experiments on ImageNet? Is it the same with that used for CIFAR-10?\n\n-\tThe author does not compare to the state-of-the-art BNN methods such as real2bin [1] and reactnet [2].\n\n[1] Training binary neural networks with real-to-binary convolutions.\n\n[2] Reactnet: Towards precise binary neural network with generalized activation functions.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "EkbzGK-7vB", "reviewer": "Reviewer_Xxt6", "summary": "This paper aims to solve the gradient approximation problem in binary neural networks, i.e. replace the non-differentiable sign function with Fourier Series, and then compensate the approximation error with noise adaptation module. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is technically sound and easy to understand. It is an interesting idea to approximate the gradient in frequency domain and I believe it is enlightening to other BNN gradient approximation articles. Specifically, Fig.1 in the main paper is inspiring and shows that even a nearly perfect approximation function such as tanh can still have an approximation error in the low-frequency domain. \n\nThe noise adaptation module is important to compensate the approximation error, and the shortcut in the module basically solves the problem when using Fourier Series that the gradient oscillates around 0 at a very high frequency.\n\nThe method does not add any extra parameters or FLOPs, and leave the main architecture unchanged. The strong experimental results on CIFAR-10 and ImageNet shows the usefulness of the proposed method. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It seems that the initial value of \\alpha in Eq.(14) and the way of decreasing it to 0 is not specified in the experiment.\n\nThe parameter T in Eq.(7) is also an important parameter. The author should specify the value of T used in the experiment and further explain how to deal with the input value outside T.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper worked on binarized neural networks and proposed to find better surrogates of the sign function in the frequency domain instead of the spatial domain. The idea is motivated, novel, and simple --- I am quite surprised such a simple idea can work fairly well, as a simple idea that works is the greatest to me. The motivation and the key to its success is that approximating the sign function in the frequency domain would drop some high-frequency terms, which cannot affect the quality of gradients much since the gradient directions are mainly determined by the low-frequency terms. The proposed noise adaptation module puts some additional technical difficulty/quality into the paper and is what I am mainly interested in (I have gone through the paper by myself and asked some questions in a separate post).

Although there were some concerns in the beginning, the authors have done a particularly good job in their rebuttal. In the end, all the reviewers hold very positive opinions and vote for acceptance. I think the idea (i.e., frequency domain approximation plus noise adaptation layer) can also be used in other areas besides BNN, since non-smooth functions are common and straight-through estimator is popular in deep learning or the entire machine learning. I am sure that this paper will be impactful and therefore I recommend it as an oral presentation. 
Decision: Accept (Oral)