BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations
Authors: Hyungjun Kim
Kyungsu Kim
Jinseok Kim
Jae-Joon Kim
Venue:  (2020)

Abstract: Binary Neural Networks (BNNs) have been garnering interest thanks to their compute cost reduction and memory savings. However, BNNs suffer from performance degradation mainly due to the gradient mismatch caused by binarizing activations. Previous works tried to address the gradient mismatch problem by reducing the discrepancy between activation functions used at forward pass and its differentiable approximation used at backward pass, which is an indirect measure. In this work, we use the gradient of smoothed loss function to better estimate the gradient mismatch in quantized neural network. Analysis using the gradient mismatch estimator indicates that using higher precision for activation is more effective than modifying the differentiable approximation of activation function. Based on the observation, we propose a new training scheme for binary activation networks called BinaryDuo in which two binary activations are coupled into a ternary activation during training. Experimental results show that BinaryDuo outperforms state-of-the-art BNNs on various benchmarks with the same amount of parameters and computing cost.

Published as a conference paper at ICLR 2020
BINARYDUO: REDUCING GRADIENT MISMATCH IN BI- NARY ACTIVATION NETWORK BY COUPLING BINARY ACTIVATIONS
Hyungjun Kim', Kyungsu Kim", Jinseok Kim, Jac-Joon Kim POSTECH, Department of Creative IT Engineering, Korea {hyung jun. kim, kyungsu. jinseok. kim, jaejoon]epostech. ac ac.kr
ABSTRACT
Binary Neural Networks (BNNs) have been garnrrin interest thanks their com- pute cost reduction and memory savings. However, BNNs suffer from perfor- mance degradation mainly due the gradient mismatch caused by binarizing activations. Previous works tried to address the gradient mismatch problem by reducing the discrepancy between activation functions used at forward pass and its differentiable approximation used backward pass, which is indirect mea- sure. this work, we use the gradient of smoothed loss function to better estimate the gradient mismatch in quantized neural network. Analysis using the gradient mismatch estimator indicates that using higher precision for activation is more effective than modifying the differentiable approximation of activation function. Based on the observation, we propose a new training scheme for binary activa- tion networks called BunaryDuo in which two binary activations are coupled into a ternary activation during training. Experimental results show that BinaryDuo outperforms state-of-the-ar BNNs on various benchmarks with the same amount of parameters and computing cost.
INTRODUCTION
Deep neural networks (DNNs) have been achieving remarkable performance improvement in vari- ous cognitive tasks. However, the performance improvement generally comes from the increase in the size of the DNN model, which also greatly increases the burden memory usage and com- puting cost. While server-level environment with large number GPUs can casily handle such a large model, deploying a large-size DNN model resource-constrained platforms such as mobile application still a challenge.
To address the challenge. several network compression techniques such quantization (Wu al. 2018b Zhou et al. 2016 Lin et al., 2016). pruning (Han et al. [201]] Hu et al. 2016 Wen et al. 2016), and efficient architecture design (Howard al.] 2017] Wu et al. 2018a Tan & Le 2019 have been introduced. For quantization, previous works tried to reduce the precision of the weights and/or activations of DNN model even 1-bit (Courbariaux al. 2016 Rastegari et al.) 2016). In Binary Neural Networks (BNNs), in which both weights and activations have 1-bit precision, high-precisio preb-sreeiioon multiplications and accumulations can be replaced by XNOR and pop-count logics which are much cheaper in terms of hardware cost. In addition, the model parameter size of a BNN 32x less than its full full-precision counterpart.
Although BNNs can exploit various hardware-friende features, their main drawback is the accuracy loss. interesting observation that binarization of activation causes much larger accuracy drop than the binarization of weights (Cai al. 20177; Mishra et al.) 2018). In particular, it can be ob- served that the performance degradation is much larger when the bit precision is reduced from 2- to 1-bit than other multi-bit cases (> (>2-bit) as shown in Table [ To explain the poor performance of binary activation networks, the gradient mismatch concept has been introduced (Lin & Talathi 2016) Cai et al.) (017) and several works proposed modify the backward pass of activation function to reduce the gradient mismatch problem (Cai et al. 2017: Darabi et al. 2018 Liu et al. 2018).
"Hyungjun Kim and Kyungsu Kim equally contributed to this work.

---

Published as a conference paper at ICLR 2020
Table Classification accuracy of quantized neural network on ImageNet dataset.
WRPN AlexNet 2x-wide (Mishra et al.
2018)
ABC-Net ResNet-18
(Lin et al.
2017)
A
32b
8b
4b
2b
1b
A
5b
3b
1b
32b
60.5
58.9
58.6
57.5
52.0
5b
65.0
62.5
54.1
8b
59.0
58.8
57.1
50.8
3b
63.1
61.0
49.1
In this work, we first show that previous approaches minimize the gradient mismatch have not been very successful in reducing accuracy loss because the target measure which the approaches tried to minimize is not effective. To back up the claim, propose new measure for better estimation of gradient mismatch. Then, we propose a new training scheme for binary activation net- work called BinaryDuo based on the observation that ternary (or higher bitwidth) activation suffers much less from gradient mismatch problem than the binary activation. We show that BinaryDuo can achieve state-of-the-ant performance on various benchmarks (VGG-7, AlexNet and ResNet). We also provide our reference implementation code online
RELATED WORKS
Binary Neural Network A BNN was first introduced by Courbariaux et al. (2016). In BNNs, both weights and activations are constrained to +1 or -1 and sign function is used to binarize weights and activations. Since sign function is not differentiable at a single point and has zero gradient everywhere else, the concept of straight-through estimator (STE) has been used for back- propagation (Bengio et al. 2013). Courbariaux et al. (2016) used HardTann function as differen- tiable approximation of the sign function so that its derivative used instead of delta function. Raste- gari et al. (2016) proposed a similar form of BNN, called XNOR-Net XNOR-Net used +a/ instead +1/-1, where a is a channel-wise full- full-precision scaling factor.
Sophisticated STES Although using STE enabled the back proppagattntionn of the quantized activation function, gradient mismatch caused by the approximation occurs across the input range of binary activation function. To mitigate the gradient mismatch problem, Darabi et al. (2018) proposed to use SwishSign function as a differentiable approximation of the binary activation function, while Liu et al. (2018) proposed to use the polynomial function.
Continuous binarization Few recent work proposed to use a continuous activation function that in- creasingly resembles binary activation function during training. thereby eliminating approximation process across activation function. [Sakr et al. (2018) used a piecewise linear function which the slope gradually increases, while Yang et al. (2019) and Gong al. (2019) proposed to use sigmoid and tanh function, respectively.
Additional shortcut Most recent models have shortcut connections across layers. Since the pa- rameter size and computing cost of shortcut connections is negligible, shortcut data often maintains full-precision even in BNN. Bi-Real-Nett (Liu al.) 2018) pointed out that these information-rich shortcut paths help improving the performance of BNNs, and hence proposed to increase the number of shortcut connections in ResNet architecture. Since additional shortcuts increase the element-wise additions only, the overhead is negligible while the effect on performance is noticeable.
SHARP ACCURACY DROP FROM 2-BIT TO 1-BIT ACTIVATION
3.1 EXPERIMENTAL CONFIRMATION OF THE POOR PERFORMANCE OF BINARY ACTIVATION
In this section, first verify that performance drop of neural network is particularly large when the bit-precision of the activation is reduced from 2-bit to 1-bit (Table I] We used the VGG- 7 (Simonyan & Zisserman 2014) network on CIFAR-10 dataset for our experiment. We trained the benchmark network with full-precision, 4-bit, 3-bit, 2-bit and 1-bit activation while keeping all weights in full-precision to solely investigate the effect of the activation quantization. Activation
2

---

Published as a conference paper at ICLR 2020
(a)
Test Accuracy [%)]
Test Accuracy [%]]
94
92
1b
90
Huge gap (>2%)
90
2b
88
88
15
26
3b
4b
FP
x1
x1.25
x1.5
x2
Activation precision
Network width
Figure 1: Training results of VGG-7 on CIFAR-10 dataset. (a) Test accuracy with various activation
precision. (b) Test accuracy of various activation precision and various network width.
quantization was carried out by clipping inputs outside of [0, and then applying uniform quantiza- tion as in Eq. [
((x)
round(elipp(r., 0) (2* 1))
Here, and (zz) are the input and output of the quantization function, and & is number of bits. For consistency, we used Eq. for binary activation function instead of the sign function which is another widely used option (Cai al. 2017 Zhou al. 2016 Mishra et al., 2018). We trained each network for 200 epochs and detailed experimental setups are described in Appendix A The best test accuracy from each precision is plotted in Figure [b. It is clearly shown that reducing activation bitwidth from 2-bit to 1 1-bit results in the substantial drop in the test accuracy.
One might argue that the sharp accuracy drop occurred because the capacity of the binary activation network was not large enough for the given task while that of the network with 2-bit or higher bitwidth was. To verify, we increased the model size by modifying the network width (not depth) for each activation precisions (Figure [p)) We found that the large accuracy gap still existed between 2-bit activation network and 1-bit activation network regardless of the network width. For example, 1 1-bit activation network with 2 times wider layers needs more parameters and more computations, shows much worse accuracy than 2-bit activation network with original width. In addition, poor performance of binary activation was also observed in different networks (e.g. AlexNet and ResNet) and different datasets as Table Therefore, we argue that the sharp accuracy drop for the binary activation stems from the inefficient training method, not the capacity of the model.
3.2 GRADIENT MISMATCH AND SOPHISTICATED STES
When training a neural network with quantized activation function such as thresholding function, STE is often used for back- propaaropa.tioon such case, we call the back-propagated gradient using STE coarse gradient following Yin al. (2019). However, training with the coarse gradient suffers from gradient mismatch problem caused by the discrepancy between the presumed and the actual activation function (Lin & Talathi 2016). The gradient mismatch makes the training process noisy thereby resulting in poor performance.
Unfortunately, it is useful to measure the amount of gradient mismatch directly because the true gradien"] of a quantized activation function is zero almost everywhere. Therefore, previous works used indirect criterion measure gradient mismatch based on the discrepancy between the actual activation function and its differentiable approximation (Darabi et al.) 2018 [iu et al.] 2018). In Liu let al.) (2018), the cumulative difference between actual activation function (f) and its differentiable approximation (g) (Eq 2 was used as a criterion for gradient mismatch.
cumulative difference = / 1f(x) g(x)
(2)
Previous works tried to improve the accuracy by designing the differentiable approximation of binary activation function which has the small cumulative difference from the binary activation function. Figure 2 shows different options for the approximation functions proposed by previous
We call the original gradient 'true gradient' to distinguish with the coarse gradient.
3

---

Published as a conference paper at ICLR 2020
Forward Tnresholding
Backward
62
ReLU1
Steep2
SwishSign
Polynomial
ReLU1 (29)
sixi)
gix)
Test Accuracy [%)
80
Steept
Steep?
Pelynomall
fixt
81x)
s/x)
R1 (1b)
SwishS
88
0.2
Cumulative Difference
Figure 2: Activation functions and their derivatives used forward and backward passes (left). The relationship between the best test accuracy and the cumulative difference when using various STEs (right). Steep4 is extended version of steep2 with slope 4.
works (Darabi et al., 2018 Liu et al. 2018). The shaded area in each activation function represents cumulative difference with thresholding function. We conducted the same experiment for binary activation network in previous section with various sophisticated STEs. For each case, the cumu- lative difference and test accuracy are shown in Figure As proposed in previous works, cumulative differences when using sophisticated STEs much lower than conventional case (ReLU1). How- ever, its effect on network performance is quite limited in our experiment. As shown in the Figure [ large accuracy gap still exists between binary activation network and 2-bit activation network the sophisticated STEs are used. From this observation, we think that current method of measuring gradient mismatch using the cumulative difference cannot reliably estimate the amount of gradient mismatch problem. The same observation was also reported by Betthee et al. (2019) recently. There- fore, in the following section, we introduce a better method for estimating the effect of gradient mismatch problem.
4 ESTIMATING THE GRADIENT MISMATCH
4.1 GRADIENT OF SMOOTHED LOSS FUNCTION
Since the true gradient of quantized activation network is zero almost everywhere, using the value of the true gradient does not provide useful measure of the gradient mismatch problem. To mitigate the issue with zero true gradient values, Yin et al. (201)) recently introduced a method using the gradient for the expected loss given the distribution of the data to analyze the effect of different STEs. However, the distribution the dataset is usually unknown and the gradient values of the layers other than the first layer also zero in multi-layered networks.
In this work, we propose to use the gradient of the smoothed loss function for gradient mismatch analysis. Using the simplest rectangular smoothing Steet. the gradient of the smoothed loss function (Srect f) along each parameter can be formulated as follows:
θ
["
f(x+8.e ds.
I f(x + +
V (arer * f)
ds
Ori
--
25
Or -
28
paite ʃ(,, x2, "n In)
zate [2 In
ds ... Dzn Dn-e
Oxy xI-E
23
25
f(0 +le
-fl 8 ((1)
fx -f(# - 8 e(m))
28
28
Ve,a.
where (i) refers to standard basis vector with value one at position
We named J f as Coordinate Discrete Gradient (CDG) since it calculates the difference between discrete points along each parameter dimension. Note that the formula for CDG is very close to the definition of the true gradient which is lim V f. While the true gradient indicates the instant rate change of loss (e - 0), CDG is based on the average rate change in each coordinate. We provide graphical explanation CDG in Appendix B
4

---

Published as a conference paper at ICLR 2020
Cosine similarity (
(b)
Cosine similarity
0.761
0.779
0.8
0.6
0.4
0.2
fc1
fc2
fc3
total
Baseline
Steep2
Steep4
SwishSign
Polynomial
precision
2-bit
Ternary
Binary
Figure 3: Cosine similarity between true/coarse gradient and CDG in a feed-forward network with 3 hidden layers. Cosine similarity value of each layer when different precision is used for activa- tion. The label total refers to the cosine similarity when gradients for each layer are concatenated. (b) Cosine similarity values when sophisticated STEs are used.
Note that the evolutionary strategy-style gradient estimation is another good candidate which can be used for the same purpose it provides unbiased gradient estimate for smoothed function in similar way (Choromanski et al. 2018). Please refer to Appendix for more information.
4.2 ESTIMATING THE GRADIENT MISMATCH USING COSINE SIMILARITY
Using CDG, we can quantitatively estimate the gradient mismatch. The cosine similarity between CDG and coarse gradient represents how similar the coarse gradient calculated by back-propagation is to CDG, and we can see the effects of 1) different precisions for activation and 2) the sophis- ticated STEs in the gradient mismatch problem. The detailed algorithm for calculating the cosine similarity between coarse gradient and CDG described in algorithm in Appendix Since the computational burden of calculating CDG is heavy even on GPU, we used toy model and dataset for experiment. We used feed-forwart neural network with hidden layers. Each hidden layer had 32 neurons and we used Gaussian distribution as the dataset as in Yin et al. (2019).
Figure shows the cosine similarity between CDG and coarse gradient in different cases. First of all, the cosine similarity between CDG and true gradient in full precision case which has smooth loss surface is almost in every layer. This implies that the CDG points similar direction to the one which the true gradient points to if the loss surface is not discrete. We also observed that the cosine similarity of the binary activation network was significantly lower than that of higher precision networks and the degradation increases as layer is further away from the last layer. This because the gradient mismatch is accumulated as the gradient propagates through layers from the end of network. On the other hand, we found that using sophisticated STEs does not improve the cosine similarity binary activation networks (Figure 3D). which is consistent with the accuracy results in Figure 2 The results indicate that the cosine similarity between coarse gradient and CDG can provide better correlations with the performance of a model than previous approaches do. In be reduced more effectively by using
addition, it can be seen that gradient mismatch problem higher precision (e.g. ternary or 2-bit) activations than using sophisticated STEs.
We also measured the cosine similarity between the estimated gradient using evolutionary strategies and the coarse gradient. The results show a similar trend to that of the results using CDG (Figure [5] in Appendix HH). Most notably, the evolutionary strategy based gradient mismatch assessment also shows that there is large gap in the cosine similarity between the ternary activation case and the binary activation case with any STEs.
5 BINARYDUO: THE PROPOSED METHOD
In this section, we introduce simple yet effective training method called BinaryDuo which couples two binary activations during training to exploit better training characteristics of ternary activation. From the observation in the previous section, we leamed that ternary activation functions suffer much less from gradient mismatch problems than binary activation functions with any kind of STE. Therefore, propose such characteristics of ternary activations while training binary acti-
5

---

Published as a conference paper at ICLR 2020
ACT1
ACT
W
0.7
ACT
0.5
W-2.2
1.1
0.7
ACT2
W.11
Figure 4: Activation functions of baseline binary (left), ternary (middle) and decoupled binary (right) models. Circle, square and rectangle denote full-precision, binary and ternary data, respectively.
vation networks. The key idea is that ternary activation function can be represented by two binary activation functions (or thresholding functions).
The BinaryDuo training scheme consists of two stages. In the first stage, a network with temary ac- tivation function is trained. This pretrained model is then deployed to binary activation network by decoupling a ternary activation into a couple of binary activations, which provides the initialization values of the target binary activation network. In the next stage, the decoupled binary network is fine-tuned to find the better weight parameters. Detailed description of each step is given as follows.
5.1 DECOUPLING TERNARY ACTIVATION TO TWO BINARY ACTIVATIONS
We first elaborate how to decouple a ternary activation to two binary activations. Figure shows three different cases of activation functions. A weighted-sum value (x) goes through activation function depicted in each case. The activation value is then multiplied by a certain weight and contributes to the next weighted-sum. For simplicity, nomaaliormaiiiiiion layer (BN) and pooling layer are omitted in the figure. The binary activation function used in the baseline network is shown in the left in Figure 4 The figure in the middle explains how ternary activation function works. The ternary activation function can produce three different values (0, 0.5 or 1) for an output as depicted in the figure. Since the input to the activation function is 0.7 in the example, the output (rectangle) becomes 0.5. The activation is then multiplied by the corresponding weight of 2.2 and contributes 1.1 to the next weighted-sum. The right figure shows how to decouple the ternary activation into two binary activations. The weighted-sum value of 0.7 now passes through two different activation functions (red solid and blue dashed), where each activation function has different threshold values (0.75 and 0.25) to mimic the ternary activation function. The two binary activation values are then multiplied by each weight that is half of the original value. Therefore, they contribute the same value of 1.1 as the ternary case to the next weighted-sum. Note that decoupling the ternary activation does not break the functionality. In other words, computation result does not change after decoupling.
As mentioned earlier, the decoupled binary activation function layer should have threshold of 0.25 in half and 0.75 in the other half to mimic the ternary activation function. In practice instead of using different threshold values layer, we shift the bias of BN layer which comes right before the activation function layer. The ternary activation function can be expressed as follows:
yt = (BN(E,
(4)
Here, fa and y/ are ternary activation function and its output as shown the middle of Figure BN(2, y, 7,B) B) denotes BN function with input æ, weight y and bias B. Similarly, two binary activation functions after decoupling can be expressed as Eq.
yb1 = (BN( 1,BN(,,,,8)) + 0.25)), yb2 = fo (BN(E, 7,88, 0.25))
(5)
Again, fo and yo are binary activation function and its output as shown in the left of Figure 4 While of BN is copied from the ternary activation case, 3 is modified during decoupling, When ternary activation function is decoupled following Eq. 3t- = (y61 + ytz)/2 becomes valid, enabling exact same computation after decoupling. Please note that computations for BN and the binary activation can be merged to a thresholding function. Therefore, modulating the BN bias with different values as in Eq. 5 does not incur additional overhead inference stage.

---

Published as a conference paper at ICLR 2020
Original
Coupled
Decoupled
3x3
2x2
4x2
North-Noom
activation
Figure 5: Example model architectures of baseline, coupled ternary model and decoupled binary model with specific width numbers. Two weights highlighted in the decoupled model are half the value the weight highlighted in the coupled model.
5.2 LAYER WIDTH RECONFIGURATION
As shown in Figure 4 the number of weights is doubled after decoupling. Therefore, to match the parameter size of the decoupled model and the baseline model, we must reduce the size of the coupled temary model. Figure 5 shows an example how we choose the network size. Each diagram shows BN-activation (ACT)-fully connected (FC)-BN-ACT layers in order from the left. In case the baseline model has 3x3 configuration for the FC layer, the layer has total of nine weights. In this case, we train the coupled ternary model with 2x2 contiguration which has a total of 4 weights. The third diagram shows the corresponding decoupled binary model. As explained in Eq. 5. a weighted-sum value generates two BN outputs with different BN biases. Since decoupling doubles the number of weights in the model, the decoupled model has weights which is still less than the number of weights in the baseline model. To make sure that the number of total weights in the decoupled binary model does not exceed that of the baseline model, we use a simple formula, Ncoupled /useiiee V2], where N denotes the number of neurons (or channels in convolution).
5.3 FINE-TUNING THE DECOUPLED MODEL
After decoupling, two weights derived from the same weight in the coupled model have the same value (1.1 in the example) since they were tied in the ternary activation case. In the fine-tuning stage, we update each weight independently two weights do not need to share the same value anymore. As mentioned earlier, decoupling itself does not change the computation result, so the same accuracy as that of the coupled ternary model is achieved before fine-tuning and experimental results show that the fine-tuning increases the accuracy even further. Detailed experimental results will be discussed in the next section. Since the initial state before fine-tuning can be already close to good local minimum when the model is decoupled, we used much smaller learning rate for fine-tuning.
that each of them can find a better value since the
EXPERIMENTAL RESULTS
6.1 VGG-7 ON CIFAR-10
We evaluate the proposed training method in two steps. First, validate the effectiveness of the BinaryDuo scheme with VGG-7 on CIFAR-10 dataset. We used the same setting with the exper- iment introduced in Figure T We first trained the coupled ternary model, then we decoupled the trained coupled ternary model as explained in section 5.1 The training results with various training schemes and activation precision are summarized in Figure 66 The three contour plots show the hyper parameter search results for each case. Please refer to Appendix A for more details on exper- imental setup. First of all, note that the best test accuracy of the coupled ternary model 89.69% which is already higher than 89.07% accuracy of the bascline binary model. In other words, using ternary activation and cutting the model size in half improved the performance of the network. By fine-tuning the decoupled binary model, we could achieve 90.44% test accuracy, which is 1.37% higher than the baseline result. It can be observed that the steep drop of accuracy from 2b activation case to lb activation case is significantly reduced with scheme.
Even though the decoupled binary model has the same (or less) number of parameters and computing cost than the baseline model, someone may wonder whether the performance improvement might
7

---

Published as a conference paper at ICLR 2020
Coupled
Decoupled
Scratch
10
89.69%
80.44%
88.95%
Learning Rate
10
50.00
Test Test Accuracy
92
89.00
10
87.00
88.00
90
1.37%
Binary
Beadline
86.00
Improvement
10
85.00
88
10"
10
10 10
10
10
10-4
10-
1b
Weight Decay
Activation precision
Figure 6: Training results of VGG-7 on CIFAR-10 dataset in the order of coupled ternary model, decoupled binary model and decoupled binary model trained from scratch (left). circles in each plot represent the top 5 test accuracy points and the red circle is for the best result. The best accuracy result is shown at the top left corner of each contour plot. Test accuracy of various models with different activation precision and training schemes (right).
Table 2: Classification accuracy BNN in various schemes.
AlexNet
ResNet-18
Accuracy
Params
Comp.
Accuracy
Params
Comp.
Network
Top-1
Top-5
(Mbit)
(FLOP)
Top-1
Top-5
(Mbit)
(FLOP)
BNN
41.8
67.1
62.3
82.3M
XNOR-Net
44.2
69.2
191
126M
51.2
73.2
33.3
164M
BNN+
46.1
75.7
191
126M
53.0
72.6
33.3
164M
Yang et al. (0199
47.9
72.5
191
126M
53.6
75.3
33.3
164M
Bi-Real- -Ntt
56.4
79.5
33.3
164M
WRPN 2x
48.3
498
283M
ABC-Net
42.7
67.6
33.3
164M
Group-Net
55.6
78.6
33.3
164M
CBCN
61.4
82.8
33.3
656M
Wang et al.
2019
59.9
84.2
36.7
183M
Ding et al.
2019)
47.8
71.5
191
126M
Bulat et al.
2019
55.6
78.5
33.3
164M
Gu et al.
20191
57.3
80.0
33.7
164M
Bethge al.
(2019)*
57.7
80.0
33.3
164M
BinaryDuo
52.7
76.0
189
119M
60.4
82.3
31.9
164M
Binary D
60.9
82.6
31.9
164M
These schemes use ResNet models with additional shortcut proposed by Liu et al. (2018).
be an outcome from the modified network topology. Therefore, we conducted another experiment rule out the possibility. We trained the decoupled binary model from scratch without using the pretrained ternary model. The result is shown in the third contour plot of Figure The best accuracy was 88, 93% which is lower than the baseline result. Hence, we conclude that the proposed training scheme plays major role for improving the performance of binary activation network.
6.2 BNN ON IMAGENET
We also compare BinaryDuo to other state-of-the-arr BNN training methods. We used the BNN version of AlexNet (Krizhevsky et al. 2012) and ResNet-18 (He et al. 2016) on ImageNet (Deng
et al. 2009) dataset for comparison. Details training conditions in Appendix D
Table shows the validation accuracy of BNN in various schemes. For fair comparison, we also report the model parameter size and computing cost at inference stage of each scheme. Computing cost (FLoating point OPerations) was calculated following the method used in Liu et al. (2018) and Wang et al. (2019). For ResNet-18 18 benchmark, report results with and without additional
8

---

Published as a conference paper at ICLR 2020
shorteut proposed in Liu et al. (2018). Regardless of the existence of additional shorteut path, BinaryDuo outperforms tta results with same level of parameter size and computing cost for both networks. BinaryDuo improved the top-1 validation accuracy by 4.8% for AlexNet and 3.2% for ResNet-18 compared to state-of-the-art results. CBCN (Liu et al., 2019) achieved slightly higher top-1 accuracy than BinaryDuo but requires more computing cost than BinaryDuo. CBCN uses similar amount of parameters and computing cost to that of the multi-bit network which uses 4-bit activations and 1-bit weights.
7 CONCLUSION AND FUTURE WORK
In this work, we used the gradient of smoothed loss function for better gradient mismatch estimation. The cosine similarity between the gradient of the smoothed loss function and the coarse gradient shows higher correlation with the performance of a model than previous approaches. Based on the analysis, we proposed BinaryDuo which is new training scheme for binary activation network. The proposed training scheme improves top-1 accuracy of BNN version of AlexNet and ResNet-18 for 4.8% and 3.2% compared to state-of-the-art results, respectively, by minimizing gradient mismatch problem during training. In the future, we plan to explore several techniques proposed for multi- bit network to improve the performance of BinaryDuo and to apply the proposed training scheme on other tasks such as speech recognition and object detection.
ACKNOWLEDGMENT
This work supported by the MSIT(Ministry of Science and ICT), Korea, under the ICT Con- silience Creative program(IITP-20199 111-200192201111007733 supervised by the ITTP(Int for Informa- tion & communications Technology Planning & Evaluation). This work was also supported by the MOTIE(Ministry of Trade, Industry & Energy (10067764) and KSRC(Korea Semiconductor Re- search Consortium) support program for the development of the future semiconductor device.
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1388 3432, 2013.
Joseph Bethge, Haojin Yang, and Christoph Meinel. Training accurate binary neural networks from scratch. In 20/9 IEEE International Conference on Image Processing (ICIP), pp. 899-903. IEFE, 2019.
Adrian Bulat, Jean Kossaifi, Georgios Tzimiropoulos, and Maja Pantic. Matrix and tensor decom- positions for training binary neural networks. arXiv preprint urXiv:1904.07833 2019.
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by hale-wave gaussian quantization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5918-5926, 2017.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang. Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: 088800000085.. 2018.
Jungwook Choi, Swagath Venkataramani, Vijayalakshmi Srinivasan, Kailash Gopalakrishnan, Zhuo Wang, and Pierce Chuang. Accurate and efficient 2-bit quantized neural networks. Proceedings of the 2nd SysML Conference, 2019.
Krzysztof Choromanski, Mark Rowland, Vikas Sindhwani, Richard E Turner, and Adrian Weller. Structured evolution with compact architectures for scalable policy optimization. arXiv preprint 17X1v:1804.02395. 2018.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- El-Yaniy, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1. arXiv preprint /602../602.000 2016.

---

Published as a conference paper at ICLR 2020
Sajad Darabi, Mouloud Belbahri, Matthieu Courbariaux, and Vahid Partovi Nia. Bnn+: Improved binary network training. arXiv preprint arXiv:1812.11800 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009.
IEEE Conference on, pp. 248-255. IEEE, 2009.
Ruizhou Ding. Ting-Wo Chin, Zeye Liu, and Diana Marculescu. Regularizing activation distribution for training binarized deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11 11408-1 1417, 2019.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. arXiv preprini arXiv: / 0903005033 2019.
Jiaxin Gu, Ce Li, Baochang Zhang, Jungong Han, Xianbin Cao, Jianzhuang Liu, and David Doer- mann. Projection convolutional neural networks for 1-bit cnns via discrete back propagation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 8344 8351, 2019.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. arXiv preprint / 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He al. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026- 1034, 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint / 2017
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi- Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv: 1607.03250, 2016.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4350-4359, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Darryl Lin, Sachin Talathi, and Sreekanth Annauureddy. Fixed point quantization of deep convolu- tional networks. In International Conference on Machine Learning, pp. 2849-2858, 2016.
Darryl D Lin and Sachin S Talathi. Overcoming challenges in fixed point training of deep convolu- tional networks. arXiv preprint arXiv: 1007. 02241, 2016.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances in Neural Information Processing Systems, pp. 345-353, 2017.
Chunlei Liu, Wenrui Ding, Xin Xia, Baochang Zhang, Jiaxin Gu, Jianzhuang Liu, Rongrong Ji, and David Doermann. Circulant binary convolutional networks: Enhancing the performance of 1-bit denns with circulant back propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2691-2699, 2019.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 722-737, 2018.
10

---

Published as a conference paper at ICLR 2020
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference on Learning Representations, 2019.
Asit Mishra, Eriko Nurvitadhi, Jeffrey Cook, and Debbie Marr. Wrpn: wide reduced-precision
networks. International Conference on Learning Representations, 2018.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pp. 525-542. Springer, 2016.
Charbel Sakr, Jungwook Choi, Zhuo Wang, Kailash Gopalakrishnan, and Naresh Shanbhag. True gradient-based training of deep binary activated neural networks via continuous binarization. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 2346 2350. IEEE, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 2014.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: /905. / 1946, 2019.
Ziwei Wang, Jiwen Lu, Chenxin Tao, Jie Zhou, and Qi Tian. Learning channel-wise interactions for binary convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 568-577, 2019.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: zero flop, zero parameter alternative to spatial con- volutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9127-9135, 2018a.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. International Conference on Learning Representations, 2018b.
Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian- sheng Hua. Quantization networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7308-7316, 2019.
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Under- standing straight-through estimator in training activation quantized neural nets. International Conference on Learning Representations, 2019.
Dongging Zhang, Jiaolong Yang, Dongqiangzi Ye. and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 365-382, 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint (FXY::1606.0616000 2016. 2016.

---

Published as a conference paper at ICLR 2020
APPENDIX
A TRAINING CONDITIONS AND RESULTS FOR VGG-7 ON CIFAR-10
For the experiments in section 3.1 we trained various VGG-7 models with different activation pre- cisions and width of layers on CIFAR- 10 dataset. Width layer indicates number of neurons in fully-connected layers number of channels in convolution layers. The baseline VGG-7 model which we used consists of 4 convolution layers with 64, 64, 128 and 128 channels and dense lay- with 512, 512 and 10 output neurons. Max pooling layer was used to reduce the spatial domain size after the second, third and fourth convolution layers. Full precision weights were used for all cases while the activation function layers except the last were modified following the activation precision.
Each model was trained for 200 epochs with learning rate scaling by 0.1 at 120th and 160th epochs. We initialized models following He et al.J (2015) and the mini-batch size was fixed to 256. We used Adam optimizer with decoupled weight decay (Loshchilov & Hutter, 2019) for easier hyper- parameter search. We tried 13 different exponentially increasing weight decay values and 10 dif- ferent exponentially increasing initial learning rates. Test accuracy from at least two runs were averaged for each combination of weight decay and initial learning rate. Figure 7 shows the test accuracy of benchmark networks with various activation precisions.
1-bit
2-bit
3-bit
4-bit
32-bit
10
89 89.07%
91.65%
91.95%
92.19%
92.51%
93.00
92.00
Learning Rate
DO
80.00
88.00
87.00
104
85.00 00
10*
10
10
10
10
10
10
10
10
10
10
10
10
10
10
Weight Decay
Figure 7: Training results of VGG-7 CIFAR-10 dataset. 5 circles represent top 5 test accuracy points and the red circle is for the best result. The best test accuracy in each contour plot is shown on top of the contour plot.
B GRAPHICAL EXPLANATION ON CDG
In this section, we elaborate the difference between true gradient and CDG using bivariate loss function. Figure shows graphical explanation of the concept of CDG. Each plot in the figure shows how to calculate the true gradient and CDG, respectively. The green contour plot in each case represents the given loss surface (C) which is a function of w' and W2. Red and blue lines in the edge shows the cross-section of the loss surface in direction of the dashed lines. True gradient at the given point can be derived by calculating the slope loss surface in each direction. Similarly, CDG can be obtained by calculating how much the loss changes when move to each direction by given step size (E). Then, by calculating the cosine similarity between the two vectors, we evaluate how similar the directions of two vectors are.
C COSINE SIMILARITY BETWEEN CDG AND COARSE GRADIENT
C.1 STEP SIZE OF CDG
When calculating the CDG, the size of in Eq. has to be carefully chosen. First, the has to be small enough to minimize the approximation error from non-linen con-linerrtty of the loss surface. At the same time. it has to be large enough to ignore the noise from rough and uneven stepwise loss surface caused by quantized activation. In addition, the optimal size depends on the number of samples used for loss caleulation. If the number of samples used for the loss calculation small, the larger
12

---

Published as a conference paper at ICLR 2020
(wg)))(W))
Glaws
Be cdg(ww)
(dg(())
cdg
2x @Lloz
gradient
6 GL/ow2
2x (og
similarity
cdg
Ws
W
gradient
Figure 8: Graphical explanation of CDG using bivariate loss function.
size of preferred to minimize the noise from the rough and uneven loss surface (Figure b). On the other hand, if the number of samples used for the loss calculation is large, the smaller size of e preferred better estimate the true gradient (Figure Ic.d). our experiment, we used one million samples and & of 0.001 which are in similar range to the dataset size and learning rate.
Although the absolute values cosine similarity may depend on the choice of с, the overall trend over different activation bit precision is maintained regardless of the value. Figure [0] shows how cosine similarity changes as the step size varies from le-4 to le-2. The gap between ternary activa- tion and binary activation was clearly shown in every layer regardless of the step size [Figure[OOa.oo and g). addition, binary activation networks with any STE showed similar cosine similarity values regardless of the epsilon value (Figure d0 and h).
(a)
(b)
Loss
LosS
W
(c)
(d)
-E 1
loss
loss
W
W
Figure 9: The examples of loss curve and coordinate discrete gradient in single dimension with different scale of and the different number samples used calculate the loss curve. The number of samples used is small (a and b) and large and d). The size of is large (a and c) and small (b and d).
13

---

Published as a conference paper at ICLR 2020
Fill Preckcon
2-hl
Terway
Riser
Baseline
BesshSign
(8)
(b)
Cosine Similarity
Cosine Similarity
0.8
0.8
0.6
0.6
0.4
0.2
1E-03
1E-02
18-04
25 04
9E-04
1E45
25-03
8E403
1E402
Step Size
Step Size (4)
(c)
Cosine Similarity
Cosine Similirtty
0.2
1E-04
Step Size
(
(
Cosine Similartty
Cosine Siintarity
0.8
0.8
0.6
0.6
0.4
0.4
1E:04
1E-03 Step Size (c)
1E-02
Step Size (+)
(9)
Cosine Similarity
0.8
0.4
0.2
1E:04
1E-03 Step Size (e)
1E-02
1E 14-04
3E-04
10.43
50-02
10-02
Step Size (4)
Figure 10: Effect step size (E) on cosine similarity between CDG and coarse gradient with dif- ferent precision (a,c,e and g) and with different sophisticated STEs (b,d,f and h). Cosine similarity trend of each layer (fel, fc2, fc3 and total) is shown in each row from top to bottom. can be observed that there is a large difference in cosine similarity between ternary activation and binary activation regardless the step size. In addition, BNNs have similar cosine similarity values regardless of STE choices and step size.
C.2 ALGORITIM FOR CALCULATING COSINE SIMILARITY
Algorithm explains how we calculated the cosine similarity between coarse gradient and CDG in Section First, generate the dataset by sampling from the standard normal distribution in i.i.d manner. Then we initialize the weight for target model and evaluating model. The target model used for caleulating the target value for computing the loss value of evaluating model. The coarse gradient of each weight matrix is calculated by back- pookppropggation with STE. For calculating the CDG of single element, we evaluate the loss at two parameter points W,+ e(i.k) and W. - e eli.)) which nearby points of current parameter point along the coordinate of the selected element. After evaluating the coarse gradient and CDG, we can estimate the gradient mismatch by calculating cosine similarity between those two.
14

---

Published as a conference paper at ICLR 2020
Algorithm 1 Calculating the cosine similarity between the coordinate discrete gradient and the coarse gradient with respect e,
Generate X E R mx from standard normal distribution NN0, 1)
Initialize Wi...., Wi-1, Wi..... Wi-1 Rm Wi. W e R Define the evaluating model F(x) Wi(f(.... (W1))))))
Define the target model (x) Wj(f(.... Waf(W(a))
L + 1
2n -.= (F(X..)) F* ().
for i = 1,, do
# Calculate the coarse gradient using STE
C...: < W
# Calculate the coordinate discrete gradient
for j = 1, 1,2..... m do
for 1: 1,2...., m do
C+ 1 Et= (F(X,, W, t e((At)) - -X)
C_ 1 Et=1 (F(X W, e eeliit)) F ()),,)))
Di.j.k < (L+ 22
end for
end for
# Calculate the cosine similarity
8, C. Dant
P.....
end for
Stotal
D
return
{81, ....... 81, Statal }
15

---

Published as a conference paper at ICLR 2020
D IMAGENET TRAINING CONDITIONS AND TRAINING CURVE
For comparison with state-of-the-arl BNN results, we used ImageNet (Deng et al. 2009) dataset which has 1.28M images for training set and 50K images for validation coupled ternary model following He et al. 2013) for all cases and the mini-batch size was fixed 256. Weights are also binarized following Rastegari al. (2016) which uses layerwise scaling factor. We also clipped weights so that weights are always between -1 and +1. For AlexNet, we used Batch-normalization layer instead of local response normalization layer. Since binarization of weights and activations serves as strong regularizer, we used 0.1 for dropout ratio in AlexNet. The maximum training epoch was 100 for AlcxNet and we used a learning rate starting at le-3 and decreasing by 1/10 at 41, 71 and 91 epochs. Weight decay 2e-5 was used and Adam optimizer was used. When fine-tuning the decoupled model, we trained the model for 40 epochs. Initial learning rate set 2e-5 and multiplied by 0.1 at 16, 26 and 36 epochs. Weight decay to le-6 when fine-tuning. For ResNet-18, we used shortcut type of 'B' as in other BNN papers. We used full precision shortcut path since its overhead is negligible as described in (Rastegari al. 2016 Liu et al. 2018). Maximum number training epoch was 120 and we did not use weight decay for ResNet-18 18. Initial learning rate was
We initialized the
also lowered
5e-3 and multiplied by 0.1 71, 91 and 111 epochs. For fine-tuning, we also trained the model for 40 epochs. Learning rate was set to 2e-3 and decayed at 21, 31 and 36 epochs. Training curves for both AlexNet and ResNet- 18 are shown in
Figure
AlexNet
ResNet-18
100
100
Pretraining
Fine-tuning
Pretraining
Fine time-tuning
vVo
Accuracy [%6]
Accuracy [%6]
Valid top-1
Valid top-1
Valid top-5
Valid top-5
Train top-1
20
top-1
Train top-5
Train
0
20
40
60
80
100
120
140
20
40
60
80
100
120
140
160
Epoch
Epoch
Figure 11: Training curves for AlexNet and ResNet-18 with BinaryDuo training scheme.
E COUPLED MODEL WITH HIGHER PRECISION
Based on the observation that the binary activation function suffers more from gradient mismatch problem than ternary activation function, we used a model with ternary activation function for cou- pled pretrained model. However, the proposed training scheme can also use the higher precision models as the pretrained model in the same manner. For example, a model with 2-bit activation function, which can be mimicked by three binary activations, can be used the pretrained model. In this case, however, the model size of the 2-bit pretrained model should be even smaller than that of the ternary pretrained model since decoupling the 2-bit activation function increases the number of weights 3x. We evaluated such case on the same benchmark which was also used in Section 6.1
Figure 12 shows the training results of the coupled 2-bit model and the corresponding decoupled binary model. Compared to the coupled ternary model reported in Section 6.1 the coupled 2-bit model achieved similar test accuracy (89.61 (1%) which is also higher than baseline result. However, the performance of the binary model which was decoupled from the 2-bit model could not achieve a comparable performance to the value of the network decoupled from the ternary network which was reported in Section 6.1 We further trained with even lower learning rate and weight decay but we could not achieve better performance. This result was interesting because decoupling 2-bit model increases the number of weights in model 3 times while decoupling ternary model doubles
16

---

Published as a conference paper at ICLR 2020
the number of weights. We suspect that the cause of this phenomenon is that the local minimum value that was found by the pretrained model becomes less useful when the model is more deformed through decoupling.
Coupled
Decoupled
93.00
89.61%
89.95%
Learning Rate
92.00
91.00
10-2
90.00
89.00
10-3
88.00
87.00
00
86.00
10-4
0-00
85.00
10-4
10
10.a
10+
Weight Decay
Figure 12: Training result of coupled 2-bit model and the corresponding decoupled model.
F BINARYDUO WITHOUT MODEL MODIFICATION
In this section, introduce variation of BinaryDuo that keep the baseline model architecture even after the decoupling. While the proposed method in Section uses the half-sized ternary model the pretrained coupled model, this approach uses quarter-sized ternary model as the pretrained model. Let us call this approach as BinaryDuo-Q. BinaryDuo-Q quadruples the number of weights of model when decoupling as shown in Figure Suppose that the original baseline network has 4x4 configuration. BinaryDuo-O Q reduces the width of all layers by half to make coupled ternary model. In the example case, coupled model has 2x2 configuration as shown in the left side of Figure 13 When decoupling, one weight from the coupled model produces four weights in the decoupled model. While the decoupled model of BinaryDuo-O Q has the same architecture as the original baseline model, the coupled ternary model has half number of parameters of the coupled ternary model used for BinaryDuo in Section Figure shows the training results of BinaryDuo- Q. As expected, the coupled ternary model with the quarter size of baseline model showed worse performance than the half-sized coupled model due to the less number of parameters. As a result, even though decoupling the quarter-size model improves the performance by 1.67%, the best test accuracy of the decoupled model using BinaryDuo-Q was 89.72% which is lower than that using BunaryDuo. Nevertheless, note that the result of BinaryDuo-Q still better than baseline result.
Coupled
Original Decoupled
2x2
4x4
Weighted sum
Binary activation
Batch-Nomm
Figure 13: Example model architectures of coupled ternary and decoupled binary model when using BinaryDuo-Q. Coupled ternary model with 2x2 configuration is decoupled binary model with 4x4 configuration.
G BETTER MULTI-BIT TRAINING
Recently, several works proposed to learn the interval and range of quantized activation function for higher accuracy (Jung et al. 2019 Choi al. 2018 2019 Zhang et al. 2018). Unfortunately,
17

---

Published as a conference paper at ICLR 2020
Coupled
Decoupled
10-1
93.00
88.05%
89.72%
Learning Rate
91.00
10
90.00
89.00
10.3
88.00
87.00
10
85.00
10-6
10
2
10
10
10
Weight Decay
Figure 14: Training results of coupled ternary model and decoupled binary model when using BinaryDuo-Q.
these techniques were not effective binary activation networks. However, we use the multi-bit network as a pretrained model for binary activation network, and hence there is chance of accuracy improvement using such interval learning methods. In the future, we plan to explore such techniques to improve the performance of BinaryDuo.
H ESTIMATING THE GRADIENT MISMATCH USING EVOLUTIONARY STRATEGY-STYLE GRADIENT ESTIMATOR
As mentioned in Section evolutionary strategy-style gradient estimator (ESG) also be used for estimating the gradient mismatch similar to CDG because both CDG and ESG provide the informa- tion of the gradient of smoothed loss funetion. In particular, among many evolutionary praatey-style gradient estimators, the antithetic evolutionary strategy gradient estimator has similar functional form with CDG as shown below.
(x) = 2No
Σ (s(x + oed)e s(x -ee)ss)) 1=1
(6)
where (e:) computed for analysis (Figure 15}. Since or in Eo the sharpness of the smoothing kernel does in CDG, we calculated the cosine similarity between ESG and the coarse gradient with various size of a. For fair comparison with CDG in terms of computational cost, we used 1024 samples for gradient estimation (N=1024 in Eq. 6)
NO T). The cosine similarity between ESG and the coarse gradient has been
Note that findings from Figure 3 and Figure 10 were that 1) large gap exists between the cosine similarities in ternary activation case and binary activation case and 2) use of sophisticated STEs does not have a significant effect on cosine similarity. Those two trends remained the same in the experiment using ESG as shown in Figure 15
It is also worthwhile to note that the cosine similarity between the true gradient (full precision case) and ESG is around 0.7 (Figure [5) while the cosine similarity between the true gradient and CDG is close to (Figure TOP. It may imply that CDG can be considered as more reliable metric than ESG given the same amount of computational cost. However, in-depth study needed to assess ad advantage and disadvantage of CDG over ESG more thoroughly.
18

---

Published as a conference paper at ICLR 2020
...... ... Full Pression
20l
Tenary
Binary
# Doseline DSSepp SSeeppt SSrshiiinn aPolynoiial
Casine Similarity
Similarity
0.8
0.6
0.4
0.2
0.2
1E-06
1E-06
1E:04
1E:06
ZE 06
10:00
1E:00
2E:00
9E-08
14.04
Cosine Similarity
Coeliee Sinnlarity
0.8
0.8
0.6
0.6
0.4
16.00
2E:06
$E.06
$E:05
16.04
(
(
Cosine Similarity
Cosine Similarity
0.6
0.4
0.2
0.2
MAMAA ÍAAIA
1E:06
1E:04
1E:06
2E 06
0E:06
1E:04
2E:08
8E
08
1E/4
(0)
(h)
Cosine Similarity
Similarity
0.8
0.8
0.6
0.6
0.4
10000
1E-06
1E-05
1E:04
15:00
20:05
50.00
10:05
56.05
10:04
0
Figure 15: Effeec of on cosine similarity between ESG and coarse gradient with different precision (a,c,e and g) and with different sophisticated STEs (b,d,f and h). Cosine similarity trend of each layer (fcl, fc2, fc3 and total) is shown in each row from top to bottom.
19


Reviews:
[{"review_id": "rJgT-SZ0tB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies activation quantization in deep networks. The authors first compare the coordinate discrete gradient and those obtained by various kinds of straight-through estimators, and found 1-bit activation networks have much poorer gradient estimation than 2-bit ones. Thus they speculate that this explains the poorer performance of 1-bit activation networks than 2-bit ones. To utilize higher precision of activation, the authors then propose to decouple a ternary activation into two binary ones, and achieve competitive results on typical image classification data sets CIFAR-10 and ImageNet.\n\nThe paper is overall well-written and easy to follow. The decoupling method is simple and straightforward. The experiments are also well conducted. One main concern is that since the computation of the decoupled binary model and the coupled ternary model are the same, why does the decoupled binary model can finally to tuned to perform better than the original ternary model? Is there any intuition or theoretical explanation? Yet another concern is that ternary activation basically can be viewed as binary+sparse activations, can it be even more computationally cheaper than the decoupled binary activation?\n\nQuestion:\n1. One line below eq (2), does STE mean the estimated gradient? How can the difference be calculated based on different things (i.e., activations and gradients)?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgd6tglcS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new measure of gradient mismatch for training binary networks, and additionally proposes a method for getting better performance out of binary networks by initializing them to behave like a ternary network.\n\nI found the new measure of gradient deviation fairly underdeveloped, and I suspect the method of converting ternary activations into binary activations works for a different reason than that proposed by the authors.\n\nThere were English language issues that somewhat reduced clarity, though the intended meaning was always understandable.\n\nDetailed comments:\n\n\"Binary Neural Network (BNN) has been gaining interest thanks to its computing cost reduction and memory saving.\" --> \"Binary Neural Networks (BNNs) have been garnering interest thanks to their compute cost reduction and memory savings.\" (will stop making English language corrections from here on)\n\n\"Therefore, we argue that the sharp accuracy drop for the binary activation stems from the inefficient training method, not the capacity of the model.\"\nThis could also be due to poor initialization in the binary case. e.g., it might make sense to initialize the binary network with bias=-0.5, so that the nonlinearity has a kink at pre-activation=0, rather than pre-activation=0.5.\n\n\"Unfortunately, it is not possible to measure the amount of gradient mismatch directly because the\ntrue gradient of a quantized activation function is zero almost everywhere. \" It *is* possible to measure the mismatch to the true gradient exactly. One could even train using the true gradient. It's just that the true gradient is useless.\n\nFig 1b -- this is a nice baseline.\n\n\"the steepest descent direction, which is the direction toward the point with the smallest loss at given distance\"\nThis is not the usual definition of steepest descent direction. If you're going to redefine this, should do so mathematically and precisely (for instance, you are going to run into trouble with the word \"distance\", since your coordinate discrete gradient more closely resembles an L\\infty-ball perturbation, rather than an L2-ball perturbation.\n\neq. 3:\nNote that this equation is equivalent to taking the true gradient of a function which has been boxcar-smoothed along each parameter. This may more closely resemble existing measures of deviation than you like.\n\nYou should also consider the relationship to an evolutionary strategies style gradient estimate, which similarly provides an unbiased gradient estimate for a smoothed function, and which allows that estimate to be computed with fewer samples (at the cost of higher error).\n\nSec. 4.2 / Figure 3:\nThe results in this section will be *highly* sensitive to the choice of epsilon. You should discuss this, specify the epsilon used, and experimentally explore the dependence on epsilon.\n\n\"The results indicate that the cosine similarity between coarse gradient and CDG can explain the relationship between gradient mismatch and performance of model better than previous approaches. \"\nDon't know that I followed this. Gradient mismatch is never formally defined, so it's hard to know what this says about its relationship. Additionally, CDG sounds more like something which is correlated with, rather than an explanation for, performance.\n\n\" cosine similarity between coarse gradient and CDG can explain the relationship between gradient mismatch and performance of model better \" --> \" cosine similarity between coarse gradient and CDG can explain the relationship between gradient mismatch and performance of model better \"\n\n\"we shift the bias of BN layer which comes right before the activation function layer. \"\nDid you try using these bias values without pre-training as a ternary network? I suspect it would work just as well!\n\n\"Please note that BN layers followed by binary activation layer can be merged to the threshold of the binary activation layer, incurring no overhead at inference stage.\"\nDid not understand this.\n\n\"it is expected that the fine-tuning increases the accuracy even further\"\nDoes it improve the accuracy further? Should state this as result, not prediction, and should have an ablation experiment showing this.\n\n\"Table 2 shows the validation accuracy of BNN in various schemes.\"\nWhy not test accuracy?\n\nFigure 6:\nWhat are the filled circles?\nWhat was the sampling grid for the HP search? The images have high spatial frequency structure that I suspect is an artifact of the interpolation function, rather than in the data.\n\n----\n\nUpdate post-rebuttal:\n\nThe authors have addressed the majority of my concerns, through both text changes and significant additional experiments. I am therefore increasing my score. Thank you for your hard work!", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1ldnneeiH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To be honest, I only read several papers on the neural network quantization. I am not familiar with this research topic, so I  provide my judgement based on my own limited knowledge rather than thorough comparison with other related works.\n\n1. The motivation is clear. The 1-bit activation networks usually deteriorates the performance greatly.\n2. The gradient mismatch for discrete variable did bring difficult for optimization. Do you mean 1-bit activation has larger gradient mismatch than other bits, at least in the defined cosine similarity by this paper?\n3. As to Eq(3), Appendix C.1 describes the way to choose step size. I understand the logic, but for the detailed method, is it cross-validation with grid search or some other tricks?\n4. Is there any relation between the decoupling method in Section 5 and the proposed estimated gradient mismatch in Section 4.2?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers suggest acceptance. Reviewers were impressed by the thoroughness of the author response. Please take reviewer comments into account in the camera ready. Congratulations!
Decision: Accept (Poster)