Weights Having Stable Signs Are Important: Finding Primary Subnetworks and Kernels to Compress Binary Weight Networks
Authors: Zhaole Sun
Anbang Yao
Venue:  (2021)

Abstract: Binary Weight Networks (BWNs) have significantly lower computational and memory costs compared to their full-precision counterparts. To address the non-differentiable issue of BWNs, existing methods usually use the Straight-Through-Estimator (STE). In the optimization, they learn optimal binary weight outputs represented as a combination of scaling factors and weight signs to approximate 32-bit floating-point weight values, usually with a layer-wise quantization scheme. In this paper, we begin with an empirical study of training BWNs with STE under the settings of using common techniques and tricks. We show that in the context of using batch normalization after convolutional layers, adapting scaling factors with either hand-crafted or learnable methods brings marginal or no accuracy gain to final model, while the change of weight signs is crucial in the training of BWNs. Furthermore, we observe two astonishing training phenomena. Firstly, the training of BWNs demonstrates the process of seeking primary binary sub-networks whose weight signs are determined and fixed at the early training stage, which is akin to recent findings on the lottery ticket hypothesis for efficient learning of sparse neural networks. Secondly, we find binary kernels in the convolutional layers of final models tend to be centered on a limited number of the most frequent binary kernels, showing binary weight networks may has the potential to be further compressed, which breaks the common wisdom that representing each weight with a single bit puts the quantization to the extreme compression. To testify this hypothesis, we additionally propose a binary kernel quantization method, and we call resulting models Quantized Binary-Kernel Networks (QBNs). We hope these new experimental observations would shed new design insights to improve the training and broaden the usages of BWNs.

Under review as a conference paper at ICLR 2021
WEIGHTS HAVING STABLE SIGNS ARE IMPORTANT: FINDING PRIMARY SUBNETWORKS AND KERNELS TO COMPRESS BINARY WEIGHT NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Binary Weight Networks (BWNs) have significantly lower computational and memory costs compared to their full-precision counterparts. To address the non- differentiable issue of B Bs,,, existing methods usually use the Straight-Through- Estimator (STE). In the optimization, they learn optimal binary weight outputs represented as combination of scaling factors and weight signs approximate 32-bit floating-point weight values, usually with layer-wise quantization scheme. In this paper, begin with empirical study of training BWNs with STE under the settings of using common techniques and tricks. We show that in the context of using batch normalization after convolutional layers, adapting scaling factors with either hand-crafted or learnable methods brings marginal or no accuracy gain final model, while the change of weight signs is crucial in the training of BWNs. Furthermore, we observe two astonishing training phenomena. Firstly, the train- ing of BWNs demonstrates the process of seeking primary binary sub-networks whose weight signs are determined and fixed at the early training stage, which is akin to recent findings on the lottery ticket hypothesis for efficient learning of sparse neural networks. Secondly, find binary kernels in the convolutional lay- ers of final models tend to be centered on a limited number of the most frequent binary kernels, showing binary weight networks may has the potential to be fur- ther compressed, which breaks the common wisdom that representing each weight with single bit puts the quantization the extreme compression. To testify this hypothesis, we additionally propose a binary kernel quantization method, and we call resulting models Quantized Binary-Kerne Networks (QBNs). We hope these new experimental observations would shed new design insights to improve the training and broaden the usages of BWNs.
INTRODUCTION
Convolutional Neural Networks (CNNs) have achieved great success in many computer vision tasks such image classification (Krizhevsky al. 2012), object detection (Girshick et al., 2014) and semantic segmentation (Long et al. 2015). However, modern CNNs usually have large number of parameters, posing heavy costs on memory and computation. To ease their deployment in resource- constrained environments, different types of neural network compression and acceleration tech- niques have been proposed in recent years, such as network pruning (Han et al.
2015 Li et al. 2017]. network quantization (Hubara et al., 2016; Rastegari et al. 2016 Zhou et al. 2016). knowl- edge distillation (Ba & Caruana Hinton et al. (2015), efficient CNN architecture engineering and searching (Howard et al. 2017 et al. 2018b; Zoph & Le 2017).
Comparatively, network quantization is more commercially attractive as it can not only benefit spe- cialized hardware accelerator designs (Sze et al. (017), but also can be readily combined with other techniques to get further improved compression and acceleration performance (Mishra & Marr 2018 Han et al.) 2016: Zhou et al. 2017). Quantization methods aim to approximate full- neural networks with low ppw-iiiciooon (low-bit) ones. In particular, the extremely quantized models called Binarized Neural Networks (BNNs) (Courbariaux et al.) 2015 2016 Rastegari et al. 2016) which force the weights or even weights and activations to have 1-bit values (+1 and 1),, bringing 32x reduction in model size and making costly 32-bit floating-point
precision (32-bit floating-poii

---

Under review as a conference paper at ICLR 2021
multiplications can be replaced by much cheaper binary wis operations. Because of this, how to train accurate BNNs either in a post-training manner or in a training from scratch manner has at- tracted increasing attention. However, training BNNs poses non-differentiable issue converting full-precision weights into binary values leads to zero gradients. To combat this issue, most existing methods use the Straight- Straight-Through- Through-Estimatee (STE). Although there are few attempts (Achterhold Chen et al. 2019] Bai et al. 2019 Hou et al. 2017) to learn BNNs without STE by using proximal gradient methods or meta-learning methods, they suffer from worse accuracy and heavier parameter tuning compared to STE based methods. In STE based methods, full- full-precision weights are retained during training. and the gradients w.E.I. them and their binarized ones are assumed to be the same. In the forward pass of the training, the full-precision weights of the currently learnt model are quantized to binary values for predication loss calculation. In the backward pass, the gradients w.r.l full-precision weights instead of binary ones are used for model update. To compensating for drastic information loss and training more accurate BNNs, most state of the art STE based meth- ods follow the formulation of (Rastegari et al. 2016) in which the binary weights are represented as a combination of scaling factors and weight signs to approximate 32-bit floating-point weight values layer-by-layen, yet also present lot of modifications. These modifications include but not limited to expanding binary weights to have multiple binary bases (Lin et al. 2017 Guo et al.) 2017), replacing hand-crafted scaling factors with learnable ones (Zhang et al.) 2018a making an ensemble of multiple binary models (Zhu et al. (019), searching tigh--erfformncee binary network architectures (Kim et al., 2020p, and designing improved regularization objectives, optimizers and
et al. |
activation functions (Cai et al. 2017 Liu et al. 2018 Helwegen et al. 2019 Martinez et al. 2020).
There are also a few works, trying to make a better understanding of the training of BNNs with STE. In (Alizadeh et al., 2019). the authors evaluate some of the widely used tricks, showing that adapting learning rate with second-momen while other tricks such as weights and gradients clipping are less important. Bethhe et al. (2019) shows the commonly used techniques such as hand-crafted scaling factors and custom gradients are also not crucial. Sajad et al. (2019) demonstrates learnable scaling factors combined into modified sign function can enhance the accuracy of BNNs. Anderson & Berg (2018) makes an interpretation of why binary models can approximate their full-precision references in terms of high-dimensional geometry. Galloway et al. (2018) validates that BNNs have surprisingly improved robustness against some adversarial attacks compared to their full-precision counterparts. In this paper, we revisit the training of BNNs, particularly Binary Weight Networks (BWNs) with STE, but in a new perspective, exploring structural weight behaviors in training BWNs.
optimizer crucial to train BNNs with STE based methods
Our main contributions are summarized as follows:
We use two popular methods (Rastegari et al. 2016) and (Zhang et al. 2018a) for an empirical study. showing both hand-crafted and learnable scaling factors are not that im- portant, while the change of weight signs plays the key role in the training of BWNs, under the settings of using common techniques and tricks.
More importantly, we observe two astonishing training phenomena: (1) the training of BWNs demonstrates the process of seeking primary binary sub-networks whose weight signs are determined and fixed at the early training stage, which is akin to recent findings of the lottery ticket hypothesis (Frankle & Carbin 2019) for training sparse neural net- works; (2) binary kernels in the convolutional layers (Conv layers) of final BWNs tend to be centered a limited number of binary kernels, showing binary weight networks may has the potential to be further compressed. This breaks the common understanding that representing each weight with a single bit puts the quantization to the extreme compres- sion.
We propose a binary kernel quantization method to compress BWNs, bringing a new type of BWNs called Quantized Binary-Kerne Networks (QBNs).
2 AN EMPIRICAL STUDY ON UNDERSTANDING BWNs' TRAINING
In this section we will briefly describe BWNs we use in experiments, implementation details, scaling factors in BWNs, full- full-precision weight norm, weight sign, and sub-networks in BWNs.
2

---

Under review as a conference paper at ICLR 2021
2.1 DIFFERENT BINARY WEIGHT NETWORKS
BWNs generally represents those networks with binary weights, and there are several different BWNs existing. Overall they use aB to replace full-precision weight W, where
sign(W)
and a is proposed to minimize ||a W|l in an either learnable or calculated way. In follow- ing experiments, we use the one implemented in XNor-Net (Rastegari et al. 20T6) and denote it as XNor-BWN, and the one implemented in LQ-Net (Zhang et al., 2018a) and denote it as LQ-BWN which is I-bit weight 32-bit activation version of LQ-Net. Other popular BWN meth- ods like DoReFa-Neet and BinaryConnect are similar to these two methods. Both XNor-BWN and LQ-BWN use STE framework, and XNor-BWN uses hand-crafted calculated scaling factors, and LQ-BWN uses learnable scaling factors.
2.2 IMPLEMENTATION DETAILS AND NOTATION
Quantization: We directly use open source codes of BWN released by authors, including XNor- BWN and LQ-BWN
Dataset and Network Structure: CIFAR-10 (Krizhevsky & Hinton 2009) and ImageNet (Rus- sakovsky et al.) are used in our experiments. We use VGG-7 (Simonyan & Zisserman, 2015) and ResNet-20 (He et al. (016)) on CIFAR-10, and ResNet18 on ImageNet. The streuture is the same as original ones.
:lyper-parameters: We the same training parameters on each network. The network is trained for 200 epochs. The learning rate is set initially to 0.02 and divided by 10 at 80 and 160 epochs. For random crop, we first use zero pad to resize the image into 40 x 40, and random crop into 32 32. For BWN trained on ImageNet, each is trained for 100 epochs. The initial learning rate is 0.1 and decays 0. at 30, 60, 90 epoch. The image is rescaled into 256 x 256 and then randomly cropped into 224 x 224. No additional data augmentations are used. For all networks, weight decay is applied to all Conv layers set to 4 x 10-5
Notations: In figures and tables, we will use the following abbreviations for clearer expression. BN: BarchNormalization, LR: Learning Rate. WD: Weight Decay. SF: Scaling Factors. FP: Full- precision. VGG-7 XNor-BWN: a VGG-7 network using the binarization algorithm of XNor-BWN. ResNet-20 Baseline: a full-precision ResNet-20 only using data augmentation and weight decay without any additional tricks. Other network structures with certain methods are similar this. Large weights, large magnitude weights, and weights with larger norm have the same meaning to indicate those weights having relatively large absolute values.
2.3 SCALING FACTORS
According to previous methods, scaling factors essential element in obtaining BWNs. How- ever, according to our experiments and analysis, we find scaling factors are not so important in training BWNs, and they can somehow be ignored without the drop in performance. Here we list four reasons to explain why scaling factors are unimportant.
simple proof: BN is a common practice be used in training BWNs. contains two operations, Normalization and Affine as shown in Equation and 3 are the affine parameters used in BN. e 5e is used in Porch to avoid the error of dividing zero. We use a simple proof to demonstrate that BN can absorb scaling factors as shown in Equatior[] This is correct during training when one scaling factor is applied to each output channel under the Conv-BN-ReLU structure.
= X- x
= Normalizo(x)
y = =Affnn(xx)) = 7xx + +
(1)
va² +c
ax ax
X v
Ya
+ 3 R
+ 3 y=y
(2)
Vo'az +
WWe use the codes DoReFa-Net to realize XNor-BWN which is the same as the original implementation. hwwppaacccessoooccee////atttttee -LQ-BWN is the 1-bit weight 32-bit activation version of I.Q-Nets. http:///tthhh.commmicccss///////
3

---

Under review as a conference paper at ICLR 2021
Proke
XNo VGG
L.Q-BWN VGG
XNor-BWN R-20
LQ-BWN R-20
The visualization of full precision weights distribution in BWNs. The X-axis indicates the full precision weight value while Y-axis indicates the frequency that the value appears in certain layer of a binary weights network. In the figure captions, VGG is VGG-7, and R-20 is ResNet-20. For VGG-7, we draw 2nd, 4th, 6th Conv layers' weight distributions (Convl, Conv3, Conv5). For ResNet-20, we display the first and the last Conv layers' weight distributions.
Figure I:
Experimental Results: In the second aspect, we directly go to experimental results. As shown Ta- in Appendix B we train different networks with and without scaling factors. The test accuracy on C0 and validation accuracy on ImageNet do not show a large difference between the two methods. Later we fix scaling factors of all layers to certain value and magnify their learning rate according the fixed scaling factors' magnitude. The performance does not change when fixing scaling factors. Thus, we conclude with proper learning rate, scaling factors are not essential to train BWNs.
Compare learnable SF and y in BN: LQ-BWN uses thannel-wise scaling factors. From the ex- find that these channel-wise scaling factors having high correlation
periments in Appendix[[] with y in the BN after corresponding binary Conv. This finding indicates that BN's can replace channel-wise SF to some extent.
Quantization Error Curve: Another purpose using scaling factors is to reduce the quantization error between full-precision weights and binary weights according to BNN survey (Qin et al., 2020). By using experiments in Appendix D we prove that the quantization error is not actually reduced by scaling factors, but weight decay helps on this reduction.
2.4 WEIGHT NORM, WEIGHT SIGN, AND SUB-NETWORKS IN BWNS
We already analyse one essential element, scaling factors, in the previous section, and another es- sential element of BWNs weights' signs. In deterministic binarization methods, full-precision weights' signs decide their binary weights' signs using a sign() function. In this section, we will discuss the relationship between weight norm, weight sign and how find primary binary sub- networks in BWNs.
Weight Histogram: We visualize the full full-precision weight distribution in different layers of dif- ferent networks as shown in Figure Different from a bi-modal distribution, it shows a centered distribution around O. This again proves that the actual distance or so-called quantization error is very large. And there are many weights close to zero behaving very unstable, which will change their signs with little perturbations. More experiments and visualizations are in Appendix
We flip the weights' signs during the inference section according to weights' full-precision norm shown in Figure 12 of Appendix We flip those weights with the largest norm and the smallest norm in two experiments. It shows that even the weights have the same norm after binarization, and the changed norm is the same for the same flipping percentage, there is still very large gap between the two results. Flipping those weights with large full-precision magnitude will cause significant performance drop compared to those weights close to zero. This reveals that weights are different where some with small norm can tolerate sign flipping, and those with large norm cannot suffer from changing signs, even though both two kinds of weights have the same norm after binarization.
Flipping Weights' Signs:
Tracing Large Weights From the last experiment, we conduct that weights with large norm are vulnerable and important during inference, however, the function of them during training remains unclear. Then we conduct two experiments to tracing these large weights during training. We also use "these large weights" to indicate these weights having the larger magnitude/larger norm in the
4

---

Under review as a conference paper at ICLR 2021
hith
Floaed Parcertag
Farcriaga
Floped
XNOr-BWN VGG
LQQBWWN VGG
XNor-BWN R-20
LQ-BWN R-20
Figure 2: Inference accuracy on training sets after flipping a certain percentage of weights' signs. We design two flipping methods, flipping those weights with larger norm (from the largest norm to the smallest norm) and flipping those weights with the smaller norm. The X-axis indicates how many percentage of weights is flipped, while the Y-axis indicates the inference accuracy. The top- left point in each figure is the un-flipped case which is the same as the result reported in Table This flipping operation is done to each binary Conv layer and each layer has the same flipping percentage.
KNor-BWN VGG
LQ-BWN VGG
Figure 3: For each group, we display three figures whose X-axis is the training epoch and Y-axis is: Left: the overlapping percentage of those largest weights' signs between the final weights and the weights during training, Middle: the overlapping percentage of those largest weights' norm between the final weights and the weights during training, Right: the hamming distance divided by the number of parameters between binarized weights during training and the final trained binarized weights, which ranges between (completely different from another) (the same).
network that has already finished training. One is trace these large weights' signs, to find when these weights' signs become the same as the ones finishing training. Another is to trace these large weights' indices, to find when they become the largest weights among all weights.
The results of VGG-7 are shown in Figure The results of ResNet-20 in Figure 9 and ResNet- 18 in 10 are placed in Appendix FF] We find those large weights mostly have been decided in the early training stage. The larger magnitude these weights finally have, the earlier they decide and fix their sign. And this rule also applies to their magnitude, that the final trained weights with larger magnitude become having a larger magnitude in the very early stage. Both curves have a similar trend to the accuracy curve's trend.
2.5 PRIMARY BINARY SUB-NESWORKS IN BWNs
We find that there are weights with the large norm, fixing their signs in the early training stage. These weights are stable and vulnerable when inversing their signs. We name these weights as Primary Binary Sub-Networks. This idea is akin to the lottery ticket hypothesis (Frankle & Carbin 2019), but the difference is our primary binary sub-networks' weights usually have fixed signs, and the rest of BWNs are not zero like the pruned networks. The primary binary sub-networks have the same norm for each weight after binarization, but different importance. The lottery ticket is based on full-pre full-precision network pruning, and it pays more attention to getting sparse networks using the retraining method, while ours is to claim the meta idea that weights with larger norm are stable and sensitive signs' changes. We will show how utilize this idea in the rest paper.
2.6 BINARY- BINARY-KERNEL DISTRIBUTION
Besides the centered distribution of full precision weights in each layer, we find that there exists another distribution of binary-kemels in each layer. For a binary-kerne with 3 x kernel size, there are 2 possible kernels in total. For easier illustrations, as shown in Figure 4 x kernels are more widely used in common CNN like VGG. ResNet, DenseNet (Huang et al. 2017). and MobileNet (Howard et al. 2017 (except the first Conv layer
use 0 to 51 to index these kernels
5

---

Under review as a conference paper at ICLR 2021
-1
-1 111,111,,1--
1
-1
1
-1 (0 1 01 0) (238)
Illustration on Indexing Binary- -ernels
XNor-BWN VGG
XNor-BWN R-20
Figure 4: The visualization of binary weight kernels in one Conv layer after assigning indices. The X-axis indicates the index of a 3x3 binary weight kemel while Y-axis indicates the frequency that the binary kernel appears in one certain Conv layer. Left Figure an example to illustrate how we index 3 x 3 kernel into the range of O to 511. Two figures on the right are from the last Conv layer of two networks. Right Figures are the visualization of binary weight kernels XNor-BWN VGG-7's last Conv layer and KNor-BWN ResNet-20's last Conv layer after assigning indices. The X-axis indicates the index of a x 3 binary weight kernel while Y-axis indicates the frequency that the certain appears in Conv layer.
C1
45
My 45 022
W/
EJR 733
Figure 5: This is pipeline to illustrate our compressing method on BWNs using 2-bit kernels. We first set those weights with larger norm into =1 and keep those weights with the smaller norm, then calculate the L2 distance with 2-bit selected binary-kernels. After sorting the distances, we assign the one with the smallest distance the original kernels. The right is two figures about the distribution of binary-kernee before and after quantization.
which is usually binarized). From Figure we can find that certain binary-kereels in favor across different layers and networks.
QUANTIZED BINARY-KERNEL NETWORKS
In this section, we will introduce Quantized Binary-Kerne Networks (QBNs). In previous sections, we have several conclusions: 1. Scaling factors are not essential to BWNs, which guide not to concentrate on designing scaling factors since good learning rates help in the most cases; 2. Weights with larger magnitude contribute to the primary binary sub-networks in BWNs, and these large weights are stable but sensitive on sign changing, determined and fixed in the early training stage; centered on limited number of the most frequent binary kernels. All these conclusions lead us to propose new compression algorithm, which will further compress BWNs into more structural and compact network, and we name this algorithm Quantized Binary- Kernel Networks (QBNs). QBN basically is to the ultimate extent to maintain the primary binary sub-networks, changing those smaller weights' signs and quantize the less frequent kernels those high frequent kernels to save space.
3. Certain binary-kernels
3.1 ALGORITHM
Before training a QBN, we first train an ordinary VGG-7 XNoN-BWN on Cifar-10 and extract its last Conv layer's binary kernel distribution. This has been already done as shown in Figure 5 Then we sort these binary kernels according to their appearance frequency and select top 2 2 .... 2 frequent binary kernels. These kernels are called selected binary-kemels Ko.1,..,29-1- In the rest of the paper, we use the selected binary kernels to indicate the kernels Ko,1....2 2 in our algorithm. In our following experiments these selected binary kernels are extracted from one single VGG-7 BWN's i last Conv layer. After pre-processing these and obtaining Ko.1....29-. we start to train QBN using Algorithm[ [ which is written with python-style pseudocodes. We the function where(A. A, B, C) from NumPy indicating that if the value satisfies then equals to otherwise equals to C.

---

Under review as a conference paper at ICLR 2021
Algorithm 1: QBN
Parameters: Quantized kernel bit number p. selected kernels Ko.1.... .2.
threshold A, weight input channel number I, output channel number O scaling factors
a 0.05
Input: W
for w W do
if (aabsw)) > AE then
w = sign(w)
end
end
for in range (T) do
for in range (O) do
for m in range (2P) do (m)) = [| Kmll 12
end
m argmin, (m)
Wij aKm.
end end
Return: W
Table I: Experiments VGG-7, ResNet-20, and ResNet-56 on Cifar10, and ResNet-18 on Ima- geNet. We put the results of baseline of full-precision networks and BWNs in Table FP indicates the first full-precision Conv layer which is not quantized according to the common practice. VGG-7 has 6 Conv layers, and we use the quantized bit numbers for each layer to indicate how many lected quantized kernels are used. Res.ett-20 and ResNet--6 have three groups, each group shares the same number of channels which 16, 32, 64 in order. We assign the same quantized bit number for each group. ResNet- 18 has four groups which have channels 64, 128, 256, 512. CR indicates Compressed Ratio. Acc is the top-1 test accuracy on Cifar-10 and top-1 validation accuracy on Im- ageNet. The accuracy is reported as the mean of the best accuracy during training of 5 runs with different random seeds. More results are displayed in Appendix
Network
Quant Bit
CR
Acc
Network
Quant Bit
CR
Acc
VGG-7
FP-6-5-43--2
x3.2
89.2%
VGG-7
FP-3-3-3-3-3
x3.0
87.6%
R-20
FP-9-9-1
x3.1
78,4%
R-20
FP-9-9-2
x2.5
78. 78.5%
R-56
FP-5-5-5
x1.8
86.6%
R-56
FP-9-6-2
x2.9
84, 1 1%
R-18
FP-4-4-4-4
x2.3
53.4%
R-18
FP-9-7-4-1
x4.6
57.3%
We scaling factors fixed 0.05 when using default learning mentioned in experimental settings of Section 2.2 We use L2 norm to calculate the distance between the full-precision kernel Wi to the selected kernels Km, where the full-precision kernel will be replaced by the selected kernel whose distance to the full-precision kernel is the shortest one during forward.
3.2 QBN EXPERIMENTS
We display our QBN experiments on in Table where we use the same experiment settings mention in Section 2.2 Besides different networks and datasets are tested, we also use different quantized bit on these networks to find how QBN can perform. When we use the quantized bit p 9, we can use less than 9-bit number to represent the binary-kernel, this provides the compression ability of QBN. We use compressed ratio (CR) which is number larger than to show the ratio between the original BWNs and the compressed model's parameters only including binarized layers. In this paper, we do not use bit quantized binary kernels, which have a high computational cost and small compressed ratio.

---

Under review as a conference paper at ICLR 2021
4 DISCUSSION ON QBN
In this section, we will discuss the experimental results of QBN and its potential usage, including model compression, kernel quantization strategies, the existence and transferability of the selected kernels, and other selection of binary-kemels.
4.1 MODEL COMPRESSION
With the discovery that BWNs contain primary binary sub-networks, we can reduce the number of parameters to represent a binary-kernel by changing the small magnitude weights' signs with bearable to the performance of BWNs. For VGG-7 on Cifar-10 10 and ResNet-18 ImageNet, we can compress their parameters an extremely small number by replacing the whole 512 types of 3x3 binary-kernel with fewer types of binary kernels from those 2 selected binary-kerells, and the compressed ratio can be higher than 5x. For ResNet-20 and ResNet-56 which are thinner and have small number of channels and parameters, they have low endurance compression, the compressed ratio can achieve to 1.5x with bearable accuracy drop (less than 3% on Cifar-10). For more aggressive compression with very low bit quantization binary-kerels,, networks with fewer parameters like ResNet-20's training stability will drop due to their limited number of parameters. The experimental results are shown in Table 3 Appendix H
4.2 CONNECTION BETWEEN PRIMARY BINARY SUB- NETWNKS
We use a hyper-parameter threshold A in Algorithm T to bridge QBN and Primary Binary Sub- Networks. When A = 0, it means we first binarize all weights, then quantize these binary- to those selected kernels. When A is large enough, it means we direetly quantize the full-precision kernels. When A AE is at a proper range of weight norm, those large weights will be first binarized to +1. Considering the weight norm usually small value (from weight visualization in Figure [ and Figure compared to 1, these large weights receive a larger penalty by changing their signs during calculating the L2 distance between full-precision kernels and the selected dinary-kernela Thus, A is a jyper-parameter deciding how many portions will be considered as large weights, in the same term, Primary Binary Sub-Networks. According to our experiments of using different A in 16 Appendix [ we find that A > O is almost better than A O. This sign() operation for all weights will eliminate the information full-precision norm. Overall, these experimental results suggest our settings of A primary binary sub-networks first helping on quantizing binary-kernels, compared to binarizing weights first.
4.3 QUANTIZATION BIT STRATEGY
When using low quantization bit for binary-kernels, the performance drop will not negligible, thus how to assign quantization bit to different layer is important. For VGG-7 and ResNet, they contain much more parameters in higher layers (layers near to the output), which have more channels, but their computational cost is similar in each layer. From the view of model compression, we find that higher layers have a higher endurance for the low-bit quantized kernels compared to lower layers (layers near the input). Thus, use low-bits in the last layer/group and use more-bits for the rest layers/groups to avoid bottlenecks.
4.4 EXISTANCE AND TRANSFERABILITT OF THE SELECTED KERNELS
To prove the existence of the selected kernels in other cases, and the transferability of our selected kernels, did experiments extracting top frequent kernels from different networks and layers and compare them with our selected kernels in Appendix[[ Then we conduct fine-tuning experi- ments for pretrained BWN. This will be further studied in Appendix|Mi
4.5 OTHER SELECTION OF KINARY-KERNEIS.
We discuss the other selection of kinnry-kernels in Appendix N For very low-bit quantization, suggest using the most frequent binary-kemell rather than those less frequent ones. For the case like quantization bit p > 4, the choice of -ernels is not essential problem.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jan Achterhold, Jan M Kohler, Anke Schmeink, and Tim Genewein. Variational network quantiza- tion. 2018.
Milad Alizadeh, Javier Fermandez-Marques, Nicholas D) Lane, and Yarin Gal. An empirical study of binary neural networks' optimisation. In ICLR, 2019.
Alexander Anderson and Cory P Berg. The high-dimensional geometry of binary neural networks. 2018.
Lei Jimmy Ba and Rich Caruana. Do deep nets really need be deep? NIPS, 2014.
Yu Bai, Yu-Xiang Wang, and Edo Liberty. Proxquant: Quantized neural networks via proximal operators. 2019.
Joseph Bethge, Haojin Yang, Marvin Bornstein, and Christoph Meinel. Back to simplicity: How to train accurate bnns from scratch? arXiv preprint arXiv:1812.01960 2019.
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. In CVPR, 2017.
Shangyu Chen, Wenya Wang, and Sinno Jialin Pan. Metaquant: Learning to quantize by learning to penetrate non-differentiable quantization. 2019.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural
networks with binary weights during propagations. In NIPS, 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training neural networks with weights and activations constrained to -1. In NIPS, 2016.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 2019.
Angus Galloway, Graham W Taylor, and Medhat Moussa. Attacking binarized neural networks. 2018.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accu- rate object detection and semantic segmentation. CVPR, 2014.
Yiwen Guo, Anbang Yao, Hao Zhao, and Yurong Chen. Network sketching: Exploiting binary structure in deep cnns. In CVPR, 2017.
Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural networks. In NIPS, 2015.
Song Han, Jeff Pool, John Tran, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang-Ting Cheng, and Roeland Nusselder. Latent weights do not exist: Rethinking binarized neural network optimization. 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253/ 2015.
Lu Hou, Quamming Yao, and James T Kwok. Loss-aware binarization of deep networks. 2017.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04861, 2017.

---

Under review as a conference paper at ICLR 2021
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4700 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yani,,, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. arXiv preprint IrX.. (0000,0061, 2016.
Dahyun Kim, Pratap Kunal Singh, and Jonghyun Choi. Learning architectures for binary networks. In ECCV, 2020.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech Report, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NIPS, 2012.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In NIPS, 2017.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capliilll and advanced training algorithm. In ECCV, 2018.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binary convolutions. 2020.
Asit Mishra and Debbie Marr. Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. 2018.
Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. Binary neural networks: survey, Pattern Recognition, pp. 107281, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Joseph. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision.
Darabi Sajad, Mouloud Belbahri, Matthicu Courbariaux, and Vahid Partovi Nia. Regularized binary network training. 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel Emer. Efficient processing of deep neural networks:a tutorial and survey. Proceedings of the IEEE, 2017.
Dongging Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Le-nets: Learned quantization for highly accurate and compact deep neural networks. In ECCV, 2018a.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018b.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights. In ICLR, 2017.
10

---

Under review as a conference paper at ICLR 2021
Shuchang Zhou, Wu Yuxin, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1xXxv:1606606160, 2016.
Shilin Zhu, Xin Dong, and Su Hao. Binary ensemble neural network: More bits per network or more networks per bit? In CVPR, 2019.
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. ICLR, 2017.
A APPENDIX
In this appendix, we will display many additional experiments due to the limited pages in the Con- text. These experiments are all introduced in the Context supplementary material to strength main idea and make our contribution more convincing.
B EXPERIMENTS ON TRAINING BWNs
In this appendix section, we display our experiments on training different networks using XNor- BWN and LQ-BWN as shown in Table
This is a performance table of different quantization methods with different network archi- tectures on different datasets with different training methods. "Baseline" means the network with full precision the baseline. "w/o WD" means without weight decay, and "w/o SF" means without using scaling factors. SF = means we fix scaling factors in layers to 1. LRx 10 means we magnify learning rate 10 times. Test testing set (avaiddatio set).
Table 2:
Acc on ImageNet) means top 1 accuracy
Network Methods
Test Acc
Network Methods
Test Acc
VGG-7 Baseline
93.8%
VGG-7 XWN
91.1%
VGG-7 LQ-BBN
92.3%
VGG-7 XNN w/o WD
90.9%
VGG-7 LQ-BWN w/o WD
90.8%
VGG-7 XNor-BWN w/o SF
91.0%
VGG-7 LQ-BWN w/o SF
92.2%
VGG-7 XNor-BWN SF=0.05
90.3%
VGG-7 XNor-BWN SF=1 LR
89.7%
VGG-7 XNor-BWN SF=0.1
89.8%
VGG-7 XNor-BWN SF=1 LR 10
91.0%
VGG-7 XNor-BWNN SF=1
85.1%
ResNet-20 Baseline
91.3%
ResNet-20 XNor-BWN
87.3%
ResNer-20 LQ-BWN
88.6%
ResNet-20 XNor-BWN w/o WD
86.0%
ResNet-20 LQ-BWN w/o WD
87.4%
ResNet-20 XNor-BWN w/o SF
87.0%
ResNet-20 LQ-BWN w/o SF
88.5%
ResNet-18 Baseline
69.6%
ResNet- 18 XNor-BWN
62.2%
ResNet-18 LQ-BWN
61.1%
ResNet- 18 XNor-BWN w/o SF
62.5%
ResNet-18 LQ-BWN w/o SF
62.4%
ResNet- 18 XNO w/o WD
61.0%
ResNet-18 LQ-BWN w/o WD
58.3%
C COMPARE LEARNABLE SF AND GAMMA IN BN
In original LQ-Net, they use an algorithm called 'Quantization Error Minimization' calculate channel-wise quantizers, which are learnable scaling factors for each channel of Conv layers in binary weight case. Similarly, there is 7 in BatchNormalization layer, which also processes the pre- activation values in a channel-wise manner. In LQ-BWN, we normalize both 7 and scaling factors in such a channel-wise manner, and then plot the relation between these two values in Figure As the figure shows, when the network goes deeper, two values behave highly correlated to each other, especially when the Conv layer is wide. Their high correlation between each other indicates the overlapping function of both.
11

---

Under review as a conference paper at ICLR 2021
Staing
C1
VGG C3
VGG C5
R-20 G1B2C1
R-20 G2B2C1
R-20 G3B2CI
Stating
R-18 GIBICI
R-18 G2BICI
R-18 G3BICI
Figure 6: The relation between the scaling factor of the LQ-BWN and the gamma in BatchNorm layer after the corresponding Conv layer channel by channel after normalizing their values. The X-axis is the scaling factor value and Y-axis is the gamma in the corresponding BatchNorm layer. The point scattered the figure combination of two values after normalization. r in the legend indicates the correlation coefficient. Texts below the figures indicates the certain layer, VGG is VGG-7, R-20 is ResNet-20,, R-18 ResNet- 18, C is Conv, G Group, and B is Block. These abbreviations will also be used in the rest of the paper.
D QUANTIZATION ERROR CURVE
In XNor-Net where XNo is first raised, the scaling factors are proposed to minimize the quantization error in calculated deterministic way. In the BNN survey (Qin et al., (020), the au- thors summarized several BWNs algorithms using "Minimizing the Quantization Error" which has a common form as shown Equatior design their methods. Thus we plot the quantization error curve of different networks shown Figure We can find that in most case the quantization error between full fueciiorn weights and binary weights is not minimized. Therefore we are con- scaling factors to reduce the quantization error,
cerned that it might not be reasonable further, might not be necessary to reduce the quantization error.
J(b. a) | abl1
a argminna, a)
(3)
E WEIGHT HISTROGRAM ON OTHER NETWORKS AND LAYERS
We visualize the full-precision weight histrograms of ResNet- 18's different layers in Figure g
F TRACING LARGE MAGNITUDE WEIGHTS
We trace the large magnitude weights' signs and their norm overlapping of ResNet-20 in Figure and ResNet- 18 in Figure I0 We also plot the hamming distance of ResNet- between the weights during training and weights after training in Figure II The experiment contents are the same as the ones in the Context.
12

---

Under review as a conference paper at ICLR 2021
12
XNo VGG
w/o WD
LQ-BWN VGG
w/o WD
30
XNor-BWN R-20
w/o WD
LQ-BWN R-20
W/o WD
XNor-BWN R-18
w/o WD
LQ-BWN R-18
w/o WD
Figure 7: L2 Distance between full-precision weights and binarized weights during training. We use the L2 distance of the trained networks at last epoch as the unit norm, and other L2 distances are divided by this unit distance to better display the increase or decrease trend. textbfw/o WD means we do the same experiments on training the network without using weight decay. Two figures are displayed together and the left one uses WD while the right one does not. The X-axis is training epochs while Y-axis is the re-scaled sum of L2 norm of all binarized layers.
Proterrl
Magcoka
Dakar
XNor-BWN R-18
LQ-BWN R-18
KNor-BWNY R-18
LQ-BWN R-18
Figure 8: Full-precision weight histograms visualization in ResNet- 18. We choose the weights from the second Conv in the first Block of each group, and the first Conv layer ConvO.
13

---

Under review as a conference paper at ICLR 2021
Pwtceetage
Humming
Ter Srie
Epoch
Epoch
Epoch
XNor-BWN R-20
Pecceetag
Herminng
Epoch
Spoch
Epoch
LQ-BWN R-20
Figure 9: XNor-BWN and LQ-BWN ResNet-20 Weights' signs tracing and weights' overlap- ping with positive weights. And the hamming distance between the weights during training and the weights after training.
G FLIPPING AND PRE-FIXING LARGE MAGNITUDE WEIGHTS AND RETRAINING
We first flip 10% of weights of trained BWN, and then fix its weights according their magnitude and retraining it in Figure 12 The small learning rate in retraining is 0.0002, which is applied to the ordinary 160th to 200th epoch training. The large learning rate is using the original learning rate when training BWN from scratch. We use three flipping strategy, flipping those largest magnitude weights, random weights, and those smallest magnitude weights.
H MODEL COMPRESSION ADDITIONAL EXPERIMENTS
We did more experiment with different quantization bit (together with different compressed ratio) on different networks and datasets shown in Fig
BINARY- KERREL DISTRIBUTION IN EACH LAYER
visualize the binary-kernel frequency of other layers and networks Figure 13 We sum
frequency of top 2" binary-kernels in each group of ResNet-18 18 with the log-scale x-axis.
J INSTABILITY TRAINING ON NETWORKS WITH VERY FEW PARAMETERS
When training QBNs on ResNet-20 with low bit quantizations, the test accuracy's variance is large, making training unstable. We write down a table on ResNet-20 with the training accuracy Standard Deviation among times in Table
K EXISTANCE AND TRANSFERABILITY OF THE SELECTED KERNELS
Selected binary- kernel- do exist, and selected binary-kernele can transfer to other different layers, networks, datasets. We extract different layers' top 2 2 ..., 2 frequent kernels percentage over all kernels and display in Table 13 shows that usually, the imbalanced distribution among B BWNs very common, where only small number of kinary-kernels take advantage percentages
14

---

Under review as a conference paper at ICLR 2021
100
100
95
90
Percentage
Percentage
80
75
75
70
dub tegid
legains
60
Th 33 Fastive
Top 03 begativ
20
40
60
100
20
40
60
80
100
Epoch
Epoch
XNor-BWN Tracing Positive Weights
XNor-B WN Tracing Negative Weights
90
Percentage
85
To
0:2 Segalive
65
Tax Frrd
65
20
40
60
80
100
20
40
50
80
100
Epoch
Epoch
XNor-BWN Positive Weights Overlapped
XNor-BWN Negative Weights Overlapped
100
90
Percentage
90
Percentage
85
80
75
75
70
To 11 Fart
70
Tre 0.31 Segal
Tex
65
Top begaria
Vepaiie
100
Epoch
Epoch
LQ-BWN Tracing Positive Weights
LQ-BWN Tracing Negative Weights
100
100
95
90
Percentage
Percentage
85
80
75
75
Tho B2
70
Segatiee
Tao 3 Fastive
65
Top 0. legatie n. beggiee
100
Epoch
Epoch
LQ-BWN Positive Weights Overlapped
LQ-BWN Negative Weights Overlapped
Figure 10: ResNet-18 Weights' signs tracing and weights' norm overlapping with both positive weights and negative weights.
15

---

Under review as a conference paper at ICLR 2021
0.30
Hamiiing Distance
0.25
Hamming Distance
0.20
0.15
0.15
0.10
0.10
20
40
60 80
100
60
80
100
Epoch
Epoch
XNor-BWN R-18
LQ-BWN R-18
Figure 11: Hamming Distance between binary weights while training and binary weights after train- ing. X-axis the epoch number, and Y-axis the hamming distance over the total binary weight numbers, zero means they become the same.
Table 3: Experiments of VGG-7, ResNet-20, and ResNet-56 on Cifarl0, and ResNet-18 on Im- ageNet. FP indicates the first full-precision conv layer which is not quamtized according to the common practice. VGG-7 has 6 conv layers, and indicate how many selected quantized kernels are used. ResNet-20 and ResNet-56 has three groups, each groups share the quantized bit number for each group. ResNet-18 has four groups which has channels of 64, 128, 256, 512. CR indicates Compressed Ratio. Acc is the top test accuracy Cifar-10, and top validation accuracy on ImageNet. The accuray is reported as the mean of the best accuracy during training among runs with different random seeds.
the quantized bit numbers for each layer to
number of channels which are 16, 32, 64 in order. We assign the same
Network
Quant Bit
CR
Acc
Network
Quant Bit
CR
Acc
VGG-7
FP-9-9-9.9-9
x1.0
92.2%
VGG-7
FP-2-2-2.2-2
x4.5
84.8%
VGG-7
FP-6-5-4-3-2
x3.2
89.2%
VGG-7
FP-3-3-3-3-3
x3.0
87.6%
VGG-7
FP-4-4-4-4-1
x3.7
89.0%
VGG-7
FP-3-2-2-1-1
x7.2
88.9%
VGG-7
FP-5-4-3-2-1
x4.9
89.0%
VGG-7
FP-5-5-5-5-5
x1.8
91.3%
R-20
FP-9-9-9
x1.0
87.6%
R-20
FP-5-3-1
x5.8
63.4%
R-20
FP-9-9-1
x3.1
78.4%
R-20
FP-9-9-2
x2.5
78.5%
R-20
FP-9-9-3
x2.0
80.6%
R-20
FP-4444
x2.2
63.8%
R-20
FP-5-5-5
x1.8
64.3%
R-20
FP-6-6-6
x1.5
81.0%
R-20
FP-9-6-3
x2.3
75.8%
R-20
FP-6-4-2
x3.5
62.8%
R-20
FP-9-4-1
x4.7
64.6%
R-56
FP-9-9-9
x1.0
89.2%
R-56
FP-9-6-2
x
84.1%
R-18
FP-9-9-9-9
x1.0
64.5%
R-18
FP-3-3-3-3
x3.0
49.3%
R-18
FP-4-4-4-4
x2.3
53.4%
R-18
FP-9-7-4-1
x4.6
57.3%
R-18
FP-5-5-5-5
x1.8
56.6%
R-18
FP-9-6-4-2
x3.4
57.5%
R-18
FP-6-6-6-6
x1.5
59.0%
R-18
FP-9-9-6-2
x2.8
60.2%
R-18
FP-7-7-7-7
x1.3
62.6%
Table 4: Experiments of QBN on ResNet-20. We use (mean + std) report the results.
Network
Quantization Bit
Acc
ResNet-20
FP-5-3-1
(63.4= 5.6)
ResNet-20
FP-4-4-1
(65.9- 2.7)%
ResNet-20
FP-6-6-3
(70.5 3.11%
ResNet-20
FP-9-6-3
(75.8 2.4)%
ResNet-20
FP-9-9-1
(78.4= 1.4)%
ResNet-20
FP-9-9-2
(78.5= 2.7)%
ResNet-20
FP-9.9-3
(80.6= 0.9)%
ResNet-20
FP-9-9-99
(88.0= 0.8)%
16

---

Under review as a conference paper at ICLR 2021
100
Accuracy
60
Accuracy
30
70
Largest
70
Largest Norm Test
Random Norm Train
Largest Norm Train
Random Norm
Largest Norm
50
Smellestt Norm Train
Largest Norm with Large LR Train
Smallesst Norm Test
Largest Norm with Large LR Test
Retraining Epoch
Retraining Epoch
XNN VGG-7 Small LR
XNor-BWN VGG-7
50
Accuracy
Largest
Largest Norm Test
Random Norm Train
Largest Norm Train Largest
50
Smallest Norm Train
30
Largest Norm with Large LR Train
Smallest Norm Test
Largest Norm with Large LR Test
40
275
325
350
375
275
Retraining Epoch
Retraining Epoch
XNor-BWN R-20 Small LR
XNor-BWN R-20
Figure 12: Flip 10% of trained BWN and fixing these weights, retrain with different learning rates, on VGG-7 and ResNet-20. The left two figures compare the flipping strategies, and the right two figures compare the learning rate magnitude, which use small learning rate and large learning rate.
Sanitatee dt
Destate
XNor-BWN R-18 GO XNo--BWN R-18 GI XNor-BWN R-18 G2 XNor-BWN R-18 G3
LQ-BWN R-18 GO LQ-BWN R-18 GI LQ-BWN R-18 G2 LQ-BWN R-18 G3
Figure 13: The sum of frequency of top 2" binary-kemels of XNor-BWN and LQ-BWN ResNet-18. X-axis is log2 scale to visualize quantization bits. GO to G3 are the groups of ResNet-18, 18, and each group contains two blocks, two layers for each block.
of all binary kernels. This proves that the QBNs algorithm can adapt to a wide range of networks and datasets.
To prove the transferability of the selected binary-kernels, Table shows that the experiments are all based on the same batch of the selected kernels, which extracted from the last Conv layer of a VGG-7 XNor-BWN without any additional tricks. The results on ResNet-20, ResNet-56 on 10, and ResNet-18 on ImageNet with such residual structures and different layers show how
17

---

Under review as a conference paper at ICLR 2021
those selected kernels perform well. We display how the selected binary-kemels throughout our experiments from the last Conv of VGG-7 appear in other networks' different layer, plotting a Figure 14 to compare their frequency.
We also apply QBNs to fine-tune pre-trained BWNs, when using relatively more quantization bit, the network can usually gain comparable performance without training from scratch. In QBN algorithm, the computational cost increases when using more quantization bits, thus we can directly fine-tune pre-train BWN. The experiments in Appendix show that we can gain uniform 7 bit QBN in one epoch, and uniform 6 bit QBN in two epochs. This provides a higher efficient way train those high quantization bit QBNs.
L THE DEGREE OF AGGREGATION OF HINARY-KERNER
We do statistic on how much percentage of the target networks' top frequent binary-kernels can be covered by our selected binary-kernch from the last Conv of XNor-BWN VGG-7. We use the first figure on the top left of Figure 14 to demonstrate that, when using 1 bit quantization on binary-kernels of KNor-BWN ResNet-18's first layers(the first group of ResNet- our selected kernels can cover 100% of XNor-BWN ResNet-18's top 2 frequent kernels. The percentage has already been processed with the number of each kernel in the layer. 100% the limit that if directly choose the most frequent kernels from this layer/group as our selected kernels. For the lowest percentage appearing when using 5-bit, there are still more than 80% kernels in the first group's top 2" frequent kemells can be represented directly by our selected kernels.
XNor-BWN R-18 GO XNor-BWN R-18 GI XNor-BWN R-18 G2 XNor-BWN R-18 G3
LQ-BWN R-18 GO
LQ-BWN R-18 G1 LQ-BWN R-18 G2 LQ-BWN R-18 G3
The ratio between the number of kernels in one layer that are in our selected binary- kernels, and that are in the top 2 frequent binary-kernels in that layer. Thus 100% is the upper bound, which means our selected binary-kernels can cover the same number of kernels that this layer's top frequent kernels cover. The X-axis is log2 scale to visualize quantization bits. GO to G3 are the groups of ResNet-18, and each group contains two blocks, two layers for each block.
Figure 14:
M FINE-TUNE WITH HIGHER QUANTIZATION BIT(ON 6 BIT, BIT)
We write Table 5 Jusing relatively higher guantization bit fine-tune ResNet-18 on ImageNet with
one epoch. The pretrained model is the normal ResNet-18 BWN at 90th epoch. We list the number of epochs that the fine-tune requires. For those lower bit, the fine-tuned performance cannot achieve to the one training from scratch.
N OTHER SELECTION OF SNNARYKKENNELS
Here list three reasons to illustrate why such way collect the binary- -ernel. If we regard QBNs as a clustering method given cluster numbers, these kemnels with high appearance frequency most likely the cluster centers. 2. We visualize these top frequent kernels as shown
18

---

Under review as a conference paper at ICLR 2021
Table ResNeet-18 on ImageNet funetuned base on pretrained BWN.
Network
Quantization Bit
Acc
ResNet-18
FP-9-9-9-99
62.93%(90 Epochs)
ResNet-18
FP-6-6-6-6
58.79%(90 Epochs Retrain 2 Epochs)
ResNet-18 FP-6-6-6-6
(8.91%((0
Epochs Retrain 10 Epochs)
ResNet-18
FP-7-7-7-7
62.28%
Epochs Retrain Epoch)
ResNet-18
FP-7-7-7-7
62.56%(90
Epochs Retrain 10 Epochs)
Figure 15: Top: The visualization Top 16 frequent binary-kernels. Down: The visualization of the least frequent 16 binary-kernels among top 2 frequent binary-kernlls, more specifically, the 113rd to 128th frequent kernels.
in Fig 15 and they appear similar conventional spatial filters, including "Moving Average Filter", "Point Detector", "Liine Detector" and so on. 3. We use the less frequent binary-kernels as the selected kernels test if our selection based on frequency
good choice. We find that using those less frequent kernels, an accuracy drop can be observed in different experiments. Those less frequent kernels are used by inverting the order of top 128 frequent kernels, eg. for 2 kernels, they are 1 125-th, 126-th, 127-th, and 128-th frequent kernels. Given the experiment results in Table we find that when using very low quantization bit, specifically less than or equal to will significantly hurt the network when using those least frequent kernels. When the quantization bit increases, the difference performances reduce, Thus we suggest to use those most frequent kernels training very low-bit QBN, or finetuning a pre-trained BWN In other cases like training from scratch with quantization bit more than 3, the frequency of the selected kernels not a strict deterministic factor.
Table 6: We train several networks using the normal top frequent binary-kernels, and those least frequent binary-kemels named "Reverse" in the table since they the order of reversed top 128 frequent kernels. 10.0% in the table indicates the training cannot converge from the beginning.
Network and Quantization Bit
Acc(Normal)
Acc(Reverse)
VGG-7 FP-3-3-3-3-3
87.0%
10.0%
VGG-7 FP-4-4--444
89.7%
90.7%
VGG-7 FP-5-5-5-5-5
90.5%
92.1%
ResNet-20 FP-9-6-3
75.9%
74.6%
ResNet-20 FP-9-9-2
78.9%
75.0%
ResNet-20 FP-9-9-3
80.6%
78.6%
ResNet-56 FP-9-6-2
84.1%
81.4%
ResNet-56 FP-5-5-5
86.6%
87.0%
O INFLUENCE OF
A
To test the influence the Hyper-Parameee quantization kernels and ResNet-56 with FP-9-6-2. We do not use ResNet-20 due to its instability training which we have discussed. The result is shown Figure 16
our algorithm, we use VGG-7 with all 5-bit
P THE SELECTED KERNELS FROM OTHER CONV LAYERS
In previous experiments, our selected binary kernels K,, use the statistic information extracted from one single VGG-7's last Conv layer and regard them constant kernels. We further different strategies to demonstrate that the selected binary kernels have a strong generalization ability to apply other networks. In the following experiments, we use the selected binary kernels from the last
19

---

Under review as a conference paper at ICLR 2021
VSG-7
ResNett55
Accuracy Percentage
Percentage
6/4
614
HaperePr (Delta
Hyper-Parameter !Delta ResNet-56
VGG-7
Figure 16: The accuracy of 5 experiments for each hyperparamter A of different values.
layer of ResNet-20, VGG-7's 3rd and 5th Conv layers, and from the last layer of ResNet-18. We use these kernels from different strategy to other different networks. The result is shown in Table
Table Using different source layers of the selected kernels. The Source Layer is where we extract the most frequent binary kernels, and target QBN is the network apply QBN to.
Source Layer
Target QBN
Quant Bit
Acc
VGG-7 3rd Layer
VGG-7
FP-5-5-5-5-5
91.0%
VGG-7 3rd Layer
VGG-7
FP-6-5-4-3-2
89.0%
VGG-7 3rd Layer
ResNet-20
FP-9-9-3
79.4%
VGG-7 3rd Layer
ResNet-20
FP-6-6-6
81.0%
VGG-7 5rd Layer
VGG-7
FP-5-5-5-5-5
91.2%
VGG-7 5rd Layer
VGG-7
FP-6-5-4-3-2
89.2%
VGG-7 5nd Layer
ResNet-20
FP-9-9-3
80.6%
VGG-7 Layer
ResNet-20
FP-6-6-6
81.4%
ResNet-18 13th Layer
VGG-7
FP-5-5-5-5-5
90.9%
ResNet-18 13th Layer
VGG-7
FP-6-5-4-3-2
88.9%
ResNet-18 13th Layer
ResNet-20
FP-9-9-3
79.0%
ResNet-18 13th Layer
ResNet-20
FP-6-6-6
81.4%
R8 17th Layer
VGG-7
FPP----555-5
91.0%
ResNet-18 17th Layer
VGG-7
FP-6-5-4-3-2
89.2%
ResNet- 18 17th Layer
ResNet-20
FP-9-9-3
79.8%
ResNet-18 18 17th Layer
ResNet-20
FP-6-6-6
81.9%
20


Reviews:
[{"review_id": "xeN_H7z0Gc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:**\nThis paper proposes some interesting observations for training BWNs. 1: The scaling factors can be removed with batch normalization used. 2: The signs of the weights with large norms are determined and fixed at the early training stage. 3:  The binary weight networks can be further compressed. Moreover, the authors provide some empirical visualizations and results to demonstrate its analysis. However, the paper seems to be incomplete and needs to be further improved. \n\n**Pros:**  \nThe observation that the signs of weights with large norms are determined and fixed at the early training stage is interesting. The idea that weights with large norms are stable and sensitive on sign changes can be utilized for improving the training of BWNs, maybe BNNs as well.\n\n**Cons:**   \n1: Extensive descriptions are very confusing. I can only list some of them. \n\n1): The contributions of the paper include the observation that binary kernels in the learned convolutional layers tend to be centered on a limited number of fixed structural patterns. However, it is still not clear to me what the “fixed structural patterns” are. \n\n2): In Sec. 3, the paper claims to “quantize the less frequent kernels to those high frequent kernels to save space” and “we sort these binary kernels according to their appearance frequency…” . However, the paper fails to explain the motivation for exploring the frequency of kernels. Some theoretical explanations are needed. \n\n3):  In Figure 4’s caption, what does “the certain appears in one certain Conv layer” mean?\n  \n4):  In Sec. 4.1, the authors propose to apply QBN on model compression.  However, it is not clear to me what the points the paper intends to express. Specifically, the authors claim that “we can reduce the number of parameters to represent a binary-kernel by changing the small magnitude weights’ signs”. Then the question comes. How can the number of parameters be reduced via changing signs? Also, the sentence “we can compress their parameters to an extremely small number by replacing the whole 3×3 binary-kernel with fewer”  is incomplete. \n\n2: Extensive symbols are undefined, which makes me hard to understand the paper properly, especially in Algorithm 1, the main algorithm of the paper. For example, definition of $n$ in calculating $E$;  definition of  “where()”;  definition of $K_m$, are not explained.\n\n3:  Extensive technical details are missing, which makes the algorithm difficult to understand. \n1):   The authors should at least explain how to generate the power-of-two number of binary kernels in details in Sec. 3.1.  \n2):   During training, the paper generates $k$-bit (i.e., k is the bitwidth) number of binary kernels. However, during testing, the learnt binary weights should be fixed ($k=1$) and why the “Quant Bit” in Table 1 can be larger than 1? This makes me very confusing.\n\n4:  This paper only binarizes weights while I am wondering whether simultaneously binarizing both weights and activations can have the similar observations. \n\n5:  There is no comparisons with current state-of-the-arts.  Also, in Sec 4.1, the paper claims that “the compressed ratio can achieve to 1.5$\\times$ with a bearable accuracy drop (less than 3% on Cifar-10)” does not stand. The 3% loss on Cifar-10 is significant. \n\n6:  There is no conclusions and future works discussions. \n\n7:  It is widely known that the scaling factors can be absorbed into BN layers. In terms of this, it should not be a contribution of this paper.\n\n8: The paper introduces extensive heuristic hyper-parameters. Specifically, it does not give any strategy to automatically determine the optimal quantized bitwidth and threshold $\\Delta$ for each layer. \n\n9:  There are lots of grammar and typo issues. For example, the words in equations should use straight format (use \\rm in latex).", "extra_scores": {}, "extra_text": {}}, {"review_id": "t4Gd-Z3BGzq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides an empirical study of binary weight networks (BWNs), where they find that 1 the commonly adopted scaling factor is not critical 2 there exists a subnetwork that stabiles early in training 3 the 3x3 filters in VGG and ResNets demonstrate a sparse distribution. They combine all the observations and propose a novel quantization algorithm that achieves more aggressive compression than standard BWNs.\n\npros:\n+ I appreciate the careful examination of design and training details of standard BWNs. The identification of a persistent subnetwork and the analysis on the sparse distribution of kernels are particularly interesting.\n\n+ The proposed quantization algorithm is interesting, which has a potential of squeezing more redundancy out of standard BWNs\n\ncons:\n- If I understand correctly, in the proposed algorithm the kernel distribution is only drawn from the last conv layer of the full precision network, which is then shared across all layers when retraining the BWN. This seems a strong assumption and needs to be justified. What's the reason to believe that the selected frequent kernels are shared across different layers?\n\n-In Algorithm 1, W = where(abs(W ) > ∆E, sign(W ), W ) is not motivated and explained well. What's the reasoning of using the threshold when computing the distance to the frequency binary kernels? \n\n-The experimental results seem to be really hard to interpret for me, and this is perhaps the weakest point of the this paper. In particular, Table 1 needs to have proper baselines. This includes the full precision, standard BWN accuracies, as well as controls which allow one to draw comparisons between the proposed algorithm and basic binarization by equating certain quantities. \n\nI suggest the authors work on the suggested improvements which will make this a much stronger contribution.\n\n*****post rebuttal updates*****\nI want to thank the authors for responding to my questions. The additional explanations are indeed helpful for clarifying my first two questions (selection of the binary kernel and the use of ∆E). However, I still have concerns about Table 1 (and Table 2). For example, I have a really hard time interpreting the significance of achieving a 3.2x CR with a loss of 3% (92.3 - 89.2 from VGG-7) in acc with the proposed method (although the paper argues that it's a \"bearable\" loss). Considering that this is the main experiment supporting the efficacy of the proposed quantization algorithm, I think the paper needs more controlled experiments to demonstrate the practical usefulness of the proposed algorithm. As a result I'm keeping my original score and hope the authors can work on the improvements for the next version. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "d2BGrqHnphy", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\nThe Authors show that scaling factors with hand-crafted or learnable methods are not so important when training Binary Weight Networks (BWNs), while the change of weight signs is crucial. They make two observations: The weight signs of the primary binary sub-networks are determined and fixed at the early training stage. Binary kernels in the convolutional layers of final models tend to be centered on a limited number of fixed structural patterns. Based on these observations, they propose a new method called binary kernel quantization to further compress BWNs. \n\nStrength bullets:\n1. They propose a binary kernel quantization to quantize the binary kernel, which effectively reduces the number of all possible kernels, and can further compress BWNs to 2-5 times.\n2. The Authors observe that the weight with the large norm, fixing their signs in the early training stage. And they compared this phenomenon with the lottery ticket hypothesis and propose the primary binary sub-networks. I think it's an interesting idea.\n3. The paper is well written and well-motivated. It contains extensive and systematic related work-study. I like it. \n\nWeakness bullets:\n1. The binary kernel quantization method is only applied to 3x3 kernels in this paper. When the size of the convolution kernel is 5x5 or 7x7，how does it work? I hope this method can extend to another cases and it also can work well. \n2. As shown in table1, why authors report the best accuracy among different seed? I think authors should report the averaged results with different random seeds instead of the best result.\n\n-----Post Rebuttal-----\n\nI have read all feedback and especially thanks to the authors' efforts on the extra experiments. I think the author has addressed my first concern. For the second one, I would prefer to see the errorbar.\n\nI tend to keep my scores unchanged. I think the findings are interesting [share similar thoughts as R4's], while the experiments part need to be improved. Although I like the ideas and observations, I don't feel especially strongly in favor of it and cannot champion it.", "extra_scores": {}, "extra_text": {}}, {"review_id": "F5V1E6gJH6V", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors show empirically that weight signs are more important than weight magnitudes. They also analyze the optimization process and the structure of optimized binary networks to note that \ni) there are clusters of weights whose weight remain almost unchanged during the optimization \nii) optimized convolutional networks have simple sub-structures  \nThey exploit the latter to propose a new quantization method that increases the compression of binary networks.\n \nstrengths:\nThe idea of studying how binary weights update during the optimization is interesting and may help understand the optimization process of binary but also real-valued networks. The experiments include numerical results for a wide range of data sets and networks. \n\nweaknesses:\nThe meaning of the given simple proof is not clear. What are gamma and epsilon in equations 1 and 2? The paper would have much more impact if, after showing that BN can absorb the redundant factors, all real-valued parameters were dropped. In some sense, the proof seems to show that the obtained result about the little influence of scaling is somehow expected. \n\nRegrading the sign flipping, observing that \"flipping weights with large full precision magnitude will cause a significant performance drop compared to those weights close to zero\" seems also something that one can expect.\n\nFinally, the presence of clusters is not really explained and shown in a very accessible way. Are there other results except from Figure 4 (whose caption sounds a bit hermetic: \"The X-axis indicates the index of a 3 x 3 binary weight kernel while Y-axis indicates the frequency that the certain appears in one certain Conv layer\")\n\nquestions: \n- what happens if all scaling factors are kept fixed?\n- is alpha usually shared by all weights in the network or is layer-specific?\n- the weights distributions in figure 1 are obtained by adding a regularization term? \n- what happens if also the first full-precision Conv layer is quantized? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: ## Description
The paper discovers interesting phenomena in training neural networks with binary weights:
- Connection between latent weight magnitude and how important its binarized version for the network performance
-training dynamics, indicating that large latent weights are identified and stabilize early on
- Observation that amongst learned binary kernel, several specific patterns prevail, up to the bits who's reversal has very little effect. This is so regardless of the architecture, the layer considered or the dataset. 
The paper further demonstrates how these observations may be used to compress binary neural networks below 1 bit per weight.

## Review Process and Decision
The reviewers welcomed the experimental investigation of new phenomena, but commented the overall technical quality of the work as somewhat substandard. The redundancy of consecutive affine transforms is known and not connected to binary weights investigation. The investigation itself lacks a more in-depth analysis. The proposed compression results appeared not convincing to reviewers since a significant drop of accuracy occurs. The AC shares these concerns and supports rejection.

## General Comments
From my perspective, the study undertaken is methodologically „wrong“. An ad-hoc training method is investigated, which is not even clearly defined in the paper (there are many „STE“ variants) and for which it is not known what it is doing, what are the real-valued weights for and whether they are needed at all (as empirically argued by Helwegen et al. (2019)). As such, the investigation makes impression of poking a black box (the training method in this case). At the same time, there are more clear learning formulations, applicable in the setting of the paper (binary weights), in particular considering the stochastic relaxation:
* Shayer et al. (2017): Learning Discrete Weights Using the Local Reparameterization Trick
* Roth et al. (2019): Training Discrete-Valued Neural Networks with Sign Activations Using Weight Distributions
* Peters et al. (2018): Probabilistic binary neural networks

These methods are approximate, but at least the optimization is well posed and it is known what do the real-valued weights represent (e.g. logits of binary weight probabilities).
From this perspective, it can be seen that latent weights close to 0 correspond to Bernoulli weights that are almost fully random (and thus only contribute noise) and are fragile to gradient steps. Therefore the model can only perform well if it learns to be robust to their state or their state becomes more deterministic (corresponding to large latent weight). So one would actually expect to see in these models phenomena similar to the observed in the paper and not bee too much surprised or astonished by them. Furthermore, there are recent works explaining STE and its latent weights as optimizing the stochastic relaxation:
* Meng et al. (2020): Training Binary Neural Networks using the Bayesian Learning Rule
* Yanush et al. (2020): Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networks. 

The authors are encouraged to make the observed phenomena more explainable by connecting to the mentioned works.

## Further Details

*  „We show that in the context of using batch normalization after convolutional layers, adapting scaling factors with either hand-crafted or learnable methods brings marginal or no accuracy gain to final model.“

From theoretical perspective, this is obvious and known to me. Practically, there could be in principle some difference due to the learning dynamics, and verifying that there is none is a useful but a weak contribution. The section devoted to this issue can be given in the appendix but is not justified in the main paper.

* „change of weight signs is crucial in the training of BWNs“

The sign determines the binary weights, so this is by definition.

* „ Firstly, the training of BWNs demonstrates the process of seeking primary binary sub-networks whose weight signs are determined and fixed at the early training stage, which is akin to recent findings on the lottery ticket hypothesis for efficient learning of sparse neural networks“

In the lottery ticket hypothesis paper it is shown explicitly that the identified sparse subnetwork changes during the learning the most rather than retains its initialization state or the state in the beginning of the training. It is therefore could be of a different nature.

Decision: Reject