Title: BNN+: IMPROVED BINARY NETWORK TRAINING

Abstract: Deep neural networks (DNN) are widely used in many applications. However, their deployment on edge devices has been difficult because they are resource hungry. Binary neural networks (BNN) help to alleviate the prohibitive resource requirements of DNN, where both activations and weights are limited to 1-bit.
We propose an improved binary training method (BNN+), by introducing a regularization function that encourages training weights around binary values. In addition to this, to enhance model performance we add trainable scaling factors to our regularization functions. Furthermore, we use an improved approximation of the derivative of the sign activation function in the backward computation. These additions are based on linear operations that are easily implementable into the binary training framework. We show experimental results on CIFAR-10 obtaining an accuracy of 86.5%, on AlexNet and 91.3% with VGG network. On ImageNet, our method also outperforms the traditional BNN method and XNOR-net, using AlexNet by a margin of 4% and 2% top-1 accuracy respectively.

1. INTRODUCTION:

Deep neural networks (DNNs) have demonstrated success for many supervised learning tasks ranging from voice recognition to object detection (Szegedy et al., 2015; Simonyan & Zisserman, 2014; Iandola et al., 2016) . The focus has been on increasing accuracy, in particular for image tasks, where deep convolutional neural networks (CNNs) are widely used. However, their increasing complexity poses a new challenge, and has become an impediment to widespread deployment in many applications; specifically when trying to deploy such models to resource constrained and lower-power devices. A typical DNN architecture contains tens to thousands of layers, resulting in millions of parameters. As an example, AlexNet (Krizhevsky et al., 2012) requires 200MB of memory, VGG-Net (Simonyan & Zisserman, 2014) requires 500MB memory. Large model sizes are further exacerbated by their computational cost, requiring GPU implementation to allow real-time inference. Such requirements evidently cannot be accustomed by edge devices as they have limited memory, computation power, and battery. This motivated the community to investigate methods for compressing and reducing computation cost of DNNs.

To make DNNs compatible with the resource constraints of low power devices, there have been several approaches developed, such as network pruning (LeCun et al., 1990) , architecture design (Sandler et al., 2018) , and quantization (Courbariaux et al., 2015; Han et al., 2015) . In particular, weight compression using quantization can achieve very large savings in memory, where binary (1bit), and ternary (2-bit) approaches have been shown to obtain competitive accuracy (Hubara et al., 2016; Zhu et al., 2016; Tang et al., 2017) . Using such schemes reduces model sizes by 8x to 32x depending on the bit resolution used for computations. In addition to this, the speed by quantizing the activation layers. In this way, both the weights and activations are quantized so that one can replace the expensive dot products and activation function evaluations with bitwise operations. This reduction in bit-width benefits hardware accelerators such as FPGAs and neural network chips.

An issue with using low-bit DNNs is the drastic drop in accuracy compared to its full precision counterpart, and this is made even more severe upon quantizing the activations. This problem is largely due to noise and lack of precision in the training objective of the networks during back-propagation (Lin et al., 2017) . Although, quantizing the weights and activations have been attracting large interests thanks to their computational benefits, closing the gap in accuracy between the full precision and the quantized version remains a challenge. Indeed, quantizing weights cause drastic information loss and make neural networks harder to train due to a large number of sign fluctuations in the weights. Therefore, how to control the stability of this training procedure is of high importance. In theory, it is infeasible to back-propagate in a quantized setting as the weights and activations employed are discontinuous and discrete. Instead, heuristics and approximations are proposed to match the forward and backward passes. Often weights at different layers of DNNs follow a certain structure. How to quantize the weights locally, and maintaining a global structure to minimize a common cost function is important (Li et al., 2017) .

Our contribution consists of three ideas that can be easily implemented in the binary training framework presented by Hubara et al. (2016) to improve convergence and generalization accuracy of binary networks. First, the activation function is modified to better approximate the sign function in the backward pass, second we propose two regularization functions that encourage training weights around binary values, and lastly a scaling factor is introduced in both the regularization term as well as network building blocks to mitigate accuracy drop due to hard binarization. Our method is evaluated on CIFAR-10 and ImageNet datasets and compared to other binary methods. We show accuracy gains to traditional binary training.

2. RELATED WORK:

We focus on challenges present in training binary networks. The training procedure emulates binary operations by restricting the weights and activations to single-bit so that computations of neural networks can be implemented using arithmetic logic units (ALU) using XNOR and popcount operations. More specifically, XNOR and popcount instructions are readily available on most CPU and GPU processing units. Custom hardware would have to be implemented to take advantage of operations with higher bits such as 2 to 4 bits. The goal of this binary training is to reduce the model size and gain inference speedups without performance degradation.

Primary work done by Courbariaux et al. (2015) (BinaryConnect) trains deep neural networks with binary weights {-1, +1}. They propose to quantize real values using the sign function. The propagated gradient applies update to weights |w| ≤ 1. Once the weights are outside of this region they are no longer updated, this is done by clipping weights between {-1, +1}. In that work, they did not consider binarizing the activation functions. BNN (Hubara et al., 2016) is the first purely binary network quantizing both the weights and activations. They achieve comparable accuracy to their prior work on BinaryConnect, and achieve significantly close performance to full-precision, by using large and deep networks. Although, they performed poorly on large datasets like ImageNet (Russakovsky et al., 2015) . The resulting network presented in their work obtains 32× compression rate and approximately 7× increase in inference speed.

To alleviate the accuracy drop of BNN on larger datasets, Rastegari et al. (2016) proposed XNOR-Net, where they strike a trade-off between compression and accuracy through the use of scaling factors for both weights and activation functions. In their work, they show performance gains compared to BNN on ImageNet classification. The scaling factors for both the weights and activations are computed dynamically, which slows down training performance. Further, they introduced an additional complexity in implementing the convolution operations on the hardware, slightly reducing compression rate and speed up gains. DoReFa-Net (Zhou et al., 2016) further improves XNOR-Net by approximating the activations with more bits. The proposed rounding mechanism allows for low bit back-propagation as well. Although they perform multi-bit quantization, their model still suffers from large accuracy drop upon quantizing the last layer.

Later in ABC-Net, Tang et al. (2017) propose several strategies, adjusting the learning rate for larger datasets. They show BNN achieve similar accuracy as XNOR-Net without the scaling overhead by adding a regularizer term which allows binary networks to generalize better. They also suggest a modified BNN, where they adopted the strategy of increasing the number of filters, to compensate for accuracy loss similar to wide reduced-precision networks (Mishra et al., 2017) . More recently, Liu et al. (2018) developed a second-order approximation to the sign activation function for a more accurate backward update. In addition to this, they pre-train the network in which they want to binarize in full precision using the hard tangent hyperbolic (htanh) activation, see Figure 2 . They use the pre-trained network weights as an initialization for the binary network to obtain state of the art performance.

3. IMPROVED BINARY TRAINING:

Training a binary neural network faces two major challenges: on weights, and on activation functions. As both weights and activations are binary, the traditional continuous optimization methods such as SGD cannot be directly applied. Instead, a continuous approximation is used for the sign activation during the backward pass. Further, the gradient of the loss with respect to the weights are small. So as training progresses weight sign remains unchanged. These are both addressed in our proposed method. In this section, we present our approach to training 1-bit CNNs in detail.

3.1. BINARY TRAINING:

We quickly revisit quantization through binary training as first presented by (Courbariaux et al., 2015) . In (Hubara et al., 2016) , the weights are quantized by using the sign function which is +1 if w > 0 and -1 otherwise.

In the forward pass, the real-valued weights are binarized to w b , and the resulting loss is computed using binary weights throughout the network. For hidden units, the sign function non-linearity is used to obtain binary activations. Prior to binarizing, the real weights are stored in a temporary variable w. The variables w are stored because one cannot back-propagate through the sign operation as its gradient is zero everywhere, and hence disturbs learning. To alleviate this problem the authors suggest using a straight through estimator (Hinton, 2012) for the gradient of the sign function. This method is a heuristic way of approximating the gradient of a neuron,

dL(w) dw ≈ dL dw w=w b 1 {|w|≤1} (1)

where L is the loss function and 1(.) is the indicator function. The gradients in the backward pass are then applied to weights that are within [-1, +1]. The training process is summarized in Figure 1 . As weights undergo gradient updates, they are eventually pushed out of the center region and instead make two modes, one at -1 and another at +1. This progression is also shown in Figure 1 . 

3.2. IMPROVED TRAINING METHOD:

Our first modification is on closing the discrepancy between the forward pass and backward pass. Originally, the sign derivative is approximated using the htanh(x) activation, as in Figure 2 . Instead, we modify the Swish-like activation (Ramachandran et al., 2017; Elfwing et al., 2018; Hendrycks & Gimpel, 2016) , where it has shown to outperform other activations on various tasks. The modifications are performed by taking its derivative and centering it around 0

EQUATION

where σ(z) is the sigmoid function and the scale β > 0 controls how fast the activation function asymptotes to -1 and +1. The β parameter can be learned by the network or be hand-tuned as a hyperparameter. As opposed to the Swish function, where it is unbounded on the right side, the modification makes it bounded and a valid approximator of the sign function. As a result, we call this activation SignSwish, and its gradient is

EQUATION

which is a closer approximation function compared to the htanh activation. Comparisons are made in Figure 2 . Hubara et al. (2016) noted that the STE fails to learn weights near the borders of -1 and +1. As depicted in Figure 2 , our proposed SignSwish activation alleviates this, as it remains differentiable near -1 and +1 allowing weights to change signs during training if necessary.

1 Note that the derivative d dx SS β (x) is zero at two points, controlled by β. Indeed, it is simple to show that the derivative is zero for x ≈ ±2.4/β. By adjusting this parameter beta, it is possible to adjust the location at which the gradients start saturating. In contrast to the STE estimators, where it is fixed. Thus, the larger β is, the closer the approximation is to the derivative of the sign function.

-1 sign(x) 1 δ(x) 1 -1 htanh(x) 1 STE(x) -1 1 SS 1 (x) -1 1 dSS 1 (x) dx -1 1 SS 5 (x) dSS 5 (x) dx

3.2.1. REGULARIZATION FUNCTION:

In general, a regularization term is added to a model to prevent over-fitting and to obtain robust generalization. The two most commonly used regularization terms are L 1 and L 2 norms. If one were to embed these regularization functions in binary training, it would encourage the weights to be near zero; though this does not align with the objective of a binary network. Instead, it is important to define a function that encourages the weights around -1 and +1. Further, in Rastegari et al. (2016) they present a scale to enhance the performance of binary networks. This scale is computed dynamically during training, using the statistics of the weights. To make the regularization term more general we introduce scaling factors α, resulting in a symmetric regularization function with two minimums, one at -α and another at +α. As these scales are introduced in the regularization function and are embedded into the layers of the network they can be learned using backpropagation.

The Manhattan regularization function is defined as

EQUATION

whereas the Euclidean version is defined as

EQUATION

where α > 0 is the scaling factor. As depicted in Figure 3 , in the case of α = 1 the weights are penalized at varying degrees upon moving away from the objective quantization values, in this case, {-1, +1}.

The proposed regularizing terms are inline with the wisdom of the regularization function R(w) = (1 -w 2 )1 {|w|≤1} as introduced in Tang et al. (2017) . A primary difference are in introducing a trainable scaling factor, and formulating it such that the gradients capture appropriate sign updates to the weights. Further, the regularization introduced in Tang et al. ( 2017) does not penalize weights that are outside of [-1, +1]. One can re-define their function to include a scaling factor as R(w) = (α -w 2 )1 {|w|≤α} . In Figure 3 , we depict the different regularization terms to help with intuition.

-3 -2 -1 1 2 3

R1(w) -3 -2 -1 1 2 3 R2(w)

Figure 3 : R (w) (left) and R 2 (w) (right) regularization functions for α = 0.5 (solid line) and α = 1 (dashed line). The scaling factor α is trainable, as a result the regularization functions can adapt accordingly.

3.3. BNN+: TRAINING METHOD:

Combining both the regularization and activation ideas, we modify the training procedure by replacing the sign backward approximation with that of the derivative of SS β activation (2). During training, the real weights are no longer clipped as in BNN training, as the network can back-propagate through the SS β activation and update the weights correspondingly.

Additional scales are introduced to the network, which multiplies into the weights of the layers. The regularization terms introduced are then added to the total loss function,

EQUATION

where L(W, b) is the cost function, W and b are the sets of all weights and biases in the network, W l is the set weights at layer l and α l is the corresponding scaling factor. Here, R(.) is the regularization function (4) or (5). Further, λ controls the effect of the regularization term. To introduce meaningful scales, they are added to the basic blocks composing a typical convolutional neural network. For example, for convolutions, the scale is multiplied into the quantized weights prior to the convolution operation. Similarly, in a linear layer, the scales are multiplied into the quantized weights prior to the dot product operation. This is made more clear in the training algorithm 1.

The scale α is a single scalar per layer, or as proposed in Rastegari et al. ( 2016) is a scalar for each filter in a convolutional layer. For example, given a CNN block with weight dimensionality (C in , C out , H, W ), where C in is the number of input channels, C out is the number of output channels, and H, W , the height and width of the filter respectively, then the scale parameter would be a vector of dimension C out , that factors into each filter.

As the scales are learned jointly with the network through backpropagation, it is important to initialize them appropriately. In the case of the Manhattan penalizing term (4), given a scale factor α and weight filter then the objective is to minimize

min α h,w |α -|W h,w || (7)

The minimum of the above is obtained when

α * = median(|W |) (8)

Similarly, in the case of the Euclidean penalty (5) the minimum is obtained when α * = mean(|W |) (9) The scales are initialized with the corresponding optimal values after weights have been initialized first. One may notice the similarity of these optimal values with that defined by Rastegari et al. (2016) , whereas in their case the optimal value for the weight filters and activations better matches the R 2 (w) goal. A difference is on how these approximations are computed, in our case they are updated on the backward pass, as opposed to computing the values dynamically.

The final resulting BNN+ training method is defined in Algorithm 1. In the following section, we present our experimental results and important training details.

Algorithm 1 BNN+ training. L is the unregularized loss function.λ and R 1 are the regularization terms we introduced. SS β is the SignSwish function we introduced and (SS β ) is its derivative. N is the number of layers. • indicates element-wise multiplication. BatchNorm() specifies how to batchnormalize the activation and BackBatchNorm() how to back-propagate through the normalization. ADAM() specifies how to update the parameters when their gradients are known. Require: a minibatch of inputs and targets (x 0 , x * ), previous weights W , previous weights' scaling factors α, and previous BatchNorm parameters θ. Ensure: updated weights W t+1 , updated weights' scaling factors α t+1 and updated BatchNorm parameters θ t+1 . {1. Forward propagation:} s 0 ← x 0 W 0 {We do not quantize the first layer.}

x 1 ← BatchNorm(s 0 , θ 0 ) for k = 1 to N -1 do x b k ← sign(x k ) W b k ← sign(W k ) s k ← α k x b k W b k {This step can be done using mostly bitwise operations.} x k+1 ← BatchNorm(s k , θ k ) end for {2. Backward propagation:} Compute g x N = ∂L ∂x N knowing x N and x * for k = N -1 to 1 do (g s k , g θ k ) ← BackBatchNorm(g x k+1 , s k , θ k ) g α k ← g s k x b k W b k + λ ∂R1 ∂α k g W b k ← g s k α k x b k g x b k ← g s k α k W b k {We use our modified straight-through estimator to back-propagate through sign:} g W k ← g W b k • (SS β ) (W k ) + λ ∂R1 ∂W k g x k ← g x b k • (SS β ) (x k ) end for (g s0 , g θ0 ) ← BackBatchNorm(g x1 , s 0 , θ 0 ) g W0 ← g s0 x 0 {We did not quantize the first layer.} {3. The update:} for k = 0 to N -1 do θ t+1 k , W t+1 k , α t+1 k ← ADAM(η, θ k , W k , α k , g θ k , g W k , g α k ) end for

4. EXPERIMENTAL RESULTS:

We evaluate our proposed method with the accuracy performance of training using BNN+ scheme versus other proposed binary networks, Hubara et al. (2016) ; Rastegari et al. (2016) ; Tang et al. (2017) . We run our method on CIFAR-10 and ImageNet datasets and show accuracy gains. They are discussed in their respective sections below.

4.1. CIFAR-10:

The CIFAR-10 data (Krizhevsky & Hinton, 2009) consists of 50,000 train images and a test set of 10,000. For pre-processing the images are padded by 4 pixels on each side and a random crop is taken. We train both, AlexNet (Krizhevsky et al., 2012) , and VGG (Simonyan & Zisserman, 2014) using the ADAM (Kingma & Ba, 2014) optimizer. The architecture used for VGG is conv(256) → conv(256) → conv(512) → conv(512) → conv(1024) → conv(1024) → fc(1024) → fc(1024) where conv(•) is a convolutional layer, and fc(•) is a fully connected layer. The standard 3 × 3 filters are used in each layer. We also add a batch normalization layer (Ioffe & Szegedy, 2015) prior to activation. For AlexNet, the architecture from Krizhevsky (2014) is used, and batch normalization layers are added prior to activations. We use a batch size of 256 for training. Many learning rates were experimented with such as 0.1, 0.03, 0.001, etc, and the initial learning rate for AlexNet was set to 10 -3 , and 3 × 10 -3 for VGG. The learning rates are correspondingly reduced by a factor 10 every 10 epoch for 50 epochs. We set the regularization parameter λ to 10 -6 , and use the regularization term as defined in (4). In these experiments weights are initialized using Glorot & Bengio (2010) initialization. Further, the scales are introduced for each convolution filter, and are initialized by sorting the absolute values of the weights for each filter and choosing the 75 th percentile value. The results are summarized in Table 1 . We evaluate the performance of our training method on two architectures AlexNet and Resnet-18 (He et al., 2016) . Following previous work, we used batch-normalization before each activation function. Additionally, we keep the first and last layers to be in full precision, as we lose 2-3% accuracy otherwise. This approach is followed by other binary methods that we compare to (Hubara et al., 2016; Rastegari et al., 2016; Tang et al., 2017) . The results are summarized in Table 2 . In all the experiments involving R 1 regularization we set the λ to 10 -7 and R 2 regularization to 10 -6 . Also, in every network, the scales are introduced per filter in convolutional layers, and per column in fully connected layers. The weights are initialized using a pre-trained model with htan activation function as done in Liu et al. (2018) . Then the learning rate for AlexNet is set to 2.33 × 10 -3 and multiplied by 0.1 at the 12 th , 18 th epoch for a total of 25 epochs trained. For the 18-layer ResNet the learning rate is started from 0.01 and multiplied by 0.1 at 10 th , 20 th , 30 th epoch.

4.3. DISCUSSION:

We proposed two regularization terms (4) and ( 5) and an activation term (2) with a trainable parameter β. We run several experiments to better understand the effect of the different modifications to the training method, especially using different regularization and asymptote parameters β. The parameter β is trainable and would add one equation through back-propagation. However, we fixed β throughout our experiments to explicit values. The results are summarized in Table 2 .

Through our experiments, we found that adding regularizing term with heavy penalization degrades the networks ability to converge, as the term would result in total loss be largely due to the regu- larizing term and not the target cross entropy loss. Similarly, the regularizing term was set to small values in Tang et al. (2017) . As a result, we set λ with a reasonably small value 10 -5 -10 -7 , so that the scales move slowly as the weights gradually converge to stable values. Some preliminary experimentation was to gradually increase the regularization with respect to batch iterations updates done in training, though this approach requires careful tuning and was not pursued further.

From Table 2 , and referring to networks without regularization, we see the benefit of using Swish-Sign approximation versus the STE. This was also noted in Liu et al. (2018) , where their second approximation provided better results. There is not much difference between using R 1 versus R 2 towards model generalization although since the loss metric used was the cross-entropy loss, the order of R 1 better matches the loss metric. Lastly, it seems moderate values of β is better than small or large values. Intuitively, this happens because for small values of β, the gradient approximation is not good enough and as β increases the gradients become too large, hence small noise could cause large fluctuations in the sign of the weights.

We did not compare our network with that of Liu et al. (2018) as they introduce a shortcut connection that proves to help even the full precision network. As a final remark, we note that the learning rate is of great importance and properly tuning this is required to achieve convergence. Table 3 summarizes the best results of the ablation study and compares with BinaryNet, XNOR-Net, and ABC-Net.

Table 3 : Comparison of top-1 and top-5 accuracies of our method BNN+ with BinaryNet, XNOR-Net and ABC-Net on ImageNet, summarized from Table 2 . The results of BNN, XNOR, & ABC-Net are reported from the corresponding papers (Rastegari et al., 2016; Hubara et al., 2016; Tang et al., 2017) . Results for ABC-NET on AlexNet were not available, and so is not reported. 

5. CONCLUSION:

To summarize we propose three incremental ideas that help binary training: i) adding a regularizer to the objective function of the network, ii) trainable scale factors that are embedded in the regularizing term and iii) an improved approximation to the derivative of the sign activation function. We obtain competitive results by training AlexNet and Resnet-18 on the ImageNet dataset. For future work, we plan on extending these to efficient models such as CondenseNet (Huang et al., 2018) , MobileNets (Howard et al., 2017) , MnasNet (Tan et al., 2018) and on object recognition tasks.

:



Figures and Tables:

Figure 1: Binary training, where arrows indicate operands flowing into operation or block. Reproduced from Guo (2018) (left). A convolutional layer depicting weight histogram progression during the popular binary training. The initial weight distribution is a standard Gaussian (right).

Figure 2: Forward and backward approximations. (Top Left) The true forward and backward functions. (Top Right) BNN training approximation. (Bottom Left) Swish function and its derivative. (Bottom Right) The modified activation, in this case SS(x) is plotted for β = 5.

Top-1 and top-5 accuracies (in percentage) on CIFAR-10, using Manhattan regularization function (4) with AlexNet and VGG. For pre-processing the dataset we follow the typical augmentation: the images are resized to 256 × 256, then are randomly cropped to 224 × 224 and the data is normalized using the mean and standard deviation statistics of the train inputs; no additional augmentation is done. At inference time, the images are first scaled to 256 × 256, center cropped to 224 × 224 and then normalized.

Table content: <table><tr><td/><td colspan="2">BNN+ Full-Precision</td></tr><tr><td>AlexNet</td><td>Top-1 86.49% Top-5 98.92%</td><td>88.58% 99.73%</td></tr><tr><td>VGG</td><td>Top-1 91.31% Top-5 99.09%</td><td>90.89% 99.76%</td></tr><tr><td>4.2 IMAGENET</td><td/><td/></tr><tr><td colspan="3">The ILSVRC-2012 dataset consists of ∼ 1.2M training images, and 1000 classes.</td></tr></table>

Top-1 and top-5 accuracies (in percentage) on ImageNet dataset, of different combinations of the proposed technical novelties on different architectures.

Table content: <table><tr><td colspan="2">Regularization Activation</td><td colspan="2">AlexNet</td><td>Resnet-18</td></tr><tr><td/><td>SS 5</td><td colspan="2">46.11 75.70</td><td>-</td><td>-</td></tr><tr><td>R 1</td><td>SS 10</td><td colspan="3">46.08 75.75 51.13 74.94</td></tr><tr><td/><td>htanh</td><td colspan="3">41.58 69.90 50.72 73.48</td></tr><tr><td/><td>SS 5</td><td colspan="3">45.62 70.13 53.01 72.55</td></tr><tr><td>R 2</td><td>SS 10</td><td colspan="3">45.79 75.06 49.06 70.25</td></tr><tr><td/><td>htanh</td><td>-</td><td>-</td><td>48.13 72.72</td></tr><tr><td/><td>SS 5</td><td colspan="3">45.25 75.30 43.23 68.51</td></tr><tr><td>None</td><td>SS 10</td><td colspan="3">45.60 75.30 44.50 64.54</td></tr><tr><td/><td>SS 20</td><td colspan="3">44.03 68.30 44.74 69.62</td></tr><tr><td/><td>htanh</td><td colspan="3">39.18 69.88 42.46 67.56</td></tr></table>

Peer review (iclr2019):
 Summary: This paper presents three small improvements for training binarized neural networks: (1) a modified straight-through estimator, (2) a novel regularizer to push weights to +/- 1, and (3) the use of scaling factors for the binarized weights. Using the methods presented, the validation accuracies on ImageNet and CIFAR-10 are improved by just under 2 percentage points. Pros: - Decent improvement in the performance of the binarized network in the end - The presented regularizers make sense and seem effective. The modified straight-through estimator seems reasonable as well, although the authors do not compare to recent work with a similar adjustment. Cons: - The paper is poorly written and confusing. It reads as if it was written in one pass with no editing or re-writing to clarify contributions or key points, or ensure consistency. - While the final numbers are acceptable, the experiments themselves could be stronger and could be presented more effectively. - The novelty of the scale factors is questionable. Questions and comments: 1. How exactly is the SS_.beta activation used? It is entirely unclear from the paper, which contradicts itself in multiple ways. Is SS_.beta used in the forward pass at all for either the weight or activation binarization? Or is only its derivative used in the backward pass? If the latter, then you are not replacing the activation anywhere but are simply using a different straight-through estimator in place of the saturated straight-through estimator (e.g., see [1]). (a) At the beginning of Section 3.3, you say that you modify the training procedure by replacing the sign binarization with the SS_.beta activation. This sounds like it is referring to the activation function at each layer; however, the pseudocode says that you are using sign() as the per-layer activation. (b) Further, Figure 4 shows that you are using the SS_.beta function to do weight binarization. However, again, the pseudocode shows that you are using sign() to do the weight binarization. 2. In [1], the authors used a similar type of straight-through estimator (essentially, the gradient of tanh instead of hard_tanh) and found that to be quite effective. You should compare to their method. Also, it*s possible that SS_.beta reduces to tanh for some choice of .beta -- is this true? 3. The use of scale factors seems to greatly increase the number of parameters in the network and thus greatly decrease the compression benefits gained by using binarization, i.e., you require essentially #scale_factors = a constant factor times the number of actual parameters in the network (since you have a scale factor for each convolutional filter and for each column of each fully-connected layer). As a result of this, what is the actual compression multiplier that your network achieves relative to the original network? 4. For the scale factor, how does yours differ from that used in Rastegari et al. (2016)? It seems the same but you claim that it is a novel contribution of your work. Please clarify. 5. Why did learning .beta not work? What was the behavior? What values of .beta did learning settle on? 6. I assume that for each layer output y_i = f(W_i x_i), the regularizer is applied as R(y_i) while at the same time y_i is passed to the next layer -- is this correct? The figures do not clearly show this and should be changed to more clearly show how the regularizer is computed and used, particularly in relation to the activation. 7. In the pseudocode: (a) What does "mostly bitwise operations" mean? Are some floating point? (b) Is this the shift-based batchnorm of Hubara et al. (2016)? 8. For Table 1: (a) I assume these are accuracies? The caption should say. (b) Why are there no comparisons to the performance of other methods on this dataset? (c) Any thoughts as to why your method performs better than the full-precision method on this dataset for VGG? 8. For Table 2: (a) Does Table 2 show accuracies on ImageNet? You need to make this clear in the caption. (b) What type of behavior do the runs that do not converge show? This seems like a learning rate problem that is easily fixable. Are there no hyperparameter values that allow it to converge? (c) What behavior do you see when you use SS_1 or SS_2, i.e., .beta = 1 or .beta = 2? Since lower .beta values seem better. (d) The regularization seems to be the most useful contribution -- do you agree? (e) Why did you not do any ablations for the scale factor? Please include these as well. 9. For Table 3, did you compute the numbers for the other approaches or did you use the numbers from their papers? Each approach has its own pros and cons. Please be clear. 10. Are there any plots of validation accuracy versus epoch/time for the different algorithms in order to ensure that the reported numbers were not simply cherry-picked from the run? I assume that you simply used the weights from the end of the 50th epoch -- correct? 11. Is there evidence for your introductory claims that *quantizing weights ... make neural networks harder to train due to a large number of sign fluctuations* and *maintaining a global structure to minimize a common cost function is important* ? If so, you should cite this evidence. If not, you should make it clear that these are hypotheses. 12. Why are there not more details about the particular architectures used? These should be included in the appendices to aid those who would like to rerun your experiments. In general, please include more experiment details in the body or appendices. Detailed comments: - R(l) is not defined in Figure 1 and thus is confusing. Also, its replacement of *Error* from the original figure source makes the figure much more confusing and less clear. - Typos: - *accustomed* (p.1) - *the speed by quantizing the activation layers* doesn*t make sense (p.1) - *obtaining* (p.4) - *asymmetric* doesn*t make sense because these are actually symmetric functions across the y-axis (p.4) - *primary difference is that this regularization ...* --> *primary difference is that their regularization ...* (p.4) - *the scales with 75th percentile of the absolute value ... * is very confusing and unclear (p.7) - *the loss metric used was the cross-entropy loss, the order of R_1.* I do not know what you*re trying to say here (p.8) - Citations: Fix the capitalization issues, typos, and formatting inconsistencies. [1] Friesen and Domingos. Deep Learning as a Mixed Convex-Combinatorial Optimization Problem. ICLR 2018. ------------------- After reading the author response, I do not think the paper does a sufficient job of evaluating the contributions or comparing to existing work. The authors should run ablation experiments, compare to existing work such as [1], and evaluate on additional datasets. These were easy tasks that could have been done during the review period but were not. If I wanted to build on top of this paper to train higher accuracy binary networks, I would have to perform all of these tasks myself to determine which contributions to employ and which are unnecessary. As such, the paper is currently not ready for publication.