Random Bias Initialization Improving Binary Neural Network Training
Authors: Xinlin Li
Vahid Partovi Nia
Venue:  (2020)
Keywords: Binarized Neural Network
Activation function
Initialization
Neural Network Acceleration

Abstract: Edge intelligence especially binary neural network (BNN) has attracted considerable attention of the artificial intelligence community recently. BNNs significantly reduce the computational cost, model size, and memory footprint.  However, there is still a performance gap between the successful full-precision neural network with ReLU activation and BNNs. We argue that the accuracy drop of BNNs is due to their geometry. 
We analyze the behaviour of the full-precision neural network with ReLU activation and compare it with its binarized counterpart. This comparison suggests random bias initialization as a remedy to activation saturation in full-precision networks and  leads us towards an improved BNN training. Our numerical experiments confirm our geometric intuition.

Under review as a conference paper at ICLR 2020
RANDOM BIAS INITIALIZATION IMPROVING BINARY NEURAL NETWORK TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Edge intelligence especially binary neural network (BNN) has attracted consider- able attention of the artificial intelligence community recently. BNNs significantly reduce the computational cost, model size, and memory footprint. However, there is still a performance gap between the successful full-precision neural network with ReLU activation and BNNs. We argue that the accuracy drop of BNNs is due to their geometry. We analyze the behaviour of the full-precision neural network with ReLU activation and compare it with its binarized counterpart. This com- parison suggests random bias initialization as a remedy to activation saturation in full-precision networks and leads us towards an improved BNN training. Our numerical experiments confirm our geometric intuition.
INTRODUCTION
Convolutional neural network has become of the most powerful tools for solving computer vi- sion, natural language processing, speech recognition, machine translation, and many other complex tasks. The most successful and widely-used recipe for deep neural network ReLU-style activa- tion function with MSRA style weight initialization (He et al. 2015). The standard sigmoid and the hyperbolic tangent were the most common activation functions, before the introductio of ReLU. ReLU-like activation functions are widely proved to be superior in terms of accuracy and conver- gence speed.
It is more common to use low-bit quantized networks such as Binary Neural Networks (BNNs) (Courbariaux et al., 2016) to implement such deep neural networks on edge devices such as cell phones, smart wearables, etc. BNNs only keeps the sign of weights and compute the sign of acti- vations (-1,+1 by applying Sign function in the forward pass. In backward propagation, BNN uses heitt--Throuhh--timme tion and update on full-precision weights. The forward and backward loop of BNN, therefore, becomes similar the full-precision neural network with hard hyperbolic tangent htanh activation. The htanh function is a piece-wise linear version of the nonlinear hyper-bolic tangent, and is known to be inferior in terms of accuracy compared to ReLU-like activation function.
(STE) to estimate the backward gradient through the Sign func-
We examine full-precision network with htanh activation to provide new look in improving BNN performance. We conclude that the bias initialization the key to mimic ReLU geometric behavior in networks with htanh activation. This conclusion challenges the common practice of deterministic bias initialization for neural networks.
Although the analysis is based on htanh function, this conclusion equally applies to BNNs that use STE, a htanh-like, back propagation scheme. Other saturating activations like hyperbolic tangent and sigmoid commonly applied in recurrent neural networks may benefit from this resolution as well.
Our novelties can be summarized in four items i) we analyze the geometric properties of ReLU and htanh activation. This provides an insight into the training efficiency of the unbounded asymmetric activation functions such as ReLU. ii) we propose bias initialization strategy as a remedy to the bounded activations such as htanh. iii) We back up our findings with experiments on full- pull-precision to reduce the performance gap between htanh and ReLU activations. iv) We show this strategy also improves BNNs, whose geometric behavior is similar to the aellsireiision neural network with htanh activation.

---

Under review as a conference paper at ICLR 2020
There are very few works that focus on the initialization strategy of the bias term of the neural network. To the best of our knowledge, we are the first to propose random bias initialization as a remedy to the saturating full fuel-pireiiion neural network, also as method to improve BNN training.
2 RELATED WORKS
Glorot et al. 2011) proposed training deep neurals network with ReLU activation, and argued that ReLU activation alleviates the vanishing gradient problem and encourages sparsity in the model. The hyperbolic tangent only allowed training of shallow neural networks.
Since AlexNet (Krizhevsky al. 2012), almost every successful neural network architectures
ReLU activation or its variants, such as adaptive ReLU, leaky ReLU, etc. Although many works reported that ReLU activation outperforms the traditional saturating activation functions, the reason for its superior performance remains an open question.
RRamachandran et al., 2017) utilized automatic search techniques on searching different activation functions. Most top novel activation functions found by the searches have an asymmetric saturating regime, which is similar to ReLUU Farhadi et al. (2019) adapts ReLU and sigmoid while training- To improve the performance of saturating activations, Xu et al. (2016) proposed penalized tanh activation, which introduces asymmetric saturating regime tanh by inserting leaky ReLU before tanh. The penalized tanh could achieve the same level of performance ReLU activating CNN. It worth to mention that similar ideas also appear in the related works of binarized neural network.
Gulcehre et al. (016)) improved the performance of saturating activations by adding random noise when the neuron is saturated, so the backward signal can easily pass through the whole model, and the model becomes easier to optimize. In this works, proposed to randomize the non-saturated regime by using random bias initialization. This initialization can guarantee all backward signals can pass through the whole model equally.
The initial work on BNN appeared in Courbariaux et al. (2016), which limits both weights and activations 11 and +1, so the weighted can be computed by bit-wise XNOR and PopCount instructions. This solution reduces memory usage and computational cost up to 32X compared with its full-precision counterpart. In the original paper, BNN tested on VGG-7 architecture. Although it is an over-parameterized architecture for CIFAR 10 dataset, there is a performance gap between BNN and full-precision with ReLU activation. We believe the different between the two activations, BNN using the sign and full-precision using ReLU, is partially responsible for this gap-
XNOR-Net (Rastegari et 2016) developed the idea of BNN and proposed to approximate the full-precision neural network by using scaling factors. They suggest inserting non- an-Binary activation (like ReI.U) after the binary convolution layer. This modification helps training considerably. Later, Tang et al. replaced replacing ReLUU activation with PReI. in XNOR-Net to improve the accuracy. Note that XNOR-Net and many relaated works require to store the full-precision activation map during the inference stage, therefore their memory occupation is significantly larger than the pure 1-bit solution like the vanilla BNN.
GEOMETRIC ANALYSIS
A typical full-precision neural network block can be described by
x x1+1 ReLU + 6
(1)
W gmxx 6² RM x' E R", 2 T
Neural networks are trained using the prok-prrpnggtton algorithm. Back propagation is composed of two components i) forward pass and ii) backward propagation. In the forward pass, the loss function 2(-) is evaluated on the current weights, and in backward propagation, the weights are updated sequentially.
3.1 FORWARD
To simplify the analysis, we assume that all weight vectors Wi have unit norm. It is a reasonable assumption when the network has batch normalization layers in which all neuron responses
2

---

Under review as a conference paper at ICLR 2020
Activated w/x' + b/j
Activated Region
Non-activated region Wix' + bj <0
w
Non act-vacttvted Region
W/x + bj =0
Figure 1: Activated and non-activated regions of ReLU (left panel). Activated region of ReLU at initialization (right panel).
W/xx bf
x
w!
bj
x/1
Wxxx + b,s
Figure 2: Geometric behavior of ReLu during forward pass, trained hyperplanes (left panel) and their geometry (right panel).
normalized, as the magnitude of the weight vectors does not affect the layer output. The j neuron response in the (i + 1)th layer are computed as
(W3x' + +b 0
Wfs' + 6 0
Wi'' + by I
First, the input data points I' are projected to the jth row vector of the weight matrix. The dot product Wi and z' are cut by the corresponding bias term bj, i.c. the output zf+1 set to zero the dot product is smaller than the threshold, Figure (left panel). A hyper-plane whose normal direction defined by Wi divides the input space into two parts: i) activated region (non-saturated regime) and ii) non-activated region (saturated regime), see Figure] [ If the data point positive side of a hyper-plane (activated region), the hyper-plane activated by o². Consequently, a positive. Otherwise, 2//+1 (+1 equals zero and remain deactivated.
x' falls on the
The weight matrix W of size m X 12 and the bias vector of size m x define m hyper-planes in the n-dimensional input space, see Figure 2 (right panel).

---

Under review as a conference paper at ICLR 2020
3.2 BACKWARD
During backward propagation, the backward gradient update on W' and x' are computed using
de
Ac +T
Jwj
de
de
de
dr
(3)
d
dxj
dr
de
dz
For the case of ReLU activation
dx dx 1+1
xx
W7z* b)
3
(4)
dWj
Wzzt + bj 0
dx i+1
[1
Wfat + + >
(5)
db3
00
W3r² + 50
dæ 1+1
[Wj
(6)
dx
0
W3 W
The activation function only allows the gradients from data point on the activated region to backward propagate and update the hyper-plane (equation
3.3 RELU ACTIVATION
From the hyper-plane analysis, we realize that ReLU activation has three ideal properties that distinguishing it from the others i) the diversity of activated regions at initialization, ii) The equality of data points at initialization, iii) The equality of hyper-planes at initialization. These may explain why ReLU activation outperforms the traditional Hyperbolic tangent or sigmoid activations. To argue each property, us suppose that the distribution of the dot products zero-centered. This assumption is automatically preserved in the batch normalization. Weight initialization techniques, like Xavier and MSRA, randomly initialize the weights to maintain the output variance.
i) Region diversity: the activated regions of hyper-planes solely depend on the direction of the weight vector, which randomly initialized. This allows different hyper-planes to learn from a different subset of data points, and ultimately diversifies the backward gradient signal. ii) Data equality: an arbitrary data point x', is located on the activated regions of approximately half of the total hyper- ppeer-saaees in layer In other words, the backward gradients from all data points pass through the approximately same amount of activation function, update hyper-planes, and propagate arbitrary hyper-plane Wj. is affected by the backward
the gradient. iii) Hyperplane equality: gradients from approximately 50% of the total data points. All hyper-planes on average receive the same amount of backward gradients. Hyper-plane equality speeds up the convergence and facilitates model optimization, see Figure T (right panel).
3.4
HTANH ACTIVATION
Similar to the ReLU activation, only the backward gradients from data points located in the activated region of hyper-plane can backward propagate through the activation function and update this hyper-plane. This analysis also applies to htanh activation. The performance gap between ReLU activation and htanh activation is caused by their different activated region distribution, see Figure Clearly, htanh activation is not as good as ReLU defining balanced and fair activated regions. However, we analyze each property for htanh as well. i) Region diversity: activated regions of htanh are not as diverse as ReLU. Activated regions of htanh cover only the area close to the origin. Assuming Gaussian data, this is dense that the majority of data points are located in. ii) Data equality: data points are not treated fairly htanh activation function. Data points that closer to the origin activate more hyper-planes than the data points far from the origin. If the magnitude
4

---

Under review as a conference paper at ICLR 2020
Non-activated region W/2 16 >
Activated Region
Non voonvactivated Region
Activated region < W 19
Nan-octivated region Wjs + 14 --
+
Nor-activated Region
w/st
Figure 3: Activated region and non-activated region htanh activation function(left panel). Acti- vated region Hard Tanh at initialization (right panel)
of data point x² is small enough, red region of Figure 3 (right panel). As a consequence, in backward gradients, few data instances affect all hyper-planes. In other words, the backward gradients from part of the training data points have a larger impact on the model than the others. We considered this imbalance ultimately affects model generalization problem since the model training focuses on a subset of the training data points close to the origin. iii) Hyperplane equality: The initial activated regions should cover similar-sized subset of the training data points overall, and this property shared both ReLU and htanh activations.
activate all hyper-planes in the same layer, the deep-
Similar analysis also applies to other activation functions with the zero-centered activated region, like Sigmoid or Hyperbolic Tangent.
4 BIAS INITIALIZATION
Here we proposed simple initialization strategy to alleviate the data inequality issue and improve activated region diversity for the htanh activation relying on our geometric insight. We argue bias initialization with uniform distribution between [-A, where Å is hyper-yaranen parameter is solution region diversity. With random bias initialization, the data points that far from the origin can activate more hyper-planes. If A max( [=xl]] + 1, all data points activate approximately the same number of hyper-planes during backward propagation, so data equality can achieved. Also, with the diverse initial activated region, different hyper-planes learn from different subset of training data points.
However, this initialization strategy comes with drawback. Hyper-plane equality longer holds when the biases are not set to zero. Hyper-planes with larger initial bias have less activated data. Therefore, choosing the optimal value of À is trade-off between the hyper-plane equality and the data equality. Experiments below shows that the validation curve becomes unsteady if X value set too high. Empirically, with batch normalization layer, À s 2 provide a good initial estimate. In this case, the activated regions covering from -3 to +3, so it allows the gradients from almost all data points propagate. It's worth to mention that the experiments showed that small À also helps to improve the performance of ResNet architecture.
5 EXPERIMENTS
The proposed bias initialization method is evaluated on the CIFAR-10. based the original implementation of the BNN (Courbariaux al. 2016). We choose the VGG-7 architecture and the ResNet architecture.
10. The network architectures
The VGG-7 architecture. is simple and over- parameterized model for CIFAR 10. This is ideal architecture to compare the performance between different activations. In the original implementa- tion, this full-precision architecture is designed to compare with BNN, the BatchNorm layers
5

---

Under review as a conference paper at ICLR 2020
Activations
VGG-7
ResNet
ReLU (Baseline)
6.98
9.45
htanh
10.91
10.63
htanh (A=0.5)
9.99
9.87
htanh (A=1.0)
9.15
10.47
htanh (A=1.5)
8.36
10.13
htanh (A=2.0)
7.98
10.23
htanh (A=2.5)
7.83
9.84
Table 1: Validation error rate % for full- full-precision training, X = 0 coincides with common determin- istic initialization.
Binary VGG-7
Binary ResNet
0.0
10.77
23.11
0.5
9.57
22.31
1.0
9.17
22.83
1.5
8.57
22.56
2.0
8.56
22.47
2.5
8.53
22.19
3.0
8.48
22.30
Table 2: Validation error rate % for Binary training, X coincides with common deterministic initialization.
inserted after each ReLU activation match the pattern of BNN. This arrangement has a negative impact on the performance of the full-precision neural network. In our full- full-precision experiments, we put the BatchNorm layers back to their original position, so the error rate of the baseline model improved from the original reported 9.0% to 6.98%. This is close to 6.50%, which is the best CIFAR 10 error rate reported on VGG-9 architecture that includes BatchNorm. Considering VGG-9 keep it the baseline for full-precision VGG. We also followed the training recipe from the original implementation, SGD optimizer with 0.9 mo- mentum and weight decay convergence caused by htanh activation.
has much more parameters than this architecture,
5 10-4, 4. We
training epochs compensate for the slow
Figure confirms that the random bias initialization strategy helps reduce the performance gap between htanh and ReLU activation. A similar effect is observed for ResNet type architectures.
We also tested the proposed bias initialization the ResNet-like architecture. The results de- picted in Figure 4 re-assures that bias initialization improves htanh and pushes it toward ReLU accuracy, see Table
Binary training that use STE is similar to htanh activation. We expect to observe similar effect in BNN training with STE gradient approximator. The validation error rate is summarized in Table In the Binary VGG-7 experiments, we reduced the accuracy gap between full- nell-precision network
CHAAKD Phe
CIAARLS anr rdu V30-13
CEARID vis IBNV Moteo)
H867 (10.))
45
25
400chs
Figure 4: Training of full-precision ResNet architecture (left panel) Binary VGG-7 architecture (middle panel), and Binary ResNet (right panel)

---

Under review as a conference paper at ICLR 2020
with ReLU activation and BNN from 4% 1.5%. The bias initialization strategy is effective to close the gap on binary ResNet architecture by almost 1,, even while the full-precision model even under-fits on CIFARIO data.
REFERENCES
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or 1. 2016.
Farnoush Farhadi, Vahid Partovi Nia, and Andrea Lodi. Activation adaptation neural networks. arXiv preprint arXiv: 19X1:9990 2019.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of foureenth international conference on artificial intelligence and statistics, pp- 315-323 2011.
Caglar Gulcehre, Marcin Moczulski, Misha Denil, and Yoshua Bengio. Noisy activation functions. In International conference on machine learning, pp. 3059-3068, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Prajit Ramachandran. Barret Zoph, and Quoc Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. European Conference on Computer Vision, pp. 525-542. Springer, 2016.
Wei Tang, Gang Hua, and Liang Wang. How to train compact binary neural network with high accuracy? In Thirly-First AAAI Conference on Artificial Intelligence, 2017.
Bing Xu, Ruitong Huang, and Mu Li. Revise saturated activation functions. arXiv preprint 100.0080000 2016.


Reviews:
[{"review_id": "H1lAIDb-Kr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for bias initialization and shows that it improves training for BNN.\n\nI vote to reject the paper. Main points against are: (1) is no theory and very limited experiments (2) Bad writing.\n\nDetailed remarks:\n - The level of english is not good enough all over the paper, example: \"It is more common to use low-bit quantized networks such as Binary Neural Networks (BNNs)״ more common then what? (I also disagree on the scientific claim)\n- The authors claim that XNOR nets and such have \"memory occupation is significantly larger than the pure 1-bit solution like the vanilla BNN.\". While this is true for training, it is not true for inference which is in many cases where one needs to use limited hardware.\n- The paper main claim is the data equality and hyperplane equality are the main strengths of ReLU, but doesn't give any justification or even intuition into why this is the case. I am not convinced that these points are important, and the paper did nothing to try to persuade me.\n- Data point equality shouldn't hold for ReLU networks with non-zero bias initialization as well.\n- The experiments show promising results but only on cifar10 and only with the outdated BNN, also as a necessary baseline it would be important to show the effect of the bias initialization on ReLU networks. \n\n\nI believe the paper shows promising initial results but needs to strengthen them considerably. It also needs to improve the writing. A better justification for the method, even if it only at an intuitive level would help considerably.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}, {"review_id": "Byg1CKm9KH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper tries to improve the training for the binary neural network.\n\nWeaknesses:\n[-] A lack of related works. There have been many related works about BNN in these years (after 2017), but the authors do not have a quick summary of them.\n[-] More reference. e.g, when authors mention 'many related works require to store the full-precision activation map during the inference stage',  some reference is necessary.\n[-] Weak Motivation: The authors argue 'We analyze the behaviour of the full-precision neural network with ReLU activation' in the abstract. However, in Section 3, I cannot find any analysis. Only writing down the backward and forward cannot be called analysis. Initialization is different from the training dynamics. Assumptions and theorems should be highlighted. \n[-] Poor writing: A lot of typos. Only in the last paragraph in Section 2, I find many typos,  e.g. 'replaced replacing ReLU activation', 'any relaated works'.\n\nQuestions:\n[.] In experiments, what structure is used for ResNet? ResNet-18-like or ResNet-110-like? (The results for these two kinds of structure are totally different for binary neural network, as the difference in the number of channels) \n[.] In experiments, the performance of the baselines seems lower than related papers? Do the authors increase the number of channels in each layer as the other people do? It can improve the result a lot, and I wonder whether the improvement still exists in this setting.\n[.] In experiments, only CIFAR10 results have been reported, but I wonder what is the error bar looks like? (Do the authors run the experiments several times and calculate the variance?)\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJeE7ZOaKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to initialize the bias terms in neural network layers, and argues that the proposed method improve the performance of binary neural networks (BNNs). The paper justifies the proposed method by analyzing the geometric properties of the ReLU and the hard tanh (htanh) activation functions, as well as by empirical results on the CIFAR-10 dataset using the (binary variants) of VGG-7 and ResNet. \n\nWhile closing the performance gap between BNNs and their full-precision counterparts is an interesting problem of practical importance, this paper has several limitations: \n\n(1) the analysis of geometric properties of ReLU/htanh is not sufficiently precise and clear;\n(2) the paper does not clearly present the connections between the htanh activation function and the straight-through estimator employed in back-propagating the gradients in training a BNN;\n(3) the experimental results are too limited on just one dateset, and only error rate on validation set is reported, however, lower error rate on validation set won't guarantee better performance on test set;\n(4) the presentation is imprecise and unpolished.\n\n\nMinor comments:\n\nSection 2: \n\"Tang et al. replaced replacing ReLU\" -> \"Tang et al. replaced ReLU\"\n\"many relaated works\" -> \"many related works\"\n\nSection 3: \nplease define the symbols used in Equation (1)\ntitle of Figure 2: \"behavior of ReLu\" -> \"behavior of ReLU\"", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The article studies the behaviour of binary and full precision ReLU networks towards explaining differences in performance and suggests a random bias initialisation strategy. The reviewers agree that, while closing the gap between binary networks and full precision networks is an interesting problem, the article cannot be accepted in its current form. They point out that more extensive theoretical analysis and experiments would be important, as well as improving the writing. The authors did not provide a rebuttal nor a revision. 
Decision: Reject