Enabling Binary Neural Network Training on the Edge
Authors: Erwei Wang
James J. Davis
Daniele Moro
Piotr Zielinski
Claudionor Coelho
Satrajit Chatterjee
Peter Y. K. Cheung
George Anthony Constantinides
Venue:  (2021)
Keywords: Binary neural network
edge computing
neural network training

Abstract: The ever-growing computational demands of increasingly complex machine learning models frequently necessitate the use of powerful cloud-based infrastructure for their training. Binary neural networks are known to be promising candidates for on-device inference due to their extreme compute and memory savings over higher-precision alternatives. In this paper, we demonstrate that they are also strongly robust to gradient quantization, thereby making the training of modern models on the edge a practical reality. We introduce a low-cost binary neural network training strategy exhibiting sizable memory footprint reductions and energy savings vs Courbariaux & Bengio's standard approach. Against the latter, we see coincident memory requirement and energy consumption drops of 2--6$\times$, while reaching similar test accuracy, across a range of small-scale models trained to classify popular datasets. We also showcase ImageNet training of ResNetE-18, achieving a 3.12$\times$ memory reduction over the aforementioned standard. Such savings will allow for unnecessary cloud offloading to be avoided, reducing latency and increasing energy efficiency while also safeguarding user privacy.

Under review as a conference paper at ICLR 2021
ENABLING BINARY NEURAL NETWORK TRAINING ON THE EDGE
Anonymous authors Paper under double-bling review
ABSTRACT
The ever-growing computational demands of increasingly complex machine learn- ing models frequently necessitate the use of powerful cloud-based infrastructure for their training. Binary neural networks are known to be promising candidates for on-device inference due to their extreme compute and memory savings over higher-precision alternatives. In this paper, we demonstrate that they also strongly robust to gradient quantization, thereby making the training of modern models on the edge practical reality. We introduce low-cost binary neural net- work training strategy exhibiting sizable memory footprint reductions and energy savings vs Courbariaux Bengio's standard approach. Against the latter, we see coincident memory requirement and energy consumption drops of 2-6x, while reaching similar test accuracy in comparable time, across a range of small-scale models trained to classify popular datasets. We also showcase ImageNet training of ResNetE-18 18, achieving a 3.12x memory reduction over the aforementioned standard. Such savings will allow for unnecessary cloud offloading to be avoided, reducing latency and increasing energy efficiency while also safeguarding privacy.
INTRODUCTION
Although binary neural networks (BNNs) feature weights and activations with just single-bit preci- sion, many models are able reach accuracy indistinguishable from that of their higher-precision counterparts (Courbariaux & Bengio 2016 Wang et al., 2019b). Since BNNs are functionally com- plete, their limited precision does not impose an upper bound on achievable accuracy (Constan- tinides] 2019). BNNs represent the ideal class of neural networks for edge inference, particularly for custom hardware implementation, due to their use of XNOR for multiplication: a fast and cheap operation to perform. Their of compact weights also suits systems with limited memory and increases opportunities for caching, providing further potential performance boosts. FINN, the sem- inal BNN implementation for filld-progrmmmable gate arrays (FPGAs), reached the highest CIFAR- 10 and SVHN classification rates to date at the time of its publication (Umuroglu et al. 2017).
Despite featuring binary forward propagation, existing BNN training approaches perform back- ward propagation using -eeecpprciioon floating-point data typscalyp float (2oat22-p0 32oofnn mak- ing training infeasible on resource-coue cooourc-constraiieed devices. The high-precision activations used between forward and backward propagation commonly constitute the largest proportion of the total memory footprint of training run (Sohoni al. 2019 Cai et al. 2020). Additionally, backward propagation with nigh-precision gradients is costly, challenging the energy limitations of edge platforms.
An understanding of standard BNN training algorithms led to ask two questions: why are high- precision weight gradients used when we are only concerned with weights' signs, and why are high- precision activations used when the computation of weight gradients only requires binary activations input? In this paper, we present low-memory, low-energy BNN training scheme based on this intuition featuring (i) the use of binary, power-of-two and 16-bit floating-point data types, and (ii) batch normalization modifications enabling the buffering of binary activations.
By increasing the viability of learning on the edge, this work will reduce the domain mismatch be- tween training and inference- part-eulaallarlyyy in conjunction with federated learning (McMahan al.)
2017 Bonawitz et al. 2019) and ensure privacy for sensitive applications (Agarwal et al. (018)) Via the aggressive energy and memory footprint reductions they facilitate, our proposals will enable

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison of applied approximations vs related low-cost neural network training works.
Weight gradients
Activation gradients
Batch
Weights
Activations
normalization
Zhou et al. 2016
int 6
int6
int6
int6
x
Gruslys et al.
(2016)
x
x
Recomputed
x
x
Ginsburg et al. (2017
f
float16
float16 int
float16
X
Graham 2017)
x
x
x
X
Bernsten ct al. (2018)
x bool
x
Wu et al.
20018b)
x bool
x
x
1
This work
bool
bool
po23
BNN-specific 4
Arbitrary precision was supported, but significant accuracy degradation was observed below 6 bits. Activations not retained between forward and backward propagation in order memory.
Power-of-two format comprising sign bit and exponent.
networks to be trained without the network access reliance, latency and energy overheads or data divulgence inherent to cloud offloading. To this end, we make the following novel contributions.
We conduct the first variable representation and lifetime analysis of the standard BNN training process, informing the application of beneficial approximations. In particular, we
binarize weight gradients owing to the lack of importance of their magnitudes, modify the forward and backward batch normalization operations such that remove the need to buffer high-precision activations and
determine and apply appropriate additional quantization schemes-power--de pfwwroof-owwwoooooooooooooo----- acti- vation gradients and reduced- preccssion doating-poiin data-taken from the literature.
Against[Courbarte Courbariaux & Bengio (2016)'s 's approach, we demonstrate the preservation of test accuracy and convergence rate when training BNNs to classify MNIST, CIFAR- 10, SVHN and ImageNet while lowering memory and energy needs by up to 5.67 x and 4.53
We provide an open-source release of our training software, along with our memory and energy estimation tools, to the community
2
RELATED WORK
The authors of all published works BNN inference acceleration to date made use of high- precisori floating-point data types during training (Courbariaux et al. [005 Courbariaux & Bengio 2016; Lin et al. 2017 Ghasemzadeh al. 2018 Liu et al. 2018 Wang al. 2019a 2020 Umuroglu al. He et al. 2020; Liu et al. 20200.. There is precedent, however, for the use of quantization when training ninabynary networks, we show in Table via side-by-side comparison of the ap- proximation approaches taken in those works along with that proposed herein.
2020
The effects of quantizing the gradients of networks with hhih--rreisse point, have been studied extensively. Zhou et al.] (2016) and Wu et al. (2018a) trained networks with fixed-point weights and activations using fixed-point gradients, reporting no accuracy loss for AlexNet classifying ImageNet with gradients wider than five bits. [een et al. (2017) and Bernstein al. (2018) focused solely on aggressive weight gradient quantization, aiming to reduce commu- nication costs for distributed learning. Weight gradients were losslessly quantized into ternary and binary formats, respectively, with forward propagation and activation gradients kept at high preci- sion. In this work, make the novel observations that activation gradient dynamic range is more important than precision, and that BNNs are more robust to approximation than higher- precision net- works. We thus propose data representation scheme more aggressive than all of the aforementioned works combined, delivering large memory and energy savings with near-lossless performance.
data, either fixed floating
higher-precision
Gradient checkpointing- the recomputation of activations during backward propagation has been proposed as a method to reduce the memory consumption of training (Chen et al.) 2016 Gruslys
Source supplied in z2ipp for review.
2

---

Under review as a conference paper at ICLR 2021
Y
X
8gn
BN
X(+1
8.X
0Y
0X1+1
W
sgn
aw
W
Figure 1: Standard BNN training graph for fully connected layer 1. "sgn," " "x" and "BN" are sign, matrix multiplication and batch normalization operations. Forward propagation dependencies are shown in black; those for backward passes are gray.
et al. al., 2016). Such methods introduce additional forward passes, however, and increase each run's duration and energy cost. Grahamm (011)) and Chakrabarti & Moseley (2099) saved memory during training by buffering activations in low-precision formats, achieving comparable accuracy to all-float 32 baselines. Wu et al. (201 8b) and Hoffer et al. (2018) reported reduced computational costs via batch normalization. Finally, Helwegen et (2019) asserted that the of both train- able weights and momenta is superfluous in BNN optimizers, proposing a weightless BNN-specific optimizer, Bop, able to reach the same level of accuracy as Adam. We took inspiration from these works in locating sources of redundancy present in standard BNN training schemes, and propose BNN-specific modifications to batch normalization allowing for activation quantization all the way down to binary, thus saving both memory and energy without inducing latency increases.
STANDARD TRAINING FLOW
For simplicity, we assume the use of a multi-layer perceptron (MLP), although the presence of convolutional layers would not change any of the principles that follow. Let W, and X, denote matrices of weights and activations, respectively, in the network's 1th layer, with OW; and ax; being their gradients. For Wi, rows and columns span input and output channels, respectively, while for X, they represent feature maps and channels. Henceforth, we use decoration to indicate low- precision data representation, with and respectively denoting binary and power-of-two encoding.
Figure [ shows the training graph of a fully connected binary layer. A detailed deseription of the standard BNN training procedure introduced by [ourbariaux & Bengio (2016) for each batch of B training samples, which we henceforth refer to as as a step, is provided in Algorithm [ Therein, "@" signifies element-wise multiplication. For brevity, implementation lack of first-layer quantization, use of a final softmax layer and the inclusion of weight gradient cancellation (Courbariaux Bengio 2016)
omit some of the intricacies of the baseline
these standard BNN practices are
not impacted by our work. We initialize weights as outlined by Glorot & Bengio (210)).
Many authors have found that BNNs require batch normalization in order avoid gradient explo- sion (Alizadeh et al. 2018 Sari et al. 2019 Qin et al. 2020), and our early experiments confirmed this to indeed be case. We thus apply as standard. Matrix products are channel-wise batch- normalized across each layer's M2 output channels (Algorithm) linns (1 to form the subsequent layer's ' inputs, X 3 constitutes the batch normalization biases. Layer- moving means μ(y)) and standard deviations (3)) are retained for use during backward propagation and inference. We forgo trainable scaling factors, commonly denoted 7; these are of irrelevance to BNNs since their activations are binarized prior to during forward propagation (line 2
VARIABLE ANALYSIS
In order to quantify the potential gains from approximation, we conducted a variable representation and lifetime analysis of Algorithm| following the approach taken (Shoni et al. (2019). Table
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Standard BNN training step.
Algorithm 2 Proposed BNN training step.
1: for < {1,... I - 1} do
Forward
1: for
{1,... L 1} do
b Forward
2: X, < sgn(X;)
2:
X; sgn Xi)
3: Wi- sgn Wi)
Wi < sgn Wi)
4:
Yi + X Wi
4: Yi +- W,
5: for m + {1.
Mi do 1 82 mL)
for + {1,
Mi do mm)
(m))
m)
(1))
3
6:
31+1
01+1
| [m)
7:
for [
1
Backward
7: for
6 [ 1, } do
D Backward
8:
for m +- {1, ... Mi do
for m {1, ..... Mi do
9:
Mi
m)
8x (m)
9:
1
mm)
Ox: fm)
1+1
Oy
(m)
v
((()
10:
dy (m)
v p(v)
(m) (m)
#
241
(m)
/ $
1
x 1+1
31+1
11:
mm(m)
Σ 0m
(m)
11:
Σ
dzt+1
(m)
12:
0x, + 0Y,T
12:
0Y < (02(
OW, X
13:
0x: WY
14: for < -1....,,.--11 do
---1)
b Update
14:
OW; + XTOY,
Wi - Optimize( Wi, @Wi,n)
15:
OW; +- sgn(äW,)
16:
3 + Optimize (3, 03,,
16:
for
<- {1.... , 1 do
Update
17: 17 LeamingRateSchecuul((()
17:
Wi < Optimize (W," aw./ M.
18:
Bi + Optimize (Br, 0B.,1) 17)
19: 17 LeamingRateSchhcuul((()]
lists the properties of all variables in Algorithm with each variable's contributions to the total footprint shown for a representative example. Variables are divided into two classes; those that must remain memory between computational phases (forward propagation, backward propagation and weight update), and those that need not. This is of pertinence since, for those in the latter category, only the largest layer's contribution counts towards the total memory occupancy. For example, aX; is read during the backward propagation of layer / only, thus 0X;-1 can safely overwrite aX; for efficiency. Additionally, Y and 0x are shown together since they are equally sized and only need to reside in memory during the forward and backward pass for each layer, respectively.
LOW-COST BNN TRAINING
As shown Table 2 all variables within the standard BNN training flow use float 32 represen- tation. In the subsections that follow, we detail the application of aggressive approximation specifi- cally tailored to BNN training. Further to this, and in line with the observation by many authors that float1 6 can be used for ImageNet training without inducing accuracy loss (Ginsburg al. 2017 Wang et al. 2018 Micikevicius et al. 2018). we also switch all remaining variables to this format. Our final training procedure is captured in Algorithm 2 with modifications from Algorithm in red and the corresponding data representations used shown in bold in Table 2 We provide both theoreti- cal evidence and training curves for all of our experiments in Appendix to show that our proposed approximations do not cause material degradation to convergence rates.
5.1 GRADIENT QUANTIZATION
Binarized weight gradients. Intuitively, BNNs should be particularly robust to weight gradient binarization since their weights only constitute signs. On line [5] of Algorithm 2 therefore, we quantize and store weight gradients in binary format, aw, for use during weight update. During the latter, we attenuate the gradients by VNz, where N, is layer I's fan-in, reduce the learning rate
4

---

Under review as a conference paper at ICLR 2021
Table 2:
Memory-rclated properties of variables used during training. To obtain the exemplary quantities of total memory given, BinaryNet was trained for CIFAR-10 classification with Adam.
Per-layer lifetime
Standard training
Proposed training
Variable
Data type
Size (MiB)
%
Data type
Size (MiB)
Saving (x)
X
x
31
111.33
26.18
bool
3.48
32.00
0x, Y
V
f1oat
50.00
11.76
float16
25.00
2.00
H
yi
X
float32
0.01
0.00
float16
0.01
2.00
(
X
float32
0.01
0.00
float16
0.01
2.00
OY
V
float32
50.00
11.76
po2 5
7.81
6.40
W
X
float32
53.49
12.58
float16
26.74
2.00
OW
X
float32
53.49
12.58
bool
1.67
32.00
X
float32
0.01
0.00
float16
0.01
2.00
0B
X
float32
0.01
0.00
float16
0.01
2.00
Momenta
X
float32
106.98
25.15
float16
53.49
2.00
Total
425.33
100.00
118.23
3.60
Variables that need not be retained between forward, backward or update phases of Algorithms and 2 OXX and Y can share memory since they are equally sized and have non non-overlapping lifetime. 5-bit power-of-twe format with 4-bit exponent.
and prevent premature weight clipping as advised by Sari et al. (2019). Since fully connected layers used an example in Algorithm N₂ Mi-1 1 in this instance.
Table shows that, with binarization, the portion of our exemplary training run's memory con- sumption attributable to weight gradients dropped from 53.49 to just 1.67 MiB, leaving the scarce resources available for more quantization-sensite variables such as W and momenta. Energy con- sumption will also decrease due the associated reduction in memory traffic.
Power-of-two activation gradients. The tolerance of BNN training to weight gradient binariza- tion further suggests that activation gradients can be aggressively approximated without causing significant accuracy loss. Unlike previous proposals, in which activation gradients were quantized into fixed- or block floating-poin point formats (Zhou et al. 2016 Wu et al. 2018a), we hypothesize that power-of -ww representation is more suitable due their typically high inter-channel variance.
We define power-of-twe quantization into i-bit "po2 k" format as po2,. = sgn( 2xxp(=)
comprising a sign bit and k 1-bit exponent exp(1)
max max [log2 + b]) with bias [log2(] [l Square brackets signify rounding to the nearest integer. With b,
b 2 we scale exp(.) layer-wise wise to make efficient use of its dynamic range. This applied to quantize matrix product gradients 0Y, line of Algorithm 2 We chose
standard, gener- ally finding this value to result high compression while inducing little loss in accuracy. While we elected not similarly approximate ox due its in the computation of quantization-sensitiee G of of σÝ = po2(0Y) nevertheless leads to sizeable reductions in total memory footprint.
reduce the energy consumption associated with lines 13 in Algorithm 2 for both of which we now have one binary and one power-of-two operand. Assuming that the target training platform has native support for only 32-bit fixed- and floating coating--pontt point arith- metic these matrix multiplications can be computed by (i) converting powers-of-two into int shifts, (ii) performing sign-flips and (iii) accumulating the int32 outputs. This consumes far less energy than the standard training method's 's all-f oat32 equivalent.
Our use of 0Y further allows
5.2 BATCH NORMALIZATION APPROXIMATION
Analysis of the backward pass of Algorithm reveals conflicting requirements for the precision of X. When computing weight gradients @W (line 13 only binary activations X are needed. For the batch normalization training (lines [T). however, high-precision X is used. As was shown in Table 2 the storage of X between forward and backward propagation constitutes the single largest portion of the algorithm's total memory. If able use X in place of X for these operations,

---

Under review as a conference paper at ICLR 2021
there will be no need to retain the high-preeision activations, significantly reducing memory footprint as a result. We achieve this goal via the following two steps.
Step I: 1 normalization. Standard batch normalization sees channel-wise 12 normalization per- formed on each layer's centralized activations. Since batch normalization is immediately followed by binarization in BNNs, however, we argue that less-costly 4 normalization is good enough in this circumstance. Replacement of batch normalization's ' backward propagation operation with our 4 norm-based version sees lines of Algorithm swapped with
Oa
(m)
V f-
dyt (m) t v v(v) - X
2441
(I)
M/
where B is the batch size. Not only does of l batch normalization eliminate all squares and square roots, also transforms one occurrence of कi41 into its binary form, 2;41
(m)
(m)
Step 2: ai+1 approximation. Since 0Y is quantized into power-of-two format immediately
(m)
after its calculation (Algorithm [ Zline| 12), we hypothesize that it should be robust to approximation.
[m) Consequently, replace the #i+1 term remaining in (1 with the product of its signs and mean
magnitude: [|47||=([11 (m)) (
Our complete batch normalization training functions are shown on lines & of Algorithm 2 which only require the storage of binary along with layer- and channel-wiss scalars. With elements of X now binarized, we not only reduce its memory cost by 32x but also save energy thanks to the corresponding memory traffic reduction.
EVALUATION
We implemented our BNN training method using Keras and TensorFlow, and experimented with the small-scale MNIST, CIFAR-I0 and SVHN datasets, as well as large-scale ImageNet, using range of network models. Our baseline for comparison was the standard BNN training method introduced by Courbariaux & Bengio (2016), and we followed those authors' practice of reporting the highest test accuracy achieved in each run. Energy consumption results were obtained using the inference energy estimator from QKeras (Coelho
et al., 2020), which we extended to also estimate the energy consumption of training. This tool assumes the use of an in-order processor fabricated on a 45 nm process and a cacheless memory hierarchy, as modeled by Horowitz/ (2014), resulting in high-level, planform-agnostic energy estimates useful for relative comparison. Note that we did not tune hyperparameters, thus it is likely that higher accuracy than we report is achievable.
For MNIST we evaluated using five-layer MLP henceforth simply denoted WMLP""with 256 neurons per hidden layer, and CNV (Umuroglu et al. a (017)) and BinaryNet (Courbariaux & Bengio, 2016) for both CIFAR-10 and SVHN. We used three popular BNN optimizers: Adam (Kingma & Ba 2015). stochastic gradient descent (SGD) with momentum and Bop (Helwegen et al.) 2019). While all three function reliably with our training scheme, we used Adam by default due to its outstanding stability in performance. Experimental setup minutiae can be found in Appendix B.
Our choice of quantization targets primarily rested on the intuition that BNNs should be more robust to approximation in backward propagation than their nigher-precision counterparts. To illustrate that this is indeed the case, we compared our method's loss when applied BNNs float net- works with identical topologies and hyperparameters. Generally, per Table 3 significantly higher accuracy degradation was observed for the non-binary networks, as expected. While our proposed BNN training method does exhibit limited accuracy degradation-a geomean drop of 1.21 percent- age points (pp) for these examples-thi comes in return for simultaneous geomean memory and energy savings of 3.66x and 3.09 respectively, as shown in Table 4 It is also interesting to note that the training cost reductions achievable for a given dataset depend on the model chosen to clas- sify it, can be seen actoss Tables and This observation is largely orthogonal to work: by applying our approach to the training of a more appropriately chosen model, one can obtain the advantages of both optimized network selection and training, effectively benefiting twice.

---

Under review as a conference paper at ICLR 2021
Table 3: Test accuracy non-binary and BNNs using standard and proposed training approaches for various models and datasets optimized with Adam. Results for our training approach applied to the former are included for reference only; we do not advocate for its use with non-binary networks.
Top-1 test accuracy
Model
Dataset
Standard training
Reference training
Proposed training
NN (%)²
BNN (%)
A (pp)
NN (%)
A (pp)²
BNN (%)
A (pp)²
MLP
MNIST
98.22
98 08.24
0.02
89.98
-8.24
96.83
-1.41
CNV
CIFAR-10
86.37
82.67
-3.70
69.88
-16.49
82.31
-0.36
CNV SVHN
97.30
96.37
-0.93
79.44
-17.86
94.22
-2.15
Nen
CIFAR-10
88.20
89.81
1.61
76.56
-11.64
88.36
-1.45
BinaryNet SVHN
96.54
97.40
0.86
85.71
-10.83
95.78
-1.62
Non-binary neural network.
Baseline: non- minbiaary network with standard training.
Baseline: BNN with standard training.
Table 4: Memory footprint and per-batch energy consumption of the standard and our proposed
training schemes for various models using the Adam optimizer.
Memory
Energy/batch
Model
Standard (MiB)
Proposed (MiB)
Saving (x)
Standard (mJ)
Proposed (mJ)
Saving (x)
MLP
7.40
2.56
2.89
2.40
0.97
2.48
CNV
128.33
27.13
4.73
144.24
52.61
2.74
BinaryNet
425.31
118.21
3.60
855.41
196.26
4.36
In order to explore the impacts of the various facets of our scheme, we applied them sequentially while training BinaryNet to classify CIFAR-10 with multiple optimizers. As shown in Table []]
choices of data types, optimizer and batch normalization implementation lead to clear tradeoffs against performance and resource costs. Major memory savings are attributable to the use of float 1 variables and through the use of our I norm-based batch normalization. The bulk of scheme's 's energy savings come from the power-of-two representation of ØY, which eliminates floating-pon point operations from lines 13 14 Algorithm We also evaluated the quantization ōY into five-bit layer-wise block floating-pon point format, denoted "int.5" in Table 5 since the individual elements are fixed-point values. With this encoding, significantly higher accuracy loss was observed than when 0Y was quantized into the proposed, equally sized power-of-two format, confirming that representation of this variable's s range is more important than its precision.
Figure 2 shows the memory footprint savings from our proposed BNN training method for different optimizers and batch sizes, again for BinaryNet with the CIFAR-10 10 dataset. Across all of these, achieved geomean reduction of 4.86 Also observable from Figure 2 is that, for all three optimizers, movement from the standard to our proposed BNN training allows the batch size used to increase by 10x, facilitating faster completion, without material increase memory consumption. With respect to energy, we saw an estimated geomean 4.49x x reduction, split into contributions attributable to arithmetic operations and memory traffic by 18.27 x and 1.71 x. Figure shows that test accuracy does not drop significantly due to our approximations. With Adam, there were small drops (gcomean 0.87 pp), while with SGD and Bop we actually saw modest improvements.
We trained ResNetE-18. mixed- peeeed-incisioon model with most convolutional layers binarized (Bethge cet al.) (019), to classify ImageNet. ResNetE-18 18 represents an exemplary instance within a broad class of ImageNet-capable networks, and believe that similar results should be achievable for models with which it shares architectural features. Setup specifics can be found in Appendix [B.2
We show the performance of this network and dataset when applying each of our proposed approxi- mations in turn, as well as with the combination we found to work best, in Table Since the Tensor Processing Units used here natively support bfloat 6 rather than f1 float 6, we switched to
7

---

Under review as a conference paper at ICLR 2021
Table Accuracy, memory and energy impacts of moving from standard our proposed data rep- resentations. We include block floating-poin point 0.X to illustrate the importance of dynamic range over precision for its representation. For these experiments, BinaryNet was trained classify CIFAR-I0.
Data type
Batch norm.
Top-1 test accuracy
Memory saving (x)²
Energy saving (x)
Optimizer
0Y
%
A (pp)
float32
float32
l2
88.74
float16
float16
88.71
-0.03
2.00
1.09
bool
float16
l2
87.93
-0.81
2.27
1.10
Adam
bool
int5²
81.12
-7.62
2.50
4.32
bool
po2
89.47
0.73
2.50
4.01
bool
po2_5
1
87.64
-1.10
2.50
4.01
bool
po2.5
Proposed
88.59
0.15
3.60
4.36
float32
float32
88.52
float16
float16
l2
88.54
0.02
2.00
1.09
bool
float16
87.35
-1177
2.31
1.10
SGD with
bool
int5
l2
81.89
6..3
2.59
4.40
momentum
bool
po2_5
l2
89.08
0.56
2.59
4.06
bool
po2
l1
88.69
0.17
2.59
4.06
bool
po2 5
Proposed
87.45
1.07
4.07
4.45
float32
3loat32
l2
91.38
float16
float16
91.36
0.02
2.00
1.09
bool
float16
l2
90.54
- 0.84
2.37
1.10
Bop
bool
int5
40.55
50.83
2.72
4.48
bool
po2.5
l2
89.34
-2.4
2.72
4.11
bool po2_5
87.81
-3.57
2.72
4.11
bool po2.5
Proposed
86.28
-5.10
4.92
4.53
Baseline: f1 oat32 @W and 0X with standard (12) batch normalization.
5-bit fixed-point elements layer-wise block floating-poinl format.
the former for these experiments. Where bfloat1 6 variables were used, these were employed across all layers; the remaining approximations were applied only to binary layers. Despite creasing the precision of power-of-two quantized 0Y by moving from & = 5 to 8, this scheme unfortunately induced significant accuracy degradation, suggesting incompatibility with large-scale disapplied for final experiment, which saw remaining three
datasets. Consequently, approximations deliver memory and energy reductions of 3.12x and 1.17x in return for a 2.25 pp drop in test accuracy. While these savings smaller than those of our small-scale experiments, we note that ResNetF--18's first convolutional layer is both its largest and is non-binary. thus its activation storage dwarfs that of the remaining layers. We also remark that, while ~2 pp accuracy drops may not be acceptable for some application deployments, sizable training resource reductions otherwise possible. The effects of binarized @W are insignificant since ImageNet's large im- ages result in proportionally small weight memory occupancy. Nevertheless, this proof of concept demonstrates the feasibility of large-scale neural network training on the edge.
7 CONCLUSION
In this paper, we introduced the first training scheme tailored specifically to BNNs. Moving first 16-bit floating -ooii representations, we selectively and opportunistically approximated beyond this based on careful analysis of the standard training algorithm presented by Courbariaux & Ben- gio. With a comprehensive evaluation conducted across multiple models, datasets, optimizers and batch sizes, we showed the generality of our approach and reported significant memory and energy reductions vs the prior art, challenging the notion that the resource constraints edge platforms present insurmountable barriers to on-device learning. In the future, will explore the potential of our training approximations in the custom hardware setting, within which we expect there to be vast energy-saving potential through the design of tailor-made arithmetic operators.
8

---

Under review as a conference paper at ICLR 2021
Adam
SGD with momentum
Boр
90
Top-1 test accuracy (%)
2
Memory (GiB)
A
A
88
A
O
86
5.23x
1544
5.44x
5.67%
3 60
4. 4.12x
4.66x
4.82X
4.07
4.56x
5.02x
594x
1.924
5 5.24
4494
5.54x
84
83
4
Energy/bateh (J)
4
4.50x
4.51X
4.52
4.36x
4.43x
4.48
4.48
4.45
4.48x
4.50X
4.50
4.53x
4.52x
4.51
4.51
100
200
400
500
1000
100
200
400
500
1000
100
200
400
500
1000
Batch size
Memory (std)
Memory (prop.)
Accuracy (std)
Accuracy (prop.)
Energy (std, ops)
Energy (std, mem.)
Energy (prop., ops)
Energy (prop., mem.)
Batch size vs training memory footprint, achieved test accuracy and per-batch training energy consumption for BinaryNet with CIFAR-10. The upper plots show memory and accuracy results for the standard and our proposed training flows. In the lower plots, total energy is split into compute- and memory-re related components, Annotations show reductions vs the standard approach.
Figure 2:
Table 6: Test accuracy. memory footprint and ver-batch energy consumption of the standard and our proposed training schemes for ResNetE-18 classifying ImageNet with Adam used for optimization.
Top-1 test accuracy
Memory
Energy/batch
Approximations
(pp)
GiB
Saving (x)²
J
Saving (x)
None All-bfloat16
58.57
57.84
185.08
58.55
0.02
29.32
1.97
162.41
1.14
bool OW only po2.8 8 0Y only
57.30
-1.27
57.80
1.00
185.08
1.00
29.56
-29.01
57.84
1.00
116.06
1.59
1 batch norm. only
57.34
-1.23
57.84
1.00
185.08
1.00
Proposed batch norm. only
57.25
-1.32
35.59
1.63
176.87
1.05
Final 2ombination
56.32
-2.25
18.54
3.12
158.44
1.17
Baseline: approximation-ffes training.
bool aW b oatt remaining variables with proposed batch normalization.
ACKNOWLEDGMENTS

---

Under review as a conference paper at ICLR 2021
REFERENCES
Naman Agarwal, Ananda Theertha Suresh, Felix Yu, Sanjiv Kumar, and H. Brendan McMahan. CpSGD: Communication-effoii and differentially-privart distributed SGD. In International Conference Neural Information Processing Systems, 2018.
Milad Alizadeh, Javier Fernández-Marqués, Nicholas D. Lane, and Yarin Gal. An empirical study of binary neural networks' optimisation. In International Conference Learning Representations, 2018.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. SignSGD: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, 2018.
Joseph Bethge, Haojin Yang, Marvin Bornstein, and Christoph Meinel. Back to simplicity: How to train accurate BNNs from scratch? arXiv preprint arXiv:1906. (9t1::/990006677, 2019.
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp. Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, H. Brendan McMahan, Timon van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design. In Conference on Machine Learning and Systems, 2019.
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tiny Transfer Learning: Towards memory- efficient on-device learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Ayan Chakrabarti and Benjamin Moseley. Backprop with approximate activations for memory- efficient network training. In Advances in Neural Information Processing Systems, 2019.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.00 066600664444... 2016.
Claudionor N. Coelho Jr., Aki Kuusela, Hao Zhuang, Thea Aarrestad, Vladimir Loncar, Jen- nifer Ngadiuba, Maurizio Pierini, and Sioni Summers. Ultra low-latency, low-area inference accelerators using heterogeneous deep quantization with QKeras and hls4ml. arXiv preprint arXiv:2006. 10159, 2020.
George A. Constantinides. Rethinking arithmetic for deep neural networks. Philosophical Transac- tions of the Royal Society A, 378(2166), 2019.
Matthieu Courbariaux and Yoshua Bengio. BinaryNet: Training deep neural networks with weights and activations constrained to +1 or -1. arXiv preprint 1602..6002.0200 2016.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. BinaryConnect: Training deep ral networks with binary weights during propagations. In Conference on Neural Information Processing Systems, 2015.
Mohammad Ghasemzadeh, Mohammad Samragh, and Farinaz Koushanfar. ReBNet: Residual bi- narized neural network. In IEEE International Symposium on Field- Prolra-rggaa Custom Com- puting Machines, 2018.
Boris Ginsburg, Sergei Nikolaev, and Paulius Micikevicius. Training of deep networks with half- precision float. In Nvidia GPU Technology Conference, 2017.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, 2010.
Benjamin Graham. Low-precision antch-nommalize activations. arXiv preprint arXiv: 1702:0170,,,231,,, 2017.
Audrunas Gruslys, Rémi Munos, Ivo Danihelka, Mare Lanctot, and Alex Graves. Memory--fficient backpropagation through time. In Advances in Neural Information Processing Systems, 2016.
10

---

Under review as a conference paper at ICLR 2021
Xiangyu He, Zitao Mo, Ke Cheng, Weixiang Xu, Qinghao Hu, Peisong Wang, Qingshan Liu, and Jian Cheng. Proxybnn: Learning binarized neural networks via proxy matrices. In European Conference Computer Vision, 2020.
Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang- Kwang-Ting Cheng, and Roeland Nusselder. Latent weights do not exist: Rethinking binarized neural network optimization. In Advances in Neural Information Processing Systems, 2019.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: Efficient and accurate normalization schemes in deep networks. In Advances in Neural Information Processing Systems, 2018.
Mark Horowitz. Computing's energy problem (and what we can do about it). In International Solid-State Circuits Conference, 2014.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Conference on Neural Information Processing Systems, 2017.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-Real Net: Enhancing the performance 1-bit CNNs with improved representational capability and advanced training algorithm. In European Conference on Computer Vision, 2018.
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. ReActNet: Towards pre- cise binary neural network with generalized activation functions. In European Conference on Computer Vision, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication--ffiiiee learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics, 2017.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. International Conference on Learning Representations, 2018
Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. Binary neural networks: survey. Pattern Recognition, 105, 2020.
Eyyüb Sari, Mouloud Belbahri, and Vahid P. Nia. How does batch normalization help binary train- ing, arXiv preprint arXiv:1909.09) 03399, 2019.
Nimit S. Sohoni, Christopher R. Aberger, Megan Leszezynski, Jian Zhang, and Christopher Ré. Low-memory neural network training: technical report. arXiv preprint urXiv:1904 10631, 2019.
Yaman Umuroglu, Nicholas Fraser, Giulio Gambardella, Michaela Blott, Philip H. W. Leong, Magnus Jahre, and Kees Vissers. FINN: A framework for fast, scalable binarized neural network inference. In ACM/SIGDA International Symposium on Field-Progrummable Gate Arrays, 2017.
Yaman Umuroglu, Yash Akhauri, Nicholas J. Fraser, and Michaela Blott. LogicNets: Co-designed neural networks and circuits for extreme-throughput applications. In International Conference on Field- Proed-Programaablee Logic and Applications, 2020.
Erwei Wang, James Davis, Peter Y. K. Cheung, and George A. Constantinides. LUTNet: Re- thinking inference in FPGA soft logic. In IEEE International Symposium on Field-Programmalle Custom Computing Machines, 2019a.
Erwei Wang, James J. Davis, Ruizhe Zhao, Ho Ho-Cheung Ng, Xinyu Niu, Wayne Luk, Peter Y. K. Che- ung. and George A. Constantinides. Deep neural network approximation for custom hardware: Where we've ve been, where we're 'e'r going. ACM Computing Surveys, 52(2), 2019b.
11

---

Under review as a conference paper at ICLR 2021
Erwei Wang, James J. Davis, Peter Y. K. Cheung, and George A. Constantinides. LUTNet: Learn- ing FPGA configurations for highly efficient neural network inference. IEEE Transactions on Computers, 2020.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Ye Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point numbers. In Advances in Neural Information Processing Systems, 2018.
Wei Wen, Cong Xu, Feng Yan, Chumpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. TernGrad: Ternary gradients reduce communication in distributed deep learning. In Advances in Neural Information Processing Systems, 2017.
Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, 2017.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. International Conference on Learning Representations, 2018a.
Shuang Wu, Guoqi Li, Lei Deng, Liu Liu, Dong Wu, Yuan Xie, and Luping Shi. /1 11-norm batch nor- malization for efficient training of deep neural networks. IEEE Transactions Neural Networks and Learning Systems, 30(7), 2018b.
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. DoReFa-Net Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprini 16Xiv:1606.06166, 060166066666600 2016.
12

---

Under review as a conference paper at ICLR 2021
A CONVERGENCE RATE ANALYSIS
A.1
THEORETICAL SUPPORT
0.8
Gradient density
0.6
0.4
0.2
Weight gradient density
Weight noise density
5
10 Epoch
15
20
Figure 3: Weight density of the sixth convolutional layer of BinaryNet trained with bool weight and po2.5 5 activation gradients using Adam and the CIFAR-10 dataset.
Bernstein et al. (2018) proved that training non-binary networks with binary weight gradients may result in similar convergence rates to those of unquantized training if weight gradient den- sity o (Iu(oW1).... m(OWs))") and weight noise density & (oWW))) ((WW)))) remain within an order of magnitude throughout a training run. Here, S is the training step size and denotes the density function of an N-element vector.
We repeated Bernstein et al.'s evaluation with our proposed gradient quantization applied during BinaryNet training with the CIFAR- 10 dataset using Adam and hyperparameters as detailed in Ap- pendix B.I The results of this experiment can be found in Figure 3 We chose to show the densities of BinaryNet's sixth convolutional layer since this is the largest layer in the network. Each batch of inputs was trained using quantized gradients aW and σÝ The trained network was then evalu- ated using the same training data to obtain the flatt 32 (unquantized) aW used to plot the data shown in Figure 3 We found that the weight gradient density ranged from 0.55-0.62, and weight noise density 0.92-0.97, therefore concluding that our quantization method may result in similar convergence rates to the unquantized baseline.
should be noted that Bernstein al.'s 's derivations assumed the of smooth objective functions. Although the forward propagation of BNNs is not smooth due to binarization, their training functions still assume smoothness due to the use of straight-through estimation.
A.2 EMPIRICAL SUPPORT
Figures 5 6 and a contain the training accuracy curves of all experiments conducted for this work. The curves of the standard and our proposed training methods are broadly similar, supporting the conclusion from Appendix A.T that proposals do not induce significant convergence rate change.
13

---

Under review as a conference paper at ICLR 2021
MLP & MNIST
CNV & CIFAR-10
CNV SVHN
1
0.8
0.6
NN, standard
0.4
BNN, standard
Top-1 training accuracy (%)
NN, reference
0.2
BNN, proposed
200
400
600
800
1000
0
200
400
600
800
1000
50
100
150
200
BinaryNet & CIFAR-10
BinaryNet & SVHN
0.8
0.6
0.4
0.2
200
400
600
800
1000
50
100
150
200
Epoch
Figure 4: Achieved training accuracy over time for experiments reported in Table
14

---

Under review as a conference paper at ICLR 2021
Adam
SGD with momentum
0.8
0.6
All-float32,1z
All-float16,1b
0.4
bool 0W, float166 0Y, l2
bool 8W, int. 0Y l2
0.2
bool
0Y, l2 bool @W.po2.5 0Y,
Top-1 training accuracy (%)
bool OW, po2_5 0Y, prop.
0
200
400
600
800
1000
200
400
600
800
1000
Bop
0.8
0.6
0.4
0.2
200
400
600
800
1000
Epoch
Figure 5: Achieved training accuracy over time for experiments reported in Table
15

---

Under review as a conference paper at ICLR 2021
Adam
SGD with momentum
Bop
Batch size 100
0.8
0.6
Standard Proposed
0.4
Batch size 200
0.8
0.6
0.4
Top-1 I training accuracy (%)
Batch size 400
0.8
0.6
0.4
Batch size 500
0.8
0.6
0.4
Batch size 1000
0.8
0.6
0.4
200
400
600
800
1000
200
400
600
800
1000
0
200
400
600
800
1000
Epoch
Figure 6: Achieved training accuracy time for experiments reported in Figure
16

---

Under review as a conference paper at ICLR 2021
0.6
0.5
Top-1 training accuracy (%)
0.4
0.3
0.2
None All-bfloat
bool OW only po2.8 oY only 11 batch norm. only Prop. batch norm. only Final combination
20
40
60
80
100
120
Epoch
Figure 7: Achieved training accuracy over time for experiments reported in Table
17

---

Under review as a conference paper at ICLR 2021
B EXPERIMENTAL SETUP
B.1 SMALL-SCALE DATASETS
We used the development- based learning rate scheduling approach proposed by Wilson et al. (2017) with an initial learning rate of 0.001 for all optimizers except for SGD with momentum, for which we used 0.1. We used batch size B 100 for all except for Bop, for which we used B 50 as recommended by Helwegen et al. (2019). MNIST and CIFAR-10 were trained for 1000 epochs; SVHN for 200.
B.2 IMAGENET
Finding development-basee learning rate scheduling work well with ResNetE-18, resorted to the fixed decay schedule described by [Bethge et al. (2019). began at 0.001 and decayed by factor of 10 epochs 70, 90 and 110. We trained for T20 epochs with B 4096.
18


Reviews:
[{"review_id": "nwvrc0YLcUI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Comments\nSummary:\n\nThe authors proposed a low-cost binary neural network training strategy exhibiting sizable memory footprint reductions and energy savings. The methods include binarizing weight gradients, modifying the forward and backward batch normalization operations and using power-of-two activation gradients and reduced-precision floating-point data. The experimental results show some improvement memory footprint reductions and energy savings vs standard approach.\n\n\nStrength:\n-- The authors carried out a relatively sufficient experimental analysis, and evaluated across multiple models, data sets, optimizers and batch sizes\n-- Storage and energy consumption based on hardware models increases the completeness and credibility of conclusions.\n-- The experiment results seem that the proposed method achieves good performance in memory footprint reductions and energy savings.\n\nWeakness:\n-- The proposed method seems like a combination of existing technical methods.\n-- There is a lack of comparison with other low-cost binary neural network training works.\n\n\nComments:\n(1)\tThere is a difference in the memory consumption in Table1 and in section 5.1 (1.67 MiB or 1.41 MiB ?). The authors may check this.\n(2)\tAs for the perspective of overall design, it’s better to emphasize the trade-offs between the importance of different variables to the overall training and the choose of the data type.\n(3)\tIt’s better to add some comparisons with other low-cost binary neural network training works.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XWf3ptGW1oC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I agree with the key contributions listed in the paper, especially the binarization of weight gradients and activations. The paper is well written and clearly articulates a contribution to the literature. The proposed BNN training scheme can have a significant practical impact. The experimental evidence is provided for several standard image classification tasks. Most of the related works are cited. The paper does not contain a theory part, but wherever possible, equations are provided to illustrate how the method works.\n\nConcerns: \nThis paper includes a detailed empirical evaluation of the proposed BNN training scheme. The major concern is that the proposed low-cost BNN training scheme can cause a nontrivial accuracy degradation (2.25%) as shown in Table 5. The tradeoff between accuracy and memory footprint/energy consumption is not carefully evaluated. For example, a smaller network model with fewer parameters and activations can be trained using the baseline BNN training scheme to reduce memory and energy consumption. Besides, the baseline networks (BinaryNet and ResNet) used for comparison are out-of-date.  Many recent works [1-3] propose new BNN architectures, which improve the accuracy of BNNs significantly. It is useful to justify the effectiveness of the proposed scheme for these SOTA BNN architectures.\n\nTo evaluate the energy consumption of the traditional and the proposed BNN training scheme, the authors assume that the two training schemes have the same convergence rate. Although the traditional BNN training scheme consumes higher power during training, it might take fewer epochs to reach the same accuracy. Therefore, the saving in energy consumption can be lower than the reported numbers. In addition, the estimated energy consumption of BNN training obtained from using QKeras is very rough, which makes the improvement in energy consumption less convincing. The authors can mainly improve the paper's strength by prototyping the proposed BNN training scheme on an embedded CPU and measuring real-world performance and power.\n\nReasons for score: In general, I like the idea of enabling low-cost BNN training by identifying unnecessary high-precision data. However, the improvement numbers presented in the paper need better justification. I would consider raising my score if the authors could address the aforementioned concerns.\n\n[1] Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm\n\n[2] ProxyBNN: Learning Binarized Neural Networks via Proxy Matrices\n\n[3] ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SB03xhDUg_a", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel method to make the training Binary Neural Networks with low-memory and low-energy by modifying the backpropagation and forward process. To this end, the paper binarized weight gradients, change batch normalization layer for removing full-precision the inputs, and utilize quantization to accelerate the whole training BNNs procedure.   \n\nThe paper targeted one of the problems in Binary Neural Networks and provided experiments as well as source codes to the proof of efficiency. This is the most significant contribution of the paper. \n\nHowever, there are the following concerns:\n- Need more experiments on the actual training time between two BNNs which are trained with quantized and full-precision gradient weight.\n- Comparison with similar works in recent BNNs.\n- What does it mean B variables in Algorithm 2 of lines: 6, 9, and 10, and how does it influence the performance of training?\n- In figure 2, the legends should be put in the figure (not in the caption). It is better to follow.\n- It would be nice to have the training time and accuracy in the large-scale dataset of ImageNet.\n\nIn conclusion, the paper addresses the novel idea for the training improvement of Binary Neural Networks in low-memory and low-energy. However, there are many concerns aforementioned.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "-n9jdfNsfqF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think this a very good contribution to ICLR given the topic and the quality of the submission (originality, contribution to the state of the art, experimental evidence, e\n\n Some of the strong points of the submission are summarized as follows, along with some points for clarification\n\n1.\tEnabling training on any embedded device (SoC, FPGA, micro-controller) is one of the holy grails for edge AI and IoT. As the authors mention, this also intersects with other domains such as federating learning privacy by design systems. The authors provide and ample motivations of the importance of this work, and some of the applications edge AI might enable, as well as the current challenges.\n2.\tThe state of the art (despite the previous comment) contextualizes the subject matter in a succinct but comprehensive manner. Although there are certain aspects that could be improved, such as including a table outlining in a clearer manner the contributions of the authors in this context.\n3.\tThe comparison with the traditional training method is clear. However, I would like to know if the authors have made an ablation study to assess whether or not the use of batch normalization would have an effect on the accuracy of the proposed models. Do you provide experimental evidence of the lack of degradation due to the use of l1 batch normalization? These two aspects are not mentioned in the next nor provided in the supplementary sections.\n4.\tThe experimental design is good, showing a careful analysis to validate the proposal and several ablation studies to assess the memory footprint reductions and its effects on the training of various models\n5.\tThe foundations for the method are presented in great detail in a formalized manner and provides sufficient elements (i.e. experiments) to assess the validity of the proposed approach.\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After reading the paper, reviews and authors’ feedback. The meta-reviewer agrees that this paper addresses an important topic. However, as the reviewers pointed out. The paper mainly builds the technique on simulated setting, and it is unclear how the method will translate to real world speedups. Past work(e.g. [1]) has also shown that many cases there could be a huge gap when the solution is not built carefully.

The paper would benefit from a prototype to demonstrate the applicability of the approach. This paper is therefore rejected.

Thank you for submitting the paper to ICLR. 

[1] Riptide: Fast End-to-End Binarized Neural Networks

Decision: Reject