Fast Binarized Neural Network Training with Partial Pre-training
Authors: Alex Renda
Joshua Wolff Fromm
Venue:  (2021)
Keywords: binarized neural network
binary
quantized
1-bit
low precision

Abstract: Binarized neural networks, networks with weights and activations constrained to lie in a 2-element set, allow for more time- and resource-efficient inference than standard floating-point networks. However, binarized neural networks typically take more training to plateau in accuracy than their floating-point counterparts, in terms of both iteration count and wall clock time. We demonstrate a technique, partial pre-training, that allows for faster from-scratch training of binarized neural networks by first training the network as a standard floating-point network for a short amount of time, then converting the network to a binarized neural network and continuing to train from there. Without tuning any hyperparameters across four networks on three different datasets, partial pre-training is able to train binarized neural networks between $1.26\times$ and $1.61\times$ faster than when training a binarized network from scratch using standard low-precision training.


Under review as a conference paper at ICLR 2021
FAST BINARIZED NEURAL NETWORK TRAINING WITH PARTIAL PRE-TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Binarized neural networks, networks with weights and activations constrained to lie a 2-element set, allow for more time- and resource-efficient inference than standard poating-point networks. However, binarized neural networks typically take more training to plateau in accuracy than their floating-poit counterparts, terms of both iteration count and wall clock time. We demonstrate a technique, partial pre-training, that allows for faster from-scratch training of binarized neural standard floating-point network for
networks by first training the network short amount of time, then converting the network to a binarized neural network and continuing to train from there. Without tuning any hyperparameters across four networks on three different datasets, partial pre-training is able to train binarized neural networks between 1.26x and 1.61x faster than when training a binarized network from scratch using standard low- wwwpprecssion training.
INTRODUCTION
Quantizing neural networks (Gupta al., 2015), constraining weights and activations to take values within some small fixed set, is a popular set of techniques for reducing the storage (Han et al., 2016) or compute (Fromm al., 2020) requirements of deep neural networks. Weights and activations can often be quantized down to as few as 8 bits with no loss in accuracy compared to a full-precision model. Further quantization often comes the expense of accuracy: it possible to binarize neural networks (Hubara et al., 2016; Rastegari et al., 2016), constraining weights and activations to take on values within a set of two elements (often 1, 13), but such binarization often lowers the accuracy of the resultant network, necessitating a tradeoff between desired compression and accuracy.
In the literature, there are two primary techniques for obtaining quantized neural network: quantizing pre-trained full- full-precision network (Banner et al., 2019; Han et al., 2016), and training quantized network from scratch (Hubara et al., 2016; Gupta et al., 2015).
Full-precision training. Quantizing full-precision network requires few or even no additional training epochs on top of training that full-precision network. Typical procedures for quantizing a full- free-icccision network range from data-blind procedures like selecting quantization bins minimize distance from the original weights (Banner et al., 2019), to data-intensive procedures such as re- training the network to be more amenable to quantization (Han et al., 2016). However without significant additional training time, quantizing pre-trained network often does not reach the highest accuracy possible for the quantized network architecture (Alizadeh et al., 2019). Further, achieving high accuracy with heavy quantization, such as binarization, often requires changing the network architecture, for instance by adding skip connections (Bethge et al., 2019); such architectural changes mean that the weights of a pre- praiee full- fuee-prrcision network may not transfer to the new architecture.
Low- low-precision training, Alternatively, training a quamtized network from scratch allows for achiev- ing high accuracy regardless of the availability pre-trained full-precision weights (Alizadeh et al., 2019). Typical procedures for training a quantized network from scratch involve tracking and optimizing latent weights, weights which are quantized during the forward pass but treated as full- precision during the backward pass (Hubara et al., 2016). However, training quantized network from scratch can be costly. Quantized networks typically require more training iterations to plateau in accuracy (Hubara et al., 2016, Figure 1; Bethge et al.. 2019, Figure 2). Further, since quantized net- works are often trained by simulating the quantized operations in poating-point (Zhang et al., 2019), low-precisio training be even more computationally expensive than the full-precision equivalent.

---

Under review as a conference paper at ICLR 2021
Research question. In this paper, we explore the question:
Can we accelerate training a binarized neural network from scratch to a given target accuracy?
Concretely, we assume that network architecture and standard training schedule are provided, but that pre-trained full- cull-precision networks are not available. We also specifically focus on achieving accuracy in the early phase of training, exposing the tradeoff between training cost and accuracy.
Partial pre-training. To answer the above research question, we evaluate a technique, partial pre-training, that allows for faster training of binarized neural networks by first training the network as a standard floating point network with standard full-precision training for short amount of time, then converting the network a binarized neural network and continuing to train from there with standard low-precision training for the remainder of the budgeted training time. We specifically evaluate partial pre-traimin training's speedup over standard low-precision training, when training a binarized neural network from scratch. We find that partial pre-training can train VGG, ResNet, and Neural Collaborative Filtering networks on CIFAR-10, ImageNet, and MovieLLens-200 between 1.26 x and 1.61 x faster than standard low-precision training.
Contributions.
We present partial pre-training, which can train binarized neural networks from scratch between 1.26x and 1.61 x faster than standard low-precision training.
We find that partial pre-training both requires fewer iterations train a given accuracy, and also that partial pre-training takes on average less time per iteration than standard low www-precision training.
We analyze the sensitivity of partial pre-training to the choice of split between full-precision and low-precision training finding that an even split, though not always optimal, nearly matches the highest accuracy achievable by any other choice of split.
All together, we find that partial tre-training is a simple and effective approach for accelerating binarized neural network training. Partial pre-training is step towards the goal of binarized neural network training procedures that can match the efficiency gains of binarized neural network inference.
2
BACKGROUND
Binarized neural networks trade off accuracy for inference efficiency. However, binarized neural networks often take longer to train than the full-precision versions of the same network architecture, both in terms of training iterations until convergence and wall-clock time per iteration.
Training iterations. Binarized neural networks tend to take more iterations to train than the full- precision versions of the same network architecture. For instance, Hubara et al. (2016, Figure I) show binarized neural network with custom architecture requiring 4x as many training iterations to plateau in accuracy as a full-precision baseline on CIFAR- 10. Bethge et al. (2019, Figure 2) similarly show binarized ResNet- 18 taking 2x as many training iterations plateau in accuracy as a full-precision baseline on ImageNet.
Wall-clock time. Beyond requiring more iterations train, binarized neural networks tend to take more wall-clock time to complete each iteration of training than full full-precision networks do. This because binarized neural networks are often trained by simulating low-precision operations with standard floating point operations (Zhang et al., 2019; Fromm et al., 2020), and require additional bookkeeping that full-precision networks do not require, such as performing the binarization of weights and activations (Hubara et al., 2016) and calculating scaling factors (Rastegari et al., 201 6). While it is theoretically possible to accelerate binarized neural network training, we are not aware of any effort to exploit binarization during the training phase. It is also not clear what the maximum speedup possible from accelerating binarized neural network training would be: Fromm et al. (2020) show an acceleration of 6.33x for VGG during inference; with the additional bookkeeping of binarized neural network training and potentially requiring higher precision gradients in the backward pass (Zhou al., 2016), real training speedups would likely be lower.
2

---

Under review as a conference paper at ICLR 2021
3 PARTIAL PRE-TRAINING
This paper focuses on accelerating the training time of binarized neural networks. Specifically, we aim reduce both the training iterations and wall clock time of training binarized neural networks. We achieve this by leveraging the faster training time- both iteration-count and sume-pe--teeree per-iteration-a of full-precision networks. This section formalizes the design space of partial pre-training algorithms, describing the exact training methodology used in the experiments in Sections 5 and 6.
Partial pre-training splits training into multiple phases. First, partial pre-training trains the network for short amount of time at full precision (32 bits) using quantization methods: weights, activations, and gradients are not quantized or clipped. Next, the binarization operators are added to the network (binarizing weights and activations). Partial pre-training then continues to train the binarized neural network using standard low-pree precision training.
To avoid requiring hyperparameter search for each different network, we prescribe standard training schedule based on the original training schedule of the full- cill-precision network (i.c., learning rate and associated decay, which assumed to be provided). Each step of partial pre-training takes 50% of the allotted training time. Within the training time for each step of partial pre-training, the network is trained using the original learning rate schedule compressed to the allotted training time.
The partial pre-training algorithm presented in Algorithm 1:
Algorithm 1 Partial pre-training.
Train the network at full precision, using the original learning rate schedule compressed down to half of the desired training time.
2. Binarize the network, inserting quantization operators into the network.
3. Train the binarized network, using the original learning rate schedule compressed down to the remaining half of the desired training time.
EXPERIMENTAL METHODOLOGY
4.1 DATASETS AND NETWORKS
We evaluate partial pre-training across variety of datasets and networks. Specifically, we evaluate partial pre-training on a CIFAR-I0 (Krizhevsky, 2009) ResNet-20 (He et al., 2016), a CIFAR-IO VGG-16 (Simonyan & Zisserman, 2014), an ImageNet (Russakovsky et al., 2015) ResNet-34, and MovieLens 20M (Harper & Konstan, 2015) Neural Collaborative Filtering (NCF) (He et al., 2017) model. Following Bethge et al. (2019), our ResNet-20 and ResNet-34 are extended with additional skip connections past every quantized convolution, rather than past every block of quantized convolutions standard for ResNets; this architectural change facilitates training binarized ResNets from scratch. Our NCF model additionally has BatchNorm (loffe & Szegedy, 2015) layers after each binarized layer. The networks otherwise use standard architectures, data augmentation schemes, and training schedules drawn from the literature. Details about the networks and their respective training regimes are presented Table 1.
4.2 TRAINING DETAILS
All networks were trained on AWS GPU instances. The CIFAR-IO and MovieLens-202 20M networks were trained on p3 p3.2xlarge instances, using an NVIDIA V100 GPU. The ImageNet networks were trained on p3.8xlarge instances, training with 4 data mata-parallel NVIDIA V100 GPUs. The vision networks were trained using custom implementations of each network in TensorFlow 2.2.0 (Abadi et al., 2016). The NCF was trained using the Py Torch 0.4 (Paszke et al., 2019) implementation included the MLPerf 0.5 training benchmark suite (Mattson al., 2020).
4.3 BINARIZATION
We evaluate partial pre-training using standard approaches neural network binarization:
3

---

Under review as a conference paper at ICLR 2021
Dutaset
Network
Optimizer
Learning rale (t= training iteration)
Test uecuracy
SCID
32: bit: 90. 75% 0.26% 1-bic 83.30% 0.25%
ResNer-20
B 0.9
0.1
e [o, 32000)
0.01
E [32000, 48000)
VGGG-6
Weight decay: 0.000]
0.001
[48000, 64000)
30% 0.19%
Nerstions: 64000
I-bin: 88.52% + 0.19%
Nesterov P = 0.9
8
0.4
t (6,30)
32 71.1%; d: 0.06% top-
ImageNer
ResNer-34
0.04
E [30, 60)
Weight dacay: 0.0001
0.004
[60, 80)
1-biz: 60.9% + 0.06% top.1
Iecaation::
0.0004
[80, 90]
Adam
MovieLens 20M
0.683 0.002
NCF
Batch size: 2048
Pg 0.089
a 0.0005
I-bit: 0.534 + 0.003 HR
387925
Table Networks, datasets, and hyperparameters. All hyperparameter choices are standard for their respective networks (He al., 2016; Goyal et al., 2017; Mattson al., 2020).
Inputs. We binarize input activations using PACT (Choi al., 2018), gradient-basee method of determining activation scale which is used in place of the ReLU activation in binarized neural networks. PACT introduces trainable parameter per layer, a, which controls the scale of the activations. The PACT activation on binarized networks has the following form:
f0, (-o0, 2 (a, x (G, 00) E (-00,
=
PACT(x)
OAACT(
S0,
OPACT(2) Or
1, x E [0,
Oa
1, x |a, 20)
0, otherwise
We initialize a = for each layer, and control its magnitude with an L2 regularization penalty with coefficient 0.0002.
Weights. We binarize weights using the sign of the weight and the trraighttthrouge estimator:
1, 0
@sig((z)
1, r [-1,11
sigu(x)
1,
0
otherwise
Full precision layers. Following standard practice (Hubara et al., 2016; Simons & Lee, 2019; Qin et al., 2020), we do not binarize inputs or weights to the first, last, or batch normalization layers, nor do binarize projection layers in ResNets (Bethge al., 2019).
4.4 EXPERIMENTS AND RESULTS
For each baseline and configuration of partial pre-training, run three independent trials, plotting the minimum, median, and maximum accuracy achieved across all three trials.
4.5 SPEEDUPS
The reported speedups of partial pre-training over low- low-precision training are calculated by sampling 100 evenly-spaced accuracies, between 50% of the maximum accuracy achievable by the binarized network (as lower bound acceptable accuracy) the accuracy which the techniques converge (with the exception of the NCF, where low-precision training does not converge to the same accuracy as partial pre-training). Speedups then calculated by determining the time train each technique to that accuracy (linearly interpolating between nearby points there is no trial at exactly that low-prscision training time accuracy), calculating the speedup as partial pre-trining training time and calculating the harmonic mean of the speedup across all target accuracies.
'The source implementation (from MLPerf) trains the NCF to an accuracy threshold of 0.635 HR @ 10, taking mean duration of 388000 iterations (Mattson al., 2020, Figure 3a). Adding BatchNorm slightly decreases the accuracy of the full-precision network, but necessary train the binarized network.
4

---

Under review as a conference paper at ICLR 2021
CIFAR-10 Nes-et Accuracy Iteration
CIFAR-10 VGG-16 Accuracy Iteration
Test Accurscy
Test. Accuracy
0.8-
0.8
0.4
0.4
0
5000
10000
15000
20000
25000
30000
5000
10000
15000
20000
25000
30000
Iterations
Herations
ImageNet ResNet-34 Accuracy vs Iteration
MovieLens 20M NCF Accuracy vs Iteration
Test Accuracy
Test Aceuracy
0.6
0.6-
0.2-
0.4
0.2
20000
40000
60000
S0000
25000
500000
75000
100000 125000 15000
Iterations
Iterations
Full-precision
Partial Pre-training
-
Low-precision
Figure 1: Accuracy versus training iterations of partial pre-training and baselines.
EXPERIMENTAL RESULTS
This section presents the accuracy achieved by partial pre-training across training times several different networks. Section 5.1 shows the accuracy achieved by partial pre-training when training for different numbers of training iterations, finding that heraaionnfor-iteratti trains faster than standard low-precision neural network training, Section 5.2 extends the analysis compare wall-clock time, finding that the overhead of simulated low-precision training leads to partial pre-training training even faster than low-precision training.
partial pre-training
5.1 ACCURACY V.S. TRAINING ITERATION
We find that partial pre-training can accelerate binarized neural network training ceration-for-iteralin binarized neural networks trained with partial pre-training require fewer training iterations to reach any target accuracy than binarized neural networks trained with standard low-precision training. Among the four networks presented in this section, find mean speedup of 1.30 of partial pre-training over standard fow-precision training when compared iteration- for-iteration.
Methodology. Each plot shows the accuracy attainable when training a neural network for the specified number of training iterations Full-precision is the accuracy attained from training full- precision network from scratch for the specified number of iterations, and is presented as an upper bound on training speed. Partial pre-training is the accuracy of the partial pre-training procedure. Low-precision is the accuracy attained from training a binarized network from scratch using standard low-precision training for the specified number of iterations. As specified in Section 4.4, each point on the graph is independent, and shows the median and min/max accuracy from three trials.
Results. Figure presents the iteration-for-itraaii accuracy of partial pre-training compared to
full-precision and low-precision training. We find that partial pre-training matches or exceeds the accuracy of low-precision training for any training time, across all networks and datasets, with the exception of after fewer than 10000 training iterations of the CIFAR-10 VGG-16. This speedup is most prominent with relatively little training time. but tapers off and eventually disappears when the network is trained for enough iterations with standard oow-iiecsi training; more analysis this behavior is provided in Section 6.
5

---

Under review as a conference paper at ICLR 2021
Full-precision training speed
Low-precision training speed
Low-precision
Dataset
Network
slowdown
CIFAR-10
ResNet-220
79.86 it/s
64.04 it/s
1.25
CIFAR-10
VGG-16
57.16 it/s
43.37 it/s
1.32 x
ImageNet
ResNet-34
9.12 it/s
7.08 it/s
1.29
MovieLens-20M
NCF
103.75 it/s
72.88 it/s
1.42x
Table 2: Slowdown of low-precision compared to full-precision training.
CIFAR-10 ResNet-20 Accuracy ve Wall-clock time
CIFAR-10 VGG-16 Accuracy vs Wall-clock time
Test Accuracy
Tost Accuracy
0.8
0.8
0.6
0.6
200
400
200
400 Seconds
600
Seconds
ImageNet ResNet-31 Accuracy vs Wall-clock time
MovieLeen--0MM NCF Accuracy vs Wall-clock Time
Tost Accuracy
Test Accuracy
0.6
0.6
0.4
0.1-
0.2
2000
7500 Seconds
12500
300
1500
Seconds
Full precision
+ Partial Pre-training
Low Low-precision
Figure 2:
Accuracy versus wall-clock time of partial pre-training and baselines. Same as Figure but with wall-clock time on the x axis rather than training iterations
5.2 ACCURACY V.S. WALL-CLOCK TIME
This section presents the accuracy attained by partial pre-training across different wall-clock training times. Table first presents the per-training-iteratio slowdown of simulated low-precision optimiza- tion, the standard practice in the literature. Figure then presents the accuracy attained at different wall-clock times, showing how the per-iteration accuracy gains from Section 5.1 and the full-precision speedups from Table 2 compound to lead to faster network training from partial pre-trat trtiiriinn....
Methodology. The training speeds in Table 2 are calculated by timing the mean time to complete training epoch across epochs, ignoring the first batches in each epoch. The speeds are presented in training iterations per second, with each iteration corresponding to processing one batch of data according to the batch sizes presented in Table
The data in Figure 2 are from the same training runs Figure 1, but with wall-clock time rather than iteration count presented on the x axis. Wall clock time is measured from just before the start of the first iteration of training until just after the last iteration of training, and is inclusive of any overhead from the full-precision to low- now-precision projection in partial pre-training.
Results. Figure presents the end-to-end wall-clock time of partial pre-training and baselines. Due to the combination of more accuracy per iteration from partial pre-training (Figure 1) and the faster training time of the full-precision training phase (Table 2), we find that partial pre-training trains binarized networks faster than standard low-precision training inclusive of all overhead from the partial pre-traine tre-training process. Specifically, in the range of data shown in Figure 2, we find that partial pre-training has harmonic mean speedup over standard low oowpprccision training of: 1.28 x for the CIFAR-10 ResNet-20, 1.26x for the CIFAR-10 10 VGG-16, 1.36x for the ImageNet ResNet- 34, and 1.61 for the MovieLLens-20M NCF.

---

Under review as a conference paper at ICLR 2021
0.825
Test Accuracy
1-bil Training Iteraiins
20RX01
0.800
0.80
160001
Tost: Accuracy
0.775
0.76
0.750
12000
0.725
8000
0.72
0%
25%
50%
75%
100%
4000
32-bit Training Time
0.68
Total training itcrations
4000
808d
12000
16000
20000
40000
28000
16000
32- Training Iterations
36000
24000
1200U
32000
20000
8000
(a) Accuracy when training with varying amounts of full-precision and low-precision iterations.
(b) Accuracy when training with equivalent training time but varying full bull-precisoonllowprreiii split.
Figure 3: Sensitivity of partial pre-training to different combinations of full- full-precision training and low-precision training times.
ANALYSIS
Section demonstrated that partial pre-training allows for faster training of binarized neural networks than standard low-precision training. This section analyzes the hyperparameter choice baked into partial pre-training algorithm splitting the training time evenly between full-precision and low-precision training, finding that
split between full-precision and Iow- low-precision training allows for generally good performance across training times. Due to compute limitations, this section only analyzes the CIFAR-I0 10 ResNet-20. We find that while this even split is not optimal in all circumstances, it is generally applicable hyperparameter choice that leads to accuracy competitive with the best choice of training time split within the given training budget.
Methodology. Figure 3a shows the accuracy from training CIFAR-10 ResNet-20 with modifica- tion of partial pre-training that has non-even split between full-precisio and low-precins wwwpprcciiion training. Each point is the median accuracy across 3 trials when first training for the number of iterations specified on the x-axis with the original learning rate schedule compressed to the number of training iterations, then projecting to low-precision and training for the number of iterations specified on the y-axis again with the original learning rate schedule compressed to the number of training iterations. Figure is generated from the same data as Figure 3a, showing the accuracy along the top-left to bottom-right diagonals of Figure 3a. That is, Figure 3b shows partial pre-training' 's accuracy as the split between full-preedd precision and low-precision training is varied for a given training budget, with each different line representing different training budget.
Results. Figure 3 shows that the even split between full- precisioc and low www-precision training baked into partial pre-training is a generally applicable hyperparameter choice that leads to accuracy competitive with the best choice of training time split within the given training budget. When training for short duration (lcading to lower overall accuracy, shown in the lower lines in Figure 3b). spending more time on the full-precision training phase leads to higher accuracy than spending more time on the low-preewt cow-precision training phase. When training for a longer duration (shown in the higher lines), spending more time on the low-precision phase leads to higher accuracy than spending more time on the full- null-precision phase. Across all training budgets, splitting time evenly between full-precision and low-precision training leads to competitive accuracy with the best choice of hyperparameters.
7 LIMITATIONS
While we find that partial pre-training accelerates binarized neural network training time across the networks and datasets experimented on Section 5, there are dimensions of the hyperparameter space that we do not explore. We do not evaluate binarization techniques other than the commonly

---

Under review as a conference paper at ICLR 2021
used PACT activation scaling (Choi et al., 2018) and static weight scaling. We also do not evaluate learning rate schedules other than compressing the original learning rate schedule from the fist phase of training, it is plausible that other learning rate schedules better tuned to the technique or network could train even faster (Renda et al., 2020), but we do not evaluate these alternative schedules.
RELATED WORK
Several other papers discuss related ideas to partial pre-training. However, this paper is the first to propose partial pre-training as a method for training binarized neural networks from scratch, to provide a concrete partial pre-training algorithm, and to systematically evaluate partial pre-training as a method of speeding up the training of binarized neural networks.
Faster binarized network training. Partial pre-training's 's speedup is due to the slow training speed of low- precision network training, both in terms of iteration count and wall-clock speed per iteration; partial pre-training would be an ineffective training method given a method that could train low-precision networks at an equivalent faster rate than the full-precision counterparts. Here survey techniques that propose speeding up low-precision network training, showing that partial pre-training not supplanted by better low-precision training techniques. Zhou et al. (2016) propose a technique that uses low-bit quantization in the backward pass in addition to the forward pass, but neither measure the iteration count slowdown induced by gradient quantization nor measure well-clock speedup of gradient quantization. De Sa et al. (2018) propose a technique to reduce quantization error between latent and quantized weights to improve asymptotic training speed; however, the proposed algorithm is only evaluated on 8-bit networks. Alizadeh et al. (2019) suggest that commonly applied weight and gradient clipping techniques constrain the learning rate that can be used during training, and propose disabling gradient clipping allow for larger learning rates and therefore faster training; however, they do not disable the quantization during training (still incurring overhead from quantization operators in the training graph), they do not evaluate the wall-clock speedup of the approach, and further the approach requires bespoke tuning of the accelerated training schedule per-network.
Full-precisiog pre-training. Alizadeh al. (2019) and Bethge et al. (2019) both find that binariz- ing a fully trained full-precision network to low- cowprrccision then training the binarized network for short amount of time allows for rapid re-training of the low-precision network. However, Alizadeh et al. and Bethge et al. dismiss the approach because the preciieciionn network that is fine- fune-t for a short amount of time does not achieve the same accuracy as a low-precision network trained from scratch for a longer duratiio. Alizadeh al. and Bethge al. also only evaluate fine-tuning from fully trained full-precision networks, rather than the partially-trained full-precision networks presented in this paper, and do not evaluate the wall-clock speedups of such an approach.
Multi- phase binarized network training. Zhou et al. (2017) propose an approach that incremen- tally quantizes neural networks by partitioning weights into disjoint groups based on the magnitude of the pre-trained weights, first binarizing weights with large magnitudes then binarizing weights with small magnitudes. Zhou et al. start this process from fully-trained cill-precision network, and do not consider observed training speedups, only accuracy gains. Verhoef et al. (2019) gradually reduce the number of bits used for weight representation over the course of training, with the goal of improving accuracy; Verhoef et al. do not evaluate the training speedup achieved by the approach, nor does their technique allow for taking advantage of faster preccisenn training.
CONCLUSION
Binarizing neural networks makes them more efficient for inference, but binarized neural networks trained from scratch typically take more iterations and more wall-clock time to plateau in accuracy than their full- full-precision counterparts. In this paper we demonstrate a technique, partial pre-training, that allows for faster training of binarized neural networks by exploiting fast full-precision training then fine-tuning at low-precision. Without introducing any new hyperparameters, we find that partial pre-training can train VGG, ResNet, and NCF networks CIFAR-10. ImageNet, and MovieLens- 20m between 1.26x x and 1.61 x faster than standard 1 1-bit training. All together, we find that partial pre- nre-training is simple and effective approach for accelerating binarized neural network training. Partial pre- tre-trainig is a step towards the goal of binarized neural network training procedures that can match the efficiency gains of binarized neural network inference.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg. Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 263-283, 2016.
Milad Alizadeh, Javier Fernández-Marqute Nicholas D. Lane, and Yarin Gal. An empirical study of binary neural networks' optimisation. In International Conference on Learning Representations, 2019. URL https: http////pppeeeeeeeeeeeeeeeeeeeee httt:/oormmmn:=JJECGGA
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. In Advances in Neural Information Processing Systems 32, pp. 9950-7958. Curran Associates, Inc., 2019.
Joseph Bethge, Haojin Yang, Marvin Bornstein, and Christoph Meinel. Back to simplicity: How to train accurate bnns from scratch?, 2019.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks, 2018.
Christopher De Sa, Megan Leszezynski, Jian Zhang, Alana Marzoev, Christopher R. Aberger, Kunle Olukotun, and Christopher Ré. High-accuracy oowpprecision training, 2018
Joshua Fromm, Meghan Cowan, Matthai Philipose, Luis Ceze, and Shwetak Patel. Riptide: Fast end-to-end binarized neural networks. In Proceedings of Machine Learning and Systems 2020, 2020.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour, 2017.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML' 15, pp. 1737-1746. JMLR.org, 2015.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. International Conference on Learning Representations (ICLR), 2016.
Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst. 5(4), December 2015. ISSN 2160-6455. doi: 10.1145/282787.. URL https:/ / //oii .org 10 1145/2827872.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In /EFE/CVF Conference Computer Vision and Patlern Recognition, 2016.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, pp. 173-182, Republic and Canton of Geneva, CHE, 2017. International World Wide Web Conferences Steering Committee. ISBN 978 7781450349930 30. doi: 10. 10.114553033912.39555699 URL https: //oii//oiii// hta.ddi.oogggg/// /1l 114 /141/113111111 912. 3052569.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Yaniv, and Yoshua Bengio. Binarized neu- ral networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4107-4115. Curran Associates, Inc., 2016. URL http:/ /ttp:://paperr.... wwarrs.niiaa nips. . cc/ wscppappr///eeee 6573 bi n red-neural- networks pdf.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.

---

Under review as a conference paper at ICLR 2021
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Peter Mattson, Christine Cheng, Cody Coleman, Greg Diamos, Paulius Micikevicius, David Patterson, Hanlin Tang, Gu-Yeon Wei, Peter Bailis, Victor Bittorf, David Brooks, Dehao Chen, Debojyoti Dutta, Udit Gupta, Kim Hazelwood, Andrew Hock, Xinyuan Huang, Bill Jía, Daniel Kang, David Kanter, Naveen Kumar, Jeffery Liao, Guokai Ma, Deepak Narayanan, Tayo Oguntebi, Gennady Pekhimenko, Lillian Pentecost, Vijay Janapa Reddi. Taylor Robie, Tom St. John, Carole-Jean Wu, Lingji Xu, Cliff Young, and Matei Zaharia. MIperf training benehmark. In Conference on Machine Learning and Systems, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vitoto, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, gegh-performacce deep learning library. In Advances in Neural Information Processing Systems 32, 2019.
Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. Binary neural networks: A survey. Pattern Recognition, 105:107281, 2020. ISSN 0031-3203, doi: https://oi..org/10.. 1011 2020 107281. URL http:: /twp:////www////// sci enceeiii rect com/ httpenceeertttlep..ppppppeecpppe http:/c/eeeeeee//001100220000000
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020. URL
attps::/cppenrewee wttp:e/wwweeeoeoonieeeeeeee Lorun?ld-SSg
net
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 2015.
Taylor Simons and Dah-Jye Lee. A review of binarized neural networks. Electronics, 8(6), 2019. ISSN 9292. doi: 10.3300eeeectomicc3866666. URL //w::///wwwwwww http://wwwwpppppttt com/ 2079-9292/42 6/661.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2014.
Bram-Ernst Verhoef, Nathan Laubeuf, Stefan Cosemans, Peter Dehacker, loannis Papistas, Arindam Mallik, and Diederik Verkest. Fq-conv: Fully quantized convolution for efficient and accurate inference, 2019.
Tianyi Zhang, Zhiqiu Lin, Guandao Yang, and Christopher De Sa. Qpytorch: A low-precision arithmetic simulation framework, 2019.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights. International Conference on Learning Representations, 2017.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, abs/ 00666666666660 C6160.
2016. URL http: //arvv/...vvv//////////ear/aaa org/
10


Reviews:
[{"review_id": "XV0UJQAVnL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper addresses the problem of slower training speed with low-precision training of neural nets. It presents a simple solution: first train with full precision on half of the budgeted trainining time, then train with low pecision in the remaining time. This achieves 1.2x - 1.6x speedup compared to low-precision training. \n\nPro:\n- The proposed idea is simple and it is nice to see that it works\n\nCons:\n- I feel there is not enough content (in terms of ideas and experiments) to warrant a full paper. Compared to other ICLR papers, the contributions seem on the low side. See suggestions below. \n- The paper mainly compares the proposed method with low-precision training, but the results would be stronger if also compared with full-precision training followed by quantization. This is especially because all the low-precision accuracies trail behind the full precision ones in the results. \n\nSuggestions:\n- Experiment with more quantization methods. Currently we do not know whether the proposed method is uniquely suited to the PACT method used, or is a general technique. \n- A 1.2x-1.6x speedup on a training process that takes 600 seconds (e.g. CIFG-10 result in Fig 2) seems not so impactful in the grander scheme of things. Even the 12500 second training time is just <4 hours. I understand the results should transfer, but the results would be more impressive if done on larger training runs.\n\nMinor questions/comments:\n- Please comment on the terminology. Is what you call low precision training similar to the mixed-precision training now implemented in TensorCore NVIDIA GPUs?\n- Table 2: what is it/s. Is it iterations per second? \n- Table 1 shows learning rate schedule for t up to 60k in CIFAR or 450k in ImageNet, but Figure 1 stops way before those points in the x-axis. This was somewhat confusing. \n- Another clarification point about Fig 1 and 2: for the proposed method, is the full precision training part of the time included in the calculation? I believe so but just want to double-check. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XCN2gwRGFVt", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed one simple method called partial pre-training to speed up the training of binary neural networks (BNN). The pros and cons are as follows:\n\nPros:\n1. The partial pre-training method is simple and easy to implement;\n2. For standard binary optimizer like the straight-through-estimator (STE), the method improves the training speed to some extent;\n\nCons:\n1. The main concern of the proposed method is kind of heuristic and there is a lack of theoretical explanation, whether rigorous or not, why this method works. \n\n2. As described in Section 7 by the author themselves, there are several apparent limitations of current evaluation, e.g., several dimensions of the hyper-parameters are not explored, other Binary optimizers are not considered, different learning rate schedules, etc. As a result, it is unconvinced that the partial pre-training could universally improve the speed as a general method. In addition, the improvement of speed-up are not very apparent especially for ResNet-20 and ResNet-34 as shown in Fig. 1 and Fig. 2, i.e., it took approximately the same time to reach the final precision even though the proposed method achieves higher accuracy before saturation. Given the inadequate evaluations and lack of theoretical explanations, this might be due to unfair comparison. \n\n3. There is a lack of explanation of the contradiction result with previous result proposed in Alizadeh et al (2019), which dismissed the approach of pre-training. Is there good explanation of such an opposite result? It would be better to show results of different split between full-precision training and low-precision training. \n\n4. Regarding pre-training for BNN, there are some related works from the Bayesian perspective. In Shayer et al. (2018), they used the result of full-precision training as the prior for the binary training, which improves the final result, as opposed to Alizadeh et al (2019). The Bayesian perspective provides an explanation of the effectiveness of a good prior. In Meng et al. (2020), they showed that STE could be viewed as Bayesian and obtained good result even with a uniform prior. Also, the posterior obtained after full-training (binary) could be used as prior to enable continual learning, which shows effectiveness of the prior. Given the above results, since the authors demonstrate that partial pre-training can increase the speed for STE, does this imply that partial pre-training provides a better prior than full pre-training? If so, why? \n\nShayer, O., Levi, D., and Fetaya, E. Learning discrete weights using the local reparameterization trick. ICLR, 2018.\nMeng, X, Bachmann. R., Khan. E. Training Binary Neural Networks using the Bayesian Learning Rule. ICML, 2020.\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "V3vuXmHW9Wb", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper suggests a method for training binary neural networks. The proposed method is to partially train with full precision and then continue with binarized training using the straight-through estimator. The method is very simple and there is very limited technical contribution, so in order to be worthy of publication it needs to be supported with compelling experimental results. Unfortunately this is not the case.\n\nThe main claim is speeding up training by a factor of 1.2/1.6. While this can help the importance of speeding up training (unless by a much larger factor) is quiet limited. The useful speedup BNN present is at inference time, and the time to train a network is of much less importance (at least for this kind of speedup).\nFrom Fig.1 I am not convinced that the speedup claims hold. You can see the test accuracy for binary training on 3 of the experiments reaches the Partial Pre-training level but it doesn't look like it completely flattened out yet. It looks like if you want take the best model on test, then you don't get any speedup\nResults on cifar-10 seem quiet poor (both full precision and 1-bit). For example \"Learning discrete weights using the local reparametrization trick\" gets 93.2 1-bit acc on cifar-10 with VGG.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5KaoPSoBUCv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Comments\nSummary:\n\nThe authors propose a fast binarized neural network training algorithm that splits the whole training process into the full precision training stage and the binary training stage. The experimental results show some improvement about training speed in terms of iterations and wall clock time. Generally, the paper is well written.\n\nStrength:\n--The idea is reasonable and the method is presented clearly.\n--The experimental results indicate that the proposed method can accelerate the convergence of the binary networks on image classification and collaborative filtering.\n\nWeakness:\n--The comparison between the proposed method and quantization-finetuning method is lacked, which seems like a closely related work.\n--The analysis of the proposed method should be also enhanced. The reason why partial pretraining can improve the training of binary neural networks should be investigated more deeply.\n\nComments:\n(1) The authors claim that the proposed method allows for faster from-scratch training of binarized neural. This seems contradictory to the partial pretraining. The authors may provide more discussion and clarification. \n(2) The improvement of the proposed method is not significant. The proposed algorithm speeds up the training marginally and cannot improve the final test accuracy. This limits the contribution.\n\nOverall, the reviewer doesn’t recommend accepting this manuscript at its form. The author may demonstrate more differences between the proposed method and the standard quantization-finetuning method.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: ## Description 

The paper asks the question whether it is possible to accelerate training a binarized neural network from scratch to a given target accuracy [by starting with training a full-precision network]. The main claimed contributions are: the idea to use *partially* pretrained networks, experimental evidence regarding the split of the training budget and measuring the speed-up.

## Review Process and Decision

All four reviewers agreed in the low rating of the paper and in the opinion that the paper is not a significant contribution. The area chair supports rejection.

## Details

It has been already observed that pre-training  in some form is needed for achieving the best accuracy: 
Rastegari (2016) XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Bulat (2019), "Improved training of binary networks for human pose estimation and image recognition"
Martinez (2020): "Training Binary Neural Networks with Real-to-Binary Convolutions"

Alizadeh, (2019 fig. 4) notice that pre-training can be viewed as a speed-up, but in their setup find that training from scratch gives a better accuracy. Bulat (2019) and Martinez (2020), on the contrary do use pre-training to achieve the best accuracy.  It is questionable whether the pre-training in these works is partial or not. I believe the result largely depends on the pre-training method used, which is not discussed in depth in the submission. More generally, some graduated optimization methods such as graduated smoothing or graduated non-convexity are known to help in finding better solutions / lead to faster optimization and in fact Bulat (2019) use pre-training with gradual transition from smooth activation to the sign function.  Relative to these points the technical contribution (one paragraph in the paper) is not significant. The empirical part of the contribution shows some effect, but does not indicate a breakthrough on its own. An investigation / design of pre-training schemes could make it more substantial.

The empirical analysis proposed does not rule out, and in fact supports, the methodology that for the best final accuracy, the full rather than partial pre-training is useful.

The gain of speed-up by a factor 1.3 (diminishing to close to 1 if we are interested in the best accuracy), is of little practical interest. In particular, a slight code optimization can give a similar speed-up without the complexity and hyperparameters involved in pre-training. The authors write  "we are not aware of any effort to exploit binarization during the training phase"
There are available public implementations that can optimize the forward pass of binary networks, in particular on GPU, while backward pass can stay in full precision. It could give a similar speed-up. In particular Courbariaux (2016) provides a GPU kernel and proposes a variant of BatchNorm with bit shifts rather than multiplications, specifically used at training time.
Making the emphasis on a relatively small speed-up that can be obtained to train sub-optimal models, in my view is not a good strategy to present this work. Rather the phenomenon that (partial) pre-training helps with the goal to improve the training methods more substantially I find of higher interest.

Finally, I agree with the reviewers that the lottery ticket hypothesis (Frankle, 2020) work speaks of the speed-up only hypothetically and its main (and fairly in-depth) contribution is in demonstrating and investigating an interesting phenomenon about training and initialization, which I do not see relevant to this submission.

Decision: Reject