Optimizing Data-Flow in Binary Neural Networks
Authors: Lorenzo Vorabbi
Davide Maltoni
Stefano Santi
Venue: Submitted to ICLR 2023 (2023)
Keywords: Binary
Networks
Quantization
1-bit

Abstract: Binary Neural Networks (BNNs) can significantly accelerate the inference time of a neural network by replacing its expensive floating-point arithmetic with bit-wise operations. Most existing solutions, however, do not fully optimize data flow through the BNN layers, and intermediate conversions from 1 to 16/32 bits often further hinder efficiency. We propose a novel training scheme that can increase data flow and parallelism in the BNN pipeline; specifically, we introduce a clipping block that decreases the data-width from 32 bits to 8. Furthermore, we reduce the internal accumulator size of a binary layer, usually kept using 32-bit to prevent data overflow without losing accuracy. Additionally, we provide an optimization of the Batch Normalization layer that both reduces latency and simplifies deployment. Finally, we present an optimized implementation of the Binary Direct Convolution for ARM instruction sets. Our experiments show a consistent improvement of the inference speed (up to $1.77$ and $1.9 \times$ compared to two state-of-the-art BNNs frameworks) with no drop in accuracy for at least one full-precision model.

Under review as a conference paper at ICLR 2023
OPTIMIZING DATA-FLOW IN BINARY NEURAL NET- WORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Binary Neural Networks (BNNs) significantly accelerate the inference time of neural network by replacing its expensive floating-point arithmetic with bit- wise operations. Most existing solutions, however, do fully optimize data flow through the BNN layers, and intermediate conversions from 1 to 16/32 bits often further hinder efficiency. We propose novel training scheme that can in- crease data flow and parallelism in the BNN pipeline; specifically, we introduce clipping block that decreases the data-width from 32 bits to Furthermore, we reduce the internal accumulator size of a binary layer, usually kept using 32- bit to prevent data overflow without losing accuracy. Additionally, we provide optimization of the Batch Normalization layer that both reduces latency and the
simplifies deployment. Finally, we present an optimized implementation Binary Direct Convolution for ARM instruction sets. Our experiments show consistent improvement of the inference speed (up to 1.77 and 1.9x x compared two state-of-the-art BNNs frameworks) with no drop in accuracy for at least one full-precision model.
INTRODUCTION
In the last decade deep neural networks (DNNs) have come demonstrate high accuracy on many datasets like ImageNet Ruusakovssy et al. (2015), outperforming legacy methods and sometimes even human (xperts(Krizhevsky al. (2012) Simonyan Zisserman (2014), S.eegeyyy et al. (2015), He et al. (2 These improvements have been achieved by bhyiineee increasing the depth and complexity of the network, leading to intensive usage of computational resources and memory bandwidth. Large DNN models run smoothly expensive GPU-based machines but cannot be easily deployed to edge devices (i.c., small mobile or loT systems), which are typically more resource-constrainee Various techniques have been introduced to mitigate this problem, including network quantization Choi al. (0188: Hubara et al.] [0166]] Lin et al. (017)): |aategari et al.) (2016); Zhou et al.] (2016), network pruning Han al. (2015); Wen et al. (2016) and efficient architecture design Howard et (01
Tan & Le 2019).
Recent work on quantization (e.g- Courbariaux et al. (2016); Hubara al. (0016) Liu et al. 2018): Martinez et al. (2020) has shown that a DNN model can be even quantized to 1-bit (also known as binarization) thus achieving a remarkable speedup compared the full precision network. The memory requirement of such a binarized DNN (BNN) is drastically reduced compared to DNN of the same structure, since a significant proportion of weights and activations can be represented by 1-bit, usually {-1. +1), }. In addition, high- precisioo replaced by faster XNOR and popcount operations.
operations can be
However, the aggressive quantization can make BNN's less accurate than their full-precision coun- terparts. Some researchers showed that the performance loss often arises from the gradient mismatch problem caused by the non-differentiable binary activation function Darabi et al. (2018): Liu et al. (2018). This con-differentiability of the quantization functions prevents gradient back back-propagation through the quantization layer. Therefore, previous works used trriightthrough-esttnas approximate the gradient on non-differentiable layers Bengio et al. (2013): Hubara et al. (2016).
(STE) to
Furthermore, to prevent that the binarization of weights and activations leads to feature maps of lower quality and capacity, a combination binary and floating-pon point layers is usually adopted. Unfortunately, each time binary layer connected to floating-poin one, the efficiency of the

---

Under review as a conference paper at ICLR 2023
pipeline is compromised by input/output layer data type conversion. In addition, the internal par- allelism of a binary layer depends on the encoding of the accumulator, which is often maintained at 32 bits to prevent overflow. In this paper
present several optimizations that allow training BNN with an inter-layer data width of 8 bits. Most prior work on BNN's emphasize overall net- work accuracy; in contrast, our aim is to preserve initial accuracy while improving efficiency. Our contributions (graphically highlighted in Figure Ti and lii)
can be summarized as follows:
a novel training scheme is proposed to improve the data-flow in the BNN pipeline (Section
3.1 : specifically, we introduce clipping block to shrink the data width from 32 to 8 bits while simultaneously reducing the internal accumulator size.
we provide (Section 3.2) an optimization of the Batch Normalization layer that decreases latency and simplifies deployment.
we optimize the Binary Direct Convolution method for ARM instruction sets (Section
To prove the effectiveness of the proposed optimizations in Section 4
we provide experimental eval- uations that show's 's the speed-up relative to state-offtth-art BNN engines like ICC. Bannink et al. 202T and DaBNN Zhang et al. (2019).
RELATED WORK
BNNs were first introduced by [ourbariiux et al. (2016), who established an end-to-end gradient pack-propagation framework for training the binary weights and activations. They achieved good success on small classification datasets including CIFAR10 Krizhevsky et al. (2009) and MNIST
Netzer et al. (201T). but encountered severe accuracy drop on ImageNet.
Many subsequent studies focused on enhancing BNN accuracy. Rastegari et al. (016)) proposed XNOR-Net, where real-valued scaling factors used multiply the binary weight kernels, and this methodology then became a representative binarization approach to bridge the gap between BNN's and their real-valued counterparts. The Bi-Real Net Liu al. (2018) added shortcuts propagate values along the feature maps, which further boosted the top-1 accuracy on ImageNet; nevertheless, the model still relies on 32-bit floating point to execute batch normalization and addition operator (as shown in Fig. Ti
One of the major weaknesses of BNN's is the gradient approximation by the STE binarization func- tion Courbariaux et al. (2016). In fact, STE computes the derivative of sign as if the binary operation was a linear function, as reported in the following formula
A(z) max( 1, min(1, x)), STE(z) = A'(x) = [
The implementation STE stated above, uses the STE with the addition that it cancels the gradients when the inputs get too large Hubara et al.) (2016). STE provides coarse approximation of the gradient that inevitably affects the testing accuracy of the BNN. To address this issue, other recent studies tried to improve the performance of BNNs by adopting proper optimization method for the guantization. Inspired by STE, many works update the parameters approximately introducing auxiliary loss functions Gu et al. (2019); Qin et al. (2020).
Besides many efforts to develop more efficient and accurate architectures, a few works have provided benchmarks real devices such as ARM processors. Based on the analysis provided in Bannink ct al. (002I). the fastest inference engines for binary neural networks, with proven benchmarks (Section 4 of Bannink et al. (2021)), are LCE and DaBNN.
DATA-FLOW OPTIMIZATIONS
As illustrated in Fig. Пa, the most commonly used BNN architectures (e.g., VGG and ResNet) have four essential blocks in each convolution/fully-coneeee
(CONV/FC) layer: sign (binarization), XNOR, popcount and Batch Normalization (BN). Since the weights, inputs and outputs are all bi- nary, the traditional popcount). XNOR and popcount are usually fused to improve efficiency. The use of Batch Nor- malization after each binarized layer is very important in BNN's pointed by Santurkar et al.
operation is replaced by XNOR and bit counting (i.e.,
2

---

Under review as a conference paper at ICLR 2023
2018 Figures Li and Lii (b and c) point out our proposed BNN optimizations during training and inference. Before discussing them in detail, we show the data-flow bottlenecks that affect existing solutions and then describe how to reduce them.
cony
POOLING
DONE
ccNV
POCT
CONV
COMY
X000
1b- Your
Reating point SN
poocouutt
Standard
XHOR poppouut
Cli to 1:22, -123]
Ploating point
Medale- Training
Prpoount
buturating
EN
aittlmens)
Dettmibedd BAN Moddle---eoeee Reforence
(ii) ResNet style block.
(i) VGG style block.
Figure 1: a) Standard BNN blocks used in Rastegari et al. (0016) and Liu al. (2088). b) BNN block with output convolution clipping used during training. c) Optimized BNN block adopted during inference. Popcount operation is performed using saturation arithmetic in order keep the data width to 8 bits at inference time. BN is replaced in by comparison in case j while in BN is 8-bit quantized.
211 we report an example of binary convolutional layer outputs for VGG and ResNet model. he ranges of activation values after popcount (green histograms) exceed the interval (-128; + 127] so adopting 8-bit encoding would lead to overflow. To prevent such a data loss, most of the existing BNN frameworks (including Bannink et al. (20TT) (/Zhng et al. (2019)) encode such data in 32-bit floating point. On the other hand, the ranges of values after BN (red histograms in Fig. are more limited.
In Figs. 21 and
In this paper, propose quantize the popcount output with 8-bit integers through a two-stages training procedure, which is designed to preserve model accuracy, In the next subsection we show how to apply this technique to VGG and ResNet models.
(i) VGG Small output layers
(ii) ResNet output layers
Figure 2: Example of output distributions after binary convolution. refers to a VGG style network whi ResNet architecture. Green shows the distribution before the BN layer and red afterward.
3.1 TWO-STAGE CLIPPING
Our training procedure selectively executes or skips clipping operation at each binary layer (row b Figs. Di and Ti blue blocks). A two-stage training method is introduced to avoid accuracy loss when clipping enabled: during first warm-up stage, the model is trained without any range constraints, while in the second stage (details are reported in Algorithm[]] I the network is trained with
We actually consider the symmetric quantization interval 1177;++27 because this choice enables substantial optimization opportunity, as reported in Appendix Jacoh al. 2018).
3

---

Under review as a conference paper at ICLR 2023
the clipping block enabled. Based the high accuracy reached at the end of the first training stage, in the second training stage the model better tolerates clipping 8-bit quantization; we experimentally found that this approach preserves the accuracy of model that does not contain clipping.
Algorithm 1:
Second stage training procedure for BNNs
Input: The full precision weights W; the input training dataset Output: BNN model with convolution output clipped
Initialize network weights W
repeat
// Forward Propagation for | to L do
Binarize floating point weights: W = sign (Wi)
Binarize floating point features previous layer: Frim = sign (FI-1)
Compute binary convolutions features: F Fil-1 bin * W bin
Clip F values to interval [-127; +127] with:
Fl out clipped maz (min (127, Fr Fowe) - 127)
Perform Batch Normalization: BN (Fupte))) oe Foue etipped v²
+ g
// Backwa rd Propagation for 1 = 1 to I. do
Compute gradients based on the binarization weights W clipped convolutions
Fo elipped and batch normalization output BN (F' clipped Update full-precision weights W
11
until convergence
3.2 BATCH NORMALIZATION OPTIMIZATION
The BN layers after the clipping also optimized/8-bii quantized further increase the data- flow of the inference pipeline. The Batch Normalization layer scales and shifts the output of the CONV/FC layer follows:
Ft
BN (Flut) =
1
+
(I)
where y, a and are learned parameters and Flut is the ouput feature of layer that is the input of BN function.
The BN optimization depends on the network model: VGG or ResNet. In both cases we show that it is possible to keep the inter-layer data type 8-bit with appropriate changes the binary layer structure.
VGG style block. When the BN layer is inserted in a pipeline similar Fig. where the following block is still binary, the BN operation can be simplified replacing multiplication and division in Eq. I with simple comparison with threshold The simplification of Eq. leads to:
+1 if BN (Fiua) 2 0
sign (BN (Flue) =
-1
otherwise
Fi + 3 020 u B4
(2)
+1 if Ft out 2 + else
(when
I 2 0 < 0)
sign (BN (Frue))
=
- if F W r else +
(when
4

---

Under review as a conference paper at ICLR 2023
The threshold of Eq. 2 can be computed offline and easily quantized to 8 bits in order to exploit the output features of layer I. Therefore, when multiple BNN modules are stacked, Batch Normalization can be replaced by threshold comparison according to Eq. 2
ResNet style block. When BNN block is placed in a ResNet style pipeline, followed by an addition operator, Fig. Tii the BN layer can be executed while both scaling and bias factors to 8 bits. As reported Fig- 3] the internal data representation of quantized BN layer is expanded to 16-bit to preserve accuracy during quantization but the input/output data type still remains within 8 bits. The complete iterative quantization procedure adopted is reported in Algorithm 2 of Appendix O The procedure iterates over floating- point layers (basically all BN layers) inside the binary blocks and, for each one: computes the quantization scale; quantizes; freezes the weights; and retrains the remaining layers.
Practiorel bits laa
Semmratic E-bit quantiation
peaclut
204
Rill puatiedd
Baaling
8-bit symmetric quantization procedure that reserves fractional/integer bits based on the range of input 32-bit floating point values. b): implementation of the BN layer with 8-bit quan- tization using an internal 16-bit representation to preserve accuracy.
Figure 3: a):
3.3 BINARY DIRECT CONVOLUTION OPTIMIZATION ON ARM
The GEMM (GEneral Matrix Multiplication) is a widely adopted method efficiently implement convolutions. However, as reported in Zhang et al. (2018b), the GEMM approach increases the memory footprint of the model, making a model's port to an embedded device more difficult. Fur- thermore, GEMM routines not always optimized for convolutions ARM devices, in particular ARMv8 (64-bit ARM architecture) and its relevant operations such as vcount and addv.
vcount takes an N-byte vector as input and outputs N-byte vector containing the number of 1s present in each input byte. addv takes an N-byte vector as input and outputs the sum of the bytes single value.
Inspired by Zhang et al. (018b) and Zhang et al. (019) we propose a hybrid direct binary convo- lution (see Fig. (4) that uses both the addv instruction and the common add operations. The binary convolution is usually composed of three different steps: binarization/bit-packiii volution. Zhang et al. (2019), executes these steps in a sequential way. contrast, we devise more friindfy approach that collapses the previous steps in one operation executed with tiling. We also devise different kernel memory layout that better fits ARMv8 SIMD processing instructions. as illustrated in Fig.
padding and con-
The implementation details of our binary convolution are reported in Fig. 5 The operation Extract sign bit executes the binarization, bit-packing and padding. Then, the (bit-wise) XNOR output is consumed by the popcount operation (vent 8-bit wise, add and addv). On the ARM architecture, the latter can be implemented with vcounf and a sequence of additions (addv instructions). We decided to implement several pair-wise additions and only a final addy instruction (which is more expen- sive). The entire convolution process does not provide intermediate outputs but instead processes the input data as a whole. is worth noting that the clipping operation can be obtained for free on ARM devices by exploiting its saturation arithmetic; all the addition operations (add and addv) can be limited to the fixed range [-127; + +127 by simply adding the postfix q to the operations and executing max to avoid - 128 value.
5

---

Under review as a conference paper at ICLR 2023
Kernel
Kernel
Input
Kernel Memory Layout: (oufchannete H. W. (n.chhamees)
Figure 4:
The x 7 input image with different channels (denoted by color) con- volved with two separate kernels to obtain 5 5 output with two output channels. To better exploit the SIMD 128-bit registers different memory layout for kernel is devised: coutchennels: Hatter W Waterr: Mechnnnll))
Input
middle
128| top left
Each elemant
Extract sign bit
Each element is L-bit wide
XIOR bitt wise
Kernel 1-bit wide
XNOR output 1-bit
Vent S-bit wise
16 bytes
16 bytes
16 bytes
TR
Sum pairs
TM
ML
Addv
Final renult
Figure 5: The x x 128 input patch is convolved (XNOR + popcount) with one kernel through the Extract sign bit, XNOR and then popcount operations. Popcount performed using vení, summing in pairs the vent output and the last step uses the addv operation. TL (top left), TM (top middle), TR (top right) and ML (middle left) indicate the position of elements inside the 3 x patch.
EXPERIMENTAL RESULTS
In this section, we first evaluate the efficiency result our approach compared to the state-of-art BNN frameworks such as LCE and DaBNN; the comparison is carried out real hardware devices like Raspberry Model 3B and 4B with 64-bit OS. Then, we present various accuracy benchmarks of the proposed two-stage training procedure focusing on CIFAR-10, SVHN and ImageNet, and to two different architectures: VGG and Resnet-18.
4.1 EFFICIENCY ANALYSIS
To validate the efficiency of our method focused the convolution macro-block of Fig. [ and compared the efficiency the proposed approach with LCE and DaBNN, which, to the best of our knowledge, are the fastest inference engines for binary neural networks.

---

Under review as a conference paper at ICLR 2023
Our assessment performed ARMv8 platforms, Raspberry Pi 3B and 4B. We implemented, differently from our predecessors, the convolution operation using ARM NEON intrinsics instead of inline assembly. Intrisics allow to produce code easier maintain and automatically fit both ARMv7 and ARMv8 platforms without losing appreciable performance compared to pure assembly targets Rpi 3B and 4B. Our solution shows clear
code. In Fig. 6 we compare implementations performance improvement for single binarized convolutions for all kernels; the performance boost more evident when the binarizatiion/iipacknn and data conversion operations are considered in addition to binary convolution only. In summary, our solution, including all the optimizations introduced in Section 3, accelerates binary convolution up to 1.77 and 1.9x compared LCE and DaBNN.
Only Binary Convolution
Input binarization (packing) padding convolution
daBNN
deBNN
UCE
Labancy Rasgbbrrry Fl 3B (ms)
THE
TH
5
Cony
height-14
height-5a
helght-28 witth=ud
helght-?
width=s0
wett==20
=4
wethh//
width=2e
woth=
withhe/
channels-:: 6C chennels- 122cchhnnett-2200 charnels-512
channels-64 64 channeit- 125 chammes-
ii) Raspberry Pi 3 benchmark.
Only Binary Convolution
Input binarization (packing) padding convolution
dabnN
0.rs
ouTs
3.0
3.5
Lelenty fheckeeyy mL)
3.0
2.0
3.5
3.04
0.5
0.0
0.0
3X5
3x3
3X5
3x3 heights
Cory
Corv
height::5
he
height:14
heigh: 56
height: 28
height:34 34
beight:7 channe
wus
witth-2
channels-6d
2hamnel-226
channels-522
c-annes-120
256
(ii) Raspberr Pi 4 benchmark.
Figure 6: Latency evaluation of our method compared to DaBNN and LCE on Raspberry 3B ii) and 4B (i) devices. The improved lower latency of our approach is accentuated when considering not only the binary convolution itself (left) but bitpacking and padding as well (right).
4.2 ACCURACY ANALYSIS
We evaluated two VGG-style networks for CIFAR-I0 and SVHN: VGG-11 Xu et al. (2019) and VGG-Small Zhang al. (2018)) which both high-capacity networks for classification. Pre- trained Larq binary models (BinaryResNetE188 and BinaryDenseNet28) were adopted to evaluate the accuracy ImageNet.
Results on CIFAR10 and SVHN. For CIFARIO the RGB images are scaled to the interval [-1.0;+1..0 and the following data augmentation was used: padding of pixels for each

---

Under review as a conference paper at ICLR 2023
VGC11 LoSs
VGGSmall Loss
VGG11 - Loss
VGCSmall Loos
Thariig
0.5
Traning
0.0
Traring
1.3
1.0
0.6
0.4
0.5
0.4
02
20
Tpoch
VSS11
VGGSmall
VSG11
VSGSmall
3.00
1.00
0.90
0.60
0.90
0.50
0.45
0.85
0.85
0.30
0.10
0.50
WVV
0.80
6.15
0.75
50
100
30
100
10
20
30
25
50
200
Pparh
Tpoch
Tpoch
Fporh
(i) CIFARIO 10 first training stage curves.
(ii) CTFAR 10 second training stage curves.
VGGI
VGGSmall
VGGSmail
Traniing
Trairing
Taning
0.20
0.75
0.30
0.8-
0.6
0.25
0.20
0.30
0
20
60
40
60
20
40
60
Spech
Epech
Epech
Epoch
VGG11 Acturacy
VCCSmell Accuracy
VGG11 Accuracy
VGGSmall Accuraty
1.00
1.20
1.00
0.95
0.95
0.95
0.85
0.80
6.75
0.75
Thaingg
0.70
Traniig
Tharing
Traming
0.70
Toxt
Tist
Test
Telk
0.25
0.55
Pporh
Tpach
Sports
Fporh
(iii) SVHN first training stage curves.
(iv) SVHN second training stage curves.
Figure 7: Training loss and testing accuracy curves for VGG11 and VGGSmall CIFARIO and SVHN of the first and second training stages.
side, a random 32 x 32 crop and a random horizontal flip. No augmentation is used at test time. The models have been trained for 140 epochs.
On SVHN the input images are scaled to the interval [-1.0; +100] and the following data augmen- tation procedure is used: random rotation (#8 degrees), zoom (0.95, 1.05)). random shift (0; 10)) and random shear ((0; 0.15)). The models have been trained for 70 epochs.
The accuracy achieved by the models is reported in Table I showing that the clip operation does not substantially affect the overall accuracy and the two-stage clipping allows to preserve the original accuracy. Fig. shows the training and validation curves on CIFAR 10 and SVHN; can note that a limited number of epochs is necessary during the second training stage.
Method
Topology
Bit-width
CIFARIO top1 %
SVHN topl %
BNN Courbariaux et al. (2016)
VGGSmall Zhang al. 2018a)
32 FP
93.8
96.5
Man/Subsidhary Network
VGGI Xu et al. 2019
32 FP
83,8
BNN
VGGSmall
1-bit
89,9
96.5
XNOR-Na Rastegari et al. 2016)
VGGSmall
1-bit
82.0
96.5
Bop Helwegen et al. 2019
VGGSmall
1-bit
91.3
MannSubsidaary Network
VGG11
1-bit
82.0
ours ours
VGGSmall
1-bit
88.8
96.1
VGG11
1-bit
83.7
95.5
Table 1: Accuracy comparison of our method with SOTA CIFARIO and SVHN.
8

---

Under review as a conference paper at ICLR 2023
Results on ImageNet. Tests were performed by using pre-trained binary versions of ResNet18 and DenseNet28 Bethge et al. (2019) taken from 200 literature of Plumerai Larg Each BNN module (refer to Fig. D has been modified according to Fig. Residual blocks seem to be more robust to clipping compared to VGG style blocks (Results are in 2
Method
Topology
Bit-width
topl %
top5
XNOR-NeI Rastegari al. (2016)
ResNet-18 ResNet-18
1-bit
51.2
73.2
Bi-Real Liu et al. 2018
1-bit
56.4
79.5
Bethge 2019
BinaryResNett188
1-bit
58.1
80.6
Bethge et al. 2019
BinaryDenseNe1288
1-bit
60.7
82.4
ours
BinaryResNetE118
1-bit
58.
80.6
ours
SinaryDenseNe128
1-bit
60.7
82.4
Table 2: Accuracy comparison of our method with SOTA on ImageNet.
CONCLUSION
This paper introduced several optimization the BNN data-flow that together achieve a speed- up of 1.77 and 1.9x compared state-of-the-art BNNs frameworks, without any aceuracy loss for at least one full-precision model. In the future, we intend to investigate: (i) the application of similar optimization techniques to ternary neworks, which naturally get higher accuracies; (ii) the simplification of the training procedure, possibly collapsing it to a single stage to further reduce training time and complexity.
REFERENCES
Tom Bannink, Adam Hillier, Lukas Geiger, Tim de Bruin, Leon Overweel, Jelmer Neeven, and Koen Helwegen. Larq compute engine: Design, benchmark and deploy state-of-the-art binarized neural networks. Proceedings of Machine Learning and Systems, 3:680-695 2021.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:
arXiv:/308.344 1308.3333 3432, 2013.
Joseph Bethge, Haojin Yang, Marvin Bornstein, and Christoph Meinel. Back to simplicity: How to train accurate bnns from scratch? arXiv preprint arXiv: (Tt6::/9000006777 2019.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: 2018.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1. arXiv preprint arXiv:1602.00 02830, 2016.
Sajad Darabi, Mouloud Belbahri, Matthieu Courbariaux, and Vahid Partovi Nia. Bnn+: Improved binary network training. 2018.
Jiaxin Gu, Ce Li, Baochang Zhang, Jungong Han, Xianbin Cao, Jianzhuang Liu, and David Doer- mann. Projection convolutional neural networks for 1-bit cnns discrete back propagation. In Proceedings of the AAAI conference artificial intelligence, volume 33, pp. 8344-8351, 2019.
Song Han, Huizi Mao, and William J Dally, Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. arXiv preprint 110.v01510 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang- Kwang-Ting Cheng, and Roe- land Nusselder. Latent weights do not exist: Rethinking binarized neural network optimization. Advances in neural information processing systems, 32, 2019.

---

Under review as a conference paper at ICLR 2023
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. Advances in neural information processing systems, 29, 2016.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anteger-aritmeetic-oe vision and pattern recognition, pp. 2704-2713, 2018.
inference. In Proceedings of the IEEE conference on computer
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ila Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Larq. Zoo literature. https://docs...
Online.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. Advances in neural information processing systems, 30, 2017.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang. Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European conference on computer vision (ECCV), pp- 722-737, 2018.
Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binary convolutions. arXiv preprint arXiv:2003.11535 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2250- 2259, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542. Springer, 2016.
Olga Russakovsky. Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)2111-553 2015. doi: 100//s 1223-015-08116y..
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? Advances in neural information processing systems, 31, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409, 1556, 2014.
Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114, PMLR, 2019.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances neural information processing systems, 29, 2016.
10

---

Under review as a conference paper at ICLR 2023
Yinghao Xu, Xin Dong, Yudian Li, and Hao Su. A main/subsidiary network framework for simpli- fying binary neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7154-7162, 2019.
Dongqing Zhang, Jiaolong Yang. Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pp. 365-382, 2018a.
Jianhao Zhang, Yingwei Pan. Ting Yao, He Zhao, and Tao Mei. dabnn: super fast inference frame- work for binary neural networks on arm devices. In Proceedings of the 27th ACM international conference on multimedia, pp. 2272-2275, 2019.
Jiyuan Zhang. Franz Franchetti, and Tze Meng Low. High performance zero-memory overhead direct convolutions. In International Conference Machine Learning, pp. 5776-5785. PMLR, 2018b.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 10X::16606.01600, 2016.
BATCH NORMALIZATION QUANTIZATION PROCEDURE
The 8-bit quantization procedure used to quantize the BN layers, as shown Fig. is reported in Algorithm The quantization procedure adopted is symmetric and keeps unaltered the zero point representation.

---

Under review as a conference paper at ICLR 2023
Algorithm 2: Procedure to quantize the BN floating point layers BNN model where convo-
lution output is clipped.
Input: The full cell-precision weights W; the input training dataset Output: BNN model with BN float layers replaced by 8-bit quantized version
for [ to L do
i is BN floating point then
Compute range of features Flur as: Range' = // N is the number of layer variables (4 for BN) for h to 1 do
min (Fone) ) max (Frwe)
Compute range of variable wi, as: Rangeni = [min (w,) :arr (w
// 1 bit is reserved for sign Compute number of bits used for range as: Range
clip ([1o92 (mar (abs (Rangew,, [0]]
(Rangew, [1] [4)))] 0, 15)
Compute number of bits used for fractional part FracBits, wi = 15 Ronge BaangeBits.g wi.
Sclect the Integer part (range) for all N weights
RangeBils,,t max (Range Bitss
Select the Fractional part for all weights FracBits 15 RongeBita for h 1 to N do
Add quantization noise to floating point weights w/. as: w/ FracBits round (2 (rraeBBsss,. w/))
Replace wA, with w/
Freeze w¹ weights and retrain the model // Export the quanti ized weights of layer I for deployment for h = to N do
13
w! qquantized, = round (2FrueBsts,, we w/
15
12


Reviews:
[{"review_id": "ejV_QSiUWA", "reviewer": "Reviewer_yPN2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper analyses the effect of the width of the data pipeline on the throughput of binary neural networks advocating for a decrease in the number of bits from 32 to 8. A series of adjustments are made to ensure, on the selected model that the performance doesn't drop when switching 32 to 8 bits (including a 2 staged training procedure). The paper also details an implementation technique for such binary convolutions on ARM CPUs. However, the proposed modifications are minor adjustments of existing techniques and are of limited novelty. Moreover, the architecture used are not representative of the current state of the art.", "summary_of_the_review": "The main concerns with the paper are: insufficient/incremental novelty and insufficient empirical results, that do not reflect the current state of the art.", "strength_and_weaknesses": "Strengths:\n- Good latency improvement on Raspberry Pi 3 and 4.\n- Overall, the paper is easy to follow.\n\nWeaknesses:\n1. Limited novelty. It is widely accepted by the community that the main remaining bottleneck in the throughput of binary network is the presence of real valued operations and data. For example, [A] removes completly the BN, while [B] proposes a binary-friendly shift based implementation. While other works investigate quantizing to int8 the BNs and the first and last layer, with [C] investigating the impact of the data width both at train and test time.\nThe proposed modification are incremental at best and are of limited novelty.\n\n2. Insufficient evaluation. Many of the recent models such as [D,E] but not limited to, make various architectural changes to the activation functions, learning various scaling factors and exotic layers. Significant testing is required with such networks that represent the current state of the art. In certain cases the gap between the tested models and current state of the art is more than 10% on ImageNet.\n\n3. Insufficient related work. The related work section is incomplete and misses important works. Most of the methods (network and methodology wise) are before 2019.\n\n4. There are results on a per-layer basis, but no aggregated ones per model. What is the speedup in this case?\n\n5. Perhaps its just a strange coincidence, but why the results from Table 2 before and after re-training have exactly the same acc? I would expect the int8 quantization + 2 staged training to have perturbed the results in either directions.\n\n6. 2-staged approaches are common for training binary networks. Where the baselines from Table 2 also trained using a stage approach for fairness?\n\n\n[A] \"BNN - BN = ?\": Training Binary Neural Networks without Batch Normalization, Chen et al, CVPR-W'21\n[B] Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or −1, Courbariaux et al, 2016\n[C] Enabling Binary Neural Network Training on the Edge, Wang et al 2021\n[D] \"ReActNet: Towards Precise Binary NeuralNetwork with Generalized Activation Functions\", Liu et al, ECCV'20\n[E] High-Capacity Expert Binary Networks, Bulat et al, ICLR'21", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "eOLpJMH6HR", "reviewer": "Reviewer_vs6q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a training scheme to increase data pipeline of binary neural networks when implemented on ARM. To this end, a two-step clipping method and 8-bit batch normalization have been proposed. The clipping method insures that the values can fit into 8 bits so that batch normalization can be performed using the same bitwidth. The proposed method has no impact on accuracy performance and its implementation on Raspberry Pi shows up to 1.9x speedup compared to prior works.\n", "summary_of_the_review": "I believe this paper suffers from the lack of motivation and also novelty. The statements of this paper don't explain why two-step clipping is required to make the inputs of BN into 8 bits while BN is in mixed-precision (8-bit and 16-bit). Besides, 8-bit batch norm has been introduced before in different works.", "strength_and_weaknesses": "Strengths:\n-- The paper provides an insightful challenges about hardware implementation challenges of binary neural networks on ARM.\n-- It provides a hardware implementation on ARM and yields up to 1.9x speedup compared to prior works.\n-- The paper is well-written and easy to understand.\n\nWeaknesses:\n-- The two-step clipping method was introduced to accommodate for 8-bit batch normalization. However, it doesn't seem to be necessary for VGG-style block. According to Eq. (2), all we need to compute is sign of the batch norm's output, which can be achieved by comparing the input of batch norm with the threshold tau. Of course, tau can be represented using 8 bits. Then, the input of batch norm doesn't need to be represented using 32 bits anymore; its 8-bit representation suffices. I am also assuming the scaling factor of binarization is merged with BN, is it a right assumption? For the ResNet-style block, the batch norm is mixed-precision not 8-bit according to Fig. 3. Then, what's the point of representing the input using 8 bits?\n\n-- 8-bit batch norm has been proposed before in literature (e.g., https://arxiv.org/pdf/1805.11046.pdf), which undermines the novelty of this work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Pa_mmQW3ED", "reviewer": "Reviewer_mSLQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tries to accelerate the inference of BNNs by:\n\n1- Quantizing the BatchNorm layer into 8-bits\n2- Simplifying the deployment of BN layer non ResNet like networks\n3- And proposing an optimized assembly implementation of the binary convolution", "summary_of_the_review": "Due to the lack of novelty I recommend to reject the paper. ", "strength_and_weaknesses": "Weaknesses: The paper lacks novelty for a conference like ICLR. Quantization of BN into 8-bits has already been proposed in previous works and the simplification of BN is obvious. The impact of their implementation of BN layer on the acceleration of BNN is not explored. \n\nStrength: The proposed assembly implementation of the binary convolution. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F6A_JlzfvK", "reviewer": "Reviewer_GiJD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes some modifications to the execution flow of BNNs for better performance on low-end hardware, specifically ARM CPUs. Firstly, the authors reduce 32-bit intermediate representations after the XNOR popcount in the BNN to 8-bit values to reduce overhead. Secondly, they replace BN with a comparison operation or use 8-bit quantized BN statistics to perform the computations in lower bitwidth arithmetic. The authors also use specific ARM instructions to implement the BNNs more efficiently.", "summary_of_the_review": "Please see the weaknesses above.", "strength_and_weaknesses": "Strengths:\n- The paper is well-written and it is easy to understand the methodologies applied.\n\nWeaknesses:\n- The paper has very limited novelty. Reducing the intermediate bitwidths to 8 bits rather than the original full-precision values is a technique that has been performed for many years in academia and industry to reduce the overheads, and the particular gradual mapping of 32-bit values to 8-bits during training is also not particularly novel. Additionally, the conversion of full-precision BN operations to 8-bit ones is a natural extension of having 8-bit input/output operands and is not a new contribution on its own. \n- The experimental results only show speedups on specific example layers (see Figure 6). But the important question is what are the \"end-to-end\" latency measurement for a \"full network\" and how does that compare with prior art? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject