BoolNet: Streamlining Binary Neural Networks Using Binary Feature Maps
Authors: Nianhui Guo
Joseph Bethge
Haojin Yang
Kai Zhong
Xuefei Ning
Christoph Meinel
Yu Wang
Venue: ICLR 2022 Submitted (2022)
Keywords: Binary Neural Networks
Hardware-Friendly Neural Architecture Design

Abstract: Recent works on Binary Neural Networks (BNNs) have made promising progress in narrowing the accuracy gap of BNNs to their 32-bit counterparts, often based on specialized model designs using additional 32-bit components. Furthermore, most previous BNNs use 32-bit values for feature maps and residual shortcuts, which helps to maintain the accuracy, but is not friendly to hardware accelerators with limited memory, energy, and computing resources. Thus, we raise the following question: How can accuracy and energy consumption be balanced in a BNN design? We extensively study this fundamental problem in this work and propose BoolNet: an architecture without most commonly used 32-bit components that uses 1-bit values to store feature maps. Experimental results on ImageNet demonstrate that BoolNet can achieve 63.0% Top-1 accuracy coupled with an energy reduction of 2.95x compared to recent state-of-the-art BNN architectures. Code and trained models are available at: (URL in the final version).

Under review as a conference paper at ICLR 2022
BOOLNET: STREAMLINING BINARY NEURAL NETWORKS USING BINARY FEATURE MAPS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent works on Binary Neural Networks (BNNs) have made promising progress in narrowing the accuracy gap of BNNs to their 32-bit counterparts, often based on specialized model designs using additional 32-bit components. Furthermore, most previous BNNs use 32-bit bit values for feature maps and residual shortcuts, which helps to maintain the accuracy, but is not friendly to hardware accelerators with limited memory, energy, and computing resources. Thus, we raise the following question: "How can accuracy and energy consumption be balanced in a BNN design?" We extensively study this fundamental problem in this work and propose BoolNet: an architecture without most commonly used 32-bit components that uses 1-bit values to store feature maps. Experimental results demonstrate that BoolNer can achieve 63.0% Top-1 accuracy on ImageNet coupled with an energy reduction of 2.95 compared to recent state-of-the-art BNN architectures. Code and trained models are available at: (URL in final version)
INTRODUCTION
The recent success of Deep Neural Networks (DNNs) is like the jewel in the crown of modern AI waves. However, the large size and the high number of operations cause the current DNNs to heavily rely on nigh-performance computing hardware, such as GPU and TPU. Training sophisticated DNN models also results in excessive energy consumption and CO2 emission, e.g., training the OpenAl's GPT-3 by Brown et al. (2020) causes as much CO2 emissions as 43 cars during their
lifetime (Patterson al. [.221) Morcover. their computational expensiveness strongly limits their applicability on resource- consurried devices such as mobile phones, 1oT devices, and embedded devices. Various works aim to solve this challenge by reducing memory footprints and accelerating inference. We can roughly categorize these works into the following directions: network pruning (Han et | 20 b). knowledge distillation (Crowley et al. 2018 Polino et al.) | 2018), compact networks (Howard et al.,2011 2017 2019, Sandler et al.) 2018 18 Ma et al. 220 8b Tan al. 2019), and low-bit quantization (Courbariaux et al. [015] Rastegari et al. 2016 Zhou et al. 2016 Hubara el al., 2016). From the latter, there is extreme case, Binary Neural Networks (BNNs), first introduced by Courbariaux et al. (016)) that uses only bit for weight and activation.
As shown in the literature (Rastegari et al. 2016), BNNs can achieve 32x memory compression and up to 58x speedup on CPU, since the conventional arithmetic operations can be replaced by bit-wise xnor and bitcount operations. However, BNNs suffer from accuracy degradation compared to their 32-bit counterparts. For instance, XNOR-Net leaves an 18% accuracy gap to ResNet-18 on ImageNet classification (Rastegari al. 2016). Therefore, recent efforts (analyzed in more detail in Section 2} mainly focus on narrowing the accuracy gap, including specific architecture design (Liu et al., 2018; Bethge et al., 2019 2020 Liu et al. 2020b). real-valued weight and activation approximation (Lin et 2l.110177a Zhuang et al. 2019), specific training recipes (Marünez et al.) (020). a dedicated optimizer (Helwegen et al., (219), leveraging neural architecture search (Bulat et al.. 2020 /Zhao et al.) 2020) and dynamic networks (Bulat et al. 2021). In the existing work, efficiency analysis usually only considers the theoretical instruction counts. However, memory usage, inference efficiency and energy consumption, which are essential to practical applications, have received little attention. Furthermore, Fromm et al. (2020); Bannink et al. (202T) point out that the theoretical complexity is often inconsistent with the actual performance in practice and measurable performance gains on existing BNN models are hard to achieve as the 32-bit components in BNNs (such as Norm, scaling, and 32-bit branches) become bottlenecks. Using 32-bit information flow (e.g..

---

Under review as a conference paper at ICLR 2022
memory oocup curing path exsodtoo
1-
+1
Method
Bitwidth
Energy
Top-1
OPs
(WZAT)
(mJ)
Acc.
(-10")
reduced manory (1-b1 featuniann
Bi-Real-Net
1/1/32
3,90
56.4%
1.63
BinaryConv
BinaryCom
BooINet (ours)
1/1/4
1.33
63.0%
1.81
BatchWoon
+3
-1
BaseNet (ours)
1/1/4
0.83
58.2%
1.54
Shifced
Sign
44
+5
BaseNel (ours)
1/1/1
0.70
53.3%
1.51
32-bi operations
BatchNorm
(c) BoolNet reduces the energy consumption by 2.9x compared to Bi- Bi-RealNett by Liu et al.
+.2
1.
Sign merged durig
-1
inference
(0088).
Design in previous work.
(b) BoolNet design.
Figure The main differences between previous work and BoolNet. BoolNet uses 1-bit feature maps and shifted sign function reducing memory requirements and the need for 32-bit operations.
32-bit identity connections, 32-bit downsampling layers are equipped by almost all latest BNNs, see Figure Tap. and multiplicationn/iiissooo operations (in BatchNorm, scaling, average pooling etc.) significantly increase the memory usage and power consumption of BNNs and are thus unfriendly to hardware accelerators. For these reasons, even if BNNs have achieved MobileNet-level accuracy with a similar theoretical number of OPs (Bethge et al. 2020; Martinez et al. (020), they still cannot be used as conveniently as compact networks (Howard et al. 2017 Sandler al. 2018).
In this paper, we extensively study the trade-off between BNN's accuracy and hardware efficiency. We propose a novel BNN architecture: BoolNet, which replaces most commonly used 32-bit components (see Section 3). First, BoolNet only uses binary feature maps in the network (see Figure Ib). Second, during inference, we fuse the BN layer into the Sign function through a lossless transformation. thereby effectively removing the Mult-Adds brought by BN. Other changes include removing com- ponents that require additional 32-bit multiplication/ciiisioo operations: (1) PReI.U, (2) average pooling, and (3) binary downsampling convolutions. We then propose a Multi-slice strategy to help alleviate the loss of representational capacity incurred by binarizing the feature maps and removing 32-bit components. We show the effectiveness of our proposed methods and the increased energy efficiency of BoolNet with experiments on the ImageNet dataset (Deng et al.] 2009). The results show the key benefit of BoolNet: reasonable accuracy coupled with higher energy efficiency over state-of-the-art BNNs (see Figure [or a brief summary and Section 3 for more details). The energy data is obtained through hardware accelerator simulation (see Section 4.4 for details). We summarize our main contributions as follows:
The first work studying the effects of 32-bit layers often used in previous works on BNNs.
A novel BNN architecture BoolNet with minimal 32-bit components for higher efficiency.
A Multi-slice strategy to alleviate the accuracy loss incurred by using 1-bit feature maps.
State-of-the-art performance on the trade-off between accuracy and energy consumption with a 2.9x lower power consumption than Bi-RealNet (Liu et al. 2018) and 6.6% higher accuracy.
2 RELATED WORK
In recent years, Efficient Deep Learning has become a research field that has attracted much attention.
Technical directions, such as, compact network design (Howard et al., 2017 [019 Sandler et al. 2018: Zhang et al. 2018; Ma et al. 2018b), knowledge distill lation (Crowley et al. 2018: Polino
et al. 2018), network pruning (Han et al. 2015a by Li et al. 2017 He al. (017). 7), and low-bit quantization (Courbariaux et al., 2015 Rastegari et al. 2016]] Liu et al., 2018 2020b, Bethhe et al.) 2020) are proposed for model compression and acceleration. The efficient models have evolved from the earliest handcrafted designs to the current use of neural architecture search to search for the best
basic block and overall network structure (Tan et al., [2019] Howard et al. 2019 Tan & Le 2019 Radosavovic et al.) 2020). The criterion of efficiency evaluation has also changed from instruction and parameter counts to more precise measurements of actual memory and operating efficiency on
the target hardware (Cai al. 2019 2018).
2

---

Under review as a conference paper at ICLR 2022
Binary Neural Networks were first introduced by Courbariaux et al. (2016) and their initial attempt only evaluated on small datasets such as MNIST (LeCun & Cortes 2010). CIFARIO Krizhevsky et al.) 2009) and SVHN (Netzer et al.] 201T). The follow-up XNOR-Net (Rastegan et al. (016) proposes channel-wise scaling factors for approximating the real-valued parameters, which achieves 51.2% top-1 accuracy on ImageNet. However, there is an 18% gap compared with its 32-bit counterpart, R8sNet-18. Therefore, recent efforts mainly focused on narrowing the accuracy gap. WRPN by Mishra et al. (2018) shows that expanding the channel width of binary convolutions can obtain better performance. ABC-Net by Lin et al. (2017a), GroupNet by ZZuang et al. (2019), and (Zhu al., 2019) use a set of k binary convolutions (referred as binary bases). instead of using a single binary convolution, to approximate 32-bit convolution. This sort of method achieves higher accuracy but increases the required memory and number of operations of each convolution by the factor k. Bi-RealNet by [iu et al. (2018) proposes using real-valued (32-bit) shortcuts to maintain a 32-bit information flow, which effectively improves the accuracy, This design strategy became a standard for later work, c.g.. Bethge et al. (2019) 2020); Liu et al. (2020b). Martinez et al. (2020)
propose using a real-valued attention mechanism and well-tuned training recipes to boost the accuracy further. Thanks to the special architecture design, the recent MeliusNet (Bethge et al.] (0200 and
ReActNet (Liu et al., (020b) achieve MobileNet-level accuracy with similar number of theoretical operations. Other attempts, such as leveraging neural architecture search (Bulat al. 2020: Zhao
et al. 2020) and dynamic networks (Bulat et al. 2022I, show that those successful methods on regular real-valued networks are also effective for BNN. Another method by [Shen et al. (2019) combines neural architecture search to dynamically increase the number of channels for more accurate BNNs. Often, with improved accuracy, 32-bit components are used more frequently as well, such as PReLU and BatchNorm after each binary convolution (Liu et al.) 2020b), a real-valued attention module (Martinez et al., 2020) and scaling factors, etc. Apart from some works that include and optimize real-time measurements on mobile devices, such as Bannink et al. (202T); Umuroglu et al. (2017), efficiency analysis in the literature often only considers the theoretical operation number. However, the memory usage and the actual energy consumption has received very little attention so far.
BOOLNET
In this section, we first revisit the latest BNNs and recap how they enhanced the accuracy by adding more 32-bit components (in Section 3.1 Afterwards, we propose to replace most commonly used 32-bit components from current BNN designs and instead use a fully binary information flow in the network (in Section 3.2). However, abandoning 32-bit information flow results in a serious degradation of the representative capacity of the network. Thus, we also present our strategies to restore the representative capacity (in Section 3.3). The focus on boolean operations and binary feature maps leads to the name of our network: BooINet.
3.1 IMPROVING ACCURACY WITH ADDITIONAL 32-BIT COMPONENTS
Recent works on BNNs have made promising progress in narrowing the gap to their 32-bit coun- terparts. The key intention is to enhance the representative capacity by fully exploiting additional 32-bit components. However, such additional 32-bit components significantly reduce the hardware efficiency (as shown by Fromm et al. (2000) and further discussed in Section 4.4). The following list summarizes the 32-bit components commonly used in the latest BNNs: (1) The channel-wise
scaling factor was first proposed by Rastegari et al. (0116) for approximating the 32-bit parameters. It increases the value range of activation and weight. (2) Bi-RealNet (Liu et al.] 2018) proposes to use a 32-bit shortcut for enclosing each binary convolution. The key advantage is that the network can maintain an almost completely 32-bit information flow (cf. Figure(ai) (3) XNOR-Net (Rastegari et al.] (016)) uses 32- bit 1x1 1 downsampling convolutions, which is also used by many subsequent methods (Liu et al. 2018, Martinez al., 2020 Bethge et al.) 2020). |Bethge et al. (201)) shows that this simple strategy can achieve about 3.6% Top-1 1 accuracy gains on ImageNet based on a binary R8 model. (4) Martinez et al. (2020): Bulat et al. (2020) 202I) show that PReLU activation effectively improves accuracy of BNNs. ReActNet (Liu et al. 2020b) constructs the RPReLU activation function and uses it before every sign function. (5) Martinez et al.) ( (2000) reuse the 32- activation in their Seal-to-Binary Net after BN with squeeze and excitation (SE) attention mechanism. This module can adaptively re-scale the outputs of each binary convolution but needs additional 32- operations.
3

---

Under review as a conference paper at ICLR 2022
training inference
training BConv
inference
Sign
32-bit
3
BConv
BConv
16-bit
Batch Sign
Shifted
3x3
BatchNorm
3
Sign
1-bit
(a) Typical binary basic block.
(b) Our binary block design without 32-bit operations.
Figure 2: Comparison between a conventional binary convolution block with 32-bit data flow (a) and proposed binary convolution block with 1-bit data flow (b).
Although these techniques effectively improve the accuracy, they increase the number of 32-bit values and floating point operations, making them not particularly efficient on hardware accelerators. They closer mixed- precision neural networks rather than being highly efficient binary neural networks, as one might expect.
mixed-precision
3.2 BASENET: REPLACING 32- BIT COMPONENTS WITH BOOLEAN OPERATIONS
To better balance accuracy and efficiency, we rethink the additional 32-bit components (Batch Normalization, 32-bit feature maps, scaling factors and PReLU) elaborated in the previous section and propose to remove or replace them with more efficient operations. We further propose new basic convolution block without 32-bit operations, as shown in Figure 2b where we rearranged the order of convolution basic block as [BinaryConv, BatchNorm, Sign]. so that all feature maps are binary. These general changes constitute our BoolNet baseline, in short BaseNet.
Integrating the BatchNorm into the Sign Function Umuroglu et al. (2017) 7) suggested to replace BatchNorm (BN) with a thresholding operation during inference on FPGAs. However their suggestion not be applied to more recent work (Hubara et al.) 2016; Rastegari et al. 2016 Liu et al., 2018 2020b; Bethge et al.] 2020), because the layer order in these works is [Sign, BinaryConv, BN surrounded by 32-bit valued shortcuts. Instead, these recent works have kept the 32-bit BatchNorm layer in both the training and testing stages. However, using a 32-bit BN right after the I-bit convolution layer decreases the computational efficiency on hardware, using more memory and energy. Thus, in the following we propose to fuse the BN layer into the Sign function during the inference stage and do use the 32-bit output of BN layer for shortcut connection.
During the training phase, the batch normalization layer normalizes feature maps with a running mean μ and running variance a. For inference, it utilizes the constant statistic mean and variance instead, which in result can be reformulated as a linear process, expressed as:
VI + +ell Ri + (B
7/44
y/
+ +
(I)
+
2
+
where and y, represent the N-dimensional input and output of a BN layer. Y and 3 are trainable scale and shift parameters, which are constant during the inference. ||... is the absolute function. We therefore simplify the formula follows:
b
Vi az, +
(z+ +
a
where a, and e denote constants in the formula. By transforming a into its sign and its absolute value, we have
||||| @ Sign (a) O (z; + c), As arranged in our basic block, Equation on (ag and (z,; + c). We thus derive a parameterized sign function as:
(3)
i is followed by sign function, and Sign(y:) only depends
Sign(y:) = ANOR(Sign(a)) Sign(r; + c))
44)
We further replace 3 by using XNOR operator so that only bit-wise operations are adopted in the inference.
4

---

Under review as a conference paper at ICLR 2022
Channel Spit
C= 612%%
Dul: 256"k
in: 512%
MS- MS-BConv
MS BCan-
(aroups k)
BineryCorv
out: 258
(groups
out 256'K
(groupe
out: 512"
MS-BCorw (groupa = k)
MS-BCone
(groups =
16-bit
258%
out: 512"K
Mult-Sliin Sign
1-btt
Channel Concate
5122K
Channel Concate
1024%K
Channel
OUE 200%
out: 512"K
out: 1024"K
(a) MS-BConv module
(b) BoolNet basic block
(c) BoolNet downsample block
Figure 3: The detailed architecture of BoolNet. To enhance the information flow, we modify the baseline architecture in two aspects: a) Reducing information loss with our multi-slices binary convolution. b) Strengthening the information propagation by using split and concatenate operations.
Further Reducing 32-bit Operations We rarely use the PReLU activation function, which is com- monly used in the recent literature (Liu al., 2018 Martinez al., 2020 Bulat et al. 202T and brings a lot of extra overhead to the hardware implementation (it is only used once before the final dense layer). We also decided not to use scaling factors as suggested by Liu al. (2018) Bethge et al. (2019). There are two components which use 32-bit operations and parameters in previous work, which are kept in 32-bit in BoolNet: the first convolution and the last dense layer. Directly replacing them with binary versions leads to a severe accuracy loss (Rastegari et al. 2016). thus leave the investigation of alternatives for these special layers for future work.
3.3 BOOLNET: ENHANCING BINARY INFORMATION FLOW
The network design changes explained in the previous section, constitute our BoolNet baseline, called BaseNet. Although it uses completely binary information flow which minimizes the energy and memory consumption, the representative capacity of BaseNet is drastically degraded compared to its 32-bit counterparts and previous BNN methods, which accumulate information in 32-bit values. To counter this reduction of representative capacity, we propose the following three ideas, which constitute our proposed BoolNet.
Multi-slice Binary Convolution. Instead of using a single 1-bit value for each 32-bit value regular BNN, our multi-slice strategy proposes of using set of l: 1-bit values. The key intention to reduce the information loss caused by the sign function. We consider the typical binarization process Sign(æ;, zero-point) as special case of single-slice numerical projection. Thus, propose a multi-slice projection strategy for binary convolution to retain more relative magnitude information. Specifically, we redesign sign function follows:
x Sign(2; b,);
where 6, indicates a set of constant bias:
+2n l:
bn
where = 0, k/2
(6)
We adopt bn to conveniently expand the channel dimension to enhance the capacity of the binary feature map. If n 0, l 1. Equation (c) degenerates to the ordinary sign function. In Equation (5). _ denotes the binary projection output with the dimension of [N, C',C' H, W], which will be ted into the subsequent binary convolution layer. The constant i also denotes the group number of the convolution. That is, by setting the number of groups to l in each convolution, the overall amount of parameters and operations each convolution is unchanged. We found three conceptually related works by Lin et al. (2017b); Zhuang et al. (2019): Zhu al. (2019). but they have a different motivation and implementation compared to our work: 1) They use a higher number (k) of 32-bit feature maps and weights close the accuracy gap to 32-bit networks. This increases the computation and memory cost of a single convolution by the factor & (A:2 in (Lin et al. (017b)). 2) These works differing composition strategies, but they usually require additional 32-bit computation, such as
5

---

Under review as a conference paper at ICLR 2022
addition and 32-bit scaling factors. For comparison, the Multi-slice Binary Convolution reduces the memory requirement by replacing each 32-bit value with 1-bit values and uses a grouped binary convolution to keep the number of operations constant. Motivated by FReLU (Ma et al. 2020p. we enhance the first multi-slices projecting module, after the input convolution of network, with a Local Adaptive Shifting module. This module consists of binary depth-wise, 3 3 convolution and a batch normalization layer and is able to adaptively change the zero points of each pixel, in a light-weight manner. For simplicity, the multi-slices binary convolution is referred to MS-BConv, subsequently. Figure 3a shows the detailed block design of MS-BConn -Coov.
Strengthening The Information Propagation in BoolNet. The layer-by-layer feature extraction and accumulation mechanism are key reasons deep neural networks have strong representative capacity. However, accumulating information from shallow to deep based on addition operation involves extra 32-bit data-flow, which is conflicted with our motivation of building hardware friendly binary neural network. To avoid 32-bit fata-flow while maintaining the accumulation mechanism, we strengthen BaseNet by reusing binary features. In ShuffleNet-V2 V2 (Ma et al. 2018a), the input tensor is divided into two equal parts, the first half is used for feature extraction, and the other half is directly copied and concatenated with the extracted features. Inspired by its characteristies of information fusion and retention, we use a similar method to enhance the information retention capability of the BoolNet block. As demonstrated in Figure 35 the feature extraction branch consists of two MS- BConv modules, and the other branch remains identity. Two branches are concatenated and followed by channel shuffle, ensuring that the features from different layers are uniformly distributed. Figure shows the downsampling block design of BoolNet, where no channel splitting is required, and it doubles the number of channels in the output. Changing this information accumulation mechanism constitutes our proposed BoolNet over the BaseNet (as referred to in Section
Rethinking the Downsample Branch. Furthermore, we modify the downsampling block compared to previous methods, which usually use the layers x2x2 AvgPool (Stride 2), 32-bit 1 X 1 Conv, BN] in this branch (Liu et al. 2018 Martinez et al. 2020). Instead, we propose to use the layers [1-bit 1x1 1 Conv, 2x2 MaxPool (Stride 2). BN, Sign] and replace the 1-bit 3x3 Conv (stride 2) in the main branch with [1-bit 3 3 Conv, 2x2 MaxPool (stride 2) Overall, this removes all 32-bit operations and 32-bit parameters from the downsample block of BoolNet, but due to space limitations, we discuss the details in the appendix (including alternatives and experimental results).
3.4 TRAINING WITH PROGRESSIVE WEIGHT BINARIZATION
Though we intend build highly efficient BNNs with fully binary information flow, this strategy makes the network more sensitive to weight initialization during training. Traditional methods tried to alleviate similar problems through two-stage training (Martinez et al. 2020 Liu et al. 2020b), which makes the training more complicated. In this paper, adopt a progressive binarization technique based on the traditional Hardtanh-STT method (Courbariaux et al. 2016). This can be understood as a smooth version of multi-stage training approach. Specifically, in the training phase, a differentiable function F(x) is used to replace sign function. The slope of this function is adjusted by a single scalar X. As the slope increases, the weight gradually changes from 32-bit 1-bit. During backward propagation, we approximate F( /A) with F(x/1), 1), to avoid the problem of gradients clipping as A decreases. In the testing phase, we use the regular sign function for inference. The whole process can be formulated as:
F(z,A) lim Hardtanh X
~ Sign(z).
To smooth the weight binarization process, we schedule during training with an exponential decay strategy A g(t), where o < 1 is the exponential decay rate of A.
EXPERIMENTS
We evaluated our baseline network BaseNet and our proposed BoolNet (as described in Sections 3.2 and 3.3, respectively) the ImageNet dataset (Deng et al. 2009). In the following section, we first present the training details for our experiments. Afterwards, we conduct an ablation study on our proposed network design changes and the Multi-slice convolution (in Section 4.2 and 4.3), analyze the energy consumption of BoolNet and other recent work on BNNs (in Section 4.4). and compare our model accuracy to state-of-the-art BNN models (in Section 4.5)

---

Under review as a conference paper at ICLR 2022
Table 1: Our ablation study on ImageNet (Deng et al. (2009) regarding accuracy, number of 32-bit
operations (FLOPs), 1-bit operations (BOPs), and model size. (OPs = FLOPs + 1/6--BPPs)
BascNet
BoolNet
Network Configuration
Top
Acc.
FLOPs
(-10%)
(-10")
BOPs
(-10%)
OPs
Model
Size
Top
Acc.
(-10%)
FLOPs
(-10")
BOPs
OPs (-10")
Model
Size
Baseline (60 opochs, CE Loss)
47.00%
1.22
1.68
1.49
3.47 MB
54.07%
2.78
1.85
3.07
5.05 MB
Multi-Slice strategy (k=4)
52.27%
1.22
1.49
56.84%
2.78
1.86
3.07
MB
+ (1) Modified downsample branch
(BuseNet has no downsample branch)
58.6618
1.23
2.48
1.62
3.84 MB
(2) Local Adaptive Shifting'
52.08%
1.25
1.69
1.51
3.47 MB
59 39.566
1.26
2.48
1.65
3.84 MB
(3) MaxPool instead stride
55.14%
1.23
1.57
59.98%
1.26
1811
MB
(4) Knowlodge distillation
56.84%
1.23
2.21
1.57
3,47 MB
61%
1.26
3.53
1.81
3.84 MB
Long training (256 spochs)
58.20%
1.23
2.21
1.57
3.47 MB
63 .0.00%%
1.26
3.53
1881
3.84 MB
Local Adupiiie Shifling not used the subsequent BaseNet experiments
Replaces the cross-entropy loss with distritutional loss by LLu et 20206
BaseNet baseline (3) (4)
BoolNet baseline + + (2) (3) + (4)
Top
Top 5
FLOPPs
BOPs
OPs
Model
Top 1
Top 5
FLOPs
BOPs
OPs (-10")
Model
k
Acc.
A
Acc.
(-10")
(-10")
(-10")
Size
Acc.
A
Acc.
(-10")
(-10")
Size
51.74%
75.39%
1.23
2.20
1.57
3.47 MB
57.62%
80.47%
1.26
3.05
1.74
3.71 MB
55.75%
+4.01
79.08%
1.23
2.20
1.57
3.47 MB
60.57%
+3.95
82.56%
1.26
3.21
1.76
3.76 MB
56.84%
+1.09
79.85%
1.23
2.21
1.57
3,47 MB
61.98%
+1.41
83.75%
1.26
3.53
1.81
3.84 MB
8
57.19%
+0.35
80.33%
1.23
2.22
1.58
3.47 MB
62.54%
+0.56
84.14%
1.26
4.16
1.91
4.01 MB
4.1 TRAINING DETAILS
Our training strategy and hyperparameters are mostly based on (Bethge et al. (2020), but we train for 256 epochs and replace the Cross- Cross-Entropy loss with the knowledge distillation approach proposed by Liu et al. (2020b) where the teacher model a 32-bit ResNet-34 (He et al. 2016). The exact hyperparameters, details and code are available in the appendix and the supplementary material.
We proposed progressive weight binarization (see Section 3.4 Equation 7 as an alternative to the
two-stage training approach taken by Liu et al. M et al. (2020). For experiments, we use o = 0.965 and thus x = 0.965', where is the number of samples processed divided by 256000, rounded down. The progressive weight binarization is replaced with the sign function during the validation pass. The two-stage training strategy aims to provide a good initialization for a BNN training, by first training a model with 1-bit activations/32-bit weights and weight decay of 10-5, and use it to initialize the training of a 1-bit activations/I weights model. We tested the effect of both strategies with plain ResNet-like model with binary feature maps. The two-stage training (trained 60 epochs in each stage- total of 120 epochs) achieved 49, 99.60% accuracy. Our progressive weight binarization achieves 48.39% when training for 60 epochs, but achieves 50.19% when training for 120 epochs. Thus we deduce that our training strategy effectively removes the need for two stage training (based on a similar total training time) and leads to similar or slightly better result.
Martinez
4.2 ABLATION ON NETWORK DESIGN AND TRAINING STRATEGIES
We conducted ablation study determine the influence of our proposed design changes and the used training techniques (see the top half of Table TT. Our results show a 4.58% (BaseNet) / 2.77% (BoolNet) increase of accuracy by using our Multi-Slice strategy with k = over our baseline training (both trained only for 60 epochs with cross-entropy loss). Although adopting our BoolNet over BaseNer increases accuracy by 4.57% / 6.38% (with/without the Multi-Slice strategy), BoolNet uses a much higher number of 32-bit operations. However, modifying the downsample branch (as proposed Section (3.3 eliminates these additional 32-bit operations in BoolNet and boosts accuracy by 1.82%. The Local Adaptive Shifting module increases accuracy by 0.90% for BooINet but does not increase the accuracy of a BaseNet (and thus was not used for our subsequent BascNet experiments). The reason is likely that the module only shifts the zero points of the very first block of BaseNet which does not aid optimization. Using a MaxPool layer (see Section (33) instead of strided convolution especially helps BaseNet to retain information (which has no traditional shortcut branch in the downsampling block).
Regarding the training techniques, found that replacing the cross-entropy loss with the knowledge distillation approach by Liu et al.) (2020b) contributes 1.70% (BaseNet) / 2.00% (BoolNet) accuracy and extending the training time from 60 to 256 epochs further increases accuracy by 1.36% (BaseNet)
7

---

Under review as a conference paper at ICLR 2022
/ 1.02% (BoolNet). We conclude that all proposed techniques (and the knowledge distillation by Liu et al. (2020b)) help to increase the accuracy of BaseNet and/or BoolNet.
4.3 ABLATION ON THE MULTI-SLICE BINARY CONVOLUTION
The previous section has already shown that our Multi-slice Binary Convolution (see Section 3.3 can reduce the accuracy loss caused by using 1-bit feature maps. However, we further evaluated the influence of the number of slices A in these convolutions based on the best configuration of the previous section but only training for 60 epochs (see the lower half of Ta. Our results show that the final accuracy increases with each increase of k. but there are diminishing returns. Doubling k from 1 to 2. from 2 to 4, and from to 8 increases accuracy by 3,95%, 1.41% ,,, and 0.56% respectively (for BoolNet). Based on the increase of the number of operations, model size, and projected memory consumption, we use h = 4 for the best trade-off between accuracy and energy efficiency for our following experiments.
4.4 ENERGY CONSUMPTION EVALUATION
This section aims to efficiently compare the energy consumption of BoolNet to classic BNN architec- tures under strictly fair design conditions. We thus implemented five BNN accelerators (BaseNet, BoolNet, XNOR Net, Bi-RealNet, ReActNet (based on a Bi-RealNet backbone)). Considering the scope of this work, we leave the details of further hardware optimization of individual accelerators for future work.
We designed the five accelerators in the RTL language. Then, the power and area of computing circuits is given by Design Compiler (DC) with TSMC 65nm process and 1GHz clock frequency. We refer to the design and implementation methods of computing units of Conti et al. (2018). Zhang jet al. (2021). For a fair comparison between the different BNNs, we keep the design of architecture, data stream, the parallelism of computing units, and total on-chip cache (192KB for feature maps and 288KB for weights) consistent and only change the bit-width of the data stream and computing units. More specifically, the parallelism of binary convolution is 64 64, and the parallelism of other units is 64 in all accelerators (except the IntConv module is 8x64). These modules are pipelined and run IGHz. When DRAM bandwidth can be fully utilized, the performance depends on the parallelism and is bounded by the convolution, so each accelerator has the same peak performance for convolution, i.e., 4096 GOPs/s. Therefore, we achieve similar throughputs between 2044 and 2237 samples per second and it is reasonable to compare the energy consumption of the whole inference process.
DC can provide the hierarchical circuit area and power of computing units, including static power (P.) and dynamic power (Pa). For each layer of the network, we know the amount (A) of each operation. According to the circuit parallelism (Pa). we can calculate the required number of cycles (C,, = A P.) and then the energy consumption according to the frequency and power (E, = C, x
(P, Pc) /10 9). For the operations with fewer cycles, the energy consumption waiting for other units is estimated by static power (E, = =CCruar C₂) x P. / 10 ). Similar to Wang et al. (021]) ; Jiang & Zokaee ) Li et al. (2021) 1 we evaluate the energy consumption of on-chip SRAM access and off-chip DRAM access by using CACTI 6.5 (CACTT) and the power calculator of DDR provided by Micron (Micron). According to the cycle accurate simulation above, we can get the access amount of SRAM and DRAM, then get the energy consumption. The above components sum the overall energy consumed by a single inference pass. More details of simulation and energy estimation can be found in A.3.
Memory access and computation are the primary factors that affect energy consumption of hardware accelcrator. In the existing BNNs, efficiency analysis only considers the thcoretical instruction counts, while the impact of memory access is often not considered. However, Yang et al. (017) has shown that computation energy only accounts for 10% of the overall energy consumption of a (32-bit) GoogleNet. Moreover, Han et al. (2015b) evaluated the energy consumption ratio of computation and memory access. In their estimation, the energy consumption of SRAM access with the same amount of data is 50x that of 32-bit addition operations, and the energy consumption of DRAM access is 6400 that of computations. A theoretical analysis of the required memory shows that the total memory by BoolNet is much lower than previous BNNs, especially during the earlier stages of the network (details presented in Figure 7b in the appendix). Thus, we expect that our design
8

---

Under review as a conference paper at ICLR 2022
Bitwidth
Energy
Top-1
OPs
Endrgy g Consump ((J))
4000
67
Energy used by
Methods
(WIA/T)
Consumption
Acc.
(-10")
16065
computation SRAM access
1581
ReActNet (Bi-Rcal)
1/1/32
3.93mJ
65.9%
1.63
Ni-RealNet
1/1/32
3.90ml
56.4%
1.63
2000
47.2
DRAM access
BoolNet, k= (ours)
XNOR-Net
1/1/32
1/1/4
1.92mJ
1.33ml
63.0%
51.2%
1.64
1.59
954.4
918.1
2230
2250.8
58.7
201.3
509.3
58.7
856.5
110.2
BaseNet, k=4 (ours)
1/1/4
0.83mJ
58.2%
1.54
XNOR Net
Bi-RaalNet
ReAdtNet
BaseNet,
BaseNet Axd
BoolNel,
BaseNet, k=1 (ours)
1/1/1
0.70mJJ
53.3%
1.51
(a) The advantage of BoolNet is reduced energy consumption. "hhe ReActNet result based on Bi-RealNet backbone is stated
(h) Energy consumption regarding
computations and access DRAM/SRAM.
the official Github repository (Liu et al. 2020a).
Figure 4: Comparison between BoolNet and state-of-the-art BNNs. The energy consumption is calculated through hardware simulations.
should achieve much higher energy efficiency due a lower memory access. Other BNNs store and read 32-bit feature maps in and from memory. On the contrary, BooINet adopts 1-bit data streams everywhere, except for the accumulation result of convolutions, which is 16-bit. However, because of our Shifted Sign network design, the result can be quantized to 1 bit immediately without cache or DRAM access. This is not as efficiently possible in other BNN designs that use 32-bit aggregation requiring the intermediate 32-bit data to be cached. Due to the small size of on-chip memory, it must be written to DRAM and read out at the next layer. The differences in the bit-width of the data path and the amount of data reading and writing result in significant gap in energy consumption (highlighted in Figure Sp. Our evaluation results (see Figure (b)) show that the energy consumption proportion between computing units and memory access is even more extreme than shown by] Yang et al. (2017); Han et al. (0115)) Our interpretation is that the simplified operations of BNNs with the bitwise operations XNOR and popcount further reduces their share of overall energy consumption. This insight shows that memory optimization is more desirable in BNNs compared to 32-bit DNNs to achieve highly energy-efficient Al accelerators.
4.5 COMPARISON TO OF-TE-OF-HE--TE BNNS
We compared our proposed networks to the state-of-the-ar BNNs, e.g., ReActNet, Bi-RealNet, XNOR-Net (Liu et al. 2018 2020b Rastegari et al. (0166). (For fair comparison we used the result of a ReActNet based on a Bi-RealNet backbone read from the official Github repository provided by Liu et al. (2020a) which is the same backbone that network uses.) We found that removing 32-bit elements from previous BNNs. leads to an energy reduction by up to 5.6x (BaseNet with k=1), but accuracy drops by 12.6% (see Table 4a Using the proposed Multi-slice strategy (k=4) reduces the accuracy drop by 4.9% and still achieves 4.7x energy reduction. Our BoolNet design further increases the accuracy by 4.8% which overall results in 2.95x reduction of energy and a 2.9% accuracy drop. Compared to the result of Bi-RealNet (Liu et al. 2018), which has been the basis for other works (Martinez al. 2020). BoolNet with k=4 provides an accuracy improvement of 6.6% (and a 2.9x energy reduction). Overall our results show that our proposed BaseNet and BoolNet can achieve significant energy reduction with little accuracy loss compared to recent state-of-the-an models.
5 CONCLUSION
In this paper, we studied how balance energy consumption and accuracy of binary neural networks. We proposed several simple yet useful strategies to remove or replace 32-bit components from BNNs. Our novel BoolNet with fully binary information flow is constructed and still maintains reasonable accuracy. Experiments on ImageNet and the hardware simulations show that (1) theoretical number of operations does not fully reveal the actual efficiency and (2) BoolNet is more energy-efficien with less computing requirements, lower memory usage and lower energy consumption. We believe this is orthogonal to the goals of previous works and a meaningful first step towards achieving extremely energy-efficieet BNNs.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Tom Bannink, Adam Hillier, Lukas Geiger, Tim de Bruin, Leon Overweel, Jelmer Neeven, and Koen Helwegen. Larq compute engine: Design, benchmark and deploy state-of-the-art binarized neural networks. Proceedings of Machine Learning and Systems, 3, 2021.
Joseph Bethge, Haojin Yang, Marvin Bornstein, and Christoph Meinel. Binarydensenet: developing an architecture for binary neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Joseph Bethge, Christian Bartz, Haojin Yang. and Christoph Meinel. Meliusnet: Can binary neural networks achieve mobilenet-level accuracy? arXiv preprint arXiv:2001 XXiv:2001.05936, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL pttips::/proceeiings hew.eeeeeidigg....e.....ee. neurips 20/ cc/papers http:/2220/2///// 111444570066ffcccboo 96 1111111111111111161111111 Paper pdf
Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. Bats: Binary architecture search. In
European Conference on Computer Vision, 2020.
Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. High-capacity expert binary networks.
In International Conference on Learning Representaations
CACTI CACTI. URL http:: www htwp./h...co/reaaacccccattttoooe Accessed: 2021-09- 29.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332,, 2018.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv: 0e00: 2019.
Francesco Conti, Pasquale Davide Schiavone, and Luca Benini. Xnor neural engine: A hardware accelerator ip for 21.6-fj/op binary neural network inference. IEEE Transactions on Computer- Aided Design of Integrated Circuits and Systems, 37(11):2940- 2951,, 2018.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural
networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123-3131, 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran EI- Yaniv,,,, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ I 2016.
or-1. arXiv preprint 16022
Elliot J Crowley, Gavin Gray, and Amos J Storkey. Moonshine: Distilling with cheap convolutions. In Advances in Neural Information Processing Systems, pp. 1888-28890 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Joshua Fromm, Meghan Cowan, Matthai Philipose, Luis Ceze, and Shwetak Patel. Riptide: Fast end-to-end binarized neural networks. Proceedings of Machine Learning and Systems, 2:379-389, 2020.
Jian Guo, He He, Tong He, Leonard Lausen, Mu Li, Haibin Lin, Xingjian Shi, Chenguang Wang, Junyuan Xie, Sheng Zha, Aston Zhang, Hang Zhang, Zhi Zhang, Zhongyue Zhang, and Shuai Zheng. GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing. arXiv preprint arXiv: arXiv: 1900. 04433, 2019.
10

---

Under review as a conference paper at ICLR 2022
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. 2015a. URL http: /eltt://arxxe arxiv. org/ 1oc0/ 00149 cite arxiv: 1ttt:11510.00119oommmm Published conference paper at ICLR 2016 (oral).
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang- Ting Cheng, and Roe- land Nusselder. Latent weights do not exist: Rethinking binarized neural network optimiza- tion. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Asso- ciates, Inc., 2019. URL lattps::/ppoceeedllll www.rsenerripippee www.eps....paappeepppeeee Pape:eaee.....dddpaae..... pdf
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1314-1324, 2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 070664461.44440 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran Ei- Yaniv,,v, and Yoshua Bengio. Binarized neural networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 41 142422, 2016.
Lei Jiang and Farzaneh Zokaee. Exma: genomics accelerator for exact-matching, In 2021 IEEE International Symposium Perfo-Performance Computer Architecture (HPCA), pp. 399-411. IEFE, 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http:://///. / yyann
Tecun. , com hetpn/exdbmmmim
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for
efficient convnets. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
net/ http://oommmmdddeeeeeeee ott.:///eee..nettttooouooreeede
Jaajun Li, Ahmed Louri, Avinash Karanth, and Razvan Bunescu, Csenn: Aggrithm-hhrdwwrr design for enn accelerators using centrosymmetric filters. In 202/ IEEE International Symposium on
Computer Architecture (HPCA), pp. 612-625. IEEE, 2021.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017a. URL https: http:tttt/////rcceddiiee
neurips poeprr/psccc/ppppr//0//71 .pdf]
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. arXiv preprint arXiv:/711.11 /7//. /294, 2017b.
11

---

Under review as a conference paper at ICLR 2022
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the Variance of the Adaptive Learning Rate and Beyond. arXiv preprint arXiv:1908.00 03265, 2019.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European conference on computer vision (ECCV), pp. 722-737, 2018.
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang- Ting Cheng. Official source code of re- actnet on github, 2020a. URL httht:///gtthhttt http:///thhbbbccoo//i//eeeeee http:///uueeccn//eettteeee Accessed: 2021-10-01.
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang- Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XIV, volume 12359 of Lecture Notes in Computer Science, pp. 159-159. Springer, 2020b. doi: 000000079788--00-588-80 URL http::tp::///m. ptettps://oioo org. 10. 1007 978- 030 58568-556666
N. Ma, X. Zhang. H. T. Zheng, and J. Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In European Conference on Computer Vision, 2a.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018b.
Ningning Ma, Xiangyu Zhang, and Jian Sun. Funnel activation for visual recognition. arXiv preprint arXiv:2007.11824, 2020.
Brais Martinez, Jing Yang. Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural net- works with real-to-binayy convolutions. In International Conference on Learning Representations, 2020.
Micron, Micron. URL Scc.u//ssppppoostscccccc///o/lee hMl22226222777722220088822222222
Accessed: 2021-09-29.
Asit Mishra, Eriko Nurvitadhi, Jeffiey Cook, and Debbie Marr. Wrpn: Wide reduced-precision networks. In International Conference Learning Representations (ICLR), 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, pp. 8026-8037, 2019.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang. Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2 104. 10350, 2021.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantiza- tion. In ICLR (Poster). OpenReview.net, 2018.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (0428-10436, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks, In European conference on computer vision, pp. 525-542. Springer, 2016.
12

---

Under review as a conference paper at ICLR 2022
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chigg--hihhhhh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520 2018.
Mingzhu Shen, Kai Han, Chunjing Xu, and Yunhe Wang. Searching for accurate binary neural architectures. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, Oct 2019.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105 61 14. PMLR, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang. Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Yaman Umuroglu, Nicholas J Fraser, Giulio Gambardella, Michaela Blott, Philip Leong, Magnus Jahre, and Kees Vissers. Finn: A framework for fast, scalable binarized neural network inference. In Proceedings of the 2017 ACM/SIGDA International Symposium Field- Goeld-Programmabee Gate Arrays, pp. 65-74, 2017.
Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning- In 202/ IEEE International Symposium Prrfo-Performanc Computer Architecture (HPCA), pp. 97-110. IEEE, 2021.
Tien-Ju Yang. Yu-Hsin Chen, Joel Emer, and Vivienne Sze. A method to estimate the energy consumption of deep neural networks. In 2017 51st asilomar conference on signals, systems, and computers, pp. 1916-1920 1920. IEEE, 2017.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-6856, 2018
Yichi Zhang, Junhao Pan, Xinheng Liu, Hongzheng Chen, Deming Chen, and Zhiru Zhang. Fracbnn: Accurate and fpga-efficient binary neural networks with fractional activations. In The 2021 ACM/SIGDA International Symposium Field- Peeerdrrggra Gate Arrays, pp. 171-182, 2021.
Tianchen Zhao, Xuefei Ning, Xiangsheng Shi, Songyi Yang, Shuang Liang, Peng Lei, Jianfei Chen, Huazhong Yang, and Yu Wang. Bars: Joint search of cell topology and layout for accurate and efficient binary architectures. arXiv preprini arXiv:201 108044 2020.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.000 /6/ 2016.
Shilin Zhu, Xin Dong, and Hao Su. Binary ensemble neural network: More bits per network or more networks per bit? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4923 4932, 2019.
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Peng Chen, Lingqiao Liu, and Ian Reid. Structured binary neural networks for image recognition. In IEEE Conference Computer Vision and Pattern Recognition, pp. 413-422 2019.
A APPENDIX
Before present further details in the following sections, present overview the total amount computation that was used during this work. We measured the total GPU hours for the three experiments in Section 4.3 of our paper. These experiments (BaseNet k=1, BaseNet k=4 BoolNet k=4) required 192, 256, and 336 GPU hours respectively, in total: 784 GPU hours.
We have recorded more than 7191 GPU hours for our ablation studies and our intermediate, initial, discarded experiments (some of which were not presented in the paper), but estimate that a further 1500-2000 hours were needed in experiments before started measuring the GPU runtime.
13

---

Under review as a conference paper at ICLR 2022
A.1 TRAINING DETAILS AND FURTHER EXPERIMENTAL RESULTS
The training strategy is mostly based on Bethge et al. 2020 More specifically, we use the RAdam optimizer by Liu et (2019) with learning rate of 0.002 without weight decay, use the cosine learning rate decay by Guo et al. (2019). and train with a batch size of 256 for 60 epochs. We only use random flippigg and cropping of images to resolution of 224 224 for augmentation and finally normalize the data according to the mean and standard deviation of the dataset. During validation we resize the images to 256 256, and then crop the center with a size of 224 x 224 (and normalize PyTorch (Paszke al.
in the same manner as during training). Our implementation is based 2019), and the code can be found the supplementary material ZIP archive. The implementations of many previous works can not be sped up with XNOR and popcount (also observed by Fromm et al. (2020)) since they use padding with zeros, which introduces third value ((-1, O, +1 in the Teature map. To circumvent this issue, use Replication padding, which duplicates the outer-most values of the feature map, thus the values are limited to (-1,+1) further difference to previous work, is progressive weight binarization technique remove the need for two-stage trainings, discussed in the following Section.
A.1.1 PROGRESSIVE WEIGHT BINARIZATION VS. TWO-STAGE TRAINING
We have introduced the progressive weight binarization strategy in Section 3 Equation 7 and discussed the results briefly Section [.1 As presented in our main paper, training with progressive weight binarization leads to higher accuracy, if we train for the same total number of epochs. However, we also conducted an experiment using linear increase (N, = instead of our proposed exponential increase (A; o²) of the slope (see Figure 5 We chose a, so the final À values are equal, i.c. if fmax represents the final epoch, then \ A 10 The learning curves show that our progressive weight binarization gains the largest advantage by only "initializing" the values during a brief initial phase the training.
6, 6 10-9)
A.1.2 CODE SUBMISSION
Within the supplementary material ZIP archive we provide our training code. We added all details needed to reproduce each of our experiments depicted in Section 4.5 of our paper in the respective folders:
BaseNet(k=1) in BaseNet_k=1
BaseNet(k=4) in BaseNet_kes k=
BoolNet(k=4) in BoolNet_kes
The complete code used for each run can be found in the subfolder sre and the exact running command is saved in src/run. sh (in cases the number of GPUs needs be provided in the environment variable NUM_GPU or replaced in the run command).
Validation Acc.
Train Acc.
Validation Acc. (Top K=5) -/-
Train Acc. (Top K=5)
Ours (linear)
Ours (exponential)
0.7
67.8
68.2
71.1
0.6
0.5
0.4
43.5.
43.7
0.3
0.2
18,7
18,6
0.1
73
7/3
0.0
04
20
40
60
20
40
60
E poch
Figure 5: The training and validation accuracy curves of our proposed Progressive Weight Binarization. An exponential increase of the slope leads to much better results, than a linear increase.
14

---

Under review as a conference paper at ICLR 2022
Validation Acc.
Train Acc.
Validation Acc. (Top k=5)
Train Acc. (Top K=5)
BaseNet (k=1)
BaseNet (k=4)
BoolNet (k=4)
0.8
-7.0
80.9
84.5
0.7
0.6
58.2
0.5
53.3
0.4
0.3
30.0
0.2
0.1 9/
13.8
0.0+0
64
128
192
256
64
128 Epoch
192
256
0
64
128
192
256
Figure 6: The training and validation accuracy curves our trainings discussed in the comparison the state-of-the-art BNNs in Section 4.5
AvgPool
Acc. (%)
MaxPool
Acc. (%)
Stride=2
Ace. (%)
k Bits
Groups
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
32
63.5
87.8
63.0
87.7
60.7
86.4
1
63.1
88.0
62.5
87.2
60.9
86.7
32
1
66.0
89.4
67.0
90.0
63.4
87.9
8
8
65.0
88.0
65.3
88.9
62.2
87.0
1
64.1
88.5
65.0
89.0
62.6
87.3
Table Our ablation study on CIFAR100 regarding different downsampling methods. The number of bits refers to both the input activation and weight binarization of the 1 x convolution in the shortcut branch.
We also added the output logs (logs/training log), data CSV (logs/datas csv), and total training time (logs/ ime). Further, we plotted the learming curves regarding training and validation accuracy (logs/acc. png- summary of all runs can be seen Figure
We use virtualized environment for PyTorch (Paszke et al. 2019) based on Ubuntu for our code setup. The hosts system thus needs support for enroot" (or docke Further, the host system requires the mvidia-contaiio
liibaary and a recent NVIDIA CUDA driver (we tested driver version 465. 27 with CUDA 11.3) for training with GPU. Since several additional requirements are needed to be installed within the Pytorch base image, provide example installation script install sh, that can be used to set up the additional requirements inside a virtual container. The install. sh script also contains complete commands (in the comments) setup container based on enroot. Note, that the ImageNet dataset also needs to be downloaded and prepared manually in the usual manner (using train and val folder for the respective split). The validation images need to be moved into labeled subfolders. The dataset then needs mounted into the container, example commands are in the file install.. sh (default path inside the container is /mnt/ www.imaaggee/e/eieemiaeoeaa (train, val) }).
A.2 ABLATION STUDY ON THE DOWNSAMPLE STRUCTURE
As described in Section
modify the x convolution in the downsampling branch in contrast
to many previous works (Rastegari et al. 2016 Liu et al.) 2018 20206 Martinez et al. 2020). While being helpful for accuracy, the 32-bit consumption, which is in conflict with our motivation. Using our multi-slice strategy with & = 8, the number of input channels for the x convolution also increases by the same factor of 8. To counter this increase of 32-bit operations, it could be an option to use 8 groups in the convolution, which would keep the number of 32-bit operations constant, compared to previous work. However, this strategy still conflicts with our motivation to remove most 32-bit operations. Furthermore, the average
convolution involves extra computing, memory and energy
https://ww..dolee https://github.rr
15

---

Under review as a conference paper at ICLR 2022
Operation
Power (m/)
Area (um²)
Operation
Power
(mrw
(um")
Area
M Upsige (bils)
1e-07
1.3 1.31e-01
6. 6.677406
85e-06
BConv
108.8
131737
Int8 Conv(1/8)
504
836269
8.39p -0
2 2.45e 06
L.49 06
7 7:40p 05
1.19 408
2 06
2.66e-08
Int Agg
43.5
53238
Momoo
bit Sign
1.4
7956
32-bit Sign
3.3
13548
0e-00
32-bit
RPReLU 137.6
310671
Int8
50.1
274606
Stage
Stage 2
Stage 3
Stage 4
BootNnt (K-1)
Noo (lx44)
Rogular BNN
(a) Energy consumption per unit operation and circuit area of commonly used components.
(b) Memory usage comparison between blocks of different stages.
Figure 7: A theoretical memory usage comparison of one convolution block between BoolNet and previous work. Actual numbers can differ during implementation. but BoolNet shows significantly lower memory usage, especially in early stages, even when using our Multi-slice strategy with k 4.
pooling layer used in previous work, requires additional 32-bit addition and division operations, which could be reduced with either using a max pooling layer or a stride of 2.
Therefore, to find good downsample module with binary data flow, we first design the downsample template as [Convy, v, BN, Sign]. In this template, x indicates the different candidate downsample operations (e.g., average pooling, max pooling, or adding stride=2 to the convolution) and the number of bits used for weights and activations in the convolution.
We conducted detailed ablation study on the CIFARI00 dataset for both & 1 and & 8 (see Table 2). The results show, that max pooling combined with 1-bit 1 x convolution (groups = 1) has the same Top-1 accuracy as average pooling combined with 32-bit Thus, we decide to use max pooling instead of average pooling, since it does not involve any 32-bit bit operations, such addition and division.
convolution (groups 8).
Based on the above analysis, we suggest using the [32-bit Conv (groups = k), AvgPool2d, BN, Sign] structure for the downsample branch if we want increase accuracy. However, if intend to build fully binary data flow, we suggest using the [1-bit Conv (groups = 1), MaxPool2d, BN, Sign] structure (independent of k) instead balance the accuracy and hardware efficiency. The latter is also the structure we used for our experiments in the main paper.
A.3 MORE DETAILS ABOUT THE ENERGY CONSUMPTION SIMULATION
In Table 3 we give an example of calculating the memory consumption among different stages of our network. Compared with regular BNNs with mixed precision data flow, the fully binary representation of BoolNet significantly lowers the memory consumption during inference process. This change leads to less memory access operations to DRAM which has a much higher power consumption than the on-chip SRAM. To the best of our knowledge, our work is the first one to study the impact of memory access energy consumption. The details of simulation and energy estimation are introduced as follow.
Overall hardware architecture. illustrative graph on the data flow between the hardware components is provided in Figure g In the typical BNN Bi-RealNet, only the convolution is binary, the shortcut branch adopts high precision, and other calculations adopt high precision, too. The corresponding accelerators we designed have different computing modules (but their parallelisms are the same, that is, the computing time of the whole block roughly the same, and the binary convolution units are exactly the same). In addition, for fair comparison, these accelerators have the same size of on-chip memory (192KB for feature map and 288KB for weight) and the same off-chip memory.
Computing unit. The binary convolution units of different BNN accelerators are exactly the same, but other calculation units of BoolNet are simpler. The first is the shorteut branch of downsample blocks. The shortcut branch of traditional BNNs are high-precision, and the high-precision convolution downsampling is adopted. Although the convolution the shorteut branch accounts only for small amount of calculation, the power consumption of precisson convolution is 37 times that of binary convolution, and the extra convolution unit also increases the complexity of the eircuit.
16

---

Under review as a conference paper at ICLR 2022
Table Theoretical minimum memory requirement of all convolution blocks (can differ depending on the implementation). k is the number of slices. The stages have different input size and thus lead different memory requirements.
Memory Usage of Woights
Stage with 64 x 56
56
Stage 2 with 128 28 x 28
BoolNet (k=1)
BoolNet (k=4)
Regular BNN
BoolNet (k=1) BoolNet (k=4)
Regular BNN
36,864
36,864
36,864
147,456
147,456
147,456
200,704-1
200,704-4
200,704-1
100,352-1
100,352-4
100,352-1
Activation
200,704
802,816
200,704
= 100,352
401,408
= 100,352
Output &
2-200,704--1
2:200,704:
2-200, 2040,704-32
2-100,352-1
2-100,352-4
2-100,352-32
Features
401,408
1,605,632
12,845,056
2000704
802,816
6,422,528
Total
638,976
2,445,312
13,082,624
448,512
1,351,680
6,670,336
Memory
Stage
with 256 x 14 x 14
Stage with 512 x 7 x 7
Usage of
BoolNet (k=1)
BoolNet (k=4) Regular BNN
BoolNet (k=1) BoolNet (k=4)
Regular BNN
Weights
589,824
589,824
589,824
2,359,296
2.359,296
2,359,296
50,176-1
50,176-4
50,176-1
25,088-1
25,088-4
25,088-1
Activation
50,176
200,704
= 50,1776
= -250088
= 100,352
= 25,088
Output &
2-50,176-1
2-50,176-4
2-50,176-32
2-25,088-1
2-25,088-4
2-25,088-32
Features
100.352
401,408
-,,21,264
50,176
2000704
1,605,632
Total
740,352
1,191,936
3,851,264
2,434,560
2,660,352
3,990,016
DRAM
BNN Computing Unit
DRAM
BNN Computing Unit
32-bit Inputs
1-bit Inputs
32-bit Zero Points
1-bit Weights
Binariration
BinseyCone
Weights
Binasy Con
Points
Panameterized Ghoss-Sign
16-bit Weights
Baidh Noemaliittion
32-bit
1-bit Residual Connection
Residual Connection
Agpreptton
Asgregation
32-bit Outputs
1-bit Outputs
(a) Bi-RealNet Data Flow Hardware
(b) BoolNet Data Flow on Hardware
Figure Hardware data flow comparison between Bi-RealNNt and BoolNet.
Secondly, regarding batch normalization and binarization, since the shortcut branch has changed from high-precision to binary, the aggregation position of the shortcut branch and the main branch has also changed, that the binarization and batch normalization calculation of typical BNN can not be simplified, and their power consumption is high. In addition, there is difference in the complexity of the aggregation operation itself (boolean logic operation vs. 32-bit addition) and the computational overhead of non-linear functions (i.e. RPReLU) added in networks such as ReActNet. These aspects show the efficiency of BoolNet.
be simplified together, while the
Energy consumption per unit. The energy consumption unit operation of some commonly used components is shown in Figure 7a| However, since the units are not operated the same number of times, the total energy consumption during one inference graph is different. For instance, the energy consumption of Int8 downsampling convolution is 37x larger than binary downsampling" Surpris- ingly, per unit operation, a 32-bit RPReLU consumes 26% more energy than a binary convolution, Int8 BN consumes about half of binary convolution, and those two components are commonly used in conjunction with binary convolutions in existing BNNs.
On-chip memory. We use CACTI 6.5 to simulate the power of on-chip SRAM. According to the requirements of the computing unit, configure the on-chip SRAM to meet the parallelism of the corresponding data reading bandwidth (64 bits for BoolNet and 2048 bits for traditional BNNs), while keeping the total storage unchanged. In addition, we split large SRAM into multiple SRAMs
*37=504 8/108.8, where 1nt8 Conv has only 1/8 of the parallel capability of BConv.
17

---

Under review as a conference paper at ICLR 2022
to meet the requirement that the read time is less than the clock eycle (Ins) of the computing unit. Finally, the simulation software can give the energy consumption of one read or one write of each SRAM unit. For each layer of the network, we know the total number of operations for each type of operation. According to the circuit parallelism, we can calculate the number of cycles. Then, according the amount of data that needs be read from (or written to) SRAM in each cycle, can get the energy that the accelerator spends to access on-chip SRAM.
Off-chip memory. Due the limited amount of on-chip memory, it inevitable to save some data to (or read from) off-chip DRAM in BNN computing. In our BoolNet design, due to the large total number of weights, all BNN accelerators need to read weights from DRAM and write to SRAM before the computation of each layer. In addition, for traditional BNN, the intermediate feature maps are larger, which cannot be completely cached on-chip. It is also necessary to save the extra part to DRAM, to read it back in the next layer. With the amount of read-write operations of data to (and from) DRAM and SRAM, the power consumption data of DRAM read-write operations (SRAM has been given by the CACTI simulation in the previous step) is also needed to estimate the overall energy consumption. We use the DDR4 Power Calculator provided by Micron, to configure a DDR UDIMM module composed of four 8Gb x16 chips, which adopts the speed grade of -075E, and the maximum transmission rate 2666MT/s. The calculator gives the average energy consumption of reading and writing data with 64 bits parallelism.
Detailed throughput. The detailed throughput of the accelerators for BaseNet, BoolNet, BiRealNet, ReActNet, XnorNet are 2125, 2044, 2237, 2237, 2237 samples per second, respectively.
18


Reviews:
[{"review_id": "p6bdxaYGiia", "reviewer": "Reviewer_nRTL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths :\n+ The first work for fully 1bit neural networks, which is very important for practical deployment on hardware.\n+ The effects of 32-bit layers in commen BNNs are analyzed and removed by specific design.\n+ A Multi-slice strategy and other tricks are proposed to alleviate the accuracy loss.\n+ State-of-the-art performance on the trade-off between accuracy and energy consumption.\n\nWeaknesses:\n- The introduced MS-BConv increases the number of channels. Will it increase the memory usage and inference time?\n- MS-BConv increases the number of channels, which is related to [1]. This related work should be included.\n[1] Searching for accurate binary neural architectures. ICCVW 2019.", "summary_of_the_paper": "The paper is the first to build fully 1bit neural networks. The proposed BoolNet achieves state-of-the-art performance on the trade-off between accuracy and energy consumption.", "summary_of_the_review": "This paper is the first work to build fully 1bit neural networks. I like this work and recommend acceptance for it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JHZLKRw4N1x", "reviewer": "Reviewer_eCWL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is an attempt to reduce the precision of 32-bit components in BNNs. To this end, a new network architecture is introduced that integrates batchnorm into sign function, modifies downsample block and skip connections to reduce their required precision, and exploit a multi-slice convolution to compensate for the loss incurred by the precision reduction techniques. \n\nStrengths:\n- The paper is well-written and easy to understand. \n- The proposed network architecture is novel and is an attempt to further reduce the complexity of neural networks for their efficient deployment.\n- There is a discussion on one employment scenario and its implementation results to support the effectiveness of the proposed architecture.\n\nWeaknesses:\n- My first concern is the accuracy performance of the proposed architecture w.r.t. the SOTA architecture (i.e., ReActNet). In the ReActNet paper, there is a network architecture called ReActNet-A which achieves the accuracy of 69.4% while requiring 0.87 OPs. If we compare performance of this architecture given the accuracy and the number of OPs with the results reported in Figure 4(a), we can see ReActNet-A significantly outperforms BoolNets and BaseNets.\n- My second concern is the implementation results of BoolNets. In industry, we don't usually design a custom hardware for a specific network architecture. On the other hand, we consider a general custom hardware that can support a wide range of computing units and layer types. Of course, in this scenario, each network architecture will show a different implementation performance on the hardware accelerator. Therefore, I believe the hardware performance of BoolNets and ReActNets should be measured on a specific hardware accelerator for a fair comparison, and the choice of hardware accelerator should be limited to well-know ones that are being used commercially. Otherwise, the design of custom hardware for a specific network can be biased to favor that network and does not constitute a fair comparison in my opinion. \n", "summary_of_the_paper": "This paper proposes methods to further reduce energy consumption of binary neural networks by removing or replacing 32-bit components (e.g., skip connections) in SOTA BNNs. More specifically, the proposed architecture (1) reduces the precision of skip connections and activation functions, (2) transforms the BatchNorm layer into a simple sign function, and (3) employs a multi-slice strategy to alleviate the loss of representational capacity incurred by binarizing the feature maps and shortcut connections. The results shows that the new model achieves 4.7x energy reduction with some accuracy degradation compared to SOTA architectures.", "summary_of_the_review": "After carefully reading the paper, I believe while this paper made novel contributions in reducing the precision of 32-bit component of BNNs, it couldn't show its advantages over existing works such as ReActNet. More specifically, ReActNet-A achieves a better accuracy and also requires a lower number of OPs, making the advantage of BoolNet limited to the implementation results. However, the implementation results on a custom hardware designed for a specific network doesn't establish a fair comparison. My suggestion is to report hardware performance of the networks on a commercial accelerator instead for a fair comparison and more reliable results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "C_PIJcMB3Bw", "reviewer": "Reviewer_pRtS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n\n1- Reducing number of FP32 operations in BNN that reduces the energy consumption. \n\n2- The author evaluated the effect of various approaches such as Multi-slice Binary Convolution for BNN on accuracy, memory footprint and number of operations ( the ablation study on ImageNet and ResNet-34 is interesting)  \n\nWeakness:\n\n1- The novelty of paper is not obvious. It seems the author uses pervious approaches and combines it with BNN to improve accuracy. \n\n2- The motivation behind using sign function for BN is not explained in the paper. Please elaborate this. \n\n3- In the table on the Figure 4, to have a fair comparison, is distillation and long training epoch considered in pervious works?\n\n4- The author did not compare BooLNet with reference [1] approach which also uses Multi-slice Binary Convolution approach. \n\n[1] Pouransari, Hadi, Zhucheng Tu, and Oncel Tuzel. \"Least squares binary quantization of neural networks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.\n\n5- The BooLNet is only evaluated on ImageNet and ResNet-34 and it is not obvious that this proposed work can be generalized for others datasets and networks. I recommend evaluate the BoolNet on the others networks such as MobileNet-V2. ", "summary_of_the_paper": "The paper proposes two novel BNNs (BaseNet and BooLNet) where most of the parameters are represented in the binary formats which is the major difference compared to pervious works where activations in layers are represented in 32-bit floats. Moreover, the proposed networks removes the BN and most of ReLu activations and replaces with sign function. To increase the knowledge capacity, the authors uses Multi-slice Binary Convolution and Local Adaptive Shifting approaches. The inference performance of BaseNet and BooLNet are evaluated and compared with the inference performance of ResNet-34 using 32-bit floating-point format. ", "summary_of_the_review": "As this reviewer mentioned, the motivation to use only binary network is good research direction, However, the novelty of this paper is not obvious and the BoolNet needs to be evaluated on other benchmarks. I recommend this paper to be rejected. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3xKjZ2hMYh", "reviewer": "Reviewer_MNqE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This reviewer has several concerns about this paper as below:\n- This reviewer is curious about the portion of overhead due to full-precision layers during BNN inference. This paper refers to a previous study (Fromm et al., 2020), but in that paper, only squeezenet was evaluated. To strengthen the motivation of this paper, this paper also includes simulation results on energy consumption, but it is a little hard for this reviewer to accept the simulated results because 1) a few networks are compared with the proposed work and 2) simulation results strongly depend on assumptions about h/w implementations. BaseNet/BoolNet with 32-but features can be comparable networks for ablation study to show the overhead of 32-bit features. Since current BNNs show better accuracy than the proposed work, this paper should strengthen and prove the motivation, why we need to eliminate the 32-bit features from BNNs completely, with deeper analysis.\n- This paper claims that existing BNNs use various 32-bit features and proposes various techniques to make neural networks that consist of only binary operations. However, some ideas seem to be similar to previous works. In addition, for some ideas, there seems to be a lack of comparisons and citations.\n  - The proposed shifted sign idea looks similar to ‘Batchnorm-activation as Threshold’ (Umuroglu et al., 2017). which is not cited and compared. This idea is not surprised for BNN researchers.\n  - Multi-slice binary convolution layers seem to make a significant impact on improved accuracy. But this idea looks similar to ABC-net (Lin, 2017) and GroupNet (Zhuang, 2019). While ABC-net is compared with some descriptions, the GroupNet (it may be improved than ABC-net) is not compared and it is simply mentioned just one time. This reviewer wants to know why GroupNet is not compared.\n  - Figure 4 uses ReActNet(Bi-Real-based) for comparison. Why don’t you use ReActNet-A that shows better accuracy and lower FLOPS? Indeed, the GitHub repo of ReActNet also includes ReActNet-A.\n\n- Minor\n  - Page 4 : with an running → with a running\n\nUmuroglu, Yaman, et al. \"Finn: A framework for fast, scalable binarized neural network inference.\" Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. 2017.\n\nLin, Xiaofan, Cong Zhao, and Wei Pan. \"Towards accurate binary convolutional neural network.\" arXiv preprint arXiv:1711.11294 (2017).\n\nZhuang, Bohan, et al. \"Structured binary neural networks for accurate image classification and semantic segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.", "summary_of_the_paper": "This paper aims to eliminate 32bit features of BNNs as possible. This paper claims that existing BNNs embed 32bit features, which can improve the accuracy of BNNs but must lead to overheads during inference. In the proposed network, full-precision batchnorm layers are replaced with 'shifted-sign' layers, full-precision scaling factors are eliminated, and multi-slice binary convolutions and 1b shortcuts are used. The network achieves 63% accuracy on ImageNet with various techniques and knowledge distillation.", "summary_of_the_review": "As this reviewer mentioned, a more concrete analysis for the motivation should be added and comparisons with previous BNN architectures are required. The current manuscript seems to be lack novelty in this reviewer’s opinion. But, this reviewer is ready to listen to other reviewers’ opinions. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: ### Description
The authors note that the most recent and successful in terms of accuracy binary networks are in fact combining binary and floating-point computations, in particular have residual full-precision paths, with their parameters, that connect all the way to the output. Although such paths are made lightweight, they can be a bottleneck with respect to the energy consumption, memory and latency. The paper proposes a novel binary neural network model that uses only binary operations (except in the first and last layers). It is proposed to estimate the energy efficiency of binary networks more accurately, using hardware design compilers.

### Decision
Reviewers came to a consensus that the proposed BNN architecture claimed in the paper to be the main novelty, while it is indeed quite distinct from the mainstream and best performing BNNs, does not propose novel solutions with respect to the total state of the art. This is our main reason for rejection. The paper makes a great effort in steering the development of BNNs towards more energy efficient models, by carefully estimating the potential energy consumption of different models, using specialized software for design and simulation of hardware needed to run particular models. This is proposed not as a practical solution for industry but rather as a way to measure the potential efficiency of different models. While this was recognized as a great effort, some questions remained regarding fairness of comparison and possibility to reproduce the results by non-experts in order that other developers could estimate and compare potential efficiency of their models. Additionally, it is unclear how power efficiency of a model with $k$ slices (BoolNet) differs from that of a $k$-bit quantized network.

### Details
First of all, I very much like the motivation for the work: to aim at a design of BNN that would be more efficient in terms of speed and energy and to measure that efficiency more precisely. I am not an expert in the hardware, however I do share the concern in this paper that full precision residual paths and blocks would incur a larger latency and higher amount of computation (more chip area, more energy,...). The fact that residual connections in BiRealNet make it necessary to read and write full precision feature maps to the global memory was non-obvious to me. The current state of the art reports binary and floating point operations and largely ignores the necessary memory operations, which as authors argue are the real performance bottleneck.

The main claimed contribution of the paper is the design of a novel model. This was the main point of concert. It is indeed innovative with respect to the current trend of the best performing binary networks to make a "pure" binary network. However it is hard to agree that removing some of the components from BiRealNet or models based on it and going back to simpler architectures which were used before can be a contribution. Specifically, plain non-residual BNNs were the very well-known pioneering works [r1, r2]. The fact that BN in front of binarization can be simplified is trivial and well known, e.g. [r3]. Using multiple binary activations through power-of-two coding or uniform thresholds has been considered multiple times [r2, r6, ABCNet]. Same for group-wise convolutions. I have not seen residual connections in the form of shuffle-net so far, but it can also be considered as a rather standard trick. Using stride instead of max pooling or average pooling is another well known trick. Many of these solutions are in fact used in recent works, e.g. [r4] has no skip-connections, combines BN with sign at the inference time and uses strided convolutions for downsampling. To summarize, the submission does not appear to propose novel modelling solutions relative to the total state of the art.
Furthermore, [r3] is closely related in that they investigated the energy efficiency (however looking at the energy consumption of individual operations only) and with a similar motivation developed binary networks without 32 bit residual connections. Please see also other references pointed out by reviewers.

### Unclear in the paper:
-What is the meaning of $F$ column in Fig4a for BaseNet / Boolnet?
-Why grouped convolution with input channels 256k and k groups has output of 256 channels in fig 3a. 
If the outputs from each group are summed, isn't it equivalent to a full convolution 256k x 256?
-What do the authors mean by 3x3 depth-wise convolution?
- The local adaptive shifting module is discussed inside the paragraph describing MS-BConv. Is it a part of MS-BConv or not?
- It seems that with $k$ slices, there is $k$ times more bits per activation used and $k$ times more bits per weight used (because the channel width is multiplied by $k$).
It must be therefore equivalent in terms of the power consumption to a network that uses $k$ bits per activation and weight. 
Using these $k$ bits to represent uniform slices rather than powers of two slices appears inferior in terms of quantization error and accuracy. Indeed, many works have successfully quantization different models down to $4$ bits without a loss in accuracy. 

### Related work: 

Rather than reviewing different methods for making networks more efficient, a deeper review of BNNs would be more helpful, in particular 
looking at the works that are closer to the hardware, such as "in-memory computing", "neuromorphic computing", [r5, r6].

### Discussion

A well-justified way to measure the potential energy efficiency of BNNs would be an excellent contribution that could standardize comparison and drive the development of BNNs in the energy-efficient direction. Unfortunately,
it does not appear at the moment that non-experts in hardware and design compilers could repeat the compilation and simulation of accelerators as the authors proposed. A simplified estimation method is needed that can be used in python for any model composed of some standard blocks. The authors seem to be in a good position to propose and validate such a simplified estimator. To start with may I suggest to clarify the following questions:
- Do we need power for the operation pipelines for different operation types (cache, global memory) that are not currently used?
- Are the arithmetic operations implemented in hardware to optimize energy or the throughput?
- Can we assume that all latencies can be masked by parallelism?
- Is it a good approximation to assume that a convolution (with an efficient implementation and large enough cache) needs to read the input only once?
- Can any coordinate-wise transform be appended to the preceding transform on the fly, canceling a read-write in between?
- What is a reasonable estimate of a cost for float32 operations? It seems from the quantization literature that all such operations can be safely quantized to e.g. 8 bit representations without a loss of accuracy.

[r1] Courbariaux et al. 2016, Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or −1] 

[r2 Hubara 2018: Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations, JMLR] 

[r3 Ding et al. 2019: Regularizing Activation Distribution for Training Binarized Deep Networks] 

[r4 Livochka et al. Initialization and Transfer Learning of Stochastic Binary Networks From Real-Valued Ones]

[r5 Baskin et al. Streaming Architecture for Large-Scale Quantized Neural Networks on an FPGA-Based Dataflow Platform]

[r6 Umuroglu et al. FINN: A Framework for Fast, Scalable Binarized Neural Network Inference]
Decision: Reject