High-Capacity Expert Binary Networks
Authors: Adrian Bulat
Brais Martinez
Georgios Tzimiropoulos
Venue: ICLR 2021 Poster (2021)

Abstract: Network binarization is a promising hardware-aware direction for creating efficient deep models. Despite its memory and computational advantages, reducing the accuracy gap between binary models and their real-valued counterparts remains an unsolved challenging research problem. To this end, we make the following 3 contributions: (a) To increase model capacity, we propose Expert Binary Convolution, which, for the first time, tailors conditional computing to binary networks by learning to select one data-specific expert binary filter at a time conditioned on input features. (b) To increase representation capacity, we propose to address the inherent information bottleneck in binary networks by introducing an efficient width expansion mechanism which keeps the binary operations within the same budget. (c) To improve network design, we propose a principled binary network growth mechanism that unveils a set of network topologies of favorable properties. Overall, our method improves upon prior work, with no increase in computational cost, by $\sim6 \%$, reaching a groundbreaking $\sim 71\%$ on ImageNet classification. Code will be made available $\href{https://www.adrianbulat.com/binary-networks}{here}$.

Published as a conference paper at ICLR 2021
HIGH-CAPACITE EXPERT BINARY NETWORKS
Adrian Bulat
Brais Martinez Samsung AI Cambridge
Samsung AT Cambridge
adrien@adrienbulat. com
brais a@samsung. com
Georgios Tzimiropoulos
Samsung AI Cambridge
Queen Mary University of London, UK g. tz inim ropoul osêquul ac uk
ABSTRACT
Network binarization is a promising hardware-aware direction for creating effi- cient deep models. Despite its memory and computational advantages, reducing the accuracy gap between binary models and their real-valued counterparts re- mains unsolved challenging research problem. To this end, make the fol- lowing contributions: (a) To increase model capacity, we propose Expert Binary Convolution, which, for the first time, tailors conditional computing to binary networks by learning to select one data-specific expert binary filter at a time con- ditioned on input features. (b) To increase representation capacity, we propose to address the inherent information bottleneck in binary networks by introducing an efficient width expansion mechanism which keeps the binary operations within the same budget. (c) To improve network design, we propose a principled binary network search mechanism that unveils a set of network topologies favorable properties. Overall, our method improves upon prior work, with no increase in computational cost, by ~ 6%, reaching a groundbreaking 71% on ImageNet classification. Code will be made available here
INTRODUCTION
promising, hardware-aware, direction for designing efficient deep learning models case is that of network binarization, in which filter and activation values are restricted to two states only:
+1 (Rastegari et al., [016]] Courbariaux (a) it compresses the weights by a factor of 32 via bit-packing, and (b) it replaces the computation- ally expensive multiply-add with bit-wise xnor and popcount operations, offering, in practice, speed-up of ~ 58x on a CPU (Rastegari et al. 2016). Despite this, how reduce the accuracy gap between a binary model and its real-valued counterpart remains an open problem and it is currently the major impediment for their wide-scale adoption.
al. (016). This comes with two important advantages:
In this work, we propose to approach this challenging problem from 3 key perspectives: 1. Model capacity: To increase model capacity, firstly introduce the first application of Con- ditional Computing (Bengio et al. 2013 2015 Yang et al. (019)) to the case of a binary networks, which call Expert Binary Convolution. For each convolutional layer, rather than learning weight tensor that is expected to generalize well across the entire input space, we learn a set of N experts each of which is tuned to specialize to portions of During inference. very light-weight gating function dynamically selects a single expert for each input sample and uses it to process the input features. Learning to select a single, tuned the input data, expert key property of our method which renders it suitable for the case of binary networks, and contrasts our approach to pre- vious works in conditional computing (Yang et al. 2019)).
2. Representation capacity: There is inherent information bottleneck in binary networks as only 2 states are used to characterize each feature, which hinders the learning of highly accurate models. To this end, for the first time, highlight the question of depth width in binary net- works and propose a surprisingly unexplored efficient mechanism for increasing the effective width of the network by preserving the original computational budget. We show that our approach leads

---

Published as a conference paper at ICLR 2021
to noticeable gains in accuracy without increasing computation.
3. Network design:
Finally, and inspired by similar work in real-valued networks Tan & Le| 2019), propose a principled approach to search for optimal directions for scaling-up up binary networks. Main results: Without increasing the computational budget of previous works, our method im- proves upon the state-of-the-art (Martinez et al. 2220) by ~ 6%, reaching groundbreaking ~ 71% on ImageNet classification.
2 RELATED WORK
2.1 NETWORK BINARIZATION
Since the seminal works of Courbariaxx et al. (2015) 2016) which showed that training fully binary models (both weights and activations) is possible, and Rastegari et al.] (2016) which reported the very first binary model of high accuracy, there has been great research effort to develop binary
models that are competitive in terms of accuracy when compared to their real-valued counterparts,
for example (Lin et al. 2017 Liu et al., 2018: Alizadeh al.) 2018 Bulat et al.) 2019 Bulat
& Tzimiropoulos 2019] Ding et al. 2019 Wang et al. 2019: Zhuang et al.) 2019 Zhu et al. 2019
Kim et al., 2020 Bulat et al.) 2020
Martinez et al. [2020]. Notably, many of these improvements
including real-valued down-sampling layers (Liu et al. 2018), double skip connections (Liu et al.,
2018 learning the scale factors (Bulat & Tzimiropoulos 2019). PReLUs (Bulat et al. 2019) and
two-stage optimization (Bulat et al. 2201)) have have been put together to build a strong baseline ine in Mar- tinez et al. (2020) which, further boosted by a sophisticated distillation and data-driven channel rescaling mechanism, yielded an accuracy of ~ 65% on ImageNet. This method, along with the recent binary NAS of Bulat et al. (2020) reporting accuracy of 66%, are to our knowledge, the state-of-the-ar in binary networks.
Our method further improves upon these works achieving an accuracy of 71% on ImageNet, crucially without increasing the computational complexity. To achieve this, to our knowledge, we propose for the first time to explore ideas from Conditional Computing (Bengio et al. 2013 2015) and learn speciiie binary expert weights which are dynamically selected during inference con- ditioned on the input data. Secondly, we are the first to identify width as an important factor for increasing the representation capacity of binary networks, and introduce surprisingly simple yet effective mechanism to enhance it without increasing complexity. Finally, although binary architec- ture design via NAS (Liu al., 20188] Real et al.) 2019) has been recently explored in (Kim al., 2020 Bulat et al., (2020), we propose to approach it from a different perspective that is more related to Tan Le (2019), which was developed for real-valued networks.
2.2 CONDITIONAL COMPUTATION
Conditional computation is a very general data processing framework which refers to using different models different parts of model conditioned the input data. [Wang et al. (0018) and Wu ct al. (018)) propose to completely bypass certain parts of the network during inference using skip connections by training policy network via reinforcement learning. [Gross al. 2017 proposes to train large models by using a mixture of experts trained independently on different partitions of the data. While speeding-up training, this approach is not end-to-end trainable nor tuned towards improving the model accuracy. Shazeer et al. (2017) trains thousands of experts that are combined using a noisy top-k expert selection while Teja Mullapudi al. (018) introduces the HydraNet in which a routing function selects and combines a subset of different operations. The later is more closely related to online network search. Chen al. (2019) uses a separate network dynamically select variable set of filters while Dai et al. (2017) learns a dynamically computed offset.
More closely related to the proposed EBConv Conditional Convolution, where Yang et al. (2019) propose to learn Mixture of Experts, i.e. a set of filters that are linearly combined using rout- ing function. In contrast, our approach learns to select a single expert at a time. This is critical for binary networks for two reasons: (1) The linear combination of binary set of weights is non- binary and, hence, a second binarization is required giving rise to training instability and increased memory consumption. In Section 5] we compare with such a model and show that approach works significantly better. (2) The additional computation to multiply and sum the weights, while negligible for real-valued networks, can lead noticeable computational increase for binary ones.
2

---

Published as a conference paper at ICLR 2021
Finally, note that our single expert selection mechanism is akin to the Gumbel max trick (Gum- bel 1948) and the Sombel-Softmax Estimator (Jang et al.] 2016 Maddison et al. 2016) previously used in various forms for NAS (Chang et al.) 2019h, multi-task learning (Guo et al. 2020) and vari-
ational auto-encoders (Jang et al. 2016). To our knowledge, the proposed ERConv is the very first adaptation for conditional computing within binary neural networks.
BACKGROUND ON BINARY NETWORKS
Following Rastegari et
2016 Bulat & Tzimiropoulos 2019 binary convolution is defined as:
BConv(x, 0) (sign(() si))(0)) a,
(1)
where x is the input, 0 the weights, + denotes the binary convolutional operation, C the Hadamard product, and a E R is learned via back propaaaoo,,,,,on, as Bulat Tzimiropoulos (2019).
The binarization is performed in two stages Bulat et al. (2019); Martinez al. (2020). During Stage I. we train a network with binary activations and valled weights. Note that the accuracy of Stage I models
that of the final fully binary model (see Table 4 During Stage II, we initialize from Stage to train a network with both weights and activations binary. When reporting results, if stage is specified, the model (weights and activations) is fully binary.
very representative
We set as baseline the Strong Baseline model (denoted as SBaseline) from Martinez et al. (2020) on top of which implemented the proposed method. We denote as Real-to-bin their full model.
Gating function
0
000
B
Convolution Op
(a) The proposed Expert Binary Convolution (b) 2D t-SNE embeddings of ImageNet validation (KBConv). Note that only single expert is active for model with experts and Top-I of a time.
63.8%.
Figure 1: (a) Schematic representation of the proposed EBConv layer, and (b) t-SNE embedding visualisation of the features before the classifier along with the corresponding expert that was acti- vated for each sample. Points located closer to each other more semantically and visually similar. Each data point coloured according to the expert selected by the last EBConv from our network. As multiple clusters emerge in the figure, it can be deduced that the experts learned preference for certain classes, or groups of classes from ImageNet. This suggests that the EBConv layer learned semantically meaningful representations of the data. Best viewed in color.
4 METHOD
4.1 EXPERT BINARY CONVOLUTION
Assume a binary convolutional layer with input x E RC.xWWXH and weight tensor e R Cin XCous xkw In contrast to normal convolution that applies the same weights to all in- put features, we propose to learn a set of expert weights (or simply experts) [θo, .,.... @ 0, C.C. XCrue Kky xkw alongside selector gating function which, given input x, selects only single expert to be applied to it. The proposed EBConv layer is depicted in Fig. Tal To learn the experts, let us first stack them in matrix + E N Cow We propose to learn the following
3

---

Published as a conference paper at ICLR 2021
function:
EBConv(x, 0) = BConv(x, (p()(x))
(2)
where v..) is gating function (returning an .N- -idimensionall vector as explained below) that im- plements the expert selection mechanism using as imput (x() which is an aggregation function of the input tensor x, and (.rr simply reshapes its argument to a tensor of appropriate dimensions.
Gating function yp: crucial component of the proposed approach is the gating function that implements the expert selection mechanism. An obvious solution would be to use a Winners- Take- All (WTA) function, however this is not differentiable. A candidate that comes in mind to solve this problem is the softargmax with temperature т: as tend to while the rest to O. However, as T + 0, the derivative of the softargmax converges to the Dirac function 8 which provides poor gradients and hence hinders the training process. This could be mitigated if high T is used, however this would require hard thresholding at test time which, for the case of binary networks, and given that the models are trained using Eq. leads large errors.
O, the entry corresponding to the max will
To mitigate the above, and distancing from reinforcement learning techniques often deployed when discrete decisions need to be made, propose, for the forward pass, use WTA function for defining (p)), as follows:
(02) f1. if argmax(2)
(3)
otherwise
Note that we define yp as y T + R i.e. as a function that returns an N N-dimensional vector which is used to multiply (element-wise) @ in Eq. 2 This is crucial as, during training, wish to back- proppgaee gradients for the non-sclected experts. To this end, we propose, for the backward pass, to use the Softmax function for approximating the gradients sp.):
ao 8 Softmax(2).
(4)
Oz
Overall, our proposal, WTA for forward and Softmax for backward, effectively addresses the mismatch during inference between training and testing while, the same time, it allows meaningful gradients to flow to all experts during training. In Section A.3.3 the appendix, we also explore the impact of adding temperature to the softmax showing how its value affects the training process. Note that backpropagating gradients for the non-selected experts applies to the gating function, only; the binary activations and weights continue use the STE introduced in (Courbariaux al. 2016 Rastegari et al. 2016).
Aggregation function ub: The purpose of this function is to give summary of the input feature tensor which will be used to select the expert. To avoid overfitting and to keep the computational cost low, we opt for simple and fast linear function:
(x)
x
10)
(5)
x is the spatial average of the i-th channel and E CxN learnable projec- tion matrix. Note that no other non-linearity was used as the WTA function is already non-linear function.
where x(:
gw
Data-specific experts: One expected property of EBConv implied by the proposed design is that the experts should specialize on portions of data. This is because, for each data sample, a single expert chosen per convolutional layer. Fig- Tb confirms this experi- mentally by t-SNE embedding visualisation the fea- tures before the classifier along with the corresponding expert that was activated for each sample of the Ima- geNet validation set.
Table 1:
Comparison on ImageNet for different number of experts. All models have the same number of BOPs, includ- ing Martinez et al. 2022)).
Accuracy (%)
#experts
Top-1
Top-5
(SBaseline) Martinez et al. 2020)
60.9
83.0
Optimization policy: As in Bulat et al. 2019). we adopt two-stage training policy where firstly the in- put features are binarized while learning real-valued weights, and then both input and weights are binarized.
4
63.8
85.1
8
64.0
85.3
4

---

Published as a conference paper at ICLR 2021
Note that the aggregation function e is kept real across
all steps since its computational cost is insignificant. Furthermore, due to the discrete decision mak- ing process early on, the training can be unstable. Therefore, stabilize the training firstly train one expert, and then use this to initialize the training of all N experts. This ensures that early on in the process any decision made by the gating function is a good decision. Overall, our optimization policy can be summarized as follows:
1. Train one expert, parametrized by Oo. using real weights and binary activations.
2. Replicate 0, to all 0,, = {1, N - 1 to initialize matrix θ
3. Train the model initialized in step 2 using real weights and binary activations.
4. Train the model obtained from step 3 using binary weights and activations.
4.2 ENHANCING BINARY INFORMATION FLOW
While the previous section addressed the issue of model capacity. in this section, we address the problem of the representation capacity of the binary activations. This issue arises due to the fact that only 2 states are used to characterize each feature, resulting in an information bottleneck which hinders the learning of highly accurate binary networks. To our knowledge, there is little prior work which explicitly tries to solve this problem (Liu et al. 2018).
Our solution surprisingly simple yet effec- tive: the only parameters can adjust in or- der to increase the representational power of bi- the resolution and the width (i.e. number of channels). The former is largely conditioned the resolution of the data, be- ing as such problem dependent. Hence, we propose the latter, which increase the net- work width. For example a width expansion of 2 can increase the number of unique con- figurations for 32 x 7 x 7 binary feature ten- sor from 232x7x7 = 21568 to 22136 However, increasing the network width directly causes quadratic increase complexity with respect to k. Hence, in order to keep the number of binary operations (BOPs) constant, we propose to use Grouped Convolutions with group size G pro- portional to the width expansion, i.e. G k:2. Note that we do not simply propose using grouped convolutions within binary networks as in (Phan et al. 22000 Bulat et al. (020).. We propose width expansion to address the inherent information bottleneck within binary networks and use grouped convolutions as a mechanism for increasing the capacity while preserving the computational budget fixed. Moreover, we note that since we are using grouped convolutions, features across groups need to somehow combined throughout the network. This can be achieved at no extra cost through the 1 x 1 convolutions used for downsampling at the end of each stage where change of spatial resolution occurs. In Section 4.3 we further propose a more effective way to achieve this, based on binary 1 x convolutions, which however add some extra complexity. Moreover, in Section 4.3 will further propose to search for the optimal group size depending on the layer location.
Table 2: Comparison on ImageNet for different number of experts and expansion rates. All mod- els have the same number of BOPs, includ- ing Martinez et al. (2020).
nary features
Accuracy (%)
Expansion
experts
Top-1
Top-5
(SBaseline)
60.9
83.0
Martinez et 2020
64.6
85.6
65.1
86.0
63.8
85.1
8
64.0
85.3
4
66.0
86.4
8 66.3
86.6
As Table 2 clearly shows, models trained with width multiplier higher than 1 offer consistent accuracy gains, notably without increasing complexity. Importantly, these gains also add up with the ones obtained by using the proposed EBConv. This is not surprising as width expansion improves representation capacity while the expert increases model capacity.
4.3 DESIGNING BINARY NETWORKS
In general, there is little work in network design for binary networks. Recently, few binary NAS techniques have been proposed (Kim et al.] 2020;; Shenn et al., 2019 Bulat et al. 2020). Despite reporting good performance, these methods have the same limitations typical of NAS methods, for
5

---

Published as a conference paper at ICLR 2021
example, having to search for optimal cell using predefined network architecture, or having to hand pick the search space. Herein, and inspired by Taan & Le (2019), we propose a mixed semi-automated approach that draws from the advantages of both automatic and manual network designing techniques. Specifically, setting the standard ResNet-18 (He et al. (016) network as a starting point, we focus on searching for optimal binary network structures, gradually exploring set of different directions (width, depth, groups, layer arrangement).
Effect of block arrangement: Starting from a ResNet-based topology in mind, we denote network with i i = {1, 2, 3, } blocks at each resolution as convolutional layers. We first investigate if e--aranggng the blocks, mainly by using a network which is heavier at later stages, can have an impact on accuracy. Note that since the number of features is doubled among stages, this re-arrangement preserves the same complexity. Table B] shows the results. As it can be observed the accuracy remains largely unchanged while the layers are re-distributed.
with each block having two
Depth vs width: In Section 4.2 we proposed an ef- ficient width expansion mechanism which is found to increase the accuracy of binary networks with- out increasing complexity. Herein, we evaluate the effect of increasing depth by adding more blocks. Fig. 2a shows the results of depth expansion. Each constellation represents a different architecture of which we vary only the number of blocks, i.e. the depth. As we may clearly see, the returns of increasing depth are diminished as complexity also rapidly increases, resulting in very heavy models. Note that previous work for the case of real-valued networks Zagoruyko & Komodakis (016) has shown that wide models can perform as well as deep ones. Our results show that, for a fixed computation budget, the proposed wide binary models with grouped convolutions actually outperform the deep ones by a large margin.
Table 3: Comparison on ImageNet between stage I models with different block arrange- ments.
Accuracy (%)
NoN1NN2N3
Top-1
Top-5
1133
63.8
86.5
1142
63.8
86.8
1124
63.7
87.4
2222
63.9
87.4
Effeet of aggregation over groups: Our efficient width expansion mechanism of Section 4.2 uses very weak way of aggregating the information across different groups. A better way is to explicitly use 1 binary convolutional layer (with no groups) after each block. The effect of adding that layer is shown in Fig. 3 Clearly, aggregation across groups via 1 x 1 convolutions offers significant accuracy gains, adding at the same time a reasonable amount of complexity.
Effect of groups: In Section 4.2 we proposed grouped convolutions as a mechanism for keeping the computations under control we increase the network width. Herein, we go one step further and explore the effect of different group sizes and their placement across the network. This, in turn, allows, with a high degree of granularity, to vary the computational budget at various points in the network while preserving the width and as such the information flow. To describe the space of network structures explored, we use the following naming convention: we denote a network with Ne, i = {1, 2, 3, 4} blocks at each resolution, corresponding width expansion E (the same E was used for all blocks) and group size G; for each convolution in these blocks as: No
G1 :::::::.. G3-
As the results from Fig. 2b and Table 4 show, increasing the number of groups (especially for the last 2 stages) results in significantly more efficient models which maintain the high accuracy (with only small decrease) compared to much larger models having the same network structure but fewer groups. Our results suggest that group sizes of 16 or 32 for the last 2 stages provide the best trade-off.
Network search strategy: In summary, the network search space used in this work consists of the following degrees of freedom: a) rearranging the blocks, b) defining the depth of the model, c) defining the width at each stage, and finally d) selecting the optimal number of groups per each stage. In order search for the optimal configuration, we gradually search in each direction separately while keeping all the others fixed. Then, we identify a set of promising search directions which we then combine to train new candidates. We repeat this step one more time and, then, from the final population of candidates we select the best models shown in Table This procedure results in
models that outperform recently proposed binary NAS methods (Bulat al. 2020 Kim al. 2020 by more than 5% while also being more computationally efficient.

---

Published as a conference paper at ICLR 2021
11
68.0
67.5
Top-1 (%)
Top-1 (%)
66
7
4:48::16
66.5
4:8.0:1
4:416:16
3.5
1.0
1.4
Bops
les
Bops
(a) Effect of depth accuracy. Each constellation represents different network from which vary only numher blocks (shown by the annotated text), Le. the depth. Increasing depth has diminishing returns. The specific networks for each constellation described in Table of Section
(b) Effect of number of groups and their placement accuracy. Networks with the nected with the same type of line. Increasing group size drastically reduces BOPs with little impact on ac- curacy. The specific networks for each constellation described in 6 Section A.1
structure
Figure 2: Effect of (a) depth and (b) groups on accuracy as a function of BOPs on Imagenet. All results reported for Stage I models. Best viewed in color.
Top-3
64
63
wkhhout 1x1
[oun]
12
1.6
1.
2.0
2.2
Bops
Figure 3: Effect of adding the 1 binary conv. layer after the grouped conv. layers. The dashed line connects same models with and without the conv. layer.
Table Comparison on ImageNet between few structures explored. All methods have approxi- mately the same number of FLOPS: 1.1 x 108, Nş: number of blocks in stage i, E: width expansion ratio, Ge group size convs at stage * denotes model trained using AT+KD (Martinez et al., 2020
Architecture
Acc. (%)
Stage II
Acc. (%)
Stage I
BOPS x10
Top-1
Top-5
Top-1
Top-5
1242-2-4:4:16:32
66.3
86.5
68.4
87.7
1.3
1262-2-4:4:16:32
66.8
86.8
69.0
88.1
1.5
1282-2-4:416::32
67.7
87.4
69.7
88.8
1242-2-4:4:8:16
67.0
87.1
68.9
88.0
1.6
1262-2-4:4:8:16
67.6
87.3
69.7
88.6
1.9
4::2-114::::166
67.8
87.5
69.7
88.7
2.2
1262-2-4:8:8:16
67.5
87.5
69.5
88.6
1.7
1262-2-4:8:8:16
70.0
89.2
71.6
90.1
1.7

---

Published as a conference paper at ICLR 2021
5 COMPARISON WITH STATE-OF-THE-AN
We compared method against the current state-of-the-art in binary networks on the ImageNet dataset (Deng et al., 2009). Additional comparisons, including on CIFAR-100 (Krizhevsky et al. 2009). can be found the supplementary material in Section A
Training: The training procedure largely follows that of Martinez et al. (020). In particular, we trained our networks using Adam optimizer (Kingma
Ba (014) for 75 epochs using a learning rate of 10-3 that is decreased by 10 at epoch 40, 55 and 65. During Stage I, we set the weight decay to 10-5 and to during Stage Furthermore, following Martinez et al. (2020), during the first 10 epochs, we apply a learning rate warm-up Goyal et al. (2017). The images augmented following varm-uppGoyya the common strategy used prior-work (He et al. 2016) by randomly scaling and cropping the images to resolution of 224 x 224px. In addition this, avoid overfitting of the given expert filters, we used Mixup (Zhang et al., 2017) with a = 0.2. For testing, we followed the standard procedure of scaling the images models were trained on 4 V100 GPUs and implemented using Py'Torch (Paszke et al. (019)..
resolution of 256px first and then center cropping them. All
Comparison against state-of-the-art: Table shows results ImageNet. When compared against methods with similar capacity (Rastegari et al., 2016 Courbariaux et al., 2015 2016 Bulat
& TTzimiropoulos, 2019; Martinez al.[20020) (bottom section of the table), our method improves on top of the currently best performing method of Martinez et al. (2020) by almost 6% in terms of top-1 accuracy, Furthermore, our approach surpasses the accuracy significantly larger and slower networks (upper section) by a wide margin.
Finally, we compared our method against two very recent works that use NAS to search for binary networks. As the results from Table 5 (middle section) show, our method outperforms them, again by a large margin, while being significantly more efficient.
In terms of computational requirements, our method maintains the same overall budget, having an equal or slightly lower number of FLOOPs and BOPs (see Table. 5 Although our method does increase the model size, by 2x for model that uses 4 experts, the run-time memory largely remains the same. For additional details see Section A.5 in the supplementary material.
Table 5: Comparison with sfute-of-th-are binary models on ImageNet. The upper section includes models that increase the network size/capacity (last column shows the capacity scaling), while the middle one binary NAS methods. denotes models trained using AT+KD (Martinez et al.) 20200) j denotes ours with an improved training scheme, see Section A.6 in supplementary material for details.
Accuracy (%)
Operations
Architecture
bits (WIA)
Top-1
Top-5
BOPS x10
FLOPS x 10
ABC-Net (M, (1 Lin al. 2017
65.0
85.9
42.5
1.3
(1/1)x5²
Struct. Approx. (Zhuang 2019
66.3
86.6
(1/l)x4
CBCN (Liu el al. 2019
82.8
(1/D)x
Ensemble (Zhu
2019
61.0
10.6
(1/1)x6
BATS (ullat et al. 2020
66.1
87.0
2.1
1.2
(1/1)
BNAS-F F
Kim et 2020
58.9
80.9
1.7
1.5
(1/1)
BNAS-G Kim
2020
62.2
83,9
3.6
1.5
(1/1)
BNN (Courbariaux el al. 2016
42.2
69.2
1.7
1.3
1/1
XNOR-NeI (Rastegan
2016
51.2
73.2
1.7
1.3
1/1
CCNN (Xu & Cheung
2019
54.2
77.9
1.7
1.3
I/1
Bi- ReaR Net
[Liu Nerl
a..
2018
56.4
79.5
1.7
1.5
I/1
Rethink. BNN (Helwegen et al.
2019
56.6
79.4
1.7
1.3
1/1
XNOR-Net++ (Bulat Izamropoulos, 2019
57.1
79.9
1.7
1.4
I/1
IR-Net QQn et al. 2020
58.1
80.0
1.7
1.3
I/1
CI-Net Waag 2019
59.9
84.2
1/1
Real-to- Bin [Martinez 2020
65.4
86.2
1.7
Ours
87.5
1.7
Ours
70.0
89.2
1.7
1.1
Ourst
71.2
90.1
1.7
1.1
1/1
8

---

Published as a conference paper at ICLR 2021
direct application of Cond- Conv Yang et al. (2019) for the case of binary networks is problematic due to the so-called "double binarization problem Le. binarization of the weights and then of their linear combination is quired. Herein, we verify this experimentally: when training a fully binarized network using Cond- Conv, noticed high degree of instability, especially during the initial phases of the training. For example, at the end of epoch I, the accuracy of the binarized CondConv model is 1% vs 20% of the one using EBConv, The final accuracy of binarized CondConv on Imagenet was 61.2% vs 63.8% compared to EBConv.
Comparison against CondConv: As mentioned in Section 4.1
Additionally, as mentioned earlier, our proposed EBConv method uses less FLOPs (no multiplica- tions required to combine the experts) and noticeably less memory at run-time (see Section A.3 in the appendix).
6 CONCLUSION
We proposed three-fold approach for improving the accuracy of binary networks. Firstly, we improved model capacity at negligible cost. To this end, we proposed E3Conv, the very first binary conditional computing layer which consists of data-specific expert binary filters and very light- weight mechanism for selecting a single expert at a time. Secondly, we increased representation capacity by addressing the inherent information bottleneck in binary networks. For this purpose, we introduced an efficient width expansion mechanism which keeps the overall number of binary operations within the same budget. Thirdly, we improved network design, by proposing a principled binary network growth mechanism that unveils a set of network topologies of favorable properties. Overall, our method improves upon prior work, with no increase in computational cost by 6%, reaching a groundbreaking ~ 71% on ImageNet classification.
REFERENCES
Milad Alizadeh, Javier Fernández-Marqués, Nicholas D Lane, and Yarin Gal. empirical study of binary neural networks' optimisation. In International Conference on Learning Representations, 2018.
Emmanuel Bengio, Pierre-Lue Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv: 2015.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 3308.3382 2013.
Adrian Bulat and Georgios Tzimiropoulos. XNOR-Net++: Improved binary neural networks. In
British Machine Vision Conference, 2019.
Adrian Bulat, Georgios Tzimiropoulos, Jean Kossaifi, and Maja Pantic. Improved training of binary networks for human pose estimation and image recognition. arXiv preprint 2019.
Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. BATS: Binary ArchitecTure search. European Conference on Computer Vision, 2020.
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by hale-wave gaussian quantization. In IEEE Conference on Computer Vision and Pattern Recogni- tion, 2017.
Jianlong Chang, Xinbang Zhang, Yiwen Guo, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Differentiable architecture search with ensemble Gumbel-Softmax. arXiv preprint 1rX1v:1905.01786. 2019.
Zhourong Chen, Yang Li. Samy Bengio, and Si Si. You look twice: GaterNet for dynamic filter selection in CNNs. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9172 9180, 2019.

---

Published as a conference paper at ICLR 2021
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. BinaryConnect: Training deep neural networks with binary weights during propagations. In Advances on Neural Information Process- ing Systems, pp- 3123-31333 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1 arXiv preprint (XX:::602.008300 2016.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In IEEE International Conference on Computer Vision, pp. 764-773, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255 2009.
Ruizhou Ding, Ting-Wi Wu Chin, Zeye Liu, and Diana Marculescu. Regularizing activation distribu- tion for training binarized deep networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Julian Faraone, Nicholas Fraser, Michaela Blott, and Philip HW Leong. SYQ: Learning symmetric quantization for efficient deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 4300-4309, 2018.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In IEEE International Conference Computer Vision. pp. 4852 482124861120 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in hour. arXiv preprint 17Xvv:770.006777 2017.
Sam Gross, Mare' Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: series of lectures, volume 33, US Government Printing Office, 1948.
Pengsheng Guo, Chen- Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. arXiv preprint arXiv:2006. 01895, 2020.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance ImageNet classification. In IEEE International Conference Computer Vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang- Ting Cheng, and Roe- land Nusselder. Latent weights do not exist: Rethinking binarized neural network optimization. Advances on Neural Information Processing Systems, 2019.
Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, and Joseph E Gonzalez. Checkmate: Breaking the memory wall with optimal tensor rematerial- ization. arXiv preprint arXivv1910.02663 2019.
Eric Jang, Shixiang Gu and Ben Poole. Categorical reparameterization with Gumbel-Softmax. 2016.
arXiv preprint
Sangil Jung, Changyong Son, Scohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
10

---

Published as a conference paper at ICLR 2021
Dahyun Kim, Kunal Pratap Singh, and Jonghyun Choi. Learning architectures for binary networks. In European Conference on Computer Vision, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances on Neural Information Processing Systems, pp. 345-353, 2017.
Chunlei Liu, Wenrui Ding, Xin Xia, Baochang Zhang, Jiaxin Gu, Jianzhuang Liu, Rongrong Ji, and David Doermann. Circulant binary convolutional networks: Enhancing the performance of 1-bit DCNNs with circulant back propagation. In IEEE Conference on Computer Vision and Pattern Recognition, pp.
2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. arXiv preprint arXiv:1806.0900 2018.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang- Ting Cheng. Bi-Real Net: Enhancing the performance of 1-bit CNNs with improved representational capability and advanced training algorithm. In European Conference on Computer Vision, pp. 747-763, 2018.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:/611.007/2 2016.
Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural net- works with real-to-binary convolutions. International Conference on Learning Representations, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems. 2019.
Hai Phan, Yihui He, Marios Savvides, Zhiqiang Shen, et al. Mobinet: A mobile binary network for image classification. In IEEE Winter Conference on Applications of Computer Vision, pp. 1462, 2020.
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In IEEE Conference Computer Vision and Patter Recognition, pp. 2250-2259, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-Net: ImageNet classification using binary convolutional neural networks. In European Conference on Computer Vision, 2016.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. AAAI Conference on Artificial Intelligence, 2019.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gater mixture- -xxture-of-experts layer. arXiv preprint arXiv: 1701006538 2017.
Mingzhu Shen, Kai Han, Chunjing Xu, and Yunhe Wang. Searching for accurate binary neural architectures. In IEEE International Conference on Computer Vision Workshops, 2019.
Mingxing Tan and Quoc V Le. EfficientNer: Rethinking model scaling for convolutional neural networks. International Conference Machine Learning, 2019.
Ravi Teja Mullapudi, William R Mark, Noam Shazeer, and Kayvon Fatahalian. HydraNets: Spe- cialized dynamic architectures for efficient inference. In IEËE Conference on Computer Vision and Pattern Recognition, pp. 8080-8089, 2018.

---

Published as a conference paper at ICLR 2021
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. SkipNet: Learning dynamic routing in convolutional networks. In European Conference on Computer Vision, 2018.
Ziwei Wang, Jiwen Lu, Chenxin Tao, Jie Zhou, and Qi Tian. Learning channel-wise interactions for binary convolutional neural networks, In IEEE Conference on Computer Vision and Pattern Recognition, pp. 568-577, 2019.
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths residual networks. In IEEE Confer- ence on Computer Vision and Pattern Recognition, 2018.
Zhe Xu and Ray CC Cheung. Accurate and compact convolutional neural networks with trained binarization. British Machine Vision Conference, 2019.
Brandon Yang, Gabriel Bender, Quoc V Le. and Jiquan Ngiam. CondConv: Conditionally pa- rameterized convolutions for efficient inference. Advances on Neural Information Processing Systems, pp. 1305-1316, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. British Machine Vision Confer- ence, 2016.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In European Conference Computer Vision, pp. 365-382, 2018.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17110:1710.09412, 2017.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. DoReFa-Nett Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv, 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. Interna- tional Conference on Learning Representations, 2017.
Shilin Zhu, Xin Dong, and Hao Su. Binary ensemble neural network: More bits per network or
more networks per bit? In IEEE Conference on Computer Vision and Pattern Recognition, pp- 4923-4932, 2019.
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and lan Reid. Structured binary neural networks for accurate image classification and semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 413-422, 2019.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 DETAILED NETWORK DEFINITIONS FOR FIG.
2
Table 6: Detailed network definitions for each of the constellations presented in Fig. The location in each architecture numbered from left right (models with more BOPs are located towards the right).
(a) Network definitions for Fig. 2a
(b) Network definitions for Fig. 2b
Constell. #
Location
Configuration
Constell.
Location
Configuration
2222-1-1:1:1:1
1142-2-4:8:16:33
1182-1-1:1:1:1
2
1142-2-4:4:16:16
1282-1-1:1:1:1
3
1142-2-4:8:8:16
1184-1-1:1:1:1
4
1142-2-4:4:8:16
1188-1-1:1:1:1
6
1288-1-1:1:1:1
1
1242-2-4:8:16:32
2
1242-2-4:4:16:16
1132-2-4:4:4:4
1242-2-4:8:8:16 3
1133-2-4:4::44
4
1242-2-4:4:8:16
1233-2-4:4:4:4
2
2233-2-4:4:4:4
1
1162-2-4:8:16:32
2282-2-4:4:4:4
2
1162-2-4:4:16:16
1188-2-4:4:4:4
1162-2-4:::8:66 3
4
1162-2-4:4:8:16
1242-2-4:4:8:16
1262-2-4:4:8:16
2
1162-2-4:4:16:16
1282-2-4:4:8:16
3
1162-2-4:8:8:16
3
1284-2-4:4:8:16
4
1162-2-4:::::66
A.2 ADDITIONAL COMPARISONS
Herein we present an extended comparison with both binary and low-bit quantization methods on ImageNet. As the results from Table show, our method significantly surpasses both the binary and the more computationally expensive low-bit quantization networks. Similar results can be observed on the CIFAR-100 (Krizhevsky al. 2009) dataset where our approach sets new state-of- of-the-------ttt result.
A.3 ABLATION STUDIES
A.3.1 REAL-VALUED DOWNSAMPLING DECOMPOSITION
The efficient width expansion mechanism of Sec- tion preserves the amount of BOPs constant for binary convolutions. However, width expansion also affects the real-valued downsampling (linear) layers. To preserve the number of FLOPs constant, as width expands, for such layer too, we propose to decom- pose into two smaller ones so that the connection between them is reduced by factor stead using [Conv(C'n Cout). we propose to use [Conv(C,,,. Can Conv Sau, Cout)]. Herein, we ex- plore few variants by adding non-linearities between them. Our results, reported in Table 9 show that the non-linear versions more expressive and bridge the gap caused by the decrease the layer's size. The proposed adaption and the original one depicted in Fig.
Table 9: Comparison ImageNet be- tween various types downsampling lay- ers (Stage I models). All decompositions reduce the complexity by 4x.
Accuracy (%)
Decomposition
3 ie. in-
Top-1
Top-5
None Linear
67.6 .4
87.2
66.5
86.6
Non-linear (ReLU)
67.2
87.2
Non-linear (PreLU)
67.5
87.3
13

---

Ours* Published as a conference paper at ICLR 2021
Table 7: Comparison with state-of-the-art binary models on ImageNet, including against methods that use low-bit quantization (upper section) and ones that increase the network size/capacity (second section). The third section compares against binary NAS methods. Last column shows the capacity scaling used, while denotes models trained using AT+KD (Martinez et al. 2020 ours with an improved training scheme, see Section A.6
denotes
Accuracy (%)
Operations
Architecture
bits (W/A)
Top-1
Top-5
BOPS x10
FLOPS x10
BWN
Courbariaux et al. 2016
60.8
83.0
1/32
DSQ
Gong
2019
63.7
1/32
TTQ
Zhu
66.6
87.2
2/32
QIL
Jung et al.
2019
2/2
HWGQ (Car el al. 20117
82.2
1/2
LQ-Net (Zhang of al.
2018
59.6
82.2
1/2
SYQ (Faraone et al.
2018
55.4
78.6
1/2
DORETA-NeI /hhou et al.
2016
62.6
84.4
1/2
ABC-Net (M, NN 1) (in al.
2017
42.2
67.6
1/1
ABC-Net (M,
5) LLn
2017)
85.9
42.5
(I/I)x5²
Struct. Approx. (Zhuang ot al. 2019
66.3
86.6
(1/1)x4
CBCN (Liu et al. 2019
61.4
82.8
(1/1)x4
Ensemble (Zhu cl al.
2019
61.0
10.6
7.8
(1/1)x6
BATS (Bulat et al. 2020
66.1
87.0
2.1
1.2
(1/I)
BNAS-F Kim 2020
58.9
80.9
1.7
1.5
(1/1)
BNAS-G Kim al.
2020
62.2
83.9
3.6
(1/1)
BNN (Courbariaux et al.
2016
42.2
69.2
1.7
1.3
1/1
XNOR-NeI (Rastegun al.
2016
51.2
73.2
1.7
1.3
1/1
CCNN Xu & Cheung
2019
54.2
77.9
1.7
1.3
1/1
Bi-Real Net (Liu
2018
56.4
79.5
1.7
1.5
I/1
Rethink. BNN (Helwegen et 2019
56.6
79.4
1.7
1.3
1/1
XNOR-Net++ (Bullt Tzimrropouoo
2019
57.1
79.9
1.4
1/1
IR-Net Qin et al. 2020
80.0
1.7
CI-Net (Wang al. 2019
84.2
Real-to-Bin (Martinez et al. 2020
65.4
86.2
1.7
1.5
1/1
Ours
67.5
87.5
1.7
1.1
1/1
70.0
89.2
1.7
1.1
1/1
Ours
71.2
90.1
1.7
1.1
1/1
Table 8: Comparison with state-of-the-art binary models on CIFAR100. Last column shows the capacity scaling used, while * denotes models trained using AT (Martinez et al. 2020
Architecture
Accuracy (%)
bits (W/A)
XNOR-Net (ResNet18) Rastegarii al. 2016)
66.
1/1
XNOR-Net (WRN40) RRstegarii et al. 2016)
73.2
1/1
CBCN (Liu al. 2019)
74.8
(1/1)x4
Real-to- Bin (Martinez et al. 2020
76.2
1/1
Ours
76.5
I/I
Ours
77.8
I/1
A.3.2 DATA AUGMENTATION
Network binarization is considered to be an extreme case of regularization Courbariaux et al. (2015). How- ever, recent work suggests that data augmentation remains an important, necessary aspect for success- fully training accurate binary networks Martinez et al. 2020)). Due to their lower representational power, Martinez et al. (2020) argues that, for the binarization stage, weaker augmentation, compared to real-valued networks, should used on large datasets such ImageNet.
Table 10: Impact temperature 7 on ac- curacy (Stage models) on ImageNet.
0.02
1
25
Top-1 I acc.
65.4
65.5
65.4
64.6
14

---

Published as a conference paper at ICLR 2021
AveragePaoling
AvcragePooling
C_in >> C_out, 1x1
C_in C_in/4,
PRoLU
PROLU
C_in/4 >> C_out, 1x1
BN PRoLU
(a) The vamilla downsampling block.
(b) The proposed non-linear decomposition: We use 2 layers, with non- non-linearity in n-between, that maps C.n. to C../ (here = 4) and then back to Cout.
Figure 4: The (a) vanilla and (b) proposed downsampling block. This module is used places inside the network where the number of channels changes between macro-modules.
As opposed to this, we found that more aggressive augmentation, similar to the one used for real- valued networks He et al. (2016) or mixup Zhang al. (2017), leads to consistently better results. For example, using mixup on top of random scaling and cropping improves the results by 0.4%. In comparison, when we trained Real-to-Bin Martinez et al.] (2020) with mixup. the accuracy dropped by 0.25% for Stage I, and 0.8% for Stage II. This suggests that, thanks to the proposed methods, are getting closer than ever to the capacity of a real-valued model (which is amenable to stronger augmentations).
A.3.3 EFFECT OF TEMPERATURE
One important component that influences the training efficiency of the gating mechanism is the softmax temperature т. As mentioned earlier, lower temperatures will produce spikier gradients while lower ones will induce the opposite. We explore the effect of various temperatures in Table I0 can be seen that our results are stable over a wide range r = [0.02, 5]. Moreover, to validate the did experiment
importance of using Eq. 4 for computing the gradients for back-propagation, where we replaced it with that of a sigmoid. Unsurprisingly, Stage accuracy drops from 65.5% 62.7%. This further highlights that the proposed form of the gating function key enabler for training higher performing models using EBConv.
A.4 NETWORK ARCHITECTURE NAMING CONVENTION
This section clarifies the naming convention used in our paper: We define network using the following notation N E - G G2 G3. Here E is the expansion rate, defined as multiplier with respect to a vanilla ResNet. For example network with the first block having 128 output channels will have expansion rate 2f N, and G {0, 2,2,3) 3} represent the number of convolutional blocks, and respectively, the number of groups used by convolutions at each stage. Note that ResNet has 4 stages. We graphically show the correspondence between this notation and the network structure in Fig.
A.5 MEMORY USAGE ANALYSIS
Model storage size: Current network binarization methods preserve the first and the last layer real-valued (Rastegari et al. 2016 Liu et al., 2018 Bulat & Tzimiropoulos) 2019). As such, for ResNet-18 binary model trained on Imagenet, predicting 1000 classes, more than 2MB of the total space is taken by these parameters. As result, our 4 expert model takes only 2x more space a device. This is still noticeably less than binary models that attempt to increase their accuracy by increasing their model size (Lin et al. 2017) or by using ensemble of binary networks (Zhu al. 2019). Full results are shown in Table
Run-time memory: In typical deep network, the memory consumed by activations far outweigh that of the parameters (Jan et al. 2019). As such even a ze 4x fold increase in the number of binary parameters (for the case of 4 experts) results in small difference due the above effect.
15

---

Published as a conference paper at ICLR 2021
G
G
G,
G
2024 groups 16
1024 groups 16
128, groups 4
256, groups 8
512, groups 8
512, groups 8
512, groups 8
512, groups 8
512, groups 8
512, groups 8
Initia block
256 groups
FC
N
N
N
N
Figure 5: The overall network architecture of our final model defined as 1262-2-4:8:88 Inline with the current practice (Rastegari et al. 2016) the first (dark-red) and last layer (light-blue) are kept real. The yellow and dark-blue rectangles represent the binary residual blocks described in Sections 4.2 and 4.3 of the main paper with the text indicating the number of output channels and the number of groups. All blocks inside a macroe, represented by rectangle with dashed lines, operate at the same resolution, with the downsampling operation taking place at the first layer via strided convolution (dark-blue).
Comparison with state-of-the-art binary models on ImageNet, including against methods that use low-bit quantization (upper section) and ones that increase the network size/capacity (second section). The third section compares against binary NAS methods. Last column shows the capacity scaling used, while denotes our model trained using AT+KD (Martinez et al. 2020). | denotes ours with an improved training scheme, see Section A.6
Table 11:
Accuracy
(%)
Operations
Architecture
Model size
# bits
BOPS
FLOPS
Top-1
Top-5
(MB)
(W/A)
x10
x10
ABC-Net (M, 5) Lin al.
2017
85,9
42.5
1.3
37.1
(1/1)x5²
Struct. Approx. (Zhuang
2019)
66.3
86.6
7.7
(1/1)x4
Ensemble (Zhu ct al. 2019
61.0
10.6
7.8
21
(1/1)x6
BNN |Courbariaxx al. 2016
42.2
69.2
1.7
1.3
3.5
I/1
XNOK-Net (Kaslegar al.
2016
51.2
73.2
1.7
1.3
3.5
I/1
Real-to-Bii Bin (Marlinez el al.
2020
1.7
4.0
Ours (num. experts = 4)
67.5
87.5
1.7
1.1
7.8
I/1
Ours (num. experts = 4)
70.0
89.2
1.7
1.1
7.8
1/1
Ourst (num. experts
71.2
90.1
1.7
1.1
7.8
I/I
Furthermore, since only a single expert active for a given input this effect further reduced. This confirmed by our measurements reported below. As simple test bed for the later leverage the built-in memory profiler from PyTorch: we measure and report the memory consumption for convolutional layer with 512 input and output channels and kernel size of 3 x 3. We set the input tensor to be of size x 512 x 16 x 16. As can be seen, since a single expert is active for a given image, our EBConv layer has minimal impact the memory usage. Bellow show the profiler output with the operations sorted in descending order, based on memory. For brevity, we show only the top 10 contributors.
Memory profiling output for normal convolutional layer, in descending order, based on memory:
Name
CPU Mem
Self CPU Mem
conv2d convolution
10 512.00 Kb
512.00 Kb
lo
_convolution
512.00 Kb
b
16

---

b Published as a conference paper at ICLR 2021
#wlddn_convolation
512.00 Kb
1
empty
512.00 Kb
512.00 Kb
size
lo b b
contiguous
as_strided
Memory profiling output for EBConv (ours), in descending order, based memory:
Name
CPU Mem
Self f CPU Mem
empty
514.02 Kb
514. 02
Kb
conv2d
512.00 Kb
b
convolution
512.00 Kb
_convolution
512.00 Kb
mklddnn_convoutton
512.00 Kb
adaptive__vv__ooll
2.00 Kb
b
mean
2.00 Kb
b
sum_out
2.00 Kb
b
addmm
16 b
16 b
softmax
16
_softmax
16
b
Memory profiling output for CondConv YYngg et al. 2019), in descending order, based on memory:
Name
CPU Mem
Self CPU Mem
matmul
9.00 Mb
b
9.00 Mb
resize_
9.00 Mb
9.00 Mb
empty
514.02 Kb
514.04 02 Kb
conv2d
512.00 Kb
b
convolution
512.00 Kb
_convolution
512.00 Kb
b
mikldmn_convolution
512.00 Kb
b
2.00 Kb
b Furthermore, the profiler outputs show, for the case of CondConv (Yang al. 2019). the addi- tional multiplication operations required to combine the experts together significantly increase the run-time memory consumption, dominating it. in fact, for low batch sizes typical scenario for models deployed on mobile devices. This further showcases the efficiency of the proposed method. We note that the numbers of BOPs and FL.OPs of our binary model will remain constant the batch size increases because the number of operations itself does not change (with the exception of the linear increase induced by the number of samples within the batch). Additionally, for batch sizes larger than 1, there will be small cost incurred for the actual reading (fetching) of the weights from the memory. However, this cost is insignificant. Finally, we note that, in most cases, when binary networks are deployed on edge devices, a batch size of is expected.
mean
2.00 Kb
A.6 IMPROVED TRAINING SCHEME WITH STRONGER TEACHER
key improvement proposed by Martinez et al. 2020) is the real-to-binary attention transfer and knowledge distillation mechanism. Therein, the authors suggest that using stronger teacher does not improve the accuracy further, hence they use real-valued ResNet- 18 model as a teacher. Here, speculate that the increase in representational capacity offered by the proposed model could benefit in fact from a stronger teacher. validate this hypothesis, we train two real-valued teacher models of different capacity (controlled by depth): scoring 72.5% % Top-1 accuracy ImageNet
17

---

Published as a conference paper at ICLR 2021
and a larger one scoring 76.0%. As the results from Table 12 show, model can exploit the knowledge contained stronger teacher network, improving the overall performance by 1.2%. Throughout the paper, mark the results obtained using the stronger teacher with +.
We note that for training we largely preserve the gradual distillation approach described in (Mrtinez al. 2020): In particular, Step I, we train full precision model with structure that matches that of our binary network. At Step II, we use the previous model as a teacher and train a student with binary activations and real-valued weights. At the end of this step, we also perform our weight expansion strategy, propagating the trained weights across all experts following the optimization procedure described in Section 4.1 Finally. we use the model produced at the previous step teacher, training fully binary network.
Table 12: Impact of the teacher used on the final accuracy of the model on ImageNet.
FP32 Teacher
Binary Student
72.5%
70.0%
76.0% %
71.2%
B SUMMARY OF PRIOR WORK COMPONENTS USED
Herein detail some of the methodological improvements proposed in prior works and also adopted for our strong baseline. We note, that most these improvements are put together to create the strong baseline introduced in (Martinez et al. 2020) which is also the starting point of our work.
B. PER PER-CHANNEL SCALING FACTORS
In order minimize the reconstruction error between the full precision and binary convolution, in Rastegari et al. (006)), channel-wise real-valued scaling factors are used to modulate the output of the binary convolutions. In Rastegaar et al. (2016), the authors proposed to calculate their values using an analytical solution that attempts to minimize the quantization error. The subsequent work of Bulat & Tzimiropoulos (2019) advocates for scaling factors learned via back-propagation by minimizing the task loss. In this work, we adopted the latter, learning one scaling factor per channel via back propaaaato..........
B.2 DOUBLE-SKIP CONNECTIONS
Originally proposed by Liu al. (2018). the double-skip connection mechanism adds skip (i.c. an identity) connection around each binary convolutional layer. This is in contrast with a typical ResNet block (He al. 2016) where the skip connection is applied idea behind it is to preserve a real- valued signal alongside the binary one, improving overall the network's capacity. We also note that network with skip connections around all binary layers will also preserve full precision data path that can improve both the gradients and the information flow.
block level. The main
B.3 PRELU ACTIVATIONS
Rastegari et al. (016) showed that, despite the non-lineat nature of binary networks, ReLU non- linearities added after the binary convolutions can further improve the model's accuracy. However, a ReLU completely eliminates negative values which in Bulat et al. (2019) is found cause training instabilities. To alleviate Bulat et al. (2019) proposes to use a PReLU (He et al. (015)) activation instead. Thanks to its negative slope, it can better preserve the full spectrum of values produced by binary convolution.
B.4 2-STAGE BNN TRAINING
Binary neural networks are notably harder to optimize in comparison with their full precision coun- terparts (Rastegari et al. 2016 Courbariaux et al. 2015 2016). Since most of the performance
18

---

Published as a conference paper at ICLR 2021
degradation comes from binarizing the signal itself (i.e. activations), (Bulat al. 2019) proposes a two-staged optimization strategy where the network is gradually binarized. During Stage I, a net- work with full precision weights and binary activations is trained. Then, in Stage II, fully binary network is trained by initializing the model from the previous stage. As detailed in Section S the training scheduler in both stages identical, with the exception of the weight decay, which for Stage II, is set to 0 Martinez et al. (2220).
B.5 REAL-TO-BINARY KNOWLEDGE DISTILLATION
reasonable objective for training highly accurate binary networks is that the features learned by binary network should closely match those of full precision one up to approximation error induced by the quantization process Rastegari et al. 2016 Martinez et al. 2020). In order to explic- itly enforce this, Martinez al. (2020) proposes to add after each block &g loss between attention maps calculated from the binary and full precision activations. The full precision guiding signal typically comes from an identically structured pretrained real-valued model. To further enhance the efficacy of this process, Martinez et al. (2020) introduces a trainable data-driven scaling factor for modulating the output of the binary convolution. We note that this process is used only top of our best models, and is marked in the tables using an "*".
Finally, we note that the gap between the real-valued and binary models is 3.5 4% (depending on the configuration). In comparison, the next best method, Martinez et al. (2020) has a gap of ~ 5%. This shows that while the proposed structure is tuned for binary networks, it will also perform well for the case of full precision networks. This is perhaps not too surprising since a model easy to binarize should also easy to train in full precision, the opposite however not always true.
C OVERALL BINARY NETWORKS STRUCTURE
Herein, we would like to add a few general notes about how a Binary Network is typically con- structed. Following (Rastegari et al. 2016) and Courbariaux et al. (2016) most works binarize all convolutional layers except for the first and last ones (i.e. the classifier) alongside the batch nor-
malization layers and the per-channel scaling factors (Rastegari et al. 2016 Lin et al. 2017 Liu
2018 Liu et al., 2019 Zhu et al. 2019; Bulat & Tzimiropoulos | Qin et al.] 2020 Wang ctt al.
iet al. 2019 Martinez al. 2020). |Rastegari al. (2016) notes that the first layer not binarized because of the low number of channels (L.e. 3), the speed-up offered by binarization
not high.
Furthermore, because the input to the network is typically real-valued, it is more natural process initially using real-valued operations. Similarly, the last layer sees smaller speedups in practice when binarized (Rastegari al., 22166) Courbariaux et al. 2016) and often, depending on the task requires outputting continuous values instead of discrete ones. Finally, the batch normalization lay- ers are kept real too since they significantly improve the training stability, and also implicitly adjust the quantization point.
We note that, as shown Table 5] the vast majority of the operations are binary, with only a small proportion of them remaining real valued.
19


Reviews:
[{"review_id": "kMIO7gnvPeO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper addresses the problem of filling the gap between the performance of binary and real-valued networks. The authors propose a series of procedures to improve the model and representation capacity of binary neural networks. Different binary-network architectures are obtained through a new network-growing approach and compared.\n\nStrengths:\nResearch on BNNs has a pretty short history if compared with studies on real-valued NN. It is a good idea to start translating some of the main tools from the standard NN literature to the binary setup. \nAccording to the authors' claim, the proposed model greatly outperforms other existing and well-known binary networks. \n\nWeaknesses:\nThe majority of the tools proposed for boosting the performance of binary networks are not new and have been already used in standard NNs. A discussion of the technical challenges associated with applying such tools to the binary setup would help understand the main contributions of the paper.   \nThe inclusion of real-valued experts seems to make the final network not completely binary and it is not clear whether the advantages of BNN (e.g. the gain on computational costs) are preserved.  \nThe network design step mainly consists of rearranging a series of pre-defined building blocks. It is not well explained how the architecture space is searched and how to interpret the results in Figure 2. \n\nQuestions:\n- is the cost of Conditional computing included in the total cost when the main results are claimed (e.g. in \"Without increasing the computational budget of previous works, our method improves upon the state-of-the-art by 6%\")? More generally, when does the fixed number of BOPs include any training step? \n- does the expert selection of the proposed method work better in the binary case than in real-valued networks?\n- are the weights in each `expert' binary or real? Is it fair to compare the obtained hybrid model with real-to-bin?\n- has Grouped Convolution with a similar scaling factor been already used somewhere?\n- what is the difference between the proposed gradient-approximation method and standard \"Straight-Through-Estimator\" (STE)?\n\n-----------------------\nI acknowledge that I read and appreciated the author's response (both parts). The authors' reply mainly answers my questions, especially regarding the difference between applying the proposed techniques to the real and binary setups.\n\nI agree with all authors comments but would tend to confirm my overall score for two reasons:\n\nthe architecture search method is not simply a block rearranging but looks more like a heuristic approach than a clear methodological contribution\nthe proposed mixing of real and binary weights may preserve the advantages of fully binary networks but, again, makes less clear the net contribution of the paper from a more theoretical perspective\nHowever, as I recognize that the paper contains significant experimental results, I would be happy to support acceptance if all other reviewers agree on that.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "j9Ik5XaDxTv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes some techniques to improve the accuracy of binary networks without adding much computational overhead. To improve model capacity, the author proposes mixture-of-experts convolution with a winner-takes-all gating mechanisms. To deal with the limited representation power of binary activations, the paper proposes utilizing group convolutions. The performance is further improved by careful selection of hyperparameters and improved training techniques.\n\nClarity: This paper is pretty clear. The methodology is well-motivated and the algorithms / experiments are described clearly.\nOriginality: To the best of my knowledge, the utilization of mixture-of-experts and group convolutions for binary neural networks is novel.\nSignificance: The propose techniques, despite being simple, achieves good performance. They can be a new baseline for future research.\n\nOverall I think this is a good paper and should be accepted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ax3OVsSLGn", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors improved the performance of BNN by adopting group convolution and data-driven expert binary networks which choose a weight group that takes part in inference. In addition, by searching for the network architecture under the condition of the same number of operations, they achieved SOTA accuracy on ImageNet dataset.\n\nThe results in the paper are strong. Especially, the SOTA accuracy (71.2%) of the proposed network which even outperforms the full-precision ResNet-18 is impressive and could be the significant result in BNN research. The adopted data-driven expert network which increases the number of parameters, but maintains that of parameters participating in inference is also interesting.\n\nBelow is my remaining concerns and questions about the experiments.     \n\n1. Let us assume the condition that the proposed BNN which has four experts processes many images (i.e. multiple batches). In worst case, expert weights which a few images (extremely four) select by gating function can be all different, which gives rise for processors to have to fetch 4 times more parameters (i.e. all trained parameters) compared to single image inference.  So, I concern if under the condition of using multiple batches there is the performance degradation in terms of inference speed or energy in hardware compared to other BNN models.\n2. Unlikely other previous works, mixup is used even on ImageNet. According to [1], it is stated that using mixup on ImageNet slightly degrades the accuracy. If an accuracy of your model trained without mixup (and if possible, accuracy of other models like Bi-Real Net or Real-to-Bin trained with mixup) is provided, the proposed model performance will be more clearly shown under the same training condition.  \n\nMinor comment:\n\n1. I could not clearly understand the message or meaning of Fig. 1b. Detailed explanation of this figure seems to be needed.\n2. I think it will be more helpful for readers to understand Fig. 2 if the authors provide clearer information about it: the detailed network information for each constellation in Fig. 2a and for type of lines in Fig. 2b.\n3. I have a question if the searched network architecture is just optimized to BNN or not. I think this can be shown by measuring the accuracy drop of your model compared to the same full-precision network architecture.\n4. As mentioned earlier, the experimental result is impressive, so it seems that other researchers or related people might want to use your model, but the stated training process seems to be little bit complicated. Are you going to make your code public?\n\nReference\n\n[1] Brais Martinez, et al. Training binary neural networks with real-to-binary convolutions. ICLR, 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JZAozrG1chk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---Paper summary---: \n\nThis paper proposes three approaches to improve the performance of BNNs. 1) Training a super-network (ensemble of BNNs) and dynamically selecting one BNNs to execute conditioned on input. 2) Widening the layers with group convolution to enhance the representational capacity. 3)  Designing the architecture using EfficientNet considering the width, depth, groups and layer arrangement configurations simultaneously.  The effectiveness of the paper has been justified on ImageNet classification but can be further strengthened. \n\n---Strength---: \n+ The performance of the paper is promising. It will serve as a strong baseline for future works.\n+ Using conditional dynamic routing to improve the BNNs capacity is interesting. Specifically, the paper proposes to learn a supernetwork (i.e., ensemble of several experts) during training and dynamically select the path during testing, which enhances the capacity while preserving the inference efficiency to some extent.\n\n---Weaknesses---: \n\n1: This paper ensembles some existing compression/NAS approaches to improve the performance of BNNs, which is not significant enough. \n+ The dynamic routing strategy (conditional on input) has been widely explored. For example, the proposed dynamic formulation in this paper has been used in several studies [2, 3]. \n+ Varying width and depth has been extensively explored in the quantization literature, especially in AutoML based approaches [Shen et al. 2019, Bulat et al. 2020], to design high capacity quantized networks. \n+ The effectiveness of the group convolution in BNNs was initially studied in [1]. Later works also incorporate the group convolution into the search space in NAS+BNNs methods [e.g., Bulat et al. 2020a] to reduce the complexity.\n\n2:  In each layer, the paper introduces a full-precision fully-connected layer to decide which expert to use. However, for deeper networks, such as ResNet-101, it will include ~100 full-precision layers, which can be very expensive especially in BNNs. As a result, it deteriorates the benefits and practicability of the dynamic routing mechanism.\n\n3:  The actual speedup, memory usage and energy consumption on edge devices (e.g., CPU/GPU/FPGA) or IoT devices must be reported. Even though the full-precision operations only account for a small amount of computations in statistics, it can have a big influence on the efficiency on platforms like FPGA. \n\n4:  This paper proposes to learn the binary gates via gradient-based optimization while exploring the network structure via EfficientNet manner. Then the problem comes. This paper can formulate the <width, depth, groups and layer arrangement> as configuration vectors and optimize them using policy gradients and so on, with the binary gates learning unified in a gradient-based framework. So what is the advantage of the \"semi-automated\" method of EfficientNet over the gradient-based optimization? In addition, how about learning a policy agent via RL to predict the gates? I encourage the authors can add comparsions and discussions with these alternatives.\n\n5: More experiments on deeper networks (e.g., ResNet-50) and other network structures (e.g., MobileNet) are needed to further strengthen the paper.\n\n\nReferences:\n\n[1]  MoBiNet: A Mobile Binary Network for Image Classification, in WACV 2020.\n\n[2]  Dynamic Channel Pruning: Feature Boosting and Suppression, in ICLR2019. \n\n[3]  Learning Dynamic Routing for Semantic Segmentation, in CVPR2020. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: ## Summary 
The paper advances the state of the art in training binary neural networks coming out to first place on ImageNet with a controlled computation budget. While any paper making a new record on ImageNet would be a serious candidate for acceptance, it is positive that this one achieves the goal by putting at work the mechanism of conditional computation, innovative for binary networks, and studying in a systematic and clear way how the network width and configuration can be varied while maintaining the computation budged.

## Review Process and Decision

The paper was thoroughly discussed by reviewers from different aspects. Several weaker spots have been identified (see below and final reviews), but no critical issues that would indicate a necessity of a major revision. In the end, reviewers agreed on acceptance although in some cases they have decided to keep their original <=5 ranking to reflect the scientific value to them from a more global perspective.
I think it is an example of well done modeling and experimental work: the work is very clear, uses sound methods, the experimental results are systematic and give interpretable evidence, which is in my experience is rather exceptional for the overall very empirical binary NNs field. I estimate high interest because of the concept of conditional computation put to work here and because of making a new record on ImageNet.

## Details

* Computation Cost

If such networks are to be deployed in low-power devices, the computation cost might need to be estimated more accurately. An example of such estimation is the work by
Ding et al. (2019) Regularizing Activation Distribution for Training Binarized Deep Networks,
where the energy and area are estimated using information from a semiconductor process design kit.

There is indeed a number of floating point operations around the binary convolutions: first and last layers, experts, skip connections with scale factors and non-linearities. The latency and cost of these operations may not be negligible on target devices. In particular Ding et al. (2019)  argue that XNOR-Net architecture is 3 times more costly because of floating point scale factors.
However the paper does a fair job in comparing in operation counts, which is a good proxy for many devices. The floating point computations needed in various places can be indeed further reduced to lower bit width, the research on quantization techniques shows this is possible and orthogonal to the contribution.

* Novelty of grouped convolutions design and search

The work of Phan et al. (CVPR 2020): Binarizing MobileNet via Evolution-based Searching
also proposed to search for best grouped convolution under computation budget constraints (evolutionary search method).
Strict budget constraint and merging results from different groups are somewhat novel and the prior work can be objectively contemporaneous.

* Clarity

Clarity of the paper has been improved by the revision. One remaining mysticism is still about the gradient estimator for the experts. The paper states: "we wish to back-propagate gradients for the non-selected experts", "allows meaningful gradients to flow to all experts during training". The problem is that since $\varphi(z)$ is binary one-hot on the forward pass, the gradient of the scalar product with $\varphi$ in (2) results in that in the backward pass only the selected expert receives the training signal and by no means all of them. This is regardless of how the gradient is propagated through $\varphi$. Maybe something is missing? I hope the authors can clarify in the final version. I do not consider it as a serious flow since this training scheme is not claimed as a contribution in any case.

One more point on the clarity: The paper claims that using experts increases the network representation power / capacity. While this seems logical, and follows the preceding work in real-valued NNs, the paper could provide additional evidence in terms of training performance of these models. Since the teacher with 76% accuracy is used in the distillation, I assume the training never reaches 100% training accuracy in any of the settings. Does the training accuracy improves with experts? This would be a helpful evidence for further work.

* Search method

The paper was further criticized for that the manual search of the architecture is a step back from automated search methods (NAS, BATS). However these methods are themselves a relaxation of discrete choices (experts, if you like), that need to keep all possible configurations at the same time, which may be less stable and too costly for real architectures and datasets. The principles of gradient-based architecture search are not entirely clear and the resulting models coming out of these methods typically give no insights regarding good (intelligent) design choices. At present, the systematic exploration with analysis of tradeoffs conducted is seen to have advantages.



Decision: Accept (Poster)