Towards Optimization-Friendly Binary Neural Network
Authors: Nianhui Guo
Joseph Bethge
Hong Guo
Christoph Meinel
Haojin Yang
Venue: Accepted by TMLR ()

Abstract: Binary neural networks (BNNs) are a promising approach for compressing and accelerating deep learning models, especially in resource-constrained environments. However, the optimization gap between BNNs and their full-precision counterparts has long been an open problem limiting their performance. In this work, we propose a novel optimization pipeline to enhance the performance of BNNs. The main approach includes three key components: (1) BNext, a strong binary baseline based on an optimization-friendly basic block design, (2) knowledge complexity, a simple yet effective teacher-selection metric taking the capacity gap between teachers and binary students under consideration, (3) consecutive knowledge distillation (CKD), a novel multi-round optimization technique to transfer high-confidence knowledge from strong teachers to low-capacity BNNs.
We empirically validate the superiority of the method on several vision classification tasks CIFAR-10/100 & ImageNet. For instance, the BNext family outperforms previous BNNs under different capacity levels and contributes the first binary neural network to reach the state-of-the-art 80.57\% Top-1 accuracy on ImageNet with 0.82 GOPS, which verifies the potential of BNNs and already contributes a strong baseline for future research on high-accuracy BNNs. The code will be publicly available at (blind URL, see supplementary material).

Published in Transactions on Machine Learning Research (12/2023)
Towards Optimization-Friend) Binary Neural Network
Nianhui Guo
nianbeui.guoohiii.cco
Hasso Plattner Institut, University of Potsdam, Germany
Joseph Bethge
joseph. whwpi de
Hasso Plattner Institud, University of Potsdam, Germany
Hong Guo
Hong.guoohhp de
Hasso Plattner Institud, University of Potsdom, Germany
Christoph Meinel
christoph.mmcinel@hpi. de
Hasso Plattner Institul, University of Potsdam, Germany
Haojin Yang
haojinn Crgang@@@pi.ce de
Hasso Plattner Institud, University of Potsdom, Germany
Reviewed on OpenReview: https: openreview. net/ forun? id= 4RgRRGXXXDDGG
Abstract
Binary neural networks (BNNs) are promising approach for compressing and accelerating deep learning models, especially in resource-constraine environments. However, the opti- mization gap between BNNs and their full-precision counterparts has long been an open problem limiting their performance. In this work, we propose novel optimization pipeline to enhance the performance of BNNs. The main approach includes three key components: (1) BNext, a strong binary baseline based on an optimization--frinde
basic block design, (2) knowledge complexity, simple yet effective teacher metric taking the capacity gap between teachers and binary students under consideration, (3) consecutive knowledge distillation (CKD), novel multi-round optimization technique to transfer high-confidence knowledge from strong teachers to low-capacity BNNs. We empirically validate the su- ImageNet.
periority of the method on several vision classification tasks CIFAR-10/10 For instance, the BNext family outperforms previous BNNs under different capacity levels and contributes the first binary neural network to reach the state-of-the-art 80.57% Top-1 accuracy ImageNet with 0.82 GOPS, which verifies the potential of BNNs and already contributes a strong baseline for future research high-accuracy BNNs. The code is pub- licly available at https::/ //gpt////gthbb com/hpi xnor xxnor/BNett.
1
Introduction
Deep neural networks (DNNs) have made remarkable progress almost all areas of AI research in recent years (He et al. 2016 Simonyan & Zisserman 2014 Szegedy al., 2015 Huang al., 2017 Tan & Le 2019 Dosovitskiy et al. 2020 Wang et al. 2020 Liu et al. 2022). However, the development of modern neural network architectures is usually accompanied by increase in computational budget, memory usage, and energy consumption, which can hinder their deployment on resouroe-constrainee edge devices such as cell phones, mini-robols, AR glasses, and aulopilot systems (Simonyan & Zisserman, 2014 Szegedy el al. 2015 He et al.) 2016 [Hann et al.] 2017; Tan & Le| 22019; Dosovitskiy et al.] 2020 Wang et al. 2020 Liu ct al. 2022). Thus, there exists significant technical gap between the evolutionary trend of modern neural networks and edge applications. To bridge this gap, necessary to co-design optimization strategies from both sides to maximize the potential of deep neural networks the edge.

---

Published in Transactions on Machine Learning Research (12/2023)
Motivated by this, various techniques have been developed to compress and accelerate modern neural net-
works, such as Network Pruming (Han et al. 2015), Knowledge Distillation (Hinton et al.]
2015),
Compact
Architecture Design (Howard et al. 2017), and low-bit Quantization [Jacob et al. 2018
Banner et al,
2019 Among them, Binary Neural Networks (BNNs) have gained attention due to their potential on edge
devices (Courbariaux el al., 2016 Rastegari el al. 2016: Liu et al., 2018: Martinez et al.
2020 Liu el al.
2020b; Zhang et al.] p021p. BNNs quantize weights and activations to 1-bit values (+1 or
-1); which can
reduce memory requirements by 32x and achieve theoretical speedup of 58x on the CPU by replacing the
floating-poiis point dot products with XNOR and bit-counting operations (Courbariaux et al. 2016 Rastegari
ct al. 2016). Despite the apparent efficiency advantages, binary neural networks (BNNs) have long faced
optimization difficulties and accuracy degradation problems (Courbariaux et al.
2016 Rastegari et al. 2016
Liu et al. 2018 Martinez el. al. 2020 Liu et al., 2020b; Zhang et al.] 2021j. While previous works have
attempted to narrow the accuracy gap to full-precision models such as ResNet-18 and ResNet-50, the lat- est BNNs still lag behind their full-precision counterparts by approximately 10% (Dosovitskiy et al.] 2020;
Liu et al. 2022 2021a). This is due to their significantly reduced representation capacity (by a factor of [Tu et al. 2022) and increased optimization difficulty. To solve the non-differentiabbiiyy problem in backpropagation, BNN training relies on gradient estimation techniques, such as the
3.4 x 10
Estimator (Courbariaux et al. 2016). Morcover, BNNs have a much coarser loss landscape, making optimiza- tion more challenging than full full-precision networks (see Fig. Tbj. These issues demand specific architect.ual designs and optimization methodologies for highly accurate BNN results.
In this paper, we first propose BNext, a novel binary ar- chitecture that serves the baseline for high-accuracy exploration. To address the limitations of previous BNNs and improve the representation capacity, we introduce a novel binary processing unit, called Info-RCP, which in- corporates an adaptive information re-coupling structure. Different from previous methods (Rastegari et al.) 2016 Martines et al.] 2020 Zhanng el al., (022)) that simply use channel-wise scaling to mitigate information loss, we find that these approaches do not consider the semantic gaps before and after binary convolution, resulting in subopti- mal information reconstruction. Our method scales and shifts the binary convolution outputs using an additional BatchNorm and PReLU layer first, followed by an atten- tion branch that combines information before and after the binary unit. We also introduce a new shortcut design for Infor-RPP to enable clement-wise attention (ELM- Attention) mechanism and enhance the block capacity, inspired by the regularized architecture design of recent vision transformers (Han et al., 2021| Liu et al. 2021ah. It should be noted that the additional computational mechanism is almost negligible. Finally, We propose an effective
DN
EN
Pobwit MM-2
RaNNa-18 18
Miclus 59
Molus
mhez--15
ReActhict BiRcal
Bonry Nat 14
1d
TR. Not
alk 1B
#Mode Faram (MB)
(a) Top-1 occuracy ImageNot
(b) 3D Lova Laudacage
Figure 1: (a) Comparison with SOTA binary neural networks on ILSVRC-2012 ImageNet re- garding accuracy and model param. (b) 3D loss landscape visualization (Li et al. 2018)) of full- precision ResNet-18 (He et al. 2016), BinaryNet- 18 (Courbariaux et al. 2016) and our BNext-
overhead introduced by the ELM--Attetiin architecture scaling strategy and construct the BNext family by stacking the basic blocks based on it. We empirically evaluate the superiority of BNext design principle by visualizing the loss landscape (Fig. Ib BNext. holds a much smoother and more complete loss landscape compared to traditional binary ResNet-18 and is pretty close to full-precision ResNet-18.
From the optimization aspect, though Knowledge Distillation (KD) is already commonly used technique in previous works Marrtinez et al. 2020: Liu et al. 2020b 2021b Zhang et al. 2021), rarely work has paid attention to the impacts and principles of teacher selections for BNNs. During our empirical exploration, we show that not all teachers teach equally to binary students and thus present a specific metric, Knowledge Complezity, as a simple yet effective indicator for teacher model selection. we show that directly transferring Knowledge from higher knowledge snowledge-compleitt teacher is less efficient for binary students, which can suffer from counter-intuitive overfitting problems. Based on the observation and proposed metric, we build consec- utive knowledge distillation (CKD), a novel multi-round optimization technique, where the high-complexity knowledge is assembled and regularized by low-complexity knowledge during the training. We evaluate the
2

---

Published in Transactions on Machine Learning Research (12/2023)
proposed optimization pipeline the challenging ImageNet classification task. As shown in Fig. la the BNext family outperforms previous BNNs under different model size levels and contributes the first binary neural network baseline to reach the 80% accuracy level on ImageNet.
Our main contributions can be summarized as follows:
1. BNext, a novel optimization-friendly binary architecture that serves as a strong baseline for high- accuracy BNN exploration.
2. Consecutive Knowledge Distillation, an effective and practical technique to alleviate the knowledge gap during binary model optimization.
3. Evaluation on ImageNet contributes the first BNN with 80.57% Top-1 accuracy, which provides a meaning strong baseline for future high-accuracy BNNs research.
2 Related Work
BNNs Optimization. Binarization of neural networks reduces computational and memory requirements, but optimizing binary neural networks (BNNs) is challenging due to their non-differentiabiiiy and loss of information (Courbariaux al., [2066). BinaryNet solved this challenge by using the trraiht-throuuhh estimation (STE) technique (Courbariaux et al. 2016), and subsequent works have attempted to improve optimization using different variants of STE. BOP proposed specialized optimizer to flip binary states [Helwegen et al. 2019), while Real2BinaryNet used knowledge distillation in a thrce-stage process (Martinez et al. 2020), and other works simplified it to a two-stage process (Liu et al. 2020b Zhang et al. 2021). Additionally, Liu et al. explored how training strategics such as optimizers and weight decay aid BNN optimization (Liu et al.) 2020b). In this paper, we analyze optimizzaiion-riinnlly module designs in BNNs further improve optimization, helping us understand why BNNs are difficult to optimize and how construct optimization-friendyy architecture.
Knowledge Distillation. In knowledge distillation (Hinton et al. 2015). dark knowledge is transferred from pre-trained teacher model to a student model. Recent works have aimed to gain deeper understanding of this technique. For instance, it has been discovered that there exists large discrepancy between the predictive distributions of the teacher and the student (Stanton et al.j 2021). Beyer (Beyer et al. 2022) highlight that cerlain implicit design decisions can significantly impact the eflicacy of distillation. Park (Pakk et al. 2021) improve the distillation process by creating student-friendyy teachers and revealing the presence of knowledge mismatch between the teacher and student. Morcover, modern binary neural network optimization
(Martinez et al. 2020: Liu et al.. 2020b Zhang et al. 2021) heavily relies knowledge distillation techniques
(Hinton et al. 2015 Shen & Savvides 2020). In particular, the soft label from a pre-trained teacher provides
more fine-gen greigea supervision signal than a one-hot label. This encourages the research community to explore the BNN-specific optimization pipeline for improved model performance.
Modern Neural Network Optimization. Modern deep neural networks (Tan & Le 2019 Dosovitskiy
al. 2020 Liu al. 2021a 2022) owe their success in image classification not only to architecture design ideas, such as attention mechanisms, but also to collection of state-of-the-art optimization techniques. Data augmentation strategies, like MixUp (Zhang et al.) 2017). CutMix (Yun et al. 2019), Rand-Augmentation (Cubuk et al. 2020), and Augmentation Repetition (Hoffer et al. 2020), have been shown to benefit the generalization of regular 32-bit networks (Riduik et al. 2022). However, il remains an open question whether these strategies can be applied to highly accurate BNNs or BNNN-spccfiic augmentation techniques are necessary. This paper presents fair and comprehensive empirical study of modern optimization techniques for BNNs in large-scale image classification tasks. The study demonstrates the importance of revisiting BNN-specific augmentation stragesees and identifies potential solutions.
3

---

Published in Transactions on Machine Learning Research (12/2023)
(6) RosNet-18
(b) Not-ryNet-15
(c) B1
(e) BiRoalNet-18 + Info-RP (ours)
(1) BiRealNot-18 + BI-M- Actenttion (OurE)
(g) BNext [owns]
BeniiBinary
Figure The loss landscape visualization of popular BNNs (contour line view) based the CIFARIO dataset. We follow the setting of (Li et al. 2018). "Info-RCP" and "ELM-Attention" are the module designs in this paper. The corresponding 3D views available in the supplementary material.
input
input
input
Hx C
Hx W
Hx Mxx c
BN
BN
Global Pool
Sign
Global Pool
Sign
Sign
Linear
BConv
QLinear
C/1
C/
ReLU
BConv
BConv
Linear
QLinear
Scale
Sigmoid
PReLU
Sigmoid
Pool
PReLU
Hin wing
H/n w/x
Whxx
x output
output
output
(a) XNOR Net
(b) Real2BinaryNet
(c) Info-RCP (ours)
Figure 3: Binary convolution module design comparison among XNOR-Net, Real2Binary.Net and Ours.
3
BNext Architecture Design
This section introduces the BNext architecture design, its motivation, and details. In Section 3.1 we visualize the loss landscape of popular binary neural networks, offering valuable insights. Additionally, we present the two core modules of B.exxt and an overview the BNext model family's structure in Section 3.2
3.1 Visualizing the Optimization Bottleneck
Binary neural networks are a type of lightweight neural network that consists of stacked 1-bit convolutions. During forward propagation, both the inputs of. and the proxy weights wf in each 1-bit convolution are binarized into #1 using the sign function before the dot product computation. In backward propagation, the gradient of the hardtanh function approximates the sign function (Hubara al.] 2016), enabling to compute the gradient of wf and optimize the weights w. using end-to-end SGD. The gradient of w is accumulated in the proxy weights w during each iteration (Courbariaux et al. 2016 mathematically formulated as follows:
This process can be
The / binarize Jetww)) (sign (x+) sign (wi)),
(1)
det
det
de Oug
1+1 wf w
Owt
awf
Dugg Dw!.
where awf = 1psg represents the Straight- Through-Estimmttio Estimation (STE) strategy. r, b, and t represent
Bif real-valued variables, binary variables, and itcration numbers, respectively.
(STE) strategy partially addressing the non-differentiable problem, Binary Neural Networks (BNNs) suffer from severe accuracy degradation due to the binary representation and gradient mismatch problem (Courbariaux al. 2016 Rastogari et al. 20166. To address these bottlenecks,
Despite the Shraigh--Throuuh-Esttmee
several subsequent designs (Rastegari al. 2016 Liu al., 20188 Martinez et al. 2020 Bethge et al. 2020
Liu et al. 2020bb have been proposed to improve capacity and optimize BNNs.
4

---

Published in Transactions on Machine Learning Research (12/2023)
To understand the existing optimization bottleneck in BNNs, first focus the recent popular module designs in binary architectures. In particular, researchers have proposed various performance--nhancing structures, such as double residual connection {Liu et al. 2018), as introduced in Real2BinaryNet (Martinez ict al. 2020 Zhang et al. 2021). and adaptive distribution reshaping (Liu et al. 2020b). However, the shared insight it behind these lechniques remains unclear.
Liu et al. (Liu et al. 2021b) studied the impact of training strategies and optimizers on BNNs by visualizing the loss landscape. Inspired by this idea, we further use it as an indicator to analyze the most popular binary architectures, including BinaryNeet (Courbariaux et al. 2016), BiRRealNet {Liu et al., 2018), Real2BinaryNet
(Martinez et al. (020)), and their full-precision counterpart ResNet-18 (He et al. 2016). To understand the relationship between binary architecture design and model optimization, We follow (Li et al. 20018) and plot the corresponding 2D (contour line view) and 3D loss landscapes for each network. The core idea behind visualization described the algorithm [ Given a pre-trained BNN characterized by its filters, symbolized as 8, we first project the filters in each layer into two orthogonal dimensions by combing 0 with independent Gaussian noises a and y follows:
Fa B, = L( + aœ + Bn, D),
(3)
Then the network prediction loss along with and 3 (ranging from -1 to 1) are recorded in Fig. 2 as proxy observation of the high-dimensional loss landscape.
Fig 2a and Fig 2b reveal that direct binarization of the network (Courbariaux et al. 2016 creates an ex- tremely discontinuous and rugged loss landscape surface. Such a highly rugged surface makes BNNs prone to converge into sub-optimal minima and sensitive to inappropriate optimization processes. In Fig. 2c we observe that the double skip-connection design proposed in BiRealNet (Liu et al. 2018) alleviates the in- formation sparsity problem to some extent and allows more continuous information flow through bypass connections. The contours are more structured and complete, although still appear rugged. Real2BinaryNet [Martinez et al. 2020) suggests reshaping the output of binary convolution in data-driven manner using a trainable attention module, which reduces the information bottleneck. In comparison to Fig 2b and 2c Real2BinaryNet further flattens the area around the global minimum, and the gap between each contour line in 2dg more uniform, implying greater robustness initialization and optimization.
Algorithm 1 Visualization the loss landscape of BNNs
Require: pretrained BNNs with filters 0, training dataset D, two random sampled Gaussian direction vectors a and 77, two scalars a and 3.
for in layers do:
for i in filters do:
of 0f T oi of nf end for end for for a in rango(-1, 1, 50) do:
filter norm Li et al. (2018)
||
> direction x
for in range(-1, 1, 50) do: 3,, = C(0 + ao + Bn, D), end for end for Plotting the 2D nad 3D loss landscape visualization using (,,
> direction y loss value
Inspired by these findings in the visualization, We have identified three core architectural design factors that inffuence the optimization friendliness of BNNs: (1) Activation binarization restricts the number of feature pallerns and diversity per layer. (2) Proper reshaping of distribution after binary convolution is essential for effective feature modeling between adjacent 1-bit processing units. (3) Effective shortcut designs can enhance information density in forward and backward propagation. Although previous works have tried improving BNNs partially from the summarized principles above, our visualization results have demonstrated clear optimization friendliness gap, and the need for further improvement, compared to the full-precision backbone,
5

---

Published in Transactions on Machine Learning Research (12/2023)
HxWxC
Sign
Global Pool
3x3 Processor
1xt Processor
3x3 Processor
1xl Processor
kak BConv
QLinear
Basic Block
BN
BN
Ch
Qlinear
Processor
PReE.U
Sigmoid
Element Attention Module
Feed Forward Network
Hin x Win x
R
If
Hf
W
If
W.W
2 2
XC
PRALU
x2
B
16
x
166
32 32
Stage 1
Stage
Stage
Stage d
3x3 Provessor
1x3 Provessor
QCanvza
8Next Basic Block
8Next Basic Block
BNext Basic Block
8Next Basic Block
BN
"Cat"
x
XN
Input Image
Stem Block
Output
Figure 4: The architecture of BNext. "Processor" is the basic binary convolution with improved Info- design, "BN" represents batch normalization. "C" indicates the base width (channel number). "N" and "M" are the number of basic blocks in each stage respectively.
which motivates us to build BNext, a strong binary baseline based on an oplimizatiin-friiee foriondy basic block design.
3.2 Building Optimizationn-Fiindde Binary Neural Networks
This section introduces the core building components and design ideas of our BNext nelwork, a binary architecture that is optimized for efficient information propagation and reduced bottlenecks.
Binary Convolution with Information Recoupling We propose a new binary convolution module with adaptive information reshaping and coupling avoid bottlenecks between 1-bit processing units. Our module enhances information propagation, shown in Figure 3c We adjust the output distribution of binary convolution during the forward pass using Batch Normalization and PReLU layer, and explicitly calibrate information flow via a shift and scale design. We use shorteut connection to merge the input and output of the main branch [BConv-BN- (N-LLB by addition, and feed the output into gap-aware snuozzo-and-Expand (SE) branch (Hu et al. 2018), which scales the main branch output. Our design, called the InfornationnReccopiing (Info-RCP) module, oflers a more robust and flexible post-convolution reshaping mechanism than existing approaches such XNOR-Net and Real2Binary-Net (Fig 3al and 3b). Our ablation study (Table 5) confirms the effectiveness of Info-RCP in improving network performance. Moreover, using Info-RCP results in more uniform and smooth loss landscape (Fig. 2e). indicating that it facilitates optimization.
Basic Block Design with ELM-Attention We propose a novel basic block design with an enhanced bypass structure, inspired by recent attention design ideas in vision transformers and our analysis in Sec.
3.1 Our design utilizes multiple Info-RCP modules stacked together form the fundamental building block (Processor), with each Processor followed by a BatchNorm layer. To relieve information bottlenecks in forward propagation, enclose each basic Processor Info-RCP-BN, using continuous residual connections. Additionally, we introduce element-wise shortcut multiplication to dynamically calibrate the output of the first 3x3 Processor. Compared to double skip-connection design (Liu et al.] 2018). this clement-wiss multiplication improves forward feature fusion and propagation in each block, significantly improving the loss landscape for BNNs. We demonstrate that combining both Info-RCP and ELLM-ttenthe yields a significantly smoother loss landscape, as shown Fig. 2g which for the first time comparable the full-precision backbone. An intuitive explanation to the synergistic smoothing effect of both modules is that this two modules work mixed channel-wise and spatial-wise attention together in the basic block design.

---

Published in Transactions on Machine Learning Research (12/2023)
Table 1: BNext architecture overview. "Q" represents the quantization bit-width of weights and activations (W/A) in Fig. for the first layer, the SE branch and the output layer, respectively.
Models
Stage Ratios (N,M) Base Width (C)
(W/A)
BNext-T
1:1:3:1
32
8/8-4/8-8/8
BNext-S
1:1:3:1
48
8/8-4/8-8/8
BNext-M
2:2:4:2
8/8-1/8-8/8
BNext-L
2:2:8:2
64
8/8-4/8-8/8
The coupled attention design largely improves the representational capacity in forward information flow and enables each basic block to recalibrate the shifted features from two orthogonal dimensions. Therefore, the basic block design implicitly rebuilds the continuous processing ability full-precision convolutional basic block.
BNext Family Construction We propose the BNext family, which comprises of basic blocks stacked under different stage design strategies. To provide fair comparison with existing works, we apply our proposed block designs to the widely-used MobileNetV1 backbone and create BNext variants by adjusting the block width and depth in each stage. Fig illustraaee the overall BNext family ar- chitectures, while Table [ presents the specific con- figurations of the BNext networks. The width of the input layer is denoted by C, while N and M represent the stage depth ratios, as shown in Fig Though the MobileNetV1 version of BNext has al- ready shown promising improvements over previous works, We build four BNext variants with increasing capacity to assess the scalability of the BNext archi- tecture design and previously unexplored accuracy boundary of BNNs.
Comneeat
Sayher Top Acuuscy
82-
CowNeetT
Suster Sop Arc. PM 2
ResNet-50
RecNot1111
60-
Emfdee UNot B4
ResNer
ConcieettS
Et
Frsshvt-50
70
14
BesNer-34
Comteot 8
1.50
1.25
1.50
1.75
2.50
1.00
1.55
1.50
1.75
2.50
Teacher Krowlodg: Comploxity Score
Teacher Srow udge Compleity Store
Figure 5: Relationship between binary model overfit- ting and the knowledge-compleet impacts under the selection of different tcachers. The two y-axes indicato the teacher (left) and student (right) test Top- accu- racy.
Algorithm Consecutive Knowledge Distillation
Require: binary student Sg, a strong teacher Tğ and a group of assistant teachers (T""...... Ty, } Training dataset D. Total epochs N. Initial assistant teacher index m = 1. AMP quantization Q, provided by PyTorch.
for in epochs do:
start training
for j, in enumerate(D) (D) do:
1.forward Sx(x), (T)) @(T
( (a),
2. calculltt L(x, T?"). Func. 5
b.bakkaard and updates, Func.
end for
if 1% (N)
=0 then:
switch reference teacher increase knowledge effectiveness
m++1,
end if end for
> end training
4 Consecutive Knowledge Distillation
Knowledge distillation (KD) a vital optimization technique for binary neural networks (BNNs) (Martinez et al. 2020 Liu et al. 2020b 2021b Zhang al. 2021). but rarely work has paid attention to the impacts and principles of teacher selections for BNNs. In this work, we propose simple yet effective metric, Knowledge
7

---

Published in Transactions on Machine Learning Research (12/2023)
Complexity (KC) incorporating accuracy and compactness for efficient teacher selection:
KC((T) = log 1+ Accuupacy(17)4
Params(T)
(4)
(ccuracy(()) + 10 10-8
where T is the selected teacher model, Params is the number of parameters (MB) and Accuray is the teacher's test top-1 accuracy. Generally speaking, KC defines the cost of teacher parameter-space complexity to the dark knowledge effectiveness.
It is commonly believed that using a stronger teacher will increase the student's accuracy upper limit. However, we empirically show in Fig. that a teacher with both high accuracy and high KC can easily cause overfitting problems in binary student optimization, where the large capacity gap and prediction discrepancy between BNNs and teachers are not considered. Similar observations have been reported PokeBNN (Zhang et al. 2021), which trys using a high-precision VIT teacher instead of ResNet-50 (Dosovitskiy et al. 2020) for binary student optimization. However, the results show that more accurale teacher does not necessarily improve the student model's accuracy, indicating that teacher accuracy alone is insufficient for optimal performance. Instead, we suggest prioritizing teachers with lower KC for BNNs' Knowledge Distillation (KD) training.
Motivated by our observation Fig. we further develop a consecutive knowledge distillation algorithm (Alg. 2) improve the optimization of BNNs. Specifically, we first build a self-pacer knowledge ensemble to control the complexity degree of teacher knowledge Тğ by adding another less confident reference teacher T into consideration. The binary student So consistently learns from two teachers during the training process. The overall loss functions are mathematically formulated as follows:
((x) = (x) KLS.,T
+ (1 ( (x)) K KLs.Try Σ. exp exp
(5)
exp (xp(=ce(((((3))))))))
(xx) =
(6)
where CE and KL indicate the Cross- Cooss-Entropy and KL-DDvergence loss (Shen Savvides 2020), respectively. The scaling factor (zx) determines the relative weight of the two terms for each input example x. It is caleulated the softmax of the cross-cntropy losses between the student model and each teacher model output distribution. Overall, the loss function aims to balance the preservation of the teacher model's capacity with the accuracy of the student model's predictions by using a reference distribution in the second term.
In order bridge the gap knowledge, we propose a novel knowledge-evolution strategy. During the training process, our model employs strong teacher Tý, which remains consistent throughout. However, the reference teacher is selected from a pre-defined group of candidates. We introduce each reference teacher in ascending order of accuracy and knowledge complexity, allowing us adapt references at different stages of the training process. This approach not only lowers the optimization difficulty but also promotes better knowledge diversity as a regularization technique. Further details on the reference teacher group settings are available in the supplementary material.
5 Experiments
In this section, we evaluate our model and methods on the ILSVRC12 ImageNet (Deng et al. 2009) and the CIFAR.
5.1 Experimental Setups
We employ the AdamW optimizer [Loscchllov & Hutter 2017 with a learning rate 10-3 and weight decays of 10-3 and 10-8 for non-binary and binary parameters, respectively. We warm up the learning rate for epochs and then reduce it using a cosine scheduler (Paszke et al., 2019). We set the backward gradient clipping range of the STE (F [-1.5, 1.5). We hard binarization (Func for activation
8

---

(Liu et al. 2020b). The models Published in Transactions on Machine Learning Research (12/2023)
Table 2: Performance comparison with modern DNN designs and SOTA binary designs on ImageNet. "W/A" indicates the bit-width of convolutions except for the input layer. "BOPs" the number of binary multiply- addition calculation (MAC) operations, "QOPs" is the INT-4/8 MAC operations in total, "OPs" is the total MAC, "Param" is the real parameter size, and "f" indicates results with post-training quantization none-binary layers. In most cases, BNext performs better than models of similar (or larger) size marked with (a) through (e).
Regular Designs
OPs (10%)
#Param (MB)
Top-1 (%)
Binary Designs
BOPs (10%)
QOPs (10%)
FLOPs (10%)
OPs (10*)
#Param (MB)
Top-1 (%)
W/A
(a)
LQ-Net-18
2/2
8.4
64.9
BNN
1.70
1.20
1.47
4.2
42.2
LQ-Net-18
4/4
16.8
69.3
XNOR-Net
1.70
1.20
1.47
4.2
51.2
(b)
MobileNetV2
8/8
0.8
3.5
68.3
XNORt-Net
1.70
1.20
1.47
4.2
57.1
(d)
Res.eet-50
8/8
4.8
24.6
74.7
(a)
Bi-RealNet-18
1.68
1.39
1.63
4.2
56.4
ResNet-18
32/32
18.0
44.6
69.6
Bi- Bi-RReaNNet-34
3.53
1.39
1.94
5.1
62.2
(c)
MobileNetV3
32/32
2.2
21.6
75.2
Bi-ReeaNeet-52
10.7
4.48
6.15
64.5
ResNet-50
32/32
38.0
97.5
76.0
(b)
MeliusNet-29
5.47
1.29
2.14
5.1
65.8
(e) DeiT--
32/32
46.0
83.9
79.7
MeliusNet-42
9.69
1.74
3.25
10.1
69.2
(e)
RegNetY-44G
32/32
40.0
80.1
80.0
MeliusNet-55
18.30
2.45
5.30
17.4
71.0
Swin-T
32/32
45.0
106.0
81.3
Real2Rinary-Net
1.67
1.56
1.82
5.1
65.4
T
32/32
45.0
114.3
82.5
BiRR18
1.68
1.63
1.89
4.2
65.9
ReActNet-A
4.82
0.87
7.4
69.4
ReActNet-Adam
4.82
0.12
0.87
7.4
70.5
Our Design
W/A
OPs
#Param
Top-1
PokeBNN 2.0x
14.14
2.27
77.2
w/o Post-Quant.
(10%)
(MB)
(%)
BNext-18 (ours)
1/1
1.64
68.4
(a)
BNext-18 f (ours)
1.68
135.2
0.43
2.2
67.9
BNext-'I' (ours)
1/1
13.3
72.4
BNext-T i (ours)
4.82
0.77
5.3
72.0
BNext-S (ours)
1/1
1.90
26.7
76.1
BNext-S (ours)
10.84
21.1
1.72
11.1
75.8
BNext-M (ours)
1/1
3.38
46.5
78.3
BNext-M (ours)
20.09
24.3
3.17
20.4
77.9
BNext-L (ours)
1/1
8.54
106.1
80.6
BNcxt-L (ours)
52.15
39.4
8.19
47.6
80.4
Table 3: Design and optimization principle comparison with existing BNNs. BNext is in line with the current works but shows a higher efficiency on binary operation ratio.
Methods
BN 32-bit Bypass KD SE BOPS/(OPS*64)
BNN Hubara et al.
18.07%
BiReal-18 Liu al.
2020aa
17.86%
Real2BinaryNee Mar finez et al. 2020
14.34%
ReActNet-A Liu al. 20206
86.50%
PokeBBNN22 Zhang ct al. 2021
V
97.15%
BNext-18]
7
V
4
61.04%
BNext-T
V 4
97.81%
BNext-Lf
V
v
99.49%
and progressive weight binarization as proposed in (Guo et al, 2021 For the evaluation of our technique, we use two different backbones: ResNet-18 (He et al. 2016) and ReActNet are trained on Nvidia DGX-A100 GPUs.
We train our model with an input resolution of 224x224 and a batch size of 512 for 512 epochs. To enhance data augmentation, we use RandAugment (7, 0.5) (Cubuk et al. 2020). For the BNext optimization,
ImageNet:
the Diversified Consecutive KD process described in Section. with ConvNext-Tiin Tiny [Liu et al. 2022 the strong teacher and [EfficientNetttBOO-B22 the candidate reference teacher groups.
TTan & Le 2019 ConyNext-Ting: as
We train the model with a batch size of 128 for 256 epochs, using standard data augmentations such as random crop, random horizontal flip, and normalization (Paszke al.] (019).. We optimize the model using cross-entropy loss Paszze al. 2019). To account for variation, we report the average of five
CIFAR:
9

---

Published in Transactions on Machine Learning Research (12/2023)
Inputs
BNex-T
ReActNet
Figure 6: Binary features between BNext-T and ReActNet. BNext-T holds more diversified binary features without checkerboard artifacts compared to ReActNet, showing improved feature modcling between adjacent 1-bit processing wnits. Best viewed color.
Table 4: A comparison state-of-the-art BNNs on CIPAR10 based on the same ResNet--18 architecture as backbone.
Method
W/A (Bitwidth)
Top-1 I Ace [%]]
Top-5 Acc [%]
Baseline (ResNet-1e
32/32
94.8
RAD Ding et al.
2019
1/1
90.5
IR-Net Qin
2020
1/1
91.5
RBNN Lin et al. 2020
1/1
92.2
ReCU Xu
2021
1/1
92.8
AdaBNN TTu et al.
(2022
1/1
93.1
BNext (ResNet-18)
(OUTS)
1/1
93.6
(+0.12)
98.8 (+0.03)
5.2 Performance Evaluation
ImageNet. We evaluate the performance of the proposed BNext family by comparing it to existing designs under different capacity levels, demonstrating the superiority our methods. To illustrate, we compare BNext-Tiny' the popular ReActNet (Liu et al., 2020b) design, where ANext-Tinyt achieves 2.6% higher accuracy with 10M fewer operations. Moreover, our BNext-Mt outperforms the previous state-of- saatte-ffthhe-ate the-art design, PokcBNN-2.0x (ResNet-50 backbone), by 0.7% in accuracy, with similar model size. Notably, BNext requires lower optimization requirements than the PokeBNN family. For instance, PokeBNN is trained with an expensive batch size of 8192 for 720 epochs [Zhang et al.) 2021 on 64 TPU-v3 chips, while BNext only uses a batch size of 512 for 512 epochs. To further evaluate the effectiveness of the BNext design, we combine our proposed techniques with a ResNet-18 backbone. We apply standard data augmentation and use only ResNet-34 as a teacher for a fair comparison. BNext-18 achieves a top-1 accuracy of 68.4%, which is only 1.2% lower than the original ResNet-18 (He al.) 2016)). This performance surpasses other designs, such BNN (Courbariaux et al.) 2016). XNOR-Net (Rastegari al.) 2016). BiReal-Net (Liu all, 2018), Real2BinaryNeet (Martinez al.) 2020), and ReActNet-BIRRs (Liu et al. 2020b). Finally, we try to contribute a previously unexplored strong baseline BNext-T. on IeageNet. For BNext-T, we do not intend to design an optimal binary architecture with the best accuracy and efficiency trade-off, but to contribute a practical baseline for future work of high-accuracy BNNs research. With the introduction of BNext-L, we have been able achieve an impressive upper boundary for BNN performance on TmageNet, reaching crucial accuracy level of 80.57%. With this result BNNs show a closer upper boundary to recent SOTA 32-bit designs, such as ConvNext (Liu et al. 2022), Swin Transformer (Liu et al. 2021a) and RegNet [Radosavovic et al. 2020). Meanwhile,
have given more detailed comparison to existing BNNs in Table. 3] where BNext family shows higher binary ratio from the aspects of binary operation ratio. For the sake of fair comparison, our calculation of OPs excludes negligible floating-poin counts as (Liu et al.] 2018
Bethge et al. 2020: Liu et. al. 2020b, 2021b Zhang et al. (022). The additional training information and corresponding optimization curves each model are available Appendix, restricted by page limitation.
CIFAR: We further explore the generalization ability BNext (with a ResNet-18 backbone) on the smaller dataset CIFAR10 (see Table. Compared to the latest designs such as AdaBNN TTu et al. 2022 ReCU
10

---

Published in Transactions on Machine Learning Research (12/2023)
Table 5: The ablation study of Info-RCP and ELM-Attention on ImageNet, showing the Top-1 1op-1/Top-5 accuracy, model size ("Param"), and total operations ("OPs").
Methods
Top-1 (%)
Top-5 (%)
Param (MB)
OPs (10")
w/o ELM-Attention
w/o Info- Recoupeigg
62.43
83.75
4.52
0.76
w/ Info-Recoupling
64.91
85.52
5.58
0.77
w/ BLM-Attention
w/o Recoopling
64.61
85.49
4.52
0.76
w/ Info- Recooplugg
65.02
85.73
5.58
0.77
Table 6: The ablation study of ELt-Attention position in each basic block on CIFAR100 datasct, showing the averaged Top-1 accuracy after runs.
Backbone
BNext- Tiny
ELM-Attention Position Top-1 Ace (%)
1th BConv 3x3
1th BConv 1x1
2th BConv 3x3
2th BConv 1x1
66.88
66.73
66.34
66.76
Backbone
BNext- 18
EIM-Attention Position
1th BConv 3x3
2th BConv 3x3
1/2th BConv 3x3
w/o
Top-1 Acc (%)
72.22
71.72
70.00
66.95
Xu el al. 2021] | and RBNN (Lin et al. 2020 BNext achieves a better performance. It surpasses one of the most recent works AdaBNN by 0.5% and closes the gap to 32-bit ResNet to 0.8%. The complete evaluation of CIFAR100 provided in the supplementary matcrial.
On-Hardware Evaluation: Our study evaluates inference elficiency of binary neural networks (BNNs) a Banana Pi M5, using the Larq Library. Despite Larq's limitations limitations-suppplet supporting only TensorFlow, we integrated it with PyTorch via the C++ extension API. In our tests, 1-bit convolutio layer with an NCHW configuration of (16, 64, 128, 128) on the Banana Pi M5 showcased a 20.1x speedup compared to native Torohh CPU implementation. We incorporated this setup with the BNext-18 model, maintaining channel constraints from Larq, and set input dimensions to [1, 3, 224, 224]. Inference times for pure Py PyTchhh C++, and ARM instruction sets were 1.27s, 0.8948s, and 0.5759s respectively. The Larq integration yielded 2.2x overall acceleration. However, highlighted in our Limitations section, the absence of efficient BNN acceleration methods and the reliance on PyTorch's C++ API, which doesn't mitigate additional floating- point operations, may not fully showcase BNNs' acceleration potential.
5.3 Ablation Study
We present comprehensive evaluation of the proposed techniques in this paper through detailed ablation study module designs, optimization schemes, data augmentation strategies, and post-quantization im- pacts. All experiments are conducted using BNext- Tiny as the base model with standard data augmentation and cross-entropy loss unless otherwise stated.
Module Designs. We evaluate the proposed Info-RCP and ELM-Attention modules through an ablation study, as shown in Table 5 The baseline model without these structures achieves only 62.43% accuracy on ImageNet. Adding ELM-Attention or Info-RCP BNext-Tiny results in accuracy gains of 2.48% and 2.18%, respectively. Combining both designs achieves the highest accuracy of 65.02%, confirming that both methods enhance the model's representative capacity. Meanwhile, futher report the corresponding overhead after using cach sub-design. Both are highly lightweight and only contribute little extra theoretical computation cost. We are also interested in the impacts of making a difference where the ELM-Attention module used. To make clear, we conducted an external evaluation using the BNext-18 and BNext- Tiny backbone on CIFAR100 datasots, Table O Applying the ELt-Attention design the first convolution module shows better modeling ability on both backbones. This shapes the architecture similar the popular transformer block. Either moving the F- module to another position or removing this design
11

---

Published in Transactions on Machine Learning Research (12/2023)
Table 7:
The ablation study on the effects of adding different training strategies to BNext-T on ImageNet dataset.
Method
Epochs
Top-1 [%]
Top-5 [%]]
KD
128
70.76
89.64
KC-Based (x) Selection
128
71.33
(+0.57)
90.02 (+0.38)
+ Consecutive KD
128
71.38
(+0.05)
90.04 (+00002)
+ Rand Augmentation Cubuk et al. (2020)
128
71.46
(+00088
90.06 (+0.02)
Long Training
512
72.40
(+0.94)
90.64 (+0.58)
Table 8: The ablation study on popular data augmentation strategies ImageNet. BNext-Tiny is selected as the baseline.
Data Augmentation
Top-1 (%)
Top-5 (%)
Baseline (BNext-Tinee
65.02
85.73
Mixup Zhang et al. (2017
65.83 (+0.81)
86.16
(+0.443
Cutmix Yun et al.
2019
64.51
(-0.51)
85.32
(-0.41)
Repeat Augment Hofter
(2020)
53.74
(-11.28)
75.65
(-10.08)
Rand Augment Cubuk et al.
2020
67.51
(+2.499
87.28
(+1.55)
Rand Augment Cubuk al. 2020 Mixup Zhang et al. 2017))
66.51
(+1.49)
86.77
(+1.04)
gives suboptimal performance. A deeper exploration of the attention mechanism could further improve BNN design, which is left for future work.
Optimization Schemes. We evaluate the effectiveness of all optimization schemes used in our experiments in Table 7 We replace the ResNet-101 teacher with a teacher selected using knowledge complexity (Efficient- BO), which results in a 0.57% accuracy gain. We further use a gap-sensitive ensemble of Efficient-BO and Efficient--33 and add Rand Augmentation, resulting in a total accuracy gain of 0. 0.13%. Though combining consecutive KD with BNext-T only contributes 0.05% improvements, this can be explained by the relatively diminutive knowledge complexity gap (KC: 0.893 1.103) between the two teachers. However, it's impera- tive to spotlight that the Consecutive KD, despite its marginal numerical impact in Table 6, has underscored pivotal revelation on mitigating the counter-intuitive overfitting issuc as portrayed in Table ! and Figure Extending the training epochs to 512 further improves the accuracy to 72.40%.
Data Augmentations.
Modern high-performance vision models have been benefiting from well-designed data augmentation strate- gies. However, the adaptation relationship to BNN has still not been effectively explored in the current research community. We evaluate the effectiveness of popular data augmentations such Mixup, Cutmix, Augment-Repeat, and Rand Augmentation, as shown in Table S Most of these choices are actually harmful to BNext. model optimization. For example, using the popular Augnent-Repeal can decrease the Top-1 accu- racy by 10.08%. Applying only Rand Augmentation achieves the best improvement of 2.49%. The adaption differences here showcase an interesting optimization gap between BNN and their 32-bit counterparts.
6 Conclusion
In this paper, propose novel optimization pipeline to enhance the cetimiizatioo-fiiendlleese of Binary Neural Networks. This is achieved by first building strong binary baseline, BNext from the aspects of smoother loss landscape and improved optimization friendlincss. Then we reconsider the vital knowledge distillation optimization and the impacts of teacher selection in binary network optimization. A simple yet effective teacher-selection metric called knowledge complexity, and a novel multi-round optimization technique, Consecutive Knowledge Distillation (CKD), proposod to alleviate the high-confidence knowledge distillation bottleneck for BNNs. The highly accurate results on ImageNet demonstrate that improved optimization friendliness plays important role in building high-accuracy binary neural networks.
12

---

Published in Transactions on Machine Learning Research (12/2023)
References
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deploymentt Advances in Neurol Information Processing Systems, 32, 2019.
Joseph Bethge, Christian Bartz, Haojin Yang, and Christoph Meinel. Meliusnet: Can binary neural networks achieve mobilenet-level accuracy? arXiv preprint or S55:20010055933 2020.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Auil, and Alexander Kolesnikov. Kuowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19925-10944, 2022.
Matthieu Courbariaux, Itay Hubara, Daniel Sondry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural net- works: Training deep neural networks with weights and activations constrained tot or-1. arXiv preprint Stf:::600.02830 2016.
Ekin D Cubuk, Barret Zoph, Jonathou Shlens, and Quoe V Le. Randaugnent: Practical automated data augmneu- tation with reduced search space. In Proceedings of the TEEE/CVF conference on computer vision and pattern recognition workshops, pp- 702 103-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. lece, 2009.
Ruizhou Ding, Ting- Wu Chin, Zeye Liu, and Diana Marculescu. Regularizing activation distribution for training bi- narized deep networks. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition, pp. 11408-11417, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. arXiv preprint ar arXim:210.. / 11929, 2020.
Nianhui Guo, Joseph Bethge, Haojin Yang, Kai Zhong, Xuefei Ning, Christoph Meinel, and Yu Wang. Boolnet: Minimizing the energy consumption of binary neural networks. arviv preprinl arXiv:2106.0690) 2021.
Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang. Demystifying local vision transformer: Sparse connectivity, weight sharing, and dynamic weight. arXie preprint arXiv:2106. 04263, 2021.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffiman coding. 2015. URL http://arxiv. wwwrxxii.rrg//e hwtp:/ogg//////aaeeeee wwtpaa10......000 00149 cite arxiv:1510.001 111100001499oom. 00149memmmt Published conference paper ICLR 2016 (oral).
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on compuler vision and pallern recognition, pp. 770 7700-77,,0 2016.
Koen Helwegen, James Widdicombe, Lukas Ceiger, Zechun Liu, Kwang- Ting Cheng, and Roeland Nusselder. La- tent weights do not exist: Rethinking binarized neural network optimization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alch-BBuc, E. Fox, and R. Garnett (eds.), Advonces Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL htttps://proceedingg. segrrcceedinggs.uuiipps... www.rpps.ccppppe/////a/.a cc/ppaper/201///11ll 6c5891099bbb05557777444446666668 Paper.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. ar Xie preprint 1503 025331, 2(7); 2015.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augmentt your batch: Improving generalization through instance repetition. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8129-8138, 8138,,18881100 2020.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXix preprint arxiv: 10e..004.000 04861, 2017.
Jie Hu, Li Shen, and Gang Sun. Squeezo-and-exxilaaii networks. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 1111171111111 2018.
13

---

Published in Transactions on Machine Learning Research (12/2023)
Gao Huang, Zhuang Liu, Laurens Van Der Maalen, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and paltern recogralion, pp. 4700 4700, 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 4114-4122, 2016.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmettcconly inference. In Proceedings of the IEEE Conference Computer Vision and Pattern Recogriition (CVPR), June 2018.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018
Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Yan Wang, Yongjian Wu, Feiyue Huang, and Chia-Wer Lin. Rotated binary neural network. Advances in neuml information processing systems, 33:7474-7485, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012 -00022, 2021a.
Zechun Liu, Baoyuan WU, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European conference on computer vision (ECCV), pp. 222-737, 2018.
Zechun Liu, Wenhan Luo, Baoyuan Wu, Xin Yang, Wei Liu, and Kwang- Swang-Ting Cheng. Bi-real net: Binarizing deep network towards real-network performance. International Journal of Computer Vision, 128(1):202 219, 2020a.
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michad Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV, volume 12359 of Lecture Noles in Computer Science, pp. 143 159. Springer, 2020b. doi: 0010//9978--005-556668
9. URL https://doi. org/10. wttpoo..10//////.......... 1007/9978-3-000-5555--
Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang-Ting Cheng, How Do Adam and Training Strategies Help BNNs Optimization? In International Conference on Machine Learning, pp. 6936-6946.
PMLR, 2021b. URL http://arxiv. http://aarxivv.og/abb////sa///// wwtp:rg///g/...o../////// 11309
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976-11986, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arxiv:1111.0055 2017.
Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Teimiropoukss. Training binary neural networks with real- to-binary convolutions. In International Conference on Learning Representations, 2020.
Dae Young Park, Moon-Hyun Cha, Daesin Kim, Bohyung Han, et al. Learning student- student-friendly teacher networks for knowledge distillation. Advances in Neural Information Processing Systems, 34:13292 13303, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, pp. 8026-8037, 2019.
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song, Forward and backward information retention for accurate binary neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognation, pp. 2250 2259, 2020.
Tijja Radosavovic, Raj Prateek Kosaraju, Ross Ginshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10428- 10436, 2020.
Mohammad Raslegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xuor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525 542. Springer, 2016.
14

---

Published in Transactions on Machine Learning Research (12/2023)
Tal Ridnik, Hussam Lawen, Emanuel Ben-Baruch, and Asaf Noy. Solving imagenel: unified scheme for training any backbone to lop results. arXiv preprind urXiv:2204.0342 2022.
Zhiqiang Shen and Marios Savvides. Meal v2: Boosting vanilla resnet-50 to 80%+ top-1 accuracy on imagenet without tricks. arXiv preprint orXiv:2009. 0044,, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXin preprint 1409 1556, 2014.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Newral Information Processing Systems, 69190666919, 2021.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Mingxiug Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-61 6114. PMLR, 2019.
Zhijun Tu, Xinghao Chen, Pengju Ren, and Yunhe Wang. Adabin: Improving binary neural networks with adaptive binary sels. Xiv preprind XrXi::2288000065 2022.
Jingdong Wang, Ke Sun, Tianheng Cheng; Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 43(10))3344-3888 -364,, 2020.
Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. In Proceedings of the IEEE/CVP international conference on computer vision, pp. 5198-5208, 2021.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regular- ization strategy train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023 2032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion. Xiviv preprint arXiv:1710.09411 2017.
Yichi Zhang, Zhiru Zhang, and Lukasz Lew. Pokebnn: binary pursuit of lightweight accuracy. ar preprint ar Xix::11 111.00333, 2021.
15

---

Published in Transactions on Machine Learning Research (12/2023)
(a) ResNet-18 18
(b) BinaryNet- 18
(c) BiRealNet- 18
(d) RealBBnaryNet-IB
12
(e) BiRealNet + Info-RCP (ours)
(f) BiRealNet-18 + ELM-Attention (ours)
(x) BNext-18 (ours)
Figure 7: The loss landscape of popular BNNs (3D view) on the CIFAR10 dataset. We follow the setting of (Li et al. 2018). "Info-RCP" and "ELM-Attention" are the module designs in this paper. Compared with previous design, the BNext-18 shows smoother loss landscape. (The corresponding contour lines are shown in Figure in the main work.)
7 Appendix
In this section, present more detailed visualization results and an ablation study, which are not listed in the main paper due to limited space.
7.1 3D Loss Landscape Visualization
We use the loss landscape visualization technique on existing binary architecture designs such as BinaryNet (Courbariaux et al.) 2016), BiRealNet (Liu et al. 2018), Real2BinaryNee (Martinez al.) 2020), and their full-precision counterpart ResNet-18 (He et al. 2016). We plotted the corresponding 3D loss landscapes for each network, shown in Fig. 7 It also shows the 3D view of the designed modules IIffo-CP and ELM- Attention) individually and our full BNext design (using both modules). It is casy to see that binarization makes the loss landscape surface really rugged but our design can alleviate this problem of binarization. As a result, the loss landscape surface BNext is already close to the previous full precision design. This explains why BNext can reach higher accuracy boundary.
7.2 Counter-Intuitive Overfitting
As explained in the main work, knowledge distillation has long been an essential choice for optimizing binary neural networks. When we move our oplimization larget from 70%-level to 80%-level binary neural network, it inevitable to seek help from higher accuracy pretrained deep neural networks. More specifically, we nced to decide which model most suitable as the teacher for the optimization of highly-accuracy BNNs, such as, BNext. We choose the teacher empirically from few popular deep neural network families, such as, ResNet, EffeientNee,, and Next. During this process, observe that the BNext design easily overfit
16

---

Published in Transactions on Machine Learning Research (12/2023)
4.0
Tran
3.5
Accura-sy Gurve on Inragalie (86)
60
2,0
50
2.5
Loss
2.0
1.5
Top-1
200
100
200
250
Epochs
Epochs
(a) BNext-L Loss Curve
(b) BNext-L Accuracy Curve
Figure 8: The optimization procedure of BNext-L model ImageNet. dataset with standard knowledge distillation technique and ConvNext-T as teacher. During the early stage of the optimization, the student always has lower training loss than testing loss. This indicates a better fitting to training set than testing set.
to strong and highly accurate teachers. In Table. model as the teacher leads to a high training accuracy of the student model, but does not generalize well and even performs worse on validation data. For example, when ConvNext-Base used as the teacher for BNext-Tiny, the student training accuracy is 3% higher than using EfficientNNet-B0 as the teacher, but the testing accuracy is 1.58% lower.
that when a highly accurate full precision
For larger students such BNext-M/L, this overfitting phenomenon still exists even with increased model capacity. Take our largest model BNext-L. with a strong teacher Conv.Next-T results in a student with sub-optimal generalization. As can see from the optimization procedure in Fig. [ the testing loss is always higher than the training loss during the early stage of training procedure. Meanwhile, the performance on the test set is less stable compared to the training curve. When reaching the same testing accuracy of 78%, the training accuracy under standard KD is already 80.23 80.23%, which is almost 4% higher than the final version with the proposed diversified consecutive KD. After 256 epochs training, the evaluation result on the test set 1.67% lower than the diversified consccutive KD results in Table. The overfitting shown in Fig. verifies the need of redesigning the optimization pipeline for BNext.
example: simply combing standard knowledge distillation
7.3 Detailed Teacher Selection for BNext Family Optimization
To solve the counter intuitive overfitting problems for BNext family optimization, we designed the diversified consecutive KD in Sec. 4 For each BNext design, we build the corresponding knowledge matrix based on a comprehensive consideration of student capacity, teacher knowledge complexity and teacher evaluation performance. The detailed teacher selections are shown Table, IO For BNext-18, we only use the standard KD setting for a fair comparison, which means that only ResNet-34 utilized during the training procedure. For BNext-T/S, we apply a gap-aware ensemble strategy for knowledge distillation (Sec. to increase the diversity of the teacher knowledge. Since the capacity of BNext-T/S performance is relatively limited, the EfficientNet-BO/ models are already sufficiently strong teachers. Consequently, no knowledge boosting strategy is utilized for them. For BNcxt-M/I. we rely on higher accuracy teachers like ConvNext-TT to explore the accuracy boundary of our BNext design. As we have observed, simply learning from this kind of high accuracy teacher from scratch under standard KD can suffer from overfitting problem. Consequently, we use the showledge-boosting stratogy (proposod in Sec. to evolve the supervision information during the training. In this way, the knowledge discrepancy between student and teacher can be decreased by slowly increasing the knowledge confidence during the training.
17

---

Published in Transactions on Machine Learning Research (12/2023)
Table 9: The counter-intuitive overfitting problem observed in the BNext model optimization under different teacher selections. The BNext-Tiny used for evaluation. Each model optimized for 128 epochs using the standard data augmentation and standard knowledge distillation technique. "KC' indicates the knowledge- complexity score, Func. 4
Teachers
KC
Teacher Test Top-1 (%)
Student Train Top-1 (%)
Student Test Top-1 (%)
None
0
76.61
65.02
ResNet-34
1.487
73.31
72.82
71.12
ResNet-50
1.539
76.13
71.59
70.90
RosNct-101
1.768
77.37
72.38
70.76
Elficien(Net-B00
0.893
77.53
72.16
71.32
EfficientNet-B2
1.103
78.06
71.91
71.25
EfficientNee-B4
1.405
79.26
73.88
70.63
ConvNext-T
1.552
82.52
73.63
70.63
ConvNext-S
1.786
83.66
74.29
70.43
ConvNext-BB
2.027
84.06
75.22
69.74
ResNext101-32x8)
2.053
79.32
75.42
69.67
Table 10: The detailed teacher settings for our BNext model optimization.
Students
Strong Teacher
Assistant Teacher Groups
BNext-18 18
ResNet-34
None
BNext-T
Efficient-B2 B2
Efficient-B0
BNext-S
Efficient-Be B2
Efficient-B0
BNext-M
Efficient-
Efficient-BO, Efficient-B2
BNext-L
ConvNext-T
Efficient-BO, Efficient- B4, ConvNext-T
7.4 BNext Model Training Procedure
The detailed training procedure of the BNext family (T/S/M/L) are shown in Fig. 10 Since each BNext design holds different teacher matrix and variant teacher scheduler, we can observe different. patterns in the training loss curve and training accuracy curve of each model. All in all, we can see that each BNext model always has a lower testing loss than training loss during the optimization. If we further compare the testing curves between Fig. [0] and Fig. Sal we can conclude that the proposed diversified consecutive KD helps BNext optimization generalize better on testing set.
7.5 BNext Post-Quantization Details
We try to maximize the efficiency of BNext family by further quantizing the input layer, the last layer and the Squeeze-and-Expant branch in each Info-RCP module. Specifically, we utilize post-training quantization for each BNext model. For the optimization settings, we use the AdamW optimizer (31=0.99, 32=0.999) to fino-tune the corresponding layers but keep the pretrained binary layers fixed. The initial learning rate 1c-7 and each model is fne-tuned for epochs on the ImageNet dataset. The data augmentation is kept line with the pre-training phase. No weight decay used during this period. We use asymmetric quantization,
18

---

Published in Transactions on Machine Learning Research (12/2023)
Table 11: runs evaluation of BNext-18 on the CIFARIO and CIFAR100 datasets.
Train Top-1 (%)
Train Top-ō (%)
Test
Test
Datasets
Top-1 (%)
Top-5 (%6
CIFARIO-O
99.99
100
93.67
99.86
CIFAR10-1
99.99
100
93.39
99.79
CIFAR10-2
99.99
100
93.59
99.83
CIFAR1O-3
99.99
93.69
99.84
CIFARIO-4
99.99
100
93.66
99.81
Avg
99.99
100
93.60
98.83
Var
0.00
0.00
0.0152
0.0007
Std
0.00
0.00
0,1233
0.027
CIFAR100-0
99.97
100
72.22
91.23
CIFAR100-1
99.98
100
72.02
91.75
CIFAR100-2
99.97
100
72.35
91.61
CIFAR100-8
99.97
100
71.97
91.51
CIFAR100-4
99.97
72.37
91.64
Avg
99.97
100
72.18
91.55
Var
2e-5
0.00
0.032
0.0379
Std
0.0045
0.00
0.1845
0.1945
which can be mathematically formulated as follows:
X round((X; - .))
+ B,
(7)
3 = min,
(8)
maxx; minxx
round(WW
3) + 3)
W
Norm(W:)),
(9)
a
2
B
min
(10)
max Seatedd min Scaled
WiiSenied
where X represents the input features and W represents weights respectively. The and g means full precision and quantization representation. The Norm indicates the absolute-mean for each output channel. The Scaled means that the variable is scaled down by its channel-wise absolute-mean.
7.6 Detailed CIFAR Evaluation Results
As have mentioned in the main pages, train BNext-18 model the CIFAR dataset (including CIFAR10 and CIFAR100) to evaluate the generalization of BNext design. Due to limited space, we only show the averaged results in the main page. Here, we present the detailed results of 5 runs on CIFAR, as shown in Table. II
7.7 Quantization Impacts.
We evaluate the impact of post-training quantization on the input convolution layer, output fully-connected layer, and layers in each SE branch, as shown in Table [2] Taking BNext-L as an example, We find that applying 8-bit quantization to weights and activations 80.47%. Further, decreasing the bit-width of weights in the SE branch 4 bits (with 8-bit activations) results in final BNext-L model with 80.37% Top-1 accuracy.
all layers only marginally reduces accuracy
19

---

Published in Transactions on Machine Learning Research (12/2023)
Table 12: Impact post-training quantization on the first input layer, the last layer, and the SE branch BNext Design. The best and second-best quantization results are highlighted.
I-S-E-O (W/A)
BNext-18
BNexi-T
BNext-S
BNext-M
BNext-L
32/32
68.37
72.36
76.06
78.27
80.57
8/8-8/8-8/8-888
68.38
72.36
76.05
78.11
80.47
8/8-4/8-4/888/8
67.94
72.04
75.75
77.97
80.37
8/84//84//8/8
67.91
71.95
75.61
77.93
80.26
8/8-1/44//-8/8
67.29
71.72
75.50
77.79
79.52
7.8 Code Submission
Within this ZIP file provide you with our training code, you are able to reproduce results if desired. We are planning to release the code and pre-trained models via Github together with the final version of this paper. (Licensing is planned with the "Apache License 2.0")) Until then, please treat this code confidentialy.
We added all the details needed to reproduce each of our BNext models depicted in Section of our paper in the respective folders:
BNext-T: atc//ssript/BBletttinny
BNext-S: stc/scriit//BNett-Sna
BNext-M: arc//ccritt/BNeettMHHddd
BNext-L: src/scrritt/BNext-Lagg
The complete code used for each run can be found in the subfolder src and the exact running command is saved in src/script. We also added the output logs (logs/training.. 1lg)..
We use virtualized environment for PyTorch (Paszke et al. 2019 based on Ananconda for our code sctup. The hosts system thus needs support for Python 3.9.13 and a recent NVIDIA CUDA driver (we tested driver version 470.82.01 with CUDA 11.4 before this submission) for training with GPU.
Note that the ImageNet dataset also needs be downloaded and prepared manually in the usual manner (using Irain and vol folder for the respective split). The validation images need to be moved into labeled subfolders.
7.9 Limitations
While the BNext design demonstrates a superior binary computation ratio compared most preceding Binary Neural Network (BNN) architectures (Table. 3) it hosts several 4-bit linear layers within each Infor- RCP module, thereby introducing additional complexity. A promising avenue for addressing this issue is refining the design of the Infor-RCP module and bolstering the optimization of the binary components, which will be explored as part of our future work. Although the employed consecutive knowledge distillation technique has provided valuable guidance for binary model optimization, the substantial GPU memory demands necessitated by maintaining multiple teacher models hampers its user-friendliness An effective remedy could encompass evolving the existing mix- precisocnn teacher models towards lower bit representations such as INT8 or even 4-bit configurations. This transition is expected to markedly diminish the computational load during the training phase and subsequently alleviate the knowledge complexity for student models. The anticipated speed-up, as posited thcoretically, awaits validation on actual hardware. Given the absence of a proficient GPU implementation for any BNN at present, efforts have been initiated to develop a GPU- accelerated version of BNext. However, as of this paper's submission, the outcomes of these endeavors remain forthcoming.
The presented work, while substantial in its contributions towards accuracy enhancement and theoretical acceleration aligned with preceding efforts, holds potential for further advancements. We optimistic
20

---

Published in Transactions on Machine Learning Research (12/2023)
about furnishing an open-source GPU implementation alongside inference speed verification results in the subsequent phases of this research, thereby propelling the practical applicability and performance verification of the BNext design.
21

---

Published in Transactions on Machine Learning Research (12/2023)
8500
950
550
Figure 9: The binary feature map diversity visualization for BNext-T. We select the binary inputs of the first binary convolution stage for visualization. The input images are resized to 1024x1024 for better comparison.
22

---

Published in Transactions on Machine Learning Research (12/2023)
0.7
3rperraare
43-
2. 64
0.2
2.6
Text Sip1 ACE Tran 1ppd
565
582
(a) BN Loss Curve
(b) BNext-T Accuracy Curve
BNext-T Temperature Curve
140% CAN mntreeeee
1.2
30
12
Tran kp3 Act
Tell 2p-
300
200
303
430
300
100
200
300
520
302
200
330
500
(d) BNext=S Loss Curve
(e) BNext-S Accuracy Curve
(1) BNext-S Temperature Curve
4.
imperdars
4.5
0.4
22
1.5
Toan Rpd Acc
330
632
500
100
200
300
400
582
300
200
(g) BNext-M Loss Curve
(h) BNext-M Accuracy Curve
(i) BNext-M Temperature Curve
4.7
Seepentor
30
0.2
1.2
38
230
2.50
4.3
166
150
205
300
400
583
202
200
(j) BNext-L Loss Curve
(k) BNext-L Accuracy Curve
BNext-L Temperature Curve
Figure 10: The training procedure of BNext family on ImageNet dataset. The temperature curve indicates the temperature schoduler for the progressive weights binarization.
23


Reviews:
[{"review_id": "2BYv0udufo", "reviewer": "Reviewer_P96B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strength]\n- The paper clearly articulates the observations and rationale behind the BNext architecture. The examination of design factors in BNNs that influence the loss landscape's smoothness is particularly enlightening. Figure 2 provides a helpful visual representation of the loss landscape, underscoring the potential advantages of BNext.\n- The research offers comprehensive comparisons with numerous existing BNNs.\n- Figure 6 distinctly demonstrates that BNext-T has the capability to produce superior binary features in contrast to ReActNet.\nGiven the same OPs or number of parameters, BNext models consistently deliver enhanced top-1 accuracy.\n\n[Weakness]\n- The reliance on CIFAR-10 and ImageNet datasets may seem a bit dated.\n- The methodology primarily focuses on supervised learning. It prompts curiosity on whether the proposed concepts could be applied to visual Transformers, which are prevalent in contemporary applications.\n- The utility of knowledge distillation seems to be more suitable for compact models operating under supervised learning guidelines.\n- The authors should consider providing a more in-depth discussion on the method's limitations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel binary neural networks (BNNs) architecture termed BNext, accompanied by training methodologies including consecutive knowledge distillation (CKD). A comparison of loss landscapes across different BNN architectures is presented, with emphasis on the assertion that Info-RCP results in a smoother loss landscape. The paper includes thorough ablation studies, and the findings suggest that BNext achieves a commendable top-1 accuracy on the ImageNet dataset. A comparative analysis between the binary features of BNext-T and ReActNet is conducted, indicating that BNext-T possesses more varied binary features devoid of checkerboard artifacts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see weakness above.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Wuvg1k9xDY", "reviewer": "Reviewer_3WQd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n- The finding that teacher selection is an important step for knowledge distillation in BNNs and that too large knowledge gaps between student and teacher hinder the optimization of BNNs.\n\n- The paper reaches SOTA and moves the Pareto-front for accuracy - model size trade-off.\n\n- The paper addresses one of the main bottlenecks in BNNs: the rough loss landscape and the inherent difficulty of optimization.\n\n- The addition of images of the first binary featuremaps is an interesting aspect. It helps corroborating the intuition that the network is indeed powerful enough to distinguish shadows and different shades of a particular color.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a new approacht to BNN design, by including an element-wise multiplication after the first binary convolution block in each basic block (processing unit) and by feeding both the input and output of a Sign-Bconv-Bn-Prelu to the squeeze-and-expand unit. With that, the authors present a new (and larger) BNN network designs that allows for better optimization due to a smoother loss landscape. Next to model design, the authors find that teacher selection is crucial in applying knowledge distillation for training BNNs, and argue that progressively using stronger teachers during training aids the optimization process. Overall, some interesting finding are reported in the paper.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "[R1] The writing quality of the paper is a MAJOR concern, many typo's. Few examples:\n- Page 2: \"Infor-RCP\" instead of \"Info-RCP\". \"we show\" with no capital.\n- Page 6: \"Fig. ?? and ??\" are not referenced, space missing between \"designLiu et al.\", paper inconsistently uses \"Sec. 6\" vs \"Section x\"\n- Page 7: \"Accuray\", \"Specifically, We\" -> no capital\n- Table 3 is nowhere referenced in text, hence I have difficulties understanding what its purpose is.\n\n[R2] Providing some sort of proof on why certain elements work is difficult in this context (understandably!), but I miss an intuitive explanation on why and how the authors think Info-RCP and ELM-attention help in smoothing the loss landscape. Specifically, it shows that the combination of the two items is key in smoothing the landscape. Both individually seem to marginally improve the smoothness of the landscape. To me, this is the main point of improvement - more experiments to elaborate on this point are required.\n\n[R3] I understand from Table 6 that Consecutive KD virtually has no effect (+0.05), while there is extra training complexity associated with using more teachers. The text, however, does not reflect on the marginal impact of this improvement. Perhaps that the Consecutive KD has more impact for larger architectures (?), even that is unclear. Further elaboration on this matter are required. \n\n[R4] I suggest adding a table to the ablation study which shows the added complexity of the different parts of the E2E architecture, with the associated accuracy gains.\n\n[R5] The ablation study only discusses using or not using the ELM-attention. It would be interesting to investigate if it makes a difference WHERE the attention module is used. Right now only the first convolution unit in the basic blocks has the attention, would it benefit to apply it to the other units as well?\n\n[R6] The paper currently does not report on-device latency as also mentioned in section 7.8. Implementing the architecture in a BNN framework like https://proceedings.mlsys.org/paper_files/paper/2021/hash/a0a81eed87dd44d6504fed5f81f6de5a-Abstract.html could help materialize this.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7pf49xLSu5", "reviewer": "Reviewer_FZQj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \nAchieved improved model performance. Analysis with loss landscape provide a window into understanding BNN optimization. \nWeakness:\nPresentation should be improve, please see below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The author proposed a novel optimization-friendly architecture based on their understanding of the loss landscape of models. They also proposed a metric for picking the proper teacher model for training student binary models. A SOTA model accuracy is achieved with their approach.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Suggest to include a short summary of how the loss landscape is visualized so readers can avoid looking it up as it is a core part of the study .\n2. P6. Please fix (Fig.?? and ??).\n3. P6. Please fix  designLiu\n4. The architecture of BNext is not clearly illustrated in Figure 4. Not sure about the Basic Block part of Figure 4, if the multiplication operation of the first segment is performed twice.\n5. Please clearly define OPs, BOPs, QOPs, and the meaning of #Param (MB)\n6. Table 4. without the context of network architectures of the models in comparison, it is difficult to tell if the comparison is fair. \n7. Table 7: it is not clear what the baseline is.\n8. Model distillation: it is difficult to tell if the result is due to a specific choice of the distillation hyper-parameters.\n9. It is difficult to tell if the proposed improvement is due to model architecture, augmentation strategy, distillation, or more extended training.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There were active discussions between the authors and Reviewer 3WQd. During the response period, the authors addressed all the reviewers' concerns and suggestions.

The revised paper looks okay to me, but I think this paper still should fit the standard of the official formatting instructions.

Now, all the citations seem to be declared by `\cite{}`. It should be `\citep{}` for better readability. For example, not "ResNet-18 He et al. (2016)", but " ResNet-18 (He et al. 2016)". Please check "4. Citations, figures, tables, references > 4.1 Citations within the text" of the official formatting instructions. 

> When the authors or the publication are included in the sentence, the citation should not be in parenthesis, using \citet{} (as in “See Hinton et al. (2006) for more information.”). Otherwise, the citation should be in parenthesis using \citep{} (as in “Deep learning shows promise to make progress towards AI (Bengio & LeCun, 2007).”).

My recommendation is "Accept", but as this paper needs to be updated to reach the official formatting instructions, I recommend minor revision. I will check whether the final paper fits the standard.
Decision: Accept (poster)