BiTAT: Neural Network Binarization with Task-Dependent Aggregated Transformation
Authors: Geon Park
Jaehong Yoon
Haiyang Zhang
Xing Zhang
Sung Ju Hwang
Yonina C. Eldar
Venue: Submitted to ICLR 2023 (2023)

Abstract: Neural network quantization aims to transform high-precision weights and activations of a given neural network into low-precision weights/activations for reduced memory usage and computation, while preserving the performance of the original model. However, extreme quantization (1-bit weight/1-bit activations) of compactly-designed backbone architectures (e.g., MobileNets) often used for edge-device deployments results in severe performance degeneration. This paper proposes a novel Quantization-Aware Training (QAT) method that can effectively alleviate performance degeneration even with extreme quantization by focusing on the inter-weight dependencies, between the weights within each layer and across consecutive layers. To minimize the quantization impact of each weight on others, we perform an orthonormal transformation of the weights at each layer by training an input-dependent correlation matrix and importance vector, such that each weight is disentangled from the others. Then, we quantize the weights based on their importance to minimize the loss of the information from the original weights/activations. We further perform progressive layer-wise quantization from the bottom layer to the top, so that quantization at each layer reflects the quantized distributions of weights and activations at previous layers. We validate the effectiveness of our method on various benchmark datasets against strong neural quantization baselines, demonstrating that it alleviates the performance degeneration on ImageNet and successfully preserves the full-precision model performance on CIFAR-100 with compact backbone networks.

Under review as a conference paper at ICLR 2023
BITAT: NEURAL NETWORK BINARIZATION WITH
AGGREGATED TRANSFORMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Neural network quantization aims to transform high-precision weights and activa- tions of a given neural network into low- low-precision weights/activations for reduced memory usage and computation while preserving the performance of the original model. However, 1-bit weight/1 -bbt activations of compactly-designe architectures often used for edge-device deployments result in severe performance degeneration. This paper proposes novel Quantization Aware Training method that can effectively alleviate performance degeneration even with extreme quanti- zation by focusing on the inter- -nter- dependencies, between the weights within each layer and across consecutive layers. To minimize the quantization impact of each weight on others, we perform orthonormal transformation of the weights at each layer by training an input-dependent correlation matrix and importance vector, such that each weight is disentangled from the others. Then, we quantize the weights based on their importance to minimize the loss of the information from the original weights/activatiinn. We further perform progressive layer-wise quantization from the bottom layer to the top, so that quantization at each layer reflects the quantized distributions of weights and activations previous layers. We validate the effectiveness of our method on various benchmark datasets against strong neural quantization baselines, demonstrating that it alleviates the perfor- mance degeneration on ImageNet and successfully preserves the full- full-precision model performance on CIFAR-100 with compact backbone networks.
backbone
INTRODUCTION
Over the past decade, deep Neural Networks (NN) have achieved tremendous success in solving various real-world problems (Creswell et al., 2018; Gidaris al., 2018; Chen al., 2020; Karras et al., 2021; Radford et al., 2021). Recently, network architectures are becoming increasingly larger based on the empirical observations of their improved performance. However, is increasingly difficult to deploy them on edge devices with limited memory and computational power. Therefore, many recent works focus on building resource-efficien networks to bridge the gap between their scale and actual permissible computational complexity/memory bounds for on-device model deployments. Several works consider designing computation- and axemory-effccieee architecture modules, while others focus compressing given neural network by either pruning its weights (Yoon & Hwang, 2017; He et al., 2020b; Lin et al., 2020a) or reducing the bits used to represent the weights and activations (Bulat al., 2021; Dbouk et al., 2020; Li et al., 2021). The latter approach, neural network quantization, is beneficial for building on-device AI systems since the edge devices oftentimes only support low bitwidth- prreiiiin parameters and/or operations. However, it inevitably suffers from the non-negligible forgetting of the encoded information from the full-precision models. Such loss of information becomes worse with extreme quantization into binary neural networks with 1-bit weights and 1-bit activations (Bulat et al., 2021; Zhuang et al., 2019; Qin et al., 2020b).
How can we then effectively preserve the original model performance even with extremely low- precision networks? To address this question, we focus on the somewhat overlooked properties of NN for quantization: the weights in a layer are highly correlated with each other and weights in consecutive layers. Quantizing the weights will inevitably affect the weights within the same layer since they together comprise transformation represented by the layer. Thus, quantizing the weights and activations at specific layer will adjust the correlation and relative importance between them. Moreover, it will also largely impact the next layer that directly uses the output of the layer, which together comprise function represented by the neural network.

---

Under review as a conference paper at ICLR 2023
weight olement
Intre-loye dependency loyer dependency
1ts
Licl (3E21 Disak m.tl 1300 Lauelal (313)
Quantoeed Training forw at obtoined We
w(++1)
nade
fneak
(
65.775
1835
05.005
Finetune W
Orthonormal
Post-traäning Quantization
Weipht
"Quntization aware Training
Transormation
Figure I: Left: An Illustration of our proposed method. Weight clements in layer is highly correlated to each other along with the weights in other layers. Our BiTAT sequentially obtains quantized weights of each layer based on the importance of disentangled weights to others using trainable orthonormal rotation matrix and importance vector. Right: Categorization relevant strong quantization methods to ours.
Despite their impact on NN quantization, such inter- dependencies have been relatively overlooked. As shown in Figure I Right, although BRECQ (Li et al., 2021) addresses the problem by considering the dependency between filters in each block, it is limited to the Post- Bost-Trainnng Quantization (PTQ) problem, which suffers from inevitable information loss, resulting in inferior performance. Most recent Quantization-Aware Training (QAT) methods (Dbouk al., 2020; Liu et al., 2020) are concerned with obtaining quantized weights by minimizing quantization losses with parameterized activation functions, disregarding cross-layer weight dependencies. To the best of our knowledge, no prior work explicitly considers dependencies among the weights for QAT.
To tackle this challenging problem, we propose new QAT method, referred to Neural Network Binarization with Task-dependent Aggregated Transformation (BiTAT), as illustrated in Figure Left. Our method sequentially quantizes the weights each layer of pre-trained model based chunk- wise depu-ddependett weight importance by training orthonormal dependency matrices and scaling vectors. After quantizing each layer, we fine-tune the subsequent full-precision layers, which utilize the quantized layer as an input for few epochs while keeping the quantized weights frozen. we aggregate redundant input dimensions for transformation matrices and scaling vectors, significantly reducing the computational cost of the quantization process. Such consideration of inter- weight dependencies allows our BiTAT algorithm to better preserve the information from a given high- precision network, allowing it to achieve comparable performance to the original full-precision network even with extreme quantization, such as binarization of both weights and activations. The main contributions of the paper can be summarized follows:
We demonstrate that weight dependencies within each layer and across layers play an essential role in preserving the model performance during quantized training.
We propose an input-dependent quantization- -aware training method that binarizes neural networks. We disentangle the correlation in the weights from across multiple layers by training rotation matrices and importance vectors, which guides the quantization process to consider the disentangled weights' importance.
We empirically validate our method on several benchmark datasets against state-of-the -trt NN quantization methods, showing that it significantly outperforms baselines with the compact neural network architecture.
2 RELATED WORK
Minimizing the quantization error. Quantization methods for deep neural networks can be broadly categorized into several strategies (Qin et al., 2020a). We first introduce the methods that aim to minimize the weight/activation discrepancy between quantized models and their high-precision counterparts. XNOR-Ne (Rastegari al., 2016) aims to minimize the least-squares error between quantized and full-precision weights for each output channel in layers. DBQ (Dbouk et al., 2020) and QIL (Jung et al., 2019) perform layerwise quantization with parametric scale or transformation functions optimized to the task. Yet, they quantize full-precision weight elements regardless of the correlation between other weights. While TSQ (Wang et al., 2018) and Real-to-Bi to-Bin (Martinez et al., 2020) propose to minimize the l distance between the quantized activations and the real- valued network's 's activations by leveraging intra-layer weight dependency, they do not consider cross-layer dependencies. PNoxyBNN (He et al., 2020a) adopts the orthogonal matrix to preserve the
2

---

Under review as a conference paper at ICLR 2023
correlation between coordinates while minimizing the quantization error. Recently, BRECQ (Li et al., 2021) and the work in a similar vein on post-training quantization (Nagel et al., 2020) consider the interdependencies between the weights and the activations by using a Taylor series-based approach. However, calculating the Hessian matrix for a large neural network is often intractable, and thus they resort to strong assumptions such as small block-diagonality of the Hessian matrix to make them feasible. BiTAT solves this problem by training the dependency matrices alongside the quantized weights while grouping similar weights together to reduce the computational cost.
Modifying the task loss function. BNN-DL (Ding et al., 2019) adds a distributional loss en- Apprentice (Mishra & Marr, 2018)
forcing the weight distributions to be quantization-friende uses knowledge distillation to preserve the knowledge of the full-precision teacher network in the quantized network. However, such methods only put a constraint on the distributional properties of the weights, not the dependencies and the values of the weight elements. CI-BCNN (Wang et al., 2019) parameterizes bitcount operations by exploring the interaction between output channels using reinforcement learning and quantizes the foottnng-poin accumulation in convolution operations based on them. However, reinforcement learning is expensive, and it still does not consider cross-layer dependencies. RBNN (Lin et al., 2020b) achieves a significantly higher cosine similarity between the full-precision weight and its binarization by constraining the model to preserve fewer angular biases.
Reducing the gradient error. Liu et al. (2018) devises a better gradient estimator for the sign func- tion used to binarize the activations and a magnitude-
gradient correction method. PCNN (Gu et al., 2019) proposes new discrete backpropagation method via projection, where the layerwise trainable function effectively projects the weights at each layer to multiple quantized weights. Re- ActNet (Liu et al., 2020) achieves oftte-of-thearr performance for binary neural networks by training a generalized activation function for compact network architecture used in Liu et al. (2018). However, their quantizer functions conduct element- unstructured compression without considering the change in other correlated weights throughout quantization training. This makes the search process converge to suboptimal solutions since task loss the only guide for finding the optimal quantized weights, which is often insufficient for high-dimensional and complex architectures. On the other hand, we can obtain petter-informe guide that compels the training procedure to spend more time searching in areas that are more likely to contain high high-peeffrming quantized weights.
3 WEIGHT IMPORTANCE FOR QUANTIZATION-AWARE TRAINING
We aim to quantize a full- precisseoi neural network into a binary neural network (BNN), where the obtained quantized network is composed of binarized 1-bit weights and activations, which preserves the performance of the original full-precision model. Let ((; W) be a L-layered neural network parameterized by a set of pre-trained weights W { un (1)
w(()), where (n()) E Rhh-ixd is the weight at layer / and do is the dimensionality of the input. Given a training dataset X' and corresponding labels V, existing QAT methods Rastegari et al. (2016); Dbouk et al. (2020); Jung et al. (2019); Bethge al. (2020); Yamamoto (2021); Park & Yoo (2020) search for optimal quantized weights by solving for the optimization problem that can be generally described as follows:
minimize Lrast (f (X; Q (W; o)) ,
(I)
W.
where Ltask is a standard task loss function, such as cross-entropy loss, and Q(:: Φ) is the weight quantization function parameterized by & which transforms a real-valued vector discrete, binary vector. Existing works quantize typically by rounding each element of the weights or activation to the nearest quantization value. This is equivalent to minimizing loss terms based on the Mean Squared Error (MSE) between the full-precision weights and the quantized weights at each layer:
arg min Q(w): == abb,, where a", b" -ER.bE(-1113
[ - -abl2
(2)
where m is the dimensionality of the target weight. For inference, w, = Q(w) is used. Us- ing this quantizer, QAT methods iteratively search for the quantized weights based on the task loss using stochastic gradient descent-based methods, and the model parameters converge into the ball-like region around the full-precision weights w. However, the region around the op- timal full-precision weights may contain suboptimal solutions with high errors. We demon- strate such inefficiency of the existing quantizer formulation through simple experiment in Figure 2. Suppose we have three input points, 1; 22, and az and full-precision weights w.
3

---

Under review as a conference paper at ICLR 2023
Quantized training of the weight using Equation 2 successfully reduces MSE between the quantized weight and the full full-precision, but the task prediction loss using Wy is nonetheless very high.
Weight Quantization
Corig (W)
00.41
-0.41
W4
N2
52
[+1.M -0An +000
0.48
Weight
1041
Prediction MSE
0.02
The main source of error comes from the inddpen- dent application of the quantization process to each weight element: Neural network weights are not in- dependent, but highly correlated, so holding the loss value constant, quantizing (perturbing) one weight will affect the others. Moreover, after quantization, the weight importances can also change significantly.
+0.74
Weigh MSE 0.32
0.25
W
+0.25
W
Qours (w:rs(w:s,V)
+0.25
Prediction MSE = 0.02
Figure 2: A simple experiment that cross- layer weight correlation is critical find well- performing quantized weights during QAT.
Both factors lead to high errors in the pre-activations. On the other hand, our proposed QAT method BiTAT, described in Section 4, achieves a quantized model with smaller error. This results from the consideration of the inter-weight dependencies, which we describe in the next subsection.
3.1 DISENTANGLING WFIGHT DEPENDENCIES VIA INPUT-DEPENDENT ORTHORNORMAL TRANSFORMATION
How can we, then, find the low-precision subspace, which contains the best-performing quantized weights on the task, by exploiting
1FAA.100 accuurccyyy
0.6
the inter- weight dependencies? The properties of the input distri- bution give us some insights into this question. Let us consider a task composed of N centered training samples (.),,..;; NNN .N. X' E R.A xdo We can obtain principal components of the train- ing samples U1...., Vd. E Idd and the corresponding coefficients Ad. 2 0, in descending order. When we optimize a single- layered neural network parameterized by (1) neurons correspond- ing to the columns of (1()) are oriented in a similar direction to the principal components with higher variances (i.e., v. than vy. where i < <) that is much more likely to get activated than the others. We apply change of basis to the column space of the weight matrix w with the bases (v (U....,, Vdo):
0.4
0.2
Top ruMs
Botiom 5 ronM
0.02
0.04
0.1 0.2 0.40.6
Whight nonn scaln
Figure 3: Solid lines: Test accu- racy of MobileNetV2 model on CIFAR- 100 dataset, after adding Gaussian noise to the top 5 rows and the bottom 5 rows of ( for all layers, considering the dependency the lower layers. Dashed lines: Not considering (3) the dependency on the lower lay- ers. The x axis is in log scale.
(0) (1) (1)
W (1) = (0) (1)
(4)
where v0 [1] ldll E dd is an orthonormal matrix. The top rows of the transformed weight matrix (1) will contain more important weights, whereas the bottom rows will contain less important ones. Therefore, the accuracy of the model will be more affected by the perturbations of the weights at top rows than ones at the bottom rows. Note that this transformation can also be applicd to the convolutional layer by "unfolding" the input image or feature map into a set of patches, which enables to convert the convolutional weights into matrix (The detailed descriptions of the orthonormal transformations for convolutional layers is provided in the supplementary file).
We can also easily generalize the method to multi- layer neural networks, by taking the inputs for the 1-th layer as the "training set",, assuming that all of the previous layer's 's weights are fixed, as follows:
{ (0(0 = (0) x. a((-")]
(5)
where 8(-) is the nonlinear transformation defined by both the non-linear activations and any layers other than linear transformation with the weights, such as pooling Batch Normalization. Then, we obtain the change-of-basis matrix (l)) for layer by using PCA on x; (!-1) The impact of transforming the weights is shown in Figure 3. We compute the principal components of each layer in the initial pre-trained model and measure the test accuracy after adding the noise to the top-5 highest-variance (dashed red) or lowest-variance components (dashed blue) per layer. While a model with perturbed high- variance components degrades the performance as the noise scale increases, a model with perturbed vowvvariance components consistently obtains high performance even with large perturbations. This shows that preserving the important weight components that respond to high-variance input components is critically important for effective neural network quantization.
4

---

Under review as a conference paper at ICLR 2023
3.2 CROSS-LAYER WEIGHT CORRELATION IMPACTS MODEL PERFORMANCE
So far, we have only described dependencies among weights within single layer. However, dependencies between the weights across different layers also significantly impact the performance as well. To validate that, we perform layerwise sequential training from the bottom layer to the top. At the training of each layer, the model computes the principal components of the target layer and adds noise to its top-5 high/low components. As shown in Figure 3, progressive training with the low-variance components (solid blue) achieves significantly improved accuracy over the end-to-end training counterpart (dashed blue) with high noise scale, which demonstrates the beneficial effect of modeling weight dependencies in earlier layers. We describe further details in the supplementary file.
4 TASK-DEPENDENT WEIGHT TRANSFORMATION FOR NN BINARIZATION
Our objective is to obtain binarized weights wy given pre-trained full-pree full-precision weights. We effec- tively mitigate performance degeneration from the binarization process by focusing on the inter-weight dependencies within each layer and across consecutive layers. Given a single-layered neural network parameterized by w(1), We first reformulate the quantization function Q in Equation with the weight correlation matrix (0) and the importance vector g so that each weight is disentangled from the others while allowing larger quantization errors on the unimportant disentangled weights (Unless otherwise stated, we omit the superscript denoting layer index):
Q(w: s, V) = arg min diag(s) (VT w- V (wTw))l 1|r + y |1 w,EQ
(6)
where V E Rdoxdo, and & E Redo is a scaling term that assigns different importance scores to each row of VT ww. We denote Q fa @ b a E RRd b (-1,1) 1)dxxdd the set of possible binarized values for w E dodo with a scalar scaling factor for each output channel, where @ is an element-wise multiplication operator, with dimensions broadcasted appropriately. We additionally include f norm adjusted by a hyperparameter y. At the same time, we want our quantized model to minimize the empirical task loss (e-g., cross-entropy loss) for given dataset. Thus we formulate the full objective in the form of bilevel optimization problem to find the best quantized weights which minmmize the task loss by considering the cross-layer weight dependencies and the relative importance among weights:
w" V arg min Crask (f (X'; W)), where w.y =(wws V). W.aV
After the quantized training, the quantized weights w. at layer 1 are determined by w (ww": s", V*). In practice, directly solving the above bilevel optimization problem imprac- tical due its excessive computational cost. We therefore consider the following relaxed problem:
a w. 8" V -arg min Lrast (f(X; ((lg(())))))))) +A diag(() V (w - a--gn(ww)) a.
(8)
+7 [[ossn(w>)]1>>
where À is a byperparameter to balance between the quantization objective and task loss. Since impossible to compute the gradients for discrete values in quantized weights, we adopt the straight-through estimator Bengio et al. (2013) that is broadly used across QAT methods: sgn(w) indicates the sign function applied elementwise to w. We follow Liu et al. (2020) for the derivative of egn(-). Finally, obtain the desired quantized weights by w a-sgn( (w") ). In order to obtain the off off-diagonal parts of the cross-layer dependency matrix V, we minimize Equation with respect to and V to dynamically determine the values (we omit X and y from this argument for readability):
Liroin (a,w, w. s, V) = Lrack (f (X::assg(ww)) V) +\ diag(s)) (s) V (w - a-sgn(w))]?
(9)
+y [[a-sgn(w) || + Reg(s, V),
2 is regulariztion term which enforces V to be orthogonal and keeps the scale of s constant. Here, o is the constant initial value of Σ, log(s.). which is non-negative importance score.
where Reg(s, V)= = IVV 1| 1
12
5

---

Under review as a conference paper at ICLR 2023
Prodipe Comporent Analys
Proteodie pararetor
C Prssmpprreles
Quaettue Weghl (Freery
w(3
W 3)
Ventar
W 33)
w(2)
block
wx22) 2)
(2)
w(1
Training guamiiee sesights fell by
WARE Reetundog
5
wrights
V(O.)
S
V(0.122)
(0)
4)
Abetuning
(0)
Quanticn wrights w
Figure 4: Quantization- aware Training with BITAT: We perform a sequential training process: quantization training of layer- rapid finetuning for upper layers. At cach layerwise quantization, we also train the importance vector and orthonormal correlation matrix, which initialized by PCA components of the current and lower layer inputs in the target block, and guide the quantization to consider the importance disentangled weights.
4.1 LAYER-PROGRESSIVE QUANTIZATION WITH BLOCK-WISE WEIGIIT DEPENDENCY
Now we extend our formulation for multi-layered neural networks considering cross-layer weight dependency. While we obtain the objective function in Equation 9, it is inefficient to perform quantization-wware training while considering the complete correlations of all weights in the given neural network. Therefore, we only consider cross cross-layer dependencies between only few consecutive layers (we denote it as a block), and initialize and V using Principal Component Analysis (PCA) on the inputs to those layers within each block.
Formally, we define a weight correlation matrix in neural network block (block)) E (E d.)x(_ d:), where i the number of layers in p(block)
v
block, similarly to the block-diagonal formulation in Li et al. (2021) to express the dependencies between weights across layers in the off- diagonal parts. We initialize s!!) and in-diagonal parts (1) by applying PCA on the input covariance matrix:
V(K)
Figure Initialization of the block correlation matrix.
N
( + (100), V < UW)
where U (
TT-1) : (10)
N
where (!)) is a column vector and the output of 1-th layer and ( x. This allows the weights at 1-th layer to consider the dependencies on the weights from the earlier layers within the same neural block, and we refer to this method as sequential quantization, so that the model alleviates the quantization errors accumulated through propagating from the lower to higher consecutive layers while preserving the performance of the quantized model. Then, instead of having one set of s and V for each layer, we can keep the previous layer's s and V and expand them. Specifically, when quantizing layer which is a part of the block that starts with the layer m, we first apply PCA on the input covariance matrix to obtain (1) and U(),, We then expand the existing s (mdl1)) and (mnd-1 to obtain D+ and E R DD as follows':
output
[1o(mit111 i { DKi,
[m(m)-1) h:s:
4 D,
[8 (mal))
[ = (
D < i,j.
(1I)
otherwise,
where D = DI- di as illustrated in Figure 4. The weight dependencies between different layers (i.e., off-diagonal areas) are trainable and zero-initialized. That is, at each layerwise quantization in the target block, we train the importance vector and orthonormal correlation matrix, where expanded areas are initialized by PCA components of the current layer inputs area. To enable the matrix multiplication of the weights with the expanded s and V, we define the expanded block weights':
w (m:t) (nn (m:--1) d:); w w0)
(12)
where PadCol(., c) zero-pads the input matrix to the right by e columns. Then, our final objective from Equation 9 with cross-layer dependencies is given as follows:
"H- indicates the i-th element of the object inside the brackets.
[A; B3 indicates vertical concatenation of the matrices A and B.

---

Under review as a conference paper at ICLR 2023
Algorithm Neural Network Binarization with Task-dependent Aggregated Transformation
Input: Pre-trained weights w(t) ..... w(t for layers, task loss function Maximum size of input- dimension group k. quantization epochs per layer Nep-
2: Output: Quantized weights =*(1) ..... w²())
3: BI..... B. DDiiide the neural network into blocks
4: for each block B do
5: []. V
6: for each layer in do
7: (!-1) + inputs for layer I
- if d > then K- K-MEANS(X k) else L4-1
Grouping permutation matrix + Initialization values for the expanded part
9:
U diag(A) (MoP"
10:
[s: V
0]
0
expand and V
11:
sa - (1:11 V &- arg sa. Lerain (sax, w. we
b Iterate for Nep epochs
12:
(1)
sa 1) sgn((()))))
Lirain
(w SrrainWlllly
g(mit), v(mil)) Lrask (s (2: (a-sg((w(()) w ((((1221))) )))
+ A (mm:)) (mmit)
w (mm))) a-sgn(w((ml)) (mil)) )|( (13)
+7 /
||, Reg g(mil) v(mi)).
Given the backbone architecture with L layers, we minimize Lerain (w))), s((), v() with respect to
w(!), s(), and (L) to find the desired binarized weights =(()) for layer / while keeping the other layers frozen. Next, we finetune the following layers using the task loss function few epochs before performing QAT on following layers, as illustrated in Figure 4. This sequential quantization proceeds from the bottom layer to the top and the obtained binarized weights are frozen during the training.
4.2 COST-EFFICIENT BITAT VIA AGGREGATED WEIGHT CORRELATION USING REDUCTION MATRIX
We derived QAT formulation which focues on the cross-layer weight dependency by learning block-wise weight correlation matrices. Yet, as the number of inputs to higher layers is often large, the model constructs dinber-dimensional (() on upper blocks, which is costly. In order to reduce the training memory footprint as well the computational complexity, we aggregate the input dimensions into several small groups based on functional similarity using k-means clustering.
IRd for each output dimension, to obtain di points Pt. P2.2,...... ....... Pd; E R&N, then aim to cluster the points to k groups using k-means clustering, each containing N/k points. Let gi E {1, [1,2,... 2.....k) k} indicate the group index of Ps. for i 1,...,d4. d. We construct the reduction matrix P E Rexd, where P.j N/R if 9, i,
First, take feature vectors, the outputs of the 1-th layer of o
and 0 otherwise. Each group corresponds a single row of the reduced ((+1) fg instead of the original dimension di x di. In practice, this significantly reduces the memory consumption of the V (down to 0.07% ). Now, we replace and V in Equation 13 to @ and V P, respectively,
initializing and V. with the grouped input covariance N Σx (Po. (Po, ()).. We describe the
full training process of our proposed method in Algorithm 1. The total number of training epochs
taken in training O (T.Nop)) where I. is the number of layers, and Nep the number of epochs for the quantizing step for each layer.
EXPERIMENTS
We validate new quantization-awar training method, BiTAT, over multiple benchmark datasets; CIFAR-10, CIFAR- Krizhevsky et al. (2009), and ILSVRC2012 ImageNet Deng et al. (2009) datasets. We MobileNet V1 (Howard et al., 2017) backbone network, which is compact neural architecture designed for mobile devices. We follow overall experimental setups from prior works Yamamoto (2021); Liu al. (2020).
7

---

Under review as a conference paper at ICLR 2023
Table Performance comparison of BITAT with baselines. We report the averaged test accuracy across three independent runs. The best results are highlighted in bold, and results of cost-expensive models (10 1 ImgNet FLOPs) are de de-emphasized in gray. We refer to several results reported from their own papers, denoted as
BITWIDTH WEIGHT/ / ACTIV,
IMONET FLOPS (x10")
IMGNET ACC
CIFAR-10 10 ACC (%)
CIFAR-100 100 ACC (%)
METHODS
ARCHITECTURE
ResNer-18
32/32
200.0
69.8
93.02
75.61
Full- pull-precisoon
MobileNet V1
32/32
56.90
70.6
66.68
MobileNet V2
32/32
31.40
71.9
94.43
68 08
BRECQ ei (2021)
MobileNet V2
4/4
3,31
66.57'
DBQ Dbouk (2020)
MobileNet
4/8
3.60
70.54*
93.77
73.20
LCQ Yamamoto (2021)
ResNet-18 18
2/2
15.00
68.91
MobilcNet V2
4/4
3,31
70.81
Me Bethie et al. (2020)
N/A
1/1
24.50
70.71
Bi-Real Net al. (2018)
ResNet-18
1/1
15.00
56.41
Real-to-Bin Bin Martinex al. (2020)
ResNet. 18
1/1
15.00
65.41
76.21
EBCony Rulat et al. (2021)
ResNer-18
1/1
11.00
71.21
76,51
ReActNei-C C Liu el al. (2020)
MobileNet V1
1/1
14.00
71.41
90.77
67.41
ReAatNet-A Liu et al. (2020)
MobileNet VI
1/1
1.20
68.26
89.73
65.51
(Ours)
MobileNet VI
1/1
14,00
69,45
BITAT-A (Ours)
MobileNer
1/1
68.51
90.21
68.36
Baselines and training details. While our method aims to solve the QAT problem, we exten- sively compare our BiTAT against various methods; Post-training Quantization (PTQ) method: BRECQ Li et al. (2021), and Quantization-awall Training (QAT) methods: DBQ Dbouk et al. (2020), EBConv Bulat et al. (2021), Bi-Real Net Liu et al. (2018). Real-to-Bin Martinez et al. (2020), LCQ Ya- mamoto (2021), MeliusNet Bethge et al. (2020), ReActNer Liu et al. (2020). Note that DBQ, LCQ, and MeliusNet, keep some crucial layers, such as xx downsampling layers, full- full-precision, leading to inefficiency at evaluation time. Due to the page limit, we provide the details on baselines and the training and inference phase during QAT including hyperparameter setups in the Supplementary file.
5.1 QUANTITATIVE ANALYSIS
We compare our BiTAT against various PTQ and QAT-based methods in Table on multiple datasets. BRECQ introduces adaptive PTQ method by focusing the weight dependency via hessian matrix computations, resulting in significant performance deterioration and excessive training time. DBQ and LCQ suggest QAT methods, but the degree of bitwidth compression for the weights and activations is limited to 2- to 8-bits, which is insufficient to meet our interest in achieving neural network binarization with 1-bits weights and activations. MeliusNet only suffers small accuracy drop, but it has a high OP count. DBQ and LCQ restrict the bit-width compression to be higher at 4 bits so that they cannot enjoy the MROR-Bitcount optimization for speedup. Although Ri-Real Net, Real-to-Bin and EBConv successfully achieve neural network binarization, over-parameterized ResNet is adopted as backbone networks, resulting in higher OP count. Moreover, except EBConv, these works still suffer from a significant accuracy drop. ReActNet binarizes all of the weights and activations (except the first and last layer) in compact network architectures while preventing model convergence failure. Nevertheless, the method still suffers from considerable performance degeneration of the binarized model. On the other hand, our BiTAT prevents information loss during quantized training up to 1 1-bits, showing a superior performance than ReActNet, 0.37 % t for ImageNet, 0.53% 1 for CIFAR- 10, and 2.31% 1 for CIFAR- 100. Note that BiTAT further achieves on par performance of the MobileNet backbone for CIFAR-100. The results support our claim on layer-wis wise quantization from the bottom layer to the top, reflecting the disentangled weight importance and correlation with the quantized weights earlier layers.
Ablation study We conduct ablation studies to analyze the effect of salient components in our proposed method in Figure 7 Left. BiTAT based on layer-wise sequential quantization without weight transformation already surpasses the performance of ReActNet, demonstrating that layer-wise progres- sive QAT through implicit reflection of adjusted importance plays a critical role in preserving the pre-trained models during quantization. We adopt intra-layer weight transformation using the input- dependent orthonormal matrix, but no significant benefits are observed. Thus, we expect that only disentangling intra-layer weight dependency is insufficient to fully reflect the adjusted importance of each weight due to a binarization of earlier weights/activations.. This evident that BiTAT considering both intra-layer and cross-layer weight dependencies achieves improved performance than the case with only latra-layer dependency. Yet, this requires considerable additional training time to compute
8

---

Under review as a conference paper at ICLR 2023
METHOD
INTRA LAYER Caoss LATEI
TRANSPORN
THANSFORM
Accuracy (16
Trin Time
(houns)
REACTNET LIU BT AL. (2020)
N/A
N/A
65.51 L 0.74
10.75
68.17 1. 0.07
3.49
0.22
3.66
Filber-ntor Tronsform
of Truncuned SVD
67,55 + 1.04
3.44
w/ Aggregaied Transform
68.36 1: 0.45
3,11
Figure 7: Left: Ablation study for analyzing core components in our method. We report the averaged performance and 95% confidence interval across independent runs and the complete BiTAT result is highlighted in gray hackground. Right: Visualization of the weight grouping during sequential quantization of BiTAT. Each 3x3 square represents convolutional filter of the topmost layer (26th, excluding the classifier) of model, and each unique color represents each group to which weight elements belong.
with chunk-wise transformation matrix. In the end, BiTAT with aggregated transformations, which is our full method, outperforms our defective variants in both terms of model performance and training time by drastically removing redundant correlation through reduction matrices. We note that using k- means clustering for aggregated correlation is also essential, another variant, BiTAT with filter-wise transformations, which flter-wisely aggregates the weights instead, results in deteriorated performance.
200
200
5.004
500-
400
400
5.000
500
300
73000
Bisgisiy
200
200
500
690
630
-0.00 00.00
500
520
500
Block
Block 3
Block 5
Figure 6: Visualization of the learned V matrix and the diag(s) of three blocks of the network, with the CIFAR-100 dataset. Notice the off-diagonal parts which represent cross-layer dependencies.
5.2 QUALITATIVE ANALYSIS
Visualization of Reduction Matrix We visualize the weight grouping for BiTAT in Figure 7 Right
to analyze the effect of the reduction matrix, which groups the weight dependencies in each layer based on the similarity between the input dimensions. Each 3x3 square represents a convolutional filter, and each unique color in weight elements represents which group each weight is assigned to, determined by the k-means algorithm, described in Section 4.2. We observe that weight elements in the same filter do not share their dependencies; rather, on average, they often belong to four-five different weight groups. Opposite to these observations, BRECQ regards the weights each filter as the same group for computing the dependencies in different layers, which is problematic since weight elements in the same filter can behave differently from each other.
Visualization of Cross-layer Weight Dependency In Figure 6, we visualize learned transformation matrices V (top row), which shows that many weight elements at each layer are also dependent on other layer weights highlighted in darker colors, verifying our initial claim. Further, we provide visualizations for their multiplications with corresponding importance vectors diag(s) VT (bottom row). Here, the row of V is sorted by the relative importance in increasing order each layer. We observe that important weights in layer affect other layers, demonstrating that cross-layer weight dependency impacts the model performance during quantized training.
6 CONCLUSION
In this work, we explored long-overlooked factors that are crucial in preventing the performance degeneration with extreme neural network quantization: the inter-weight dependencies. That is, quantization of a set of weights affect the weights for other neurons within each layer, as well as weights in consecutive layers. Grounded by the empirical analyses of the node interdependency, we propose a Duantization-Aware Training (QAT) method for binarizing the weights and activations of a given neural network with minimal loss of performance. Specifically, we proposed orthonormal transformation of the weights at each layer to disentangle the correlation among the weights to minimize the negative impact of quantization on other weights. Further, we learned scaling term to allow varying degree of quantization error for each weight based on their measured importance, for layer-wise quantization. Then we proposed an iterative algorithm to perform the layerwise quantization in a progressive manner. We demonstrate the effectiveness of our method in neural network binarization on multiple benchmark datasets with compact backbone networks, largely outperforming state-of-the-art baselines.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: (30i. 100.333883333 3432, 2013.
Joseph Bethge, Christian Bartz, Haojin Yang, Ying Chen, and Christoph Meinel. MeliusNet: Can binary neural networks achieve MobileNet-level accuracy? Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Adrian Bulat, Georgios Tzimiropoulos, and Brais Martinez. High-capacity expert binary networks. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.
1597-1607. PMLR, 2020.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran Ei- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1, 2016.
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 2018.
Hassan Dbouk, Hetul Sanghvi, Mahesh Mehendale, and Naresh Shanbhag. DBQ: A differentiable branch quantizer for lightweight deep neural networks. Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition (CVPR), 2009.
Ruizhou Ding, Ting- Wu Chin, Zeye Liu, and Diana Marculescu. Regularizing activation distribution for training binarized deep networks. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 18011103307700 077282, 2018
Jiaxin Gu, Ce Li, Baochang Zhang, Jungong Han, Xianbin Cao, Jianzhuang Liu, and David Doer- mann. Projection convolutional neural networks for 1-bit cans via discrete back propagation. In Proceedings of the AAAI National Conference on Artificial Intelligence (AAAI), 2019.
Xiangyu He, Zitao Mo, Ke Cheng, Weixiang Xu, Qinghao Hu, Peisong Wang, Qingshan Liu, and Jian Cheng. Proxybnn: Learning binarized neural networks via proxy matrices. In Proceedings of the European Conference on Computer Vision (ECCV), 2020a.
Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruning criteria for deep convolutional neural networks acceleration. Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition (CVPR), 2020b.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 000..00444..111 2017.
Sangil Jung, Changyong Son, Seohyung Lee, JinWoo Son, Jac-Joon Han. Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Tero Karras, Miia Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2015.
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. BRECQ: Pushing the limit of post-training quantization by block reconstruction. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2020a.
Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Yan Wang, Yongjian Wu, Feiyue Huang, and Chi Lin. Rotated binary neural network. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit CNNs with improved representational capability and advanced training algorithm. Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. ReActNet: Towards pre- cise binary neural network with generalized activation functions. Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binary convolutions. Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Asit Mishra and Debbie Marr. Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. Proceedings of the International Conference on Learning Repre- sentations (ICLR), 2018
Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. Proceedings of the International Conference on Machine Learning (ICML), 2020.
Eunhyeok Park and Sungjoo Yoo. PROFIT: A novel training method for sub-4-bit MobileNet models. Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. Binary neural networks: A survey. Proceedings of the International Conference on Pattern Recognition (ICPR), 2020a.
Haotong Qin, Ruihao Gong. Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. Binary neural networks: A survey, Pattern Recognition, 2020b.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv: 103. 103.00020, 2021.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. Proceedings of the European Conference on Computer Vision (ECCV), 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bileNetV2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Peisong Wang, Qinghao Hu, Yifan Zhang, Chunjie Zhang, Yang Liu, and Jian Cheng. Two-step quantization for low-bit neural networks. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2018
Ziwei Wang, Jiwen Lu, Chenxin Tao, Jie Zhou, and Qi Tian. Learning channel-wise interactions for binary convolutional neural networks. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

---

Under review as a conference paper at ICLR 2023
Kohei Yamamoto. Learnable companding quantization for accurate low-bit neural networks. Proceed- ings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Jachong Yoon and Sung Ju Hwang. Combined group and exclusive sparsity for deep neural networks. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Linggiao Liu, and Ian Reid. Structured binary neural networks for accurate image classification and semantic segmentation. In Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition (CVPR), 2019.
12


Reviews:
[{"review_id": "lSOZswBFTYz", "reviewer": "Reviewer_Ed2K", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a QAT method to alleviate performance degeneration with binarization by focusing on the inter-weight dependencies, between the weights within each layer and across consecutive layers. To minimize the quantization impact of each weight on others, the authors perform an orthonormal transformation of the weights at each layer by training an input-dependent correlation matrix and importance vector, such that each weight is disentangled from the others. Then, the authors quantize the weights based on their importance to minimize the loss of information from the original weights/activations. The authors further perform progressive layer-wise quantization from the bottom layer to the top, so that quantization at each layer reflects the quantized distributions of weights and activations at previous layers. ", "summary_of_the_review": "See Strength And Weaknesses. ", "strength_and_weaknesses": "The strengths are as follows:\n(1) Benefiting from the proposed method, the performance of BNN is improved to a certain extent, including on ImageNet and CIFAR100 datasets;\n(2) The paper is well-written and visualized.\n\nBut there are some significant weaknesses:\n(1) My biggest concern is that this paper can almost be seen as a direct application of the techniques and ideas from the PTQ method BRECQ to binarized perception training, which not only limits the novelty of the method but may even be motivated wrongly.\nSince in the PTQ task, the weights of the model need to be quantized without retraining, methods such as BRECQ (such as AdaRound, QDrop, etc.) are devoted to evaluating the impact of quantization on the model. where BRECQ takes into account the intra-block weight dependency and reconstructs the quantized model block-wise. With QAT, however, a fundamental difference is that the weights are aware of the influence of the quantizer and optimized. An empirical conclusion is that the lower the quantization bit width, the less similar the weights of the trained quantized model are to the original model. Some 1-bit binarization work even pointed out that panorama pre-weights after retraining in binary neural networks should be optimized to a bimodal distribution, completely independent of full accuracy.\nTherefore, the paper first needs to prove theoretically or empirically that the dependency of weights in 1-bit QAT still needs to be maintained in the context of being highly discretized, rather than being optimized to obtain new dependencies. Moreover, in binarization-aware quantization, the back-propagation-based optimization of weights actually takes into account the effect of each weight quantization on the global prediction (although discretization causes gradient errors). On this basis, why limit the dependency of weights?\n(2) Moreover, compared with many recent works, the modified work is far from realizing SoTA, such as [1][2][3][4]. The accuracy of ReActNet-A and the proposed BiTAT-A is even far lower than what was reported in the original paper [1]. Also, it would be unwise to compare QAT with PTQ as a comparison method unless the QAT method has a tiny training computational cost comparable to PTQ.\n\n[1] Zechun Liu, et al. ReActNet: Towards precise binary neural network with generalized activation functions. ECCV, 2020.\n[2] Zechun Liu, et al. How do adam and training strategies help bnns optimization. ICML, 2021.\n[3] Zhijun Tu, et al. Adabin: Improving binary neural networks with adaptive binary sets. arXiv, 2022.\n[4] Sheng Xu, et al. Recurrent bilinear optimization for binary neural networks. arXiv, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Y5caKbFP6uL", "reviewer": "Reviewer_2eTK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to consider the important correlations between weights and activations in each binary NN layer. Second, the authors train a correlation matrix to optimize the cross-layer weight correlations. They further propose to apply QAT to learn the correlations and progressively quantize the weights from sequential layers rather than quantizing the weights from each layer independently.", "summary_of_the_review": "Though the paper is well written, there are likewise specific novel ideas. However, hopefully, the author can answer a few concerns I raised, and these concerns make the paper borderline.", "strength_and_weaknesses": "Strengths:\n\n- The paper is well written and easy to follow.\n- The proposed idea of principle components initialization is novel.\n- The experiments are well conducted.\n\nWeaknesses:\n\n- The experimental results are not convincing enough. One of the claimed contributions is the enhancement of intra-layer and cross-layer dependencies through the learnable correlation weight matrix. But, we can see in figure 7 left, there is only marginal improvements compare to the baseline, i.e., imagenet top-1 68.17% → 68.36%. \nSecond, the reported result based on ReActNet-A (in table 1) is about  0.89% lower (68.26% → 69.4%) than that reported in the ReActNet paper,  which is inconsistent with the claim “… it significantly outperforms baselines with the compact neural network architecture. “ Considering that there is rarely reproducibility problems reported with the ReActNet’s codes (training scripts, hyper-parameters, pre-trained models all available.), I would recommend that the author be able to give a suitable explanation.\n\n- I was wondering how much additional overhead the layer-wise progressive QAT will create compared to the classical baseline methods and combined with the marginal accuracy gain, is the extra overhead worth it?\n\n- It will be very interesting to see the performance on a second backbone, e.g., BiRealNet-18. The official ReActNet code actually also supports it, and this model is smaller, thus easier to train.\n\n- I recommend establishing a mathematical justification for the weight projection using the basis of the activation values. I am not sure for this to be convincing enough.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "7XAKK2Pcmx", "reviewer": "Reviewer_3kJB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel quantization-aware training scheme for the binary neural network. The novelty of this paper is the formulation of error minimization during quantization considering inter-weight dependencies between the weights within each layer and across consecutive layers. Based on the principle component analysis for activation, we can estimate the importance of feature characteristics, and QAT should optimize the quantization error to minimize the critical error of the generated feature after the quantization. The task-dependent weight transformation is well formularized, and the layer-wise progressive quantization is proposed on top of the transformation, resulting in high accuracy after QAT. ", "summary_of_the_review": "Personally, I like the idea of this paper, and the quality of the paper is quite good. However, the idea seems not yet validated enough. I can't ensure we are able to apply the idea beyond the toy-level scale problems. It would be better to provide additional extensive studies. ", "strength_and_weaknesses": "The relationship between weight/activation quantization and the information within generated feature map has been explored in several previous studies, but this paper proposes an intuitive formulation of task-dependent weight correlation. The writing is well organized, so the complex part of the idea becomes straightforward via the development of paragraphs. \n\nHowever, this paper has a few limitations that adversely affect the final evaluations. First, while the paper mainly focuses on the binary neural network, isn't it possible to extend the proposed method for multi-bit quantization? While binary quantization has a lot of advantages, multi-bit precision has substantial practical importance; thereby, presenting the corresponding results would be better. Second, BiTAT-A networks show outstanding accuracy in the CIFAR-100 dataset, but the gain becomes negligible in the ImageNet dataset. I highly doubt that the benefit of the proposed method is shrunk in a large, complex dataset. Can we still exploit the benefit of the proposed method in a large network for a complex dataset? Supportive material is required. Last, can we apply the proposed method for other complicated vision tasks, e.g., object detection or super-resolution? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-FODIGNXGgQ", "reviewer": "Reviewer_axSm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to improve binary neural network performance through layer-wise quantization and orthonormal transformation of weights. ", "summary_of_the_review": "The paper is not ready for publication, and additional experiments are needed to identify the reason behind the improvement and make their claim solid.", "strength_and_weaknesses": "Strength:\nThe work gives some improvement over ReActNet on MobileNet V1-like architecture. \n\nWeakness:\nThe authors only show improvement in single-model architecture, and the difference is relatively small. \nIt is not clear which component is responsible for the improvement, and it is also not clear whether pre-training is necessary.\n\nActionable feedback:\n1. Additional experiments on other model architectures.\n2. The difference with and without aggregated transformation training is relatively small, given the variance of the experiment results. More experiments are also needed here.\n3. An ablation experiment on a model without pre-training would make the paper stronger.\n4. Please fix the citation format of the appendix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject