Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers
Authors: Yiwei Lu
Yaoliang Yu
Xinlin Li
Vahid Partovi Nia
Venue: NeurIPS 2023 poster (2023)
Keywords: Neural network quantization
Model compression
Conditional gradient algorithm

Abstract: In neural network binarization, BinaryConnect (BC) and its variants are considered the standard. These methods apply the sign function in their forward pass and their respective gradients are backpropagated to update the weights. However, the derivative of the sign function is zero whenever defined, which consequently freezes training. Therefore, implementations of BC (e.g., BNN) usually replace the derivative of sign in the backward computation with identity or other approximate gradient alternatives. Although such practice works well empirically, it is largely a heuristic or ``training trick.'' We aim at shedding some light on these training tricks from the optimization perspective. Building from existing theory on ProxConnect (PC, a generalization of BC), we (1) equip PC with different forward-backward quantizers and obtain ProxConnect++ (PC++) that includes existing binarization techniques as special cases; (2) derive a principled way to synthesize forward-backward quantizers with automatic theoretical guarantees; (3) illustrate our theory by proposing an enhanced binarization algorithm BNN++; (4) conduct image classification experiments on CNNs and vision transformers, and empirically verify that BNN++ generally achieves competitive results on binarizing these models.

Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers
Yiwei Lu"
Yaoliang Yu School of Computer Science University of Waterloo Vector Institute yaoliang yu@uaaterloo.ca
School of Computer Science
University of Waterloo Vector Institute yiwei. www.wwwweww....oooomo ca
Xinlin Li
Vahid Partovi Nia Huawei Noah's Ark Lab vahid. www.dupparttviinncuuueee...occ
Huawei Noah's 's Ark Lab xinlin. 1i1@huawei.cm
Abstract
neural network binarization, BinaryConnect (BC) and its variants are consid- ered the standard. These methods apply the sign function in their forward pass and their respective gradients are backpropagated to update the weights. How- ever, the derivative of the sign function is zero whenever defined, which con- sequently freezes training. Therefore, implementations of BC (e.g., BNN) usu- ally replace the derivative of sign in the backward computation with identity other approximate gradient alternatives. Although such practice works well em- pirically, is largely heuristic "training trick." We aim at shedding light on these training tricks from the optimization perspective. Building from existing theory ProxConnect (PC, generalization of BC), (1) equip PC with different forward-backward quantizers and obtain ProxConnect++ (PC++) that includes existing binarization techniques special cases; (2) derive prin- cipled way to synthesize forward-backward quantizers with automatic theoretical guarantees; (3) illustrate our theory by proposing an enhanced binarization algo- rithm BNN++ (4) conduct image classification experiments on CNNs and vision transformers, and empirically verify that BNN++ generally achieves competitive results on binarizing these models.
Introduction
The recent success of numerous applications in machine learning is largely fueled by training big models with billions of parameters, e.g., GPTss in large language models [7, 8], on extremely large datasets. However, such models continue scale up, end-to-end training even fine-tuning becomes prohibitively expensive, due to the heavy amount of computation, memory and storage required. Moreover, even after successful training, deploying these models on resource-limited devices or environments that require real-time inference still poses significant challenges.
A common way tackle the above problems is through model compression, such as pruning [44, 47, 51]. reusing attention [6], weight sharing [57], structured factorization [49], and network quantiza- tion [16, 30, 32, 38). Among them, network quantization (i.e., replacing full- full-precision weights with lower-precision ones) is a popular approach. In this work we focus on an extreme case of network quantization:
binarization, i.e., constraining a subset of the weights to be only binary (i.e., +1), with
Work done during intersship at Huawei Noah's Ark Lab.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the benefit of much reduced memory and storage cost, as well as inference time through simpler and faster matrix-vector multiplications, which is one of the main computationally expensive steps in transformers and the recently advanced vision transformers [14, 34, 52].
For neural network binarization, BinaryConnect [BC, 11] is considered the de facto standard. BC applies the sign function to binarize the weights in the forward pass, and evaluates the gradient at the binarized weights using the Straight Through Estimator [STE, 4J². This widely adopted training trick has been formally justified from an optimization perspective: Dockhorn et al. [13], among others, identify BC as a nonconvex counterpart of dual averaging, which itself is a special case of the generalized conditional gradient algorithm. Dockhorn et al. [13] further propose ProxConnect (PC) as an extension of BC, by allowing arbitrary proximal quantizers (with sign being a special case) in the forward pass.
However, practical implementations [e.g.. 2, 12, 22] usually apply approximate gradient of the sign function on top of STE. For example, Hubara et al. [22] employ the hard tanh function as an approximator of sign. Thus, in the backward pass, the derivative of sign is approximated by the indicator function 1. the derivative of hard tanh. Later, Darabi et al. [12] consider the sign-Swish function as more accurate and flexible approximation in the backward pass (but still employs the sign in the forward pass).
Despite their excellent performance in practice, approximate gradient approaches cannot be readily understood in the PC framework of Dockhorn et al. [13], which does not equip any quantization in the backward pass. Thus, the main goal of this work is further generalize PC and improve our understanding of approximate gradient approaches. Specifically, we introduce PC++ that comes with a pair of forward- baakkddd proximal quantizers, and we show that most of the existing approx- imate gradient approaches are special cases of our proximal quantizers, and hence offering a formal justification of their empirical success from an optimization perspective. Moreover, inspired by our theoretical findings, we propose a novel binarization algorithm BNN++ that improves BNN+ [12] on both theoretical convergence properties and empirical performances. Notably, our work provides direct guidance on designing new wwward-bbckwwer proximal quantizers in the PC++ family, with immediate theoretical guarantees while enabling streamlined implementation and comparison of a wide family of cxisting quantization algorithms.
Empirically, we benchmark existing PC++ algorithms (including the new BNN++) on image clas- sification tasks on CNNs and vision transformers. Specifically, we perform weight (and activation) binarization various datasets and models. Moreover, we explore the fully binarized scenario, where the dot-product accumulators are also quantized to 8-bit integers. In general, we observe that BNN+ is very competitive against existing approaches on most tasks, and achieves 30x reduction in memory and storage with a modest 5-10% accuracy drop compared to full precision training.
We summarize main contributions in more detail:
quantizers and introduce ProxConnect++ (PC++) that includes existing binarization techniques as special cases.
We generalize ProxConnect with
We derive principled way synthesize tees. Moreover, we design a new BNN++ variant to illustrate our theoretical findings.
quantizers with theoretical guaran-
We empirically compare different choices of forward-backward quantizers on image classification benchmarks, and confirm that BNN++ is competitive against existing alternatives.
2 Background
In neural network quantization, we aim at minimizing the usual (nonconvex) objective function {(w) with discrete weights w:
min f(w), WEQ
(I)
where Q C Rd is discrete, nonconvex quantization set such as Q = [=1]d. The acquired discrete weights w E Q are compared directly with continuous full precision weights, which we denote
Note that we refer to STE as its original detinition by Bengio et al. [4] for binarizing weights, and other variants of STE (e.g.. in BNN) as approximate gradient.

---

as w" for clarity. While our work easily extends to most discrete set Q, we focus on Q = [ since this binary setting remains most challenging and leads the most significant savings. Existing binarization schemes can be largely divided into the following two categories.
Post- Training Binarization (PTB): we can formulate post-training binarization schemes as the following standard forward and backward pass:
W; (w(w))
W+1 w Th V((w,).
where Pe is the projector that binarizes the continuous weights w deterministically (e.g., the sign function) or stochastically'. and Ve(w;; denotes a sample (sub)gradient at w We point out that PTB is merely post-processing step, i.e., the binarized weights w, do not affect the update of the continuous weights w,, which are obtained through normal training. As result, there is no guarantee that the acquired discrete weights W, is a good solution (either global local) eq. (1),
Binarization-Aware Training (BAT): then recall the more difficult binarization-awann train- ing scheme BinaryConnect (BC), first initialized by Courbariaux et al. [11]:
W Pw(W)),
W w. T. V((we).
where we spot that the gradient is evaluated at the binarized weights w, but used to update the continuous weights w This approach is also known as Straight Through Estimator [STE, 4]. Note that it is also possible to update the binarized weights instead, effectively performing the proximal gradient algorithm solve (1), shown by Bai al. [2]:
Po(w?),
Wi+1
Th V((..)
This method is known ProxQuant, and will serve baseline in experiments.
2.1 ProxConnect
Dockhorn et al. [13] proposed ProxConnect (PC) broad generalization of BinaryConnect in (2):
P (w]), W = w Ve(w:)
where Me : 1+ Ex=1 Tr. Th 0 is the step size, and P is the proximal quantizer:
PM (w) == argmin 1 w 21/² + r(z), and
2μ
MP (w) == min 2n ||
z r(z).
In particular, when the regularizer LQ (the indicator function of Q). P PQ (for any μi) and we recover BC in (2). Dockhorn et al. [13] showed that the PC update (3) amounts to applying the generalized conditional gradient algorithm a smoothened dual of the regularized problem:
min (((w) w
r r(w))
As min (-w"") M (w"),
w"
where f*(w") max (w, f(w) is the Fenchel conjugate of f. The theory behind PC thus formally justifies STE from an optimization perspective. We provide a number of examples of the proximal quantizer P in Appendix A.
Another natural cousin of PC is the reversed PC (rPC):
W, Pw"()))
wh+1 W, Vl(w)),
which is able to exploit the rich landscape of the loss by evaluating the gradient the continuous weights w.. Thus, we also include it as a baseline in our experiments.
We further discuss other related works in Appendix B.
We only consider deterministic binarization in this paper.

---

3 Methodology
One popular heuristic explain BC is through the following reformulation of problem (I):
min (Po(w")).
Applying (stochastic) "gradient" update the continuous weights obtain:
W w 7) P')(w)) Ve(Po(w:)).
Unfortunately, the derivative of the projector Pq is O everywhere except the origin, where the derivative actually does not exist. BC [11], (2), simply "pretended" that P I. Later works propose to replace the troublesome P'o by the derivative of functions that approximate PQ: e-g.. the hard tanh in BNN [22] and the sign-Swish in BNN+ [12]. Despite their empirical success, is not clear what is the underlying optimization problem or if it is possible to also replace the projector inside Ve, i.e., allowing the algorithm to evaluate gradients at continuous weights, a clear advantage demonstrated by Bai al. [2] and Dockhorn al. [13]. Moreover, the theory established in PC, through a connection to the generalized conditional gradient algorithm, does not apply to these modifications yet, which gap that aim to fill in this section.
3.1 ProxConnect++
To address the above-mentioned issues, we propose to study the following regularized problem: min ((T(w")) + r(w"),
w"
as relaxation of the (equivalent) reformulation of (1):
min ((Pe(w")) + (ww).
In other words, T : Rd + Rd is some transformation that approximates PQ and the regularizer
r Rd + R approximates the indicator function 'Q- Directly applying ProxConnect in (3) we obtain*:
W. PKu (w?), W = w - T'w,) VE((T..)).
(5)
Introducing the forward and backward proximal quantizers:
F = To P.;
B = T'o P
(6)
we can rewrite the update (5) simply as:
w w w The B (w]) VE(F.
is clear that the original ProxConnect corresponds to the special choice
PK
By
Of course, one may now follow the recipe in (6) to design new hrwaaddbbckkward quantizers. We call this general formulation in (7) ProxConnect++ (PC++), which covers a broad family of algorithms.
Conversely, the complete characterization of proximal quantizers in Dockhorn et al. [13] allows us also to reverse engineer T and from manually designed forward and backward quantizers. As we will see, most existing quantizers, and thus their empirical success can be justified from an optimization perspective. In- deed, for simplicity, let us restrict all quantizers to univariate ones that apply component-wise. Then, the following result proven in Appendix C.
quantizers turn out to be special cases of our proximal
Corollary 1. A pair of forward-backward quantizers (F,B) admits the decomposition in (6) (for some smoothing parameter and regularizer r) iff both F and B are functions of P(w) = [(w) (F)) which is proximal (i.e., monotone, compact-valued and with closed graph).
We assume throughout that T. and any function whose derivative we use, are locally Lipschitz so that their generalized derivative always defined, see Rockafellar and Wets [50].

---

FP (forward)
(forward)
BNN (forward)
BNN+ (forward)
BNN+ (forward)
input to layer
fp (p-weights
Forward Quantizer
bn- On-weights
output of layer n
Gp-1
W
F!
evaluate gradient
back-prop
update
0on.
B
ÜWe
Backward Quantizer
FP (hackward)
PC (backward)
BNN (hackward)
BNN+ (backward)
BNN+ (hackward)
Figure 1: Forward and backward pass for ProxConnect++ algorithms (red/blue arrows indicate the forward/backward pass), where fp denotes full precision, bn denotes binary and back-prop denotes backpropagation.
Importantly, with Dockhorn et al. [13] for PC directly carries over to PC++ (see Appendix C for details). Let us further illustrate the convenience of Corollary by some examples.
proximal quantizers, the convergence results established by
Example 1 (BNN), Hubara et al. [22] proposed BNN with the choice
F= sign and B = 1-11))
which satisfies the decomposition in (6). Indeed, let
T(w) min max(-1,
(8)
1 w sign(w)(1 - 1/) [/l > (/ <
(w) =
(9)
sign(w),
Since is constant over [-1, 1), applying Corollary we deduce that the proximal quantizer P if exists, must coincide with F over the support of B. Applying monotonicity of Pi: we may complete the reverse engineering by making the choice over |w] > as indicated above. can easily verify the decomposition in (6):
sign = B= B
Thus, BNN is exactly BinaryCommect applied the transformed problem in (4), where the transfor- mation is the so-called hard lanh in (8) while the regularizer r is determined (implicitly) by the proximal quantizer PK in (9).
To our best knowledge, this is the first time the (regularized) objective function that BNN aims to optimize has been identified. The convergence properties BNN hence follow from the general result of Dockhom et al. [13] ProxConnect, see Appendix C.
5

---

Table 1: Variants of ProxConnectt+.
Forward Quantizer
Backward Quantizer
Algorithm
identity
identity
FP
Pg
identity
BC
L
identity
PC
Po
[-1.1]
BNN
P
VSS
BNN+
SS
VSS
BNN++
Example (BNN+). Darabi et al. [12] adopted the derivative of the sign-Swish (SS) function as backward quantizer while retaining the sign function the forward quantizer:
((w) VSS(w) = 111111110 pw
tanh
1AW 2
(4 F= sign,
where μ a hyperparameter that controls how well SS approximates the sign. Applying Corollary we find that the derivative of SS (as backward) coupled with the sign (as forward) do not admit the decomposition in (6), for any regularizer r. Thus, we are not able to find the (regularized) objective function (if exists) underlying BNN+
We conclude that BNN+ cannot be justified under the framework of PC++. However, it is possible to design variant of BNN+ that does belong to the PC++ family and hence enjoys the accompanying theoretical properties:
Example 3 (BNN++). We propose that simple fix of BNN+ would be replace its sign forward quantizer with the sign-Swish (SS) function:
F(ww: SS(w)
tanh' ((W)
tanh(
which is simply the primitive B. 5. In this case, the algorithm simply reduces to PC++ applied on (4) with (and hence essentially stochastic gradient descent). Of course, we could also compose with proximal quantizer arrive the pair (F o B PH)), which effectively reduces to PC++ applied on the regularized objective in (4) with nontrivial r. We call this variant BNN++
We will demonstrate in the next section that BNN+ is more desirable than BNN+ empirically.
More generally, we have the following result designing new forward-backe quantizers:
Corollary 2. If the forward quantizer is continuously differentiable (with bounded support), then one can simply choose the backward quantizer as the derivative of the forward quantizer.
This follows from Corollary since P(w) = w is clearly proximal. Note that the BNN example does not follow from Corollary 2. In Appendix F, we provide additional examples of howaardbackwwrr quantizers based on existing methods, and we show that Corollary 2 consistently improves previous practices.
In summary: (1) ProxConnect++ enables us to design many choices of T and (2) it also allows to reverse engineer T and from existing forward- backward quantizers, which helps us to better understand existing practices, (3) with our theoretical tool, design BNN+ algorithm, which enjoys immediate convergence properties. Figure visualizes ProxConnect++ with variety of forward-backward quantizers.
quantizers with infinite
Experiments
In this section, we perform extensive experiments to benchmark PC++ on CNN backbone models and the recently advanced vision transformer architectures three settings: (a) binarizing weights only (BW); (b) binarizing weights and activations (BWA), where we simply apply similar forward- backward proximal quantizer to the activations; and (c) binarizing weights, activations, with 8-bit dot-product accumulators (BWAA) [43].

---

Table 2: Binarizing weights (BW), binarizing weights and activation (BWA) and binarizing weights, activation, with 8-bit accumulators (BWAA) on CNN backbones. We consider the fine-tuning (FT) pipeline and the end-to-end (E2E) pipeline. We compare five variants of ProxConnect++ (BC, PC, BNN, BN and BNN++) with FP, PQ, and rPC in terms of test accuracy. For the end-to-end pipeline, omit the results for BWAA due training divergence and report the mean of five runs with different random seeds.
ProxConnect++
Dataset
Pipeline
Task
FP
PQ
rPC
BC
PC
BNN
BNN+
BNN++
BW
92.01 %
89.94%
89.98%
90.31%
90.31%
90.35
90.27%
90.40 %
FT
BWA
92.01% %
88.79%
83.55%
89.39%
89.95%
90.01%
89.99%
90.22%
CIFAR-10
BWAA
92.01 %
85.39%
81.10%
89.11% 1%
89.21%
89.32%
89.55%
90.01 %
BW
92.01% %
81.59%
81.82%
87.51%
88.05%
89.92%
89.39%
90.03%
E2E
BWA
92.01% %
81.51%
81.60%
86.99%
87.26%
89.15%
89.02%
89.91 %
BW 78.87%
66.77%
69.22%
71.35%
71.29%
71.41%
70.22%
72.33 %
FT
BWA
78.87%
56.21%
58.19%
65.99%
65.61%
66.02%
55.22%%
68.03%
ImageNet-Il IK
BWAA
78.87%
53.29%
55.28%
58.18%
59.21%
59.77%
59.10%
63.02% %
BW
78.87%
63,23%
66.39%
67.45%
67.51%
67.49%
66.99%
68.11%
E2E
BWA
78.87%
61.19%
64.17%
65.42%
65.31%
65.29%
65. 65.98%
66.08%
4.1 Experimental settings
Datasets: We perform image classification on CIFAR-10/100 datasets [27] and ImageNet-IK
dataset [28]. Additional details on our experimental setting can be found in Appendix D.
Backbone architectures: (1) CNNs: we evaluate CIFAR-10 classification using ResNet20 [18]. and ImageNet- 1K with ResNet-50 [18]. We consider both fine-tume fine-tuning and mn-to-end training: (2) Vision transformers: we further evaluate our algorithm on two popular vision transformer models: ViT [14] and DeiT [52]. For ViT, we consider ViT-B model and fine-tuning task across all models For DeiT, we consider DeiTT-B, DeiT--,, and DeïT-T, which consist of 12, 6, 3 building blocks and 768, 384 and 192 embedding dimensions, respectively; we consider fine-tuning task on ImageNet-16 pre-trained model for CIFAR datasets and end-to-end training ImageNet-Iee IK dataset.
Baselines: For ProxConnect++,, we consider the 6 variants in Table
With different choices of the forward quantizer Fü' and the backward quantizer B,", we include the full precision (FP) baseline and 5 binarization methods:
(BC) [11], Prox ProxConnect (PC) [13], Binary Neural Network (BNN) [22]. the original BNN+ [12], and the modified BNN++ with F!' = SS. Note that we linearly increase μ in BNN++ to achieve full binarization in the end. We also compare ProxConnect++ with the ProxQuant and reverseProx
baselines.
Hyperparameters: We apply the same training hyperparameters and tme-tune/end-to-end training for 100/300 epochs across all models. For binarization methods: (I) PQ (ProxQuant): similar to
Bai et al. [2], we apply the LinearQuantizer (LQ), see (10) in Appendix A, with initial po 0.01 and linearly increase to pr 10; (2) rPC CevverseroxCooneed
we use the same LQ for rPC; (3)
ProxConnect++: for PC, we apply the same LQ; for BNN+, we choose μ (no need to increase as the forward quantizer is sign); for BNN++ we choose Ho = 5 and linearly increase to pr = 30 achieve binarization the final step.
Across all the experiments with random initialization, we report the mean of three runs with different random seeds. Furthermore, provide the complete results with error bars in Appendix G.
4.2
CNN as backbone
We first compare PC++ against baseline methods on various tasks employing CNNs:
(1) Binarizing weights only (BW), where we simply binarize the weights and keep the other com- ponents (i.e., activations and accumulations) in full precision.
'Note that we use pre-trained models provided by Dosovitskiy et al. [14] on the 2m IK for fine-tuning ViT-B model the CiaageNet-IKKCIFAA datasets, respectively.
7

---

Table 3: Our results on binarizing vision transformers (binarizing weights only). We compare five variants of ProxConnect++ (BC, PC, BNN, BNN+ and BNN++) with FP, PQ. and rPC. End-to-end training tasks are marked as bold (i.e., ImageNet- for DeiT-T/S/B), where the results are the mean of five runs with different random seeds.
ProxConnect+
Model
Dataset
FP
PQ
BC
PC
BNN
BNN+
BNN++
CIFAR-10
98.13%
85.06%
86.22%
87.97%
90.12%
89.08%
88. 12%
90.24%
VIT-B
CIFAR- 100
87.14%
72.07%
73.52%
76.35%
78.13%
77.23%
77.10%
79.22% %
ImageNet-IK IK
77.91%
57.65%
55.33%
63.24%
66.33%
65.31%
63.55%
66.33%
CIFAR-10
94.86%
82.76%
82.25%
83.10%
85.15%
86.12%
85.91%
86.41 %
DeiT-T
CIFAR-100
72.37%
54.55%
55.66%
59.65%
60.15% %
60.06%
60.33% 59.77%
ImageNet-1K
72.20%
61.23%
60.35%
63.22%
65.
65.00%
66.67%
67.34%
CIFAR-10
95.10%
81.67%
80.23%
84.85%
85.13%
85.09%
85. 16%
86.19%
DeiT-S
CIFAR-100
73.19%
45.55%
46.66%
60. 12%
61.59%
60.55%
60. 17%
62.98%
ImageNet-IK
79.91%
69.87%
68.74%
73. 16%
73.51%
73.77%
73.23%
73.53% %
CIFAR-10
98.72%
85.22%
86.35%
88, 88.95%
90.53%
90.21%
89.03%
90.67% %
DefT-B
CIFAR-100
86.65%
72.11%
73.40%
75.40%
78.55 %
76.22%
76.51%
78.30%
ImageNet-1K 81.81%
72.54%
70.11%
76.55%
76.61% %
75.60%
76.63%
76.74% %
Table Results on binarizing vision transformers (BW, BWA, and BWAA) on DeiT-T. We compare 5 variants of ProxConnect++ (BC, PC, BNN, BNN+, and BNN++) with FP, PQ, and rPC. End-to- end training tasks are marked as bold (i.e., ImageNet- I), where we omit the results for BWAA due to training divergence and the reported results are the mean of five runs with different random seeds.
ProxConnect++
Dataset
Task
FP
PQ
rPC
BC
PC
BNN
BNN+
BNN++
BW
94.85%
82.76%
82.25%
83.10%
85.15%
86.12%
85.91
86.41 %
CIFAR-10
BWA
94.85%
82.56%
82.02%
82.89%
85.01%
85.99%
85.66%
86.12% %
BWAA
94.85%
81.34%
80.97%
82.08%
84.31%
84.87%
84.72%
85.31 %
BW 72.37%
54.55%
55.66%
59.65%
60.15%
60.06%
59.77%
60.33 %
CIFAR-100
BWA
72.37%
53.77%
54.98%
59.21%
59.71%
59.66%
59.12%
59,85 %
BWAA
72.37%
52.15%
54.36%
58.15%
59.01
58.72%
58, 15%
59.06 %
BW
72.20%
61.23%
60.35%
63.23%
66.15%
65.00%
66.67%
67.34 %
ImageNet-1K
BWA
72.20%
60.01% %
58.77%
63.1
65.29%
63.75%
65.29%
65.65 %
(2) Binarizing weights and activations (BWA), while keeping accumulation in full precision. Sim- ilar to the weights, we apply the same aerwaddbbakwwar proximal quantizer to binarize activa- tions.
(3) Binarizing weights, activations, with 8-bit accumulators (BWAA), BWAA is more desirable in certain cases where the network bandwidth is narrow, e.g., in homomorphic encryption. To achieve BWAA, addition to quantizing the weights and activations, we follow the implemen- tation of WrapNet [43] and quantize the accumulation of each layer with additional cyclic function. In practice, we find that with 1-bit weights and activations, the lowest bits we can successfully employ to quantize accumulation is 8, while any smaller choice would raise high overflow rate and cause the network to diverge. Moreover, BWAA highly relies on a good ini- tialization and cannot be successfully trained end-to-end in evaluation (and hence omitted).
Note that for the fine-tuning pipeline, we initialize the model with their corresponding pre-trained full precision weights. For the end-to-end pipeline, we utilize random initialization. We report our results in Table 2 and observe: (1) the PC family outperforms baseline methods (i.e., PQ and rPC), and achieves competitive performance on both small and larger scale datasets; (2) BNN++ performs consistently better and is more desirable among the five variants of PC++, especially on B WA and BWAA tasks. advantage over BNN+ further validates our theoretical guidance.
4.3 Vision transformer as backbone
Next, we perform similar experiments on the three tasks on vision transformers.
8

---

Table 5: Ablation study on the effect of the scaling factor, normalization, pre-training, and knowl- edge distillation. Experiments are performed on 10 with ViT-B.
Method
Scaling
Normalization
Pre-train
KD
Accuracy
X
X
0.10%
X
X
12.81%
PC
X
X
66.51
X
88.53%
/
90.13% /
X
X
1.50%
BNN++
X
X
X
77.22%
23.55%
X
89.05%
90.22%
Implementation on vision transformers: While network binarization is popular for CNNs, its application for vision transformers is still rare". Here we apply four protocols for implementation:
UtFP IBNN+ DDPTB
100
50
(1) We keep the mean 8n of full precision weights w for each layer as a scaling factor (can be thus absorbed into F(r) for the binary weights w,. Such an approach keeps the
DeiT-T DeiT-S DeiT-B VIT-B
range of w, during binarization and significantly reduces training difficulty without additional computation.
Figure 2: Comparison between Full Precision (FP) model, BNN++ and Post-training Binarization (PTB) on the fine-tuning task on CIFAR-10.
(2) For binarized vision transformer models, LayerNorm important to avoid gradient explosion. Thus, we add one more LayerNorm layer at the end of each attention block.
(3) When fine-tuning a pre-trained model (full precision). the binarized vision transformer usually suffers from bad initialization. Thus, a few epochs of pre-training on the binarized vision transformer is extremely helpful and can make fine-tuning much more efficient and effective.
(4) We apply the knowledge distillation technique in BiBERT [45] to boost the performance. We use full precision pre-trained models as the teacher model.
Main Results: We report the main results of binarizing vision transformers in Table (BW) and Table (BW, BWA, BWAA), where we compare ProxConnect++ algorithms with the FP, PQ. and rPC baselines on fine-tuning and end-to-end training tasks. We observe that: (1) ProxConnect++ variants generally outperform PQ and rPC and able binarize vision transformers with less than 10% accuracy degradation on the BW task. In particular, for end-to-end training, the best performing ProxConnect++ algorithms achieve z 5% accuracy drop: (2) Among the five variants, we confirm BNN++ is also generally better overall for vision transformers. This provides evidence that our Corollary allows practitioners easily design many and choose the one that performs best empirically; (3) With clear underlying optimization objective, BNN+ again outperforms BNN+ across all tasks, which empirically verifies our theoretical findings on vision transformers; (4) In general, we find that weight binarization achieves about 30x reduction in memory footprint, e.g., from 450 MB to 15 MB for ViT-B
Ablation Studies:
We provide further ablation studies to gain more insights and verify our binariza- tion protocols for vision transformers.
Post-training Binarization: in Figure 2, we verify the difference between PTB
bina- training) on the fine-tuning task on CIFAR- 10 across different
rization) and BAT (binarization-wwer models. Note that we use BNN+ as a demonstration of BAT. We observe that without optimization during fine-tuning, the PTB approach fails in general, thus confirming the importance of considering BAT for vision transformers.
Effect of binarizing protocols: here we show the effect of the four binarizing protocols mentioned at the beginning, including scaling the binarized weights using the mean of full precision weights
'Notably, He et al. [19] also consider binarizing vision transformers, which we compare our implementation details and experimental results against in Appendix E.

---

Ardiiiuturr
Blck
Norn
Rack
Black 11
Namz
Blckk
37%
Back
47%
Blck
97%
Back
875
Dleck
Figure 3:
Results of binarizing different components (blocks) of ViT-B architecture on CIFAR-IO. Warmer color indicates significant accuracy degradation after binarization.
(scaling). adding additional LayerNorm layers (normalization), BAT on the full precision pre-trained models (pre-train) and knowledge distillation. We report the results Table 5 and confirm that each protocol is essential to binarize vision transformers successfully.
Which block should one binarize: lastly, we visualize the sensitivity of each building block to bina- rization vision transformers (i.e., ViT-B) on CIFAR-10 Figure 3. We observe that binarizing blocks near the head and the tail of the architecture causes significant accuracy drop.
5
Conclusion
In this work we study the popular approximate gradient approach in neural network binarization. By generalizing ProxConnect and proposing PC++, provide a principled way understand forward- backward quantizers and cover most existing binarization techniques as special cases. Furthermore, PC++ enables
easily design the desired quantizers (e.g.. the new BNN++) with automatic theoretical guarantees. We apply PC++ to CNNs and vision transformers and compare its variants extensive experiments. We confirm empirically that PC++ overall achieves competitive results, whereas BNN++ is generally more desirable.
Broader impacts and limitations
We anticipate our work to further enable training and deploying advanced machine learning models to resource limited devices and environments, and help reducing energy consumption and carbon footprint at large. We do not foresee any direct negative societal impact. One limitation we hope to address in the future is to build theoretical framework that will allow practitioners quickly evaluate different howward-backward quantizers for variety of applications.
Acknowledgments and Disclosure of F Funding
We thank the reviewers and the area chair for thoughtful comments that have improved our final draft. We thank Arash Ardakani, Ali Mosleh and Marzieh Tahaei for their early participation in this project. YY gratefully acknowledges NSERC and CIFAR for funding support.
10

---

References
[1] M. Alizadeh, A. Behboodi, M. van Baalen, C. Louizos, T. Blankevoort, and M. Welling. "Gradient L1 Regularization for Quantization Robustness". arXiv e-prints. 2020.
[2] Y. Bai, Y.-X. Wang, and E. Liberty. "ProxQuant: Quantized Neural Networks via Proximal Operators". . In: International Conference on Learning Representations. 2018.
[3] R. Banner, Y. Nahshan, and D. Soudry. "Post training 4-bit quantization of convolutional networks for rapid-deployment.. In: Advances in Neural Information Processing Systems. 2019, pp. 7948-7956.
[4] Bengio, N. Léonard, and A. Courville. "Estimating or propagating gradients through stochastic neurons for conditional computation". arXiv preprint arXiv:1308.3433 2013.
[5] Y. Bhalgat, Lee, M. Nagel, T. Blankevoort, and N. Kwak. "LSQ+: Improving low-bit quanti- zation through learnable offsets and better initialization", In: IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) Workshops. 2020, pp. 5978-2986
[6] S. Bhojanapalli, A. Chakrabarti, A. Veit, M. Lukasik, H. Jain, Liu, Y.W. Chang, and S. Kumar. "Leveraging redundancy in attention with Reuse Transformers". arXiv preprint arXiv:21 10.02210.0062... 2021.
[7] S. Biderman et al. "Pythia: A suite for analyzing large language models across training and scaling". arXiv preprint arXiv:2304.01373 (2023).
[8] T. Brown et al. "Language models are few-shot learners". Advances in neural information processing systems, vol. 33 (2020), pp. 1877-1901.
[9] Y. Cai. Z. Yao, Z. Dong, A. Gholami, M. W. Mahoney, and K. Keutzer. "Zeroq: A novel zero shot quantization framework". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, pp. 13169-13178.
[10] J. Choi, Z. Wang, S. Venkataramani, P. I.-J Chuang, V. Srinivasan, and K. Gopalakrish- nan. "Pact: Parameterized clipping activation for quantized neural networks". arXiv preprint arXiv:1805. 180.00088... 2018.
[11] M. Courbariaux, Y. Bengio, and J.-P. David. "Binaryconnect: Training deep neural networks with binary weights during propagations". In: Advances in Neural Information Processing Systems. 2015.
[12] S. Darabi, M. Belbahri, M. Courbariaux, and V. P. Nia. "Regularized binary network train- ing". In: NeurlPS Workshop on Energy Efficient Machine Learning and Cognitive Computing. 2019.
[13] T. Dockhorn, Y. Yu, Sari, M. Zolnouri, and V. Partovi Nia. "Demystifying and Generalizing BinaryConnect", In: Advances in Neural Information Processing Systems. 2021, pp. 13202- 13216.
[14] A. Dosovitskiy et al. "An image is worth 16x16 words: Transformers for image recognition at scale". In: International Conference on Learning Representations. 2021.
[15] S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha. "Learned step size quantization". arXiv preprint 1900:1102..08133 2019.
[16] A. Ghaffari, M. S. Tahaei, M. Tayaranian, M. Asgharian, and V. Nia. "Is Integer Arith- metic Enough for Deep Learning Training?" In: Advances in Neural Information Processing Systems. 2022.
[17] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan. "Deep Learning with Lim- ited Numerical Precision".. In: Proceedings of the 32nd International Conference on Machine Learning (ICML). 2015, pp. 1737-1746.

---

[18] K. He, X. Zhang. S. Ren, and J. Sun. "Deep residual learning for image recognition". In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 770- 778.
[19] Y. He, Z. Lou, L. Zhang, W. Wu, B. Zhuang, and H. Zhou. "BiViT: Extremely Compressed Binary Vision Transformer". arXiv preprint MEXiv:2211.0799) 2022.
[20] K. Helwegen, J. Widdicombe, L. Geiger, Z. Liu, K.-T. Cheng, and R. Nusselder. "Latent weights do not exist: Rethinking binarized neural network optimization". Advances in neural information processing systems, vol. 32 (2019).
[21] Z. Hou and S.-Y. Kung. "Multi-Dimennionall Model Compression of Vision Transformer". arxiv preprint arXiv:2201.0000 00043. 2021.
[22] I. Hubara, M. Courbariaux, Soudry, R. El-Yaniv, and Y. Bengio. "Binarized neural net- works". In: Advances in Neural Information Processing Systems. 2016.
[23] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. "Quantization and Training of Neural Networks for Efficient Integer-Aritmmeti-Onnl, Infer- ence". In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.
[24] S. Jain, A. Gural, M. Wu, and C. H. Dick. "Trained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks". arXiv preprint arXiv: net::9903.000000 00006. 2019.
[25] S. Jung, C. Son, S. Lee, J. Son, J.-J Han, Y. Kwak, S. J. Hwang, and C. Choi. "Learning to quantize deep networks by optimizing quantization intervals with task loss". In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019, pp. 4350-4359,
[26] H. Kim, J. Park, C. Lee, and J.-J. Kim. "Improving accuracy of binary neural networks using unbalanced activation distribution". In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition. 2021, pp. 7862-7871.
[27] A. Krizhevsky. "Learning multiple layers of features from tiny images". Tech. rep. University of Toronto, 2009.
[28] A. Krizhevsky, I. Sutskever, and G. E. Hinton. "Imagenet classification with deep convo- lutional neural networks". In: Advances in Neural Information Processing Systems. 2012, pp. 1097-1105.
[29] Y. Li, R. Gong, X. Tan, Y. Yang, P. Hu, Q. Zhang, F. Yu, W. Wang, and S. Gu. "BREQQ Pushing the Limit of Post- bost-irainnng Quantization by Block Reconstruution". In: International Conference Learning Representations, 2021.
[30] Z. Li, T. Yang, P. Wang, and J. Cheng. "Q-ViT: Fully Differentiable Quantization for Vision Transformer". arXiv preprint arXiv:2201.077 107703. 2022.
[31] M. Lin, R. Ji, Z. Xu, B. Zhang, Y. Wang. Y. Wu, F. Huang, and C.-W. Lin. "Rotated bi- nary neural network". Advances in neural information processing systems, vol. 33 (2020), pp. 7474-7485.
[32] Y. Lin, T. Zhang, P. Sun, Z. Li, and S. Zhou. "FQ-ViT: Fully Quantized Vision Transformer without Retraining". arXiv preprint arXiv:2 13824. 2021.
[33] C. Liu, P. Chen, B. Zhuang, C. Shen, B. Zhang, and W. Ding. "SA-BNN: State-aware binary neural network". In: Proceedings of the AAAI Conference Artificial Intelligence. Vol. 35. 3. 2021, pp. 2091-2099.
[34] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. "Swin transformer: Hierarchical vision transformer using shifted windows". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
12

---

[35] Z. Liu, Z. Shen. S. Li, K. Helwegen, D. Huang, and K.-T. T. Cheng. "How do adam and training strategies help bnns optimization". In: International conference on machine learning. PMLR. 2021, pp. 6936-6946.
[36] Z. Liu, Z. Shen, M. Savvides, and K.-T. Cheng. "Reactnet: Towards precise binary neural network with generalized activation functions". In: Computer Vision-ECCW 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16. Springer. 2020, pp. 143-159.
[37] Z. Liu, B. Wu, W. Luo, X. Yang, W. Liu, and K.-T. Cheng. "Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm". In: Proceedings of the European conference computer vision (ECCV). 2018, pp. 722-737.
[38] Z. Liu, Y. Wang, K. Han, W. Zhang, S. Ma, and W. Gao. "Post-training quantization for vision transformer". Advances in Neural Information Processing Systems (2021), ) pp. 28092-28103.
[39] C. Louizos, M. Reisser, T. Blankevoort, E. Gavves, and M. Welling. "Relaxed Quantization for Discretized Neural Networks". In: International Conference on Learning Representations (ICLR). 2019.
[40] Martinez, J. Yang. Bulat, and G. Tzimiropoulos. "Training binary neural networks with real-to-binary convolutions". In: International Conference on Learning Representations. 2019.
[41] M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort. "Up or down? Adaptive rounding for post-training quantization". In: International Conference on Machine Learning. 2020, pp. 7197-7206.
[42] M. Nagel, M. Baalen, T. Blankevoort, and M. Welling. "Data-free quantization through weight equalization and bias correction". In: Proceedings of the IEEE International Confer- ence Computer Vision. 2019, pp. 1325-1334.
[43] Ni, H. H. Chu, O. Castañeda Fernández, P.-y. Chiang, C. Studer, and T. Goldstein. "Wrap- net: Neural net inference with altra-low-precistie arithmetic". In: International Conference on Learning Representations ICLR 2021. OpenReview. 2021.
[44] B. Pan, R. Panda, Y. Jiang, Z. Wang. R. Feris, and A. Oliva. "IA-RED" Interpretability- Aware Redundancy Reduction for Vision Transformers". In: Advances Neural Information Processing Systems. 2021, pp. 14898-24911
[45] H. Qin, Y. Ding, M. Zhang, Q. Yan, A. Liu, Dang. Z. Liu, and X. Liu. "Bibert: Accurate fully binarized bert".. arXiv preprint arXiv:2203.0633 2022.
[46] H. Qin, R. Gong, X. Liu, M. Shen, Z. Wei, Yu, and Song. "Forward and backward information retention for accurate binary neural networks". In: Proceedings the conference computer vision and pattern recognition. 2020, pp. 2250-2259.
the IEEF/CVF
[47] Y. Rao, W. Zhao, B. Liu, J. Lu, Zhou, and C.-J. Hsich. "Dynamicvit: Efficient vision trans- formers with dynamic token sparsification". In: Advances in Neural Information Processing Systems. 2021, pp. 13937-13949.
[48] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. "Xnor-net: Imagenet classification using binary convolutional neural networks". In: European conference on computer vision. Springer. 2016, pp. 525-542.
[49] H. Ren, H. Dai, Z. Dai, M. Yang. J. Leskovec, Schuurmans, and B. Dai. "Combiner: Full attention transformer with sparse computation cost". Advances in Neural Information Processing Systems (2021), pp. 22470-22482.
[50] R. T. Rockafellar and R. J.-B. Wets. "Variational Analysis". Springer, 1998.
13

---

[51] M. S. Ryoo, A. Piergiovanni. A. Amab, M. Dehghani, and A. Angelova. "TokenLeamer: What Can 8 Learned Tokens Do for Images and Videos?" arXiv preprint arXiv:2106.11297 2021.
[52] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. "Training data- efficient image transformers & distillation through attention". In: International Conference Machine Learning. 2021.
[53] Z. Tu, X. Chen, P. Ren, and Wang. "Adabin: Improving binary neural networks with adap- tive binary sets". In: European conference computer vision. Springer. 2022, pp. 379-395.
[54] P. Wang, Q. Chen, He, and Cheng. "Towards accurate post-training network quantiza- tion via sit-sp and stitching". In: International Conference Machine Learning. 2020, pp. 9847-9856.
[55] S. Xu, Y. Li. T. Ma, B. Zeng, B. Zhang, P. Gao, and J. Lu. "TerViT: An Efficient Ternrry Vision Transformer". arXiv preprint arXiv:2201 .08050. 2022.
[56] Z. Xu, M. Lin, J. Liu, J. Chen, L. Shao, Y. Gao, Y. Tian, and R. Ji. "Recu: Reviving the dead weights in binary neural networks". In: Proceedings of the IEEE/CVF international conference on computer vision. 2021, pp. 5198-5208.
[57] J. Zhang, H. Peng, K. Wu, M. Liu, B. Xiao, J. Fu, and Yuan. Viin Compressing Vision Transformers with Weight Multiplexing", In: JEEE/CVE Conference on Computer Vision and Pattern Recognition. 2022, pp. 12145-12154
[58] X. Zhao, Y. Wang. X. Cai, C. Liu, and L. Zhang. "Linear symmetric quantization of neu- ral networks for oowpprccision integer hardware". In: International Conference on Learning Representations. 2019.
14

---

Appendix for Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers
A More on Proximal Quantizers
Dockhorn et al. [13] gave a complete characterization of the proximal quantizer P,: (multi-valued) mapping P is proximal quantizer (of some underlying regularizer r) iff it is monotone, compact- valued and with a closed graph. We now give a few examples to illustrate the ubiquity proximal quantizers, well the generality of PC:
Identity function: apparently, choosing P/ as the identity function recovers the full preci- sion training.
P. Pq: as Q = [+1}, this choice recovers exactly BC in (2).
Pl L. This is the general piecewise linear quantizer designed by Dockhorn et al. [13].
Recall that Q = {ox 1k= where = -1, 92 = +1, such that p2 = 0 the middle point. By introducing parameters p, Q 0, we define two shifts:
horizontal: qi = q1- q" = P2 A (4 +)
Q2 P2 V (42 -), Q2
vertical:
P2 91 V
92 (p2+2).
Then, we define Lg the piece-wist linear map (that simply connects the points by straight lines):
91
dg w² s
91 (w
91
M2
of <2
Li )===)===
(10)
+ (wo
p2)
22 -2
if < w' S 5.
92.
i $
For the middle points, Lg("") ) can be regarded as the intermediate state between the identity function and Po such that, where Li(w") may take any value within the two limits. Note that p controls the discretization vicinity, such that in practice, p is linearly increased over time to fulfill binary weights in the end. We visualize examples of Lg (ww") in Figure 4.
Lg(w))
Lg(w))
1
1
1
pa
w"
1
02
1
1
2
1
1
1
-1
-1
-1
(a) p 0, 0.2.
(b) p 0 == 0.2.
(c) 0.2, e
Figure 4: Different instantiations of the proximal map Lg in (10) for Q (111))
B Related works
computer vision, vision transformers have become one of the most pop- ular backbone architectures. Dosovitskiy et al. [14] is the first to modify the transformer model to enable images as input, namely the ViT model. Specifically, Dosovitskiy et al. [14] translates an image to a sequence of flattened image patches as input, and applies self-attention mechanism to retrieve patch-wise information in the feature representation. Touvron al. [52] further equips VIT with knowledge distillation and proposes DeiT that generalizes well on smaller models and datasets. Liu et al. [34] further proposes Swin as a hierarchical vision transformer that computes representation with shifted windows.
Vision Transformer.
15

---

In vision transformers, the main computation overhead is the multi-heas attention layer, whose cost quadratic with the length of the image patches. As result, such models are general expensive to train. To reduce the computational cost, different compression techniques have been explored. For instance, Pan et [44] performs dynamic pruning for less important patches; Bhojanapalli et al. [6] reuses attention scores computed for one layer in multiple building blocks; Hou and Kung [21] applies multi-dimensional model compression. In this paper, we focus on an alternative approach, namely network quantization.
Network Quantization.
We consider two possible scenarios of network quantization:
(1) tast-training Quantization: We first discuss the easier
quantization methods. Such approaches usually quantize the full- precciseeiioonn pre-trained model and directly apply it for inference. Post-training quantization is widely used in CNNs [1, 3, 9, 29, 41, 42, 54]. Liu et al. [38] is the first explore PTQ for vision transformers. It optimizes the quantization intervals and considers ranking information in the loss function. However, it only considers quantization to a 6-bit model without severe performance degradation. For lower-bit quantization, it is essential to leverage training.
(2) Quantization-Aware Training (QAT): different from post-training quantization, quantization- aware training leverage quantization during pre-training or fine-tuning. Thus it can be formulated as an optimization problem for learning the optimal quantized weights [5, 10, 15. 17, 23-25, 39, 58). Compared with PTQ. QAT can obtain less accuracy drop in low-bit quantization compared to the full-precision model. Li et al. [30] and Xu et al. [55] demonstrate that QAT requires a unique design to quantize vision transformers and it is possible perform quantization to 3 bit without severe performance degradation. [19] further performs binarization with softmax-aware binarization and information preservation.
Binarization Techniques: (1) Here we summarize existing binarization approaches that can be either justified improved by PC++ Some existing implementations [31, 37, 46] set the forward quantizer as sign and design the backward quantizer in an ad hoc fashion (based on graphic approx- imation of the sign function), e.g., Liu et al. [37] applies piece-wise polynomial approximation; Lin et al. [31] improves [37] with dynamic polynomial approximation: Qin et al. [46] designs dynamic error decay estimator based on the tanh function. These methods cannot be justified with PC++ but could be improved by designing new forward quantizers using our Corollary 2. We dis- cuss these new variants in Appendix and compare them with our methods. Other implementations [36, 53] applies shift transformation on both forward and backward quantizers, which belongs to the PC++ family.
(2) Architecture design that can be further integrated into our PC+ as future work: Xu et al. [56] designs rectified clamp unit address "dead weights"; Rastegari et al. [48] applies the absolute mean of weights and activations; Martinez et al. [40] uses real-to-binary attention matching and data-driven channel re-scaling; Kim al. [26] proposes shifted activation function.
(3) Optimization refinement, again could be integrated into our framework as future work: Liu et al. [33] utilizes state-aware gradient state; Liu et al. [35] provides a weight decay scheme; and Helwegen et al. [20] proposes Bop, a new optimizer for BNNs.
C Additional Theoretical Results
Corollary 1. A pair of forward hawkardbackwward quantizers (F, B) admits the decomposition in (6) (for some smoothing parameter a and regularizer r) iff both F and B are functions of P(w)
1 1 dF(ww,, which is proximal (i.e., monotone, colpact-valued and with closed graph).
B(w)
Proof. We first recall the decomposition in (6): F. := To P Suppose first that (F, B) satisfies the above decomposition. Clearly, both F and B are functions of P P" Moreover.
T= P.
(19)
F'(w)
p (w)
T' PM
PM (w)
B(w)
T'
PM
16

---

and thus
10
dF(w) (" P-
((w)
which is clearly proximal.
Conversely, let ((w) = _ ((w) dF(w) be proximal. Taking (generalized) derivative we obtain
F'(w)
P' (w)
((w)
Since a function of P. say T' P. performing integration we obtain
==Top
up to some immaterial constant (that can be absorbed into T). Thus, (F, B) satisfies the decomposi- tion (6).
The following convergence guarantee for PC++ follows directly from the results in Dockhorn et al. [13]:
Theorem 1. Fix any w, the iterates in (7) satisfy:
Σ n=llw, - w, Ve(Tw>)) r(w+) - r(w)) As-1(w) A.(w) + Σ A (11)
where (,(w):: == r=]]w) r, (w(+)) - (w Wr+1, w is the Bregman divergence induced by the (possibly nonconvex) function r, (w() == MT+1 1ir+1-r(ww r(w) + |w (Recall that μ4 == 1+ - 17-.)
The summand on the left-hand side of (11) is related to the duality gap, which is natural measure of stationarity for the nonconvex problem (4). Indeed, it reduces to the familiar ones when convexity present:
Theorem 2. For convex c T and any w, the iterates in (7) satisfy:
min E[f(w,)- f(w)] s E._ 9F E(A.-1(w) -w)
E. (w.)).
(12)
If is also convex, then
min
B([(w=)-](w) S _r DE (w)
(13)
and
[.[[(::) s(w)] -(W))
S Etsr 4--
EE. F[A.-I(w)
A w -
11
(14)
where w, Si nr w. and f = (c the regularized and transformed objective.
E.e. dr
The right-hand sides of (13) and (14) diminish iff 0 and E, 7/- (assuming boundedness of the stochastic gradient). We note some trade-off choosing the step size 7/; both the numerator and denominator of the right-hand sides of (13) and (14) are increasing w.r.t. 7). The same conclusion can be drawn for (12) and (11), where A, also depends on 7, (through the accumulated magnitude of w 1)
D Additional Experimental Settings
Hardware and package: All experiments were run on GPU cluster with NVIDIA V100 GPUs. The platform we use PyTorch. Specifically, we apply ViT and DeiT models implemented in Pytorch Image Models (timm) ?
"https://ttin :ttp:://tiii fast.ai/
17

---

Bi-R (i-Realforwarr)
R-BNN (fowwrdd)
Poly+ (forward)
EDE (forward)
EDE+ (forward)
ReActNet (forward)
Hi-Realbackkwar)
R-BNN (hackward)
Polys<(ackward)
(hackward)
(backward)
ReActNet (hockward)
Figure 5: Forward and backward pass for 6 additional ProxConnect++ algorithms.
Table 6: Results for additional ProxConnect++ algorithms on binarizing vision transformers (bina- rizing weights only), where the results are the mean of five runs with different random seeds.
ProxConnect++
Model
Dataset
FP
Bi-Real
R-BNN
Poly+
EDE
EDE+
ReAct
BNN++
CIFAR-10
94.85%
84.11% %
84.54%
85.31%
84.99%
85.57%
85.35%
86.41 %
DeiT-T
CIFAR-100
72.37%
59.01%
59.02%
60.00%
59.32%
60.04%
60.11%
60.33 %
ImageNet-1K
72.20%
64.55%
64.59%
64.97%
64.28%
65.01%
65.37%
67.34% %
CIFAR-I0
95.09%
85.01%
86.07%
84.99%
85.37%
85.33%
85.91%
86.19 %
DeiT-S
CIFAR-100
73.19%
59.66%
59.75%
60.14%
60.09%
61.17%
61.09%
62.98 %
ImageNet-IK IK
79.90%
70.51
70.87%
71.36%
70.66%
72.99%
72.53%
73.53 %
Pre-trained models: In this work, we applied pre-trained full precision models for fine-tuning tasks. Here we specify the links to the models we used (note that we choose patch size equal to 16 across all models):
ViT-B (ImageNet-IK):
https://storage http:/tttrrrrggggoogpp.......ooo ouge://..6-2222200000001-0000012
0 0--
ViT-B http://wwww.//ateee/////////ooea
DeiT-T (ImageNet-1K);
natch16__244a11111bcf
DeiT-S (ImageNet-IK):
http:///g.faappppiccccceeeeo///r patch16_224--a (0____224ccc6515..pppp
DeïT-B (ImageNet-IK):
http:://hl.faappllllccclaarerres patch16_224-1 222222522f4...p-.mmmm pth.
E
Comparison with BiVIT
He et al. [19] propose BiVIT, which considers the same binarization task on vision transformers (specifically, Swin-T and Nest-T). He et al. [19] follow different implementation with softmax- aware binarization and information preservation. To fairly compare with this work, we follow the same setting and PC++ Swin-T and NesT-T ImageNet- IK. We observe that BNN+ achieves 71.3% Top-1 accuracy (BiViT:70.8%) and 69.3% Top-1 accuracy BBiViT:68.7% respec- tively on Swin-T and NesT-T. Note that BiVIT simply applies BNN as the main algorithm and may be further improved with PC++ algorithms.
Additional forward-baccwarrd quantizers
In this section, we summarize additional forward-hacwwart quantizers, which can either im- proved or justified with our PC++ framework. Specifically, we find that:
18

---

Table 7: Error bar for binarizing weights (BW), binarizing weights and activation (BWA) and bina- rizing weights, activation, with 8-bit accumulators (BWAA) on CNN backbones. We consider the end-to-end (E2E) pipeline. We compare five variants of ProxConnect++ (BC, PC, BNN, BNN+ and BNN++) with FP, PQ, and rPC. For the end-to-end pipeline, we report the mean of five runs with different random seeds.
ProxConnect++
Dataset
Task
FP
PQ
rPC
BC
PC
BNN
BNN+
BNN++
92.01%
81.59%
81.82%
87.51%
88.05%
89.92%
89.39%
90.03%
BW
:0.19
10.11
10.16
10.07
10.05
10.11
10.13
10.06
CIFAR-10
92.01%
81.51%
81.60%
86.99%
87.26%
89.15%
89.02%
89.91%
BWA
#0.13
10.16
10.09
#0.11
+0.23
10.08
#0.16
10.09
78.87%
63.23%
66.39%
67.45%
67.51%
67.49%
66.99%
68.11%
BW
+0.06
+0.11
+0.22
+0.04
+0.09
+0.12
#0.26
+0.02
ImageNet-Il IK
78.87%
61.19%
64.17%
65.42%
65.31%
65.29%
65.98%
66.08%
BWA
+0.88
+0.22
10.19
+0.22
+0.17
#0.21
10.55
+0.13
Table 8: Error bar binarizing vision transformers (BW and BWA) ImageNet- 1K. We consider the end-to-end (E2E) pipeline. We compare five variants of ProxConnect++ (BC, PC, BNN, BNN+, and BNN++) with FP, PQ, and rPC. The results are the mean of five runs with different random seeds.
ProxConnect++
Model
Task
PQ
rPC
BC
PC
BNN
BNN+
BNN++
72.20%
61.23%
60.35%
63.22%
66.15%
65.00%
66.67%
67.34%
BW
#0011
#0.07
#0.19
+0.21
#0.11
#0.15
#0.09
#0.07
DeiT-T
72.20%%
60.01%
58.77%
62.13%
65.29%
63.75%
65.29%
65.65%
BWA
+0.13
+0.12
+0.08
+0.06
+0.19
+0.18
+0.06
+0.03
79.91%
69.8 69.87%
68. 74%
73.16%
73.51%
73.77%
73.23%
73.53%
DeïT-S
BW
:0.21
10.26
10.16
10.19
10.22
1:0.88
#0.11
10.13
DeiT-B
81.81%
72.54%
70.11%
76.55%
76.61%
75.60%
76.63%
76.74%
BW
#0.17
00155
+0.23
+0.07
+0.24
10.17
+0.13
+0.07
Bi-Real Net [37]/R-BNN [31]: F(w) = sign. B(w) VF(w), where F(w) is a piewise polynomial function. We simply choose F F(w) and arrive at our legitimate variant Poly+. Note that we gradually increase the coefficient of F(w) such that we ensure full binarization at the end of the training phase.
EDE in IR-Net [46]: F(w) sign, B(w) = Vg(w) kt(1 (anh"((w)) where & and I are control variables varying during the training process, such that g(w) s sign at the end of training. Again, we choose F g(w) and arrive our new legitimate variant EDE+,
ReActNet [36] can be well justified and is special case of PC++.
We visualize these forward-backwand quantizers and our new variants Figure Moreover, we perform experiments vision transformers examine the performance of additional quantizers and their modified variants. We report the results in Table 6 and observe that (1 Our new proposed Poly+ and EDE+ always outperform the original algorithms and further confirm that our PC+ framework merits theoretical and empirical justifications; (2)BNN+ still outperforms other algorithms on all tasks.
G Additional results for end-to-end training
Finally, we provide the error bars for our main experiments in Table 7 and Table 8 for CNN back- bones and vision transformer backbones, respectively.
19


Reviews:
[{"review_id": "0oC2e5ojG5", "reviewer": "Reviewer_YJmg", "summary": "This paper studied binary neural networks which extend the existing theory of ProxConnect(PC) to ProxConnect++ and explored the fully binarized scenario, where the dot-product accumulators are also quantized to 8-bit integers. The authors also proposed BNN++ with non-linear forward and backward approximation to the sign function. Authors did experiments on CIFAR10 and ImageNet datasets.", "questions": "The authors use the non-linear approximation of the sign function in the forward pass, will that yield other real-valued outputs besides the binary values?\n\nLine 170 is confusing: “BNN++ is more desirable than BNN+ empirically.” The author means BNN++ is better than PC++?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Binarizing weights and activations in neural networks is a challenging problem worth studying. This paper proposed an in-depth study of forward and backward functions for binary neural networks and proposed a new method that performs outstanding accuracy on CIFAR10 and ImageNet datasets.", "weaknesses": "The authors only compared their method with several BNN methods. But many classic and famous BNN works are not even mentioned, such as XNOR-Net. And some papers studying the backward approximation are missing, e.g., the second-order approximation to the sign function proposed in Bi-Real Net should be compared in Fig.1. I would suggest authors compare or at least mention the binary neural networks to make the paper more comprehensive.\n\nTo name a few: \n\n[1] Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. (CVPR)\n\n[2] Zhijun Tu, Xinghao Chen, Pengju Ren, and Yunhe Wang. Adabin: Improving binary neural networks with adaptive binary sets. (ECCV)\n\n[3] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. (ECCV)\n\n[4] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. (ECCV)\n\n[5] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. (CVPR)\n\n[6] Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binary convolutions. (ICLR)\n\n[7] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. (ECCV)\n\n[8] Chunlei Liu, Peng Chen, Bohan Zhuang, Chunhua Shen, Baochang Zhang, and Wenrui Ding. Sa-bnn: State-aware binary neural network. (AAAI)\n\n[9] Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Yan Wang, Yongjian Wu, Feiyue Huang, and Chia-Wen Lin. Rotated binary neural network. (NeurIPS) \n\n[10] Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang-Ting Cheng. How do adam and training strategies help bnns optimization. (ICML)\n\n[11] Hyungjun Kim, Jihoon Park, Changhun Lee, and Jae-Joon Kim. Improving accuracy of binary  neural networks using unbalanced activation distribution. (CVPR)\n\n[12] Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang-Ting Cheng, and Roeland Nusselder. Latent weights do not exist: Rethinking binarized neural network optimization. (NeurIPS)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Additional References**: \n\nWe would first like to thank Reviewer YJmg for providing the additional references, especially Bi-Real net, R-BNN, IR-Net, and ReActNet, which are closely related to our work and expanding our PC++ family. We agree that these are important papers that deserve proper discussion in our work. Here we extend our discussion and we will add them to our paper (we follow the Reviewer's citation order here):\n\n> Existing works propose different methods for improving the performance of BNN and they can be roughly categorized into three classes: \n>\n> (1) different *gradient approximation* that can be either improved or justified with PC++: \n>\n> Some existing implementations [3][5][9] set the forward quantizer as $\\text{sign}$ and design the backward quantizer in an ad hoc fashion (based on graphic approximations of the sign function), e.g., [3] applies a piece-wise polynomial approximation; [9] improves [3] with a dynamic polynomial approximation; [5] designs a dynamic error decay estimator based on the $\\text{tanh}$ function. These methods in their original forms do not belong to PC++, but they could be brought into the PC++ family by designing new forward quantizers using our Theorem 1:\n>\n> - Bi-Real/R-BNN: $\\mathsf{F}(\\mathbf{w})=\\text{sign}$, $\\mathsf{B}(\\mathbf{w})=\\nabla F(\\mathbf{w})$, where $F(\\mathbf{w})$ is a piewise polynomial function. We simply choose $\\mathsf{F} =F(\\mathbf{w})$ and arrive at our legitimate variant **Poly+**. Note that we gradually increase the coefficient of $F(\\mathbf{w})$ such that we ensure full binarization at the end of the training phase.  \n> - EDE in IR-Net: $\\mathsf{F}(\\mathbf{w})=\\text{sign}$, $\\mathsf{B}(\\mathbf{w})=\\nabla g(\\mathbf{w})=kt(1-\\text{tanh}^2(t\\mathbf{w}))$, where $k$ and $t$ are control variables varying during the training process, such that $g(\\mathbf{w})\\approx \\text{sign}$ at the end of training. Again, we choose $\\mathsf{F} =g(\\mathbf{w})$ and arrive at our new legitimate variant **EDE+**.\n>\n> Other implementations [2][7] apply shift transformation on both forward and backward quantizers, which belong to the PC++ family.\n>\n> We visualize the forward-backward quantizers of [3][5][7][9], our new Poly+ and EDE+ in Figure 1 of the global response. Moreover, we perform experiments on vision transformers to examine the performance of these 6 additional quantizers in Table 1, we observe that:\n> - Our new proposed Poly+ and EDE+ always outperform the original algorithms and further confirm that our PC++ framework merits theoretical and empirical justifications.\n>- BNN++ still outperforms other algorithms on all tasks.\n> \n >(2) architecture design that can be further integrated into our PC++ as future work: [1] designs a rectified clamp unit to address \"dead weights\"; [4] applies the absolute mean of weights and activations; [6] uses real-to-binary attention matching and data-driven channel re-scaling; [11] proposes a shifted activation function. \n> \n> (3) optimization refinement, which again could be integrated into our framework as future work: [8] proposes to rescale the backward gradient on binary activations in order to stabilize BNN training; [10] provides a weight decay scheme; and [12] proposes Bop, a new optimizer for BNNs.  \n> \n**Questions**: \n\n(1) **Full binarization:** In our experiments (lines 195-196), we linearly increase $\\mu$ in BNN++ to achieve full binarization in the end. To avoid real-valued outputs, we also performed a sanity check to confirm the final weights are 100% binary.\n\n(2) **Regarding the relationship between BNN++ and PC++**:\n\nHere we clarify that BNN++ is an algorithm that belongs to the broader PC++ family:\n- Firstly, we want to emphasize that PC++ is a family of algorithms where the forward/backward quantizers satisfy the condition in Theorem 1; \n- Secondly, as we discussed in Example 2, BNN+ cannot be justified under the framework of PC++, thus does not belong to PC++. \n- Thirdly, our new variant BNN++ is designed to be a legitimate PC++ algorithm and is empirically shown to perform better than BNN+.\n- Similarly to BNN+, references [3][5][9] provided by the Reviewer use different backward quantizers and the same $\\text{sign}$ function, thus cannot be justified by PC++. Using our Theorem 1, we design new algorithms Poly+ and EDE+ that belong to the PC++ family. These new variants are able to explore the loss landscape with gradients evaluated at more fine-grained weights, especially in the initial phase of training (as opposed to evaluating the gradient at full quantized weights).\n\n"}, {"review_id": "pcfpe4nbvd", "reviewer": "Reviewer_WcKF", "summary": "This paper proposes a new framework for training neural networks with binary weights, which generalizes ProxConnect and takes it as a special case. In the new framework, forward and backward quantizers are defined. A consistency result of the two quanziters is derived (Theorem 1). Extensive experiments are conducted to evaluate the usefulness of the new framework.", "questions": "See above.", "limitations": "See above.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is very well-written and easy to the follow. I love the clarity of the paper.\n2. The theoretical framework is sensible and the analysis is rigorous. I did not find a technical flaw.\n3. The empirical evaluation is sufficient. The code is provided for reproduction.", "weaknesses": "1. The proposed framework seems a direct generalization of ProxConnect. Thus, it's novelty is somewhat limited.\n2. It is not straightforward for practitioners how to use the framework to design new proximal quantizers. Please provide practical guidelines and tips.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first thank Reviewer WcKF for the positive review and great questions, which we address blew:\n\n**Novelty on theory:** \n- We agree with Reviewer WcKF (and also Reviewer 5jjv) that PC++ is an extension of PC, and we are more than happy to credit the theoretical convergence property of PC++ to Dockhorn et al.\n- As pointed out by Reviewer 5jjv, our paper also contributes to designing the unifying framework of forward-backward quantizers, deriving a sufficient and necessary condition, and demonstrating the practical usefulness of our framework.\n\n**Design new proximal quantizers:**\n\n(1) Prior to our work, existing implementations mostly designed the backward quantizer in an ad hoc fashion (based on graphic approximations of the sign function), while Dockhorn et al. showed that any (continuous) monotonic function can be used as a backward quantizer. Our Theorem 1 allows practitioners to easily design compatible forward and backward quantizers. \n\n(2) We will add the following corollary to facilitate the design of F-B quantizers: \n\n> **If the forward quantizer is a continuously differentiable function (with bounded support), then one can simply choose the backward quantizer as the derivative of the forward quantizer.** This follows from Theorem 1 since $\\mathsfit{P}(w) \\equiv w$ is clearly proximal. Note that the BNN example does not follow from this Corollary (but still follows from the more general Theorem 1). \n\n(3) Aside from the BNN++ example, we give more examples here on how to derive new F-B quantizers from existing implementations (visualization for existing algorithms and new ones are presented in Figure 1 of the global response):\n\n- Bi-Real/R-BNN: $\\mathsf{F}(\\mathbf{w})=\\text{sign}$, $\\mathsf{B}(\\mathbf{w})=\\nabla F(\\mathbf{w})$, where $F(\\mathbf{w})$ is a piewise polynomial function. We simply choose $\\mathsf{F} =F(\\mathbf{w})$ and arrive at our legitimate variant Poly+. Note that we gradually increase the coefficient of $F(\\mathbf{w})$ such that we ensure full binarization at the end of the training phase.  \n- EDE in IR-Net: $\\mathsf{F}(\\mathbf{w})=\\text{sign}$, $\\mathsf{B}(\\mathbf{w})=\\nabla g(\\mathbf{w})=kt(1-\\text{tanh}^2(t\\mathbf{w}))$, where $k$ and $t$ are control variables varying during the training process, such that $g(\\mathbf{w})\\approx \\text{sign}$ at the end of training. Again, we choose $\\mathsf{F} =g(\\mathbf{w})$ and arrive at our new legitimate variant EDE+.\n\nEmpirically, we find both variants outperform their original form in Table 1 of the global response. \n\n(4) With numerous examples, we have demonstrated that PC++ helps us understand and improve existing gradient approximation methods. We find that using PC++ to design new proximal quantizers based on existing ones is straightforward according to the corollary above. Furthermore, given any other forward quantizers (e.g., higher-order approximation of $\\text{sign}$), we can easily derive their corresponding backward quantizers.   "}, {"review_id": "aYrrMnDa1u", "reviewer": "Reviewer_5jjv", "summary": "Thanks to authors for submitting their work to NeuRips 2023. After nicelly flowing introduction the lines 47-58 lay goals of the paper and its contributions out in the context of recent advances, cf. Dockhorn et al. [13], PC (ProxConnect). In particular, the paper generalizes PC to forward-backward binarization, as opposed to PC using only forward binarization, labelling this more general approach PC++. As a practical result of presented theory the fully binarized (8-bit integers) BNN++ is shown to be competitive against existing approaches on most tasks, including experiments on several datasets and architectures (incl. CNN and ViT). It achieves 30x reduction in memory and storage with a modest 5-10% accuracy drop compared to full precision training. Binarization results on vision transformers, rare up until now, is especially timely and promising given the recent deployment of the transformers in deep learning. Architecture ablation experiments conducted demonstrate practicality of the hereby developed theory and increase potential impact of the paper.\n\nBesides Theorem 1, being rather a straightforward extension of previous work Dockhorn et al. [13], including convergence guarantees applied on PC++ in appendix, I value authors used this general result to present a unifying and pragmatic framework for binarization of the NN in response to increasing energy demands for training and fine-tuning the recent large transformer models. I find paper timely and relevant contribution to research community. ", "questions": "- Would it be possible to Increase an extend of randomly seeded pool of experiments for Table 3,4? - reported averages on 3 runs together with rather diminishing differences between columns may not be significant. Because the results are used to draw performance claims, as on line 250, 251, “In particular, for end-to-end training, the best performing ProxConnect++ algorithms achieve ≈ 5% accuracy drop”,  the extended experiments are suggested to be reported in camera ready version. Especially as it concerns ViT, treated rarely before and thus of high impact and interest to community.\n\n- The Figure 2., p.8, presents well why more involved techniques that post-training binarization (PTB) are needed. Would it be worth to use it in the beginning of the paper as a motivation figure? Accompanied with similar figure of memory and storage requirements, perhaps.\n\n- What does Table 2 show? (accuracy?) While it is commented in the text it is not clear from Table captions. I recommend to factor it in for the sake of higher clarity.\n\n- Line 136: “proved” —> “proven”", "limitations": "Suggestion: For a camera ready version notes on limitation of the method are suggested to be brought back to main body of the paper from the Appendix G.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ $\\textbf{[Accessibility, Approach]}$ The goal of the paper is achieved in very accessible and neat form. Proving Theorem 1 (the main result) introduces a compelling and general framework for designing binarization algorithms for NNs.\n\n+ ProxConnect (3) ++  >>> nice idea of formally adding transformation T that “cancels out” (formally in the Theorem 1 sense) in backward step, so the original theory of PC, Dockhorn et al. [13], applies.\n\n+ $\\textbf{[Unifying framework]}$ anAdditional and very impactful merit of the paper is its unifying approach of currently quite scattered and uneasy to access research of neural networks binarization. Especially natural inclusion of the transformers is very timely, convenient and impactful addressed by only few previous works, e.g. Y. He et al. [19].\n\n+ $\\textbf{[Demonstration of practical usefulness]}$ (Architecture) Ablation experiments present practical usefulness of hereby developed theory (Theorem 1) (line: 283-284) and increasing the impact of the paper on further research.\n", "weaknesses": "- Is the extension of previous work of Dockhorn et al. [13] novel enough? As noted above in Summary and Strength sections, formally adding transformation $T$ that “cancels out” (formally in the Theorem 1 sense) in backward step, so the original theory of PC, Dockhorn et al. [13], applies is smart yet rather straightforward (also demonstrated in the proof in Appendix). Some may see this rather a direct consequence of the previous work, that should be fully contributed to the respective authors :-)... .Despite this caveat I believe the paper's focus on application of this theory outweighs the theoretical merits and yields a work with potential high impact on further advances in the field. I leave to authors without impacting my decision whether or not to recognise previous work of Dockhorn et al. [13] more and consider renaming the Theorem 1 to \"Corollary\" to refer directly to previous work. \n\n- Extend of experiments in terms of datasets and modes is well selected. Yet number of runs to avoid statistical error, especially in Tables $3$ and $4$, is insufficient and should be higher. See Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would first like to sincerely thank Reviewer5jjv for appreciating our contribution and providing valuable suggestions. Below we address your concerns:\n\n**(1) Novelty on theory:** \nWe agree that PC++ is an extension of PC, and we have followed the reviewer's suggestion to label the convergence result in the appendix as a Corollary and credit it more directly to Dockhorn et al. \n\n**(2) More runs to avoid statistical error**:\nWe have increased the randomly seeded pool of experiments from 3 to 5 for Table 3 in the main paper. We confirm that the updated Table (denoted as Table 2 in the global response) reports very similar results with 3 runs and our conclusion remains the same. Due to limited time, we can only finish Table 3 in a week but we will update Table 4 as well in our final draft.\n\n**(3) Figure 2**:\nWe agree that Figure 2 would serve as a nice motivation for this work. We have modified the original Figure 2 and provided an additional figure on memory footprint (Figure 2 in the global response) to illustrate: \n> BAT is necessary for our context as it performs much better than PTB and reduces the same memory footprint during inference.\n\n(4) Yes, Table 2 shows accuracy. We will add the caption to our final draft.\n\n(5) We have changed \"proved\" to \"proven\" on line 136.\n\n(6) Thank you for the suggestion. We will move the limitations back to the main paper in the final draft.\n"}, {"review_id": "mpEqfiDojo", "reviewer": "Reviewer_fayS", "summary": "The paper generalizes ProxConnect with forward-backward quantizers and introduces ProxConnect++ that includes some binarization techniques as special cases. With the derived ProxConnect++, the paper proposes BNN++ to illustrate the effectiveness of ProxConnect++. Experiments show the advantages of BNN++ on image classification benchmarks with CNNs and vision transformers. ", "questions": "The paper presents a unified framework ProxConnect++. It is claimed that ProxConnect++ can include many binarization techniques as special cases. I have two concerns for the unified framework.\n\nFirst, it is desirable to present a unified framework. So one can analyze existing algorithms in a unified way. As a unified framework, the most important thing is to propose new superior algorithms following the framework. However, the paper does not give more insights into developing new algorithms. Note that BNN++ is also inspired by the exiting forward and backward quantizers. \n\nSecond, the paper compares with some variants of ProxConnect++ and shows that BNN++ performs the best. It is better to include other state-of-the-art binarization techniques in the experiments. \n", "limitations": "The authors addressed the limitations of their work. There is no negative societal impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper generalizes ProxConnect and presents the unified framework ProxConnect++, which includes some binarization techniques as special cases. With ProxConnect++, one may design new forward-backward quantizers. Extensive experiments on CNNs and vision transformers demonstrate the advantages of the proposed ProxConnect++.", "weaknesses": "The paper claims that it develops a unified framework ProxConnect++, which allows to design various forward-backward quantizers. However, the forward and backward quantizers used in the paper are limited to the existing ones. To show the advantages of ProxConnect++, it should design new forward and backward quantizers, which outperform the existing ones. The paper only gives an example BNN++, which is also inspired the existing forward and backward quantizers. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would first like to thank Reviewer fayS for the great review and questions. Below we address your concerns:\n\n**Design new proximal quantizers:**\n\n(1) Prior to our work, existing implementations mostly designed the backward quantizer in an ad hoc fashion (based on graphic approximations of the sign function), while Dockhorn et al. showed that any (continuous) monotonic function can be used as a backward quantizer. Our Theorem 1 allows practitioners to easily design compatible forward and backward quantizers. \n\n(2) We will add the following corollary to facilitate the design of F-B quantizers: \n\n> **If the forward quantizer is a continuous differentiable function (with bounded support), then one can simply choose the backward quantizer as the derivative of the forward quantizer.** This follows from Theorem 1 since $\\mathsfit{P}(w) \\equiv w$ is clearly proximal. Note that the BNN example does not follow from this Corollary (but still follows from the more general Theorem 1).\n\n(3) Aside from the BNN++ example, we give more examples here on how to derive new F-B quantizers from existing implementations (visualization for existing algorithms and new ones are presented in Figure 1 of the global response):\n\n- Bi-Real/R-BNN: $\\mathsf{F}(\\mathbf{w})=\\text{sign}$, $\\mathsf{B}(\\mathbf{w})=\\nabla F(\\mathbf{w})$, where $F(\\mathbf{w})$ is a piewise polynomial function. We simply choose $\\mathsf{F} =F(\\mathbf{w})$ and arrive at our legitimate variant Poly+. Note that we gradually increase the coefficient of $F(\\mathbf{w})$ such that we ensure full binarization at the end of the training phase.  \n- EDE in IR-Net: $\\mathsf{F}(\\mathbf{w})=\\text{sign}$, $\\mathsf{B}(\\mathbf{w})=\\nabla g(\\mathbf{w})=kt(1-\\text{tanh}^2(t\\mathbf{w}))$, where $k$ and $t$ are control variables varying during the training process, such that $g(\\mathbf{w})\\approx \\text{sign}$ at the end of training. Again, we choose $\\mathsf{F} =g(\\mathbf{w})$ and arrive at our new legitimate variant EDE+.\n\nEmpirically, we find both variants outperform their original form in Table 1 of the global response. \n\n(4) With numerous examples, we have demonstrated that PC++ helps us understand and improve existing gradient approximation methods. We find that using PC++ to design new proximal quantizers based on existing ones is straightforward according to the corollary above. Furthermore, given any other forward quantizers (e.g., higher-order approximation of $\\text{sign}$), we can easily derive their corresponding backward quantizers.   \n\n**Sota binarization techniques:**\n\n(1) In Appendix E, we compare with another Sota binarization work on vision transformers called BiViT, where we observe BNN++ outperforms BiViT under the same experimental setting.\n\n(2) We add additional results on comparison with other SOTA BNN methods, including Bi-Real Net, Rotated RNN, IR-Net (EDE), and ReActNet. As stated in our previous paragraph, we find that using our PC++ framework, we can improve existing methods with better F-B quantizers design. Furthermore, we find that BNN++ still outperforms other algorithms across all tasks.\n\n(3) In this paper we mainly study the design of forward-backward quantizers.  To verify our theoretical framework on vision transformers, we already applied several empirical tricks to improve the performance of baseline methods and our methods, including the layer-wise scaling factor, additional layernorm function, knowledge distillation, and additional pertaining. \n\n(4) Nevertheless, we are aware of several other techniques such as attention matching, channel re-scaling, shifted activation, state-aware gradient update, and so on, which may boost the performance of PC++ algorithms. We identify this as a limitation of our work and we leave it as future work to further improve our established baselines by combining and adapting some of these techniques."}]

Author rebuttal:
We would like to thank all reviewers again for their extremely informative reviews that helped us improve the paper. Here we want to provide additional figures and tables (in the new one-page PDF file) that we will add to our final draft:

(1) **Figure 1:** According to Reviewer YJmg's suggestions, we have carefully reviewed more existing methods and summarized additional forward-backward quantizers here, which can be either *improved* or *justified* with our PC++ framework. Specifically, we find that:
- Bi-Real Net, R-BNN, and Error Decay Estimator (EDE) in their original forms do not belong to PC++, but they can be improved by modifying the forward quantizers (we will show the improvement in the next table). Thus, we propose the modified Poly+ and EDE+.
- ReActNet is a special case of PC++.

(2) **Table 1:** Following Figure 1, we perform experiments on vision transformers to examine the performance of additional quantizers (and their modified variants), we observe that:
- Our new proposed Poly+ and EDE+ always outperform the original algorithms and further confirm that our PC++ framework merits theoretical and empirical justifications.
- BNN++ still outperforms other algorithms on all tasks.

(3) **Figure 2:** Following the suggestions by Reviewer 5jjv, we modify the original Figure 2 and provide an additional figure on memory footprint to serve as a motivation for this paper: 
> BAT is necessary for our context as it performs much better than PTB and reduces the same memory footprint during inference.

(4) **Table 2:** As suggested by Reviewer 5jjv, we increase the randomly seeded pool of experiments from 3 to 5 for Table 3 in the main paper. We confirm that the updated Table reports very similar results with 3 runs and our conclusion remains the same. 

Meta-review: All reviewers lean towards acceptance. The paper puts forward a principled method to design forward-and-backward quantizers for binarized networks. Besides good theorical contributions, empirical results show noticeable and consistent improvements. The authors are encouraged to incoporate missing citations suggested by the reviewers, and potentially add them for experimental comparisons if appropriate.
Decision: Accept (poster)