Title: Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural network quantization

Model compression

Conditional gradient algorithm
Abstract: In neural network binarization, BinaryConnect (BC) and its variants are considered the standard. These methods apply the sign function in their forward pass and their respective gradients are backpropagated to update the weights. However, the derivative of the sign function is zero whenever defined, which consequently freezes training. Therefore, implementations of BC (e.g., BNN) usually replace the derivative of sign in the backward computation with identity or other approximate gradient alternatives. Although such practice works well empirically, it is largely a heuristic or ``training trick.'' We aim at shedding some light on these training tricks from the optimization perspective. Building from existing theory on ProxConnect (PC, a generalization of BC), we (1) equip PC with different forward-backward quantizers and obtain ProxConnect++ (PC++) that includes existing binarization techniques as special cases; (2) derive a principled way to synthesize forward-backward quantizers with automatic theoretical guarantees; (3) illustrate our theory by proposing an enhanced binarization algorithm BNN++; (4) conduct image classification experiments on CNNs and vision transformers, and empirically verify that BNN++ generally achieves competitive results on binarizing these models.

Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers

Abstract

In neural network binarization, BinaryConnect (BC) and its variants are considered the standard. These methods apply the sign function in their forward pass and their respective gradients are backpropagated to update the weights. However, the derivative of the sign function is zero whenever defined, which consequently freezes training. Therefore, implementations of BC (e.g., BNN) usually replace the derivative of sign in the backward computation with identity or other approximate gradient alternatives. Although such practice works well empirically, it is largely a heuristic or 'training trick.' We aim at shedding some light on these training tricks from the optimization perspective. Building from existing theory on ProxConnect (PC, a generalization of BC), we (1) equip PC with different forward-backward quantizers and obtain ProxConnect++ (PC++) that includes existing binarization techniques as special cases; (2) derive a principled way to synthesize forward-backward quantizers with automatic theoretical guarantees; (3) illustrate our theory by proposing an enhanced binarization algorithm BNN++; (4) conduct image classification experiments on CNNs and vision transformers, and empirically verify that BNN++ generally achieves competitive results on binarizing these models.

1 Introduction

The recent success of numerous applications in machine learning is largely fueled by training big models with billions of parameters, e.g., GPTs in large language models [7, 8], on extremely large datasets. However, as such models continue to scale up, end-to-end training or even fine-tuning becomes prohibitively expensive, due to the heavy amount of computation, memory and storage required. Moreover, even after successful training, deploying these models on resource-limited devices or environments that require real-time inference still poses significant challenges.

Acommonwaytotackle the above problems is through model compression, such as pruning [44, 47, 51], reusing attention [6], weight sharing [57], structured factorization [49], and network quantization [16, 30, 32, 38]. Among them, network quantization (i.e., replacing full-precision weights with lower-precision ones) is a popular approach. In this work we focus on an extreme case of network quantization: binarization, i.e., constraining a subset of the weights to be only binary (i.e., ± 1 ), with

the benefit of much reduced memory and storage cost, as well as inference time through simpler and faster matrix-vector multiplications, which is one of the main computationally expensive steps in transformers and the recently advanced vision transformers [14, 34, 52].

For neural network binarization, BinaryConnect [BC, 11] is considered the de facto standard. BC applies the sign function to binarize the weights in the forward pass, and evaluates the gradient at the binarized weights using the Straight Through Estimator [STE, 4] 2 . This widely adopted training trick has been formally justified from an optimization perspective: Dockhorn et al. [13], among others, identify BC as a nonconvex counterpart of dual averaging, which itself is a special case of the generalized conditional gradient algorithm. Dockhorn et al. [13] further propose ProxConnect (PC) as an extension of BC, by allowing arbitrary proximal quantizers (with sign being a special case) in the forward pass.

However, practical implementations [e.g., 2, 12, 22] usually apply an approximate gradient of the sign function on top of STE. For example, Hubara et al. [22] employ the hard tanh function as an approximator of sign . Thus, in the backward pass, the derivative of sign is approximated by the indicator function 1 [ -1 , 1] , the derivative of hard tanh . Later, Darabi et al. [12] consider the sign-Swish function as a more accurate and flexible approximation in the backward pass (but still employs the sign in the forward pass).

Despite their excellent performance in practice, approximate gradient approaches cannot be readily understood in the PC framework of Dockhorn et al. [13], which does not equip any quantization in the backward pass. Thus, the main goal of this work is to further generalize PC and improve our understanding of approximate gradient approaches. Specifically, we introduce PC++ that comes with a pair of forward-backward proximal quantizers, and we show that most of the existing approximate gradient approaches are special cases of our proximal quantizers, and hence offering a formal justification of their empirical success from an optimization perspective. Moreover, inspired by our theoretical findings, we propose a novel binarization algorithm BNN++ that improves BNN+ [12] on both theoretical convergence properties and empirical performances. Notably, our work provides direct guidance on designing new forward-backward proximal quantizers in the PC++ family, with immediate theoretical guarantees while enabling streamlined implementation and comparison of a wide family of existing quantization algorithms.

Empirically, we benchmark existing PC++ algorithms (including the new BNN++) on image classification tasks on CNNs and vision transformers. Specifically, we perform weight (and activation) binarization on various datasets and models. Moreover, we explore the fully binarized scenario, where the dot-product accumulators are also quantized to 8-bit integers. In general, we observe that BNN++ is very competitive against existing approaches on most tasks, and achieves 30x reduction in memory and storage with a modest 5-10% accuracy drop compared to full precision training.

We summarize our main contributions in more detail:

We generalize ProxConnect with forward-backward quantizers and introduce ProxConnect++ (PC++) that includes existing binarization techniques as special cases.

We derive a principled way to synthesize forward-backward quantizers with theoretical guarantees. Moreover, we design a new BNN++ variant to illustrate our theoretical findings.

We empirically compare different choices of forward-backward quantizers on image classification benchmarks, and confirm that BNN++ is competitive against existing alternatives.

2 Background

In neural network quantization, we aim at minimizing the usual (nonconvex) objective function ℓ ( w ) with discrete weights w :

where Q ⊆ R d is a discrete, nonconvex quantization set such as Q = {± 1 } d . The acquired discrete weights w ∈ Q are compared directly with continuous full precision weights, which we denote

as w ∗ for clarity. While our work easily extends to most discrete set Q , we focus on Q = {± 1 } d since this binary setting remains most challenging and leads to the most significant savings. Existing binarization schemes can be largely divided into the following two categories.

Post-Training Binarization (PTB): we can formulate post-training binarization schemes as the following standard forward and backward pass:

where P Q is the projector that binarizes the continuous weights w ∗ deterministically (e.g., the sign function) or stochastically 3 , and ˜ ∇ ℓ ( w ∗ t ) denotes a sample (sub)gradient of ℓ at w ∗ t . We point out that PTB is merely a post-processing step, i.e., the binarized weights w t do not affect the update of the continuous weights w ∗ t , which are obtained through normal training. As a result, there is no guarantee that the acquired discrete weights w t is a good solution (either global or local) to eq. (1).

Binarization-Aware Training (BAT): we then recall the more difficult binarization-aware training scheme BinaryConnect (BC), first initialized by Courbariaux et al. [11]:

where we spot that the gradient is evaluated at the binarized weights w t but used to update the continuous weights w ∗ t . This approach is also known as Straight Through Estimator [STE, 4]. Note that it is also possible to update the binarized weights instead, effectively performing the proximal gradient algorithm to solve (1), as shown by Bai et al. [2]:

This method is known as ProxQuant, and will serve as a baseline in our experiments.

2.1 ProxConnect

Dockhorn et al. [13] proposed ProxConnect (PC) as a broad generalization of BinaryConnect in (2):

where µ t := 1 + ∑ t -1 τ =1 η τ , η t > 0 is the step size, and P µ t r is the proximal quantizer:

In particular, when the regularizer r = ι Q (the indicator function of Q ), P µ t r = P Q (for any µ t ) and we recover BC in (2). Dockhorn et al. [13] showed that the PC update (3) amounts to applying the generalized conditional gradient algorithm to a smoothened dual of the regularized problem:

where f ∗ ( w ∗ ) := max w ⟨ w , w ∗ ⟩ -f ( w ) is the Fenchel conjugate of f . The theory behind PC thus formally justifies STE from an optimization perspective. We provide a number of examples of the proximal quantizer P µ t r in Appendix A.

Another natural cousin of PC is the reversed PC (rPC):

which is able to exploit the rich landscape of the loss by evaluating the gradient at the continuous weights w ∗ t . Thus, we also include it as a baseline in our experiments.

We further discuss other related works in Appendix B.

3 Methodology

One popular heuristic to explain BC is through the following reformulation of problem (1):

Applying (stochastic) 'gradient' to update the continuous weights we obtain:

Unfortunately, the derivative of the projector P Q is 0 everywhere except at the origin, where the derivative actually does not exist. BC [11], see (2), simply 'pretended' that P ′ Q = I . Later works propose to replace the troublesome P ′ Q by the derivative of functions that approximate P Q , e.g., the hard tanh in BNN [22] and the sign-Swish in BNN+ [12]. Despite their empirical success, it is not clear what is the underlying optimization problem or if it is possible to also replace the projector inside ˜ ∇ ℓ , i.e., allowing the algorithm to evaluate gradients at continuous weights, a clear advantage demonstrated by Bai et al. [2] and Dockhorn et al. [13]. Moreover, the theory established in PC, through a connection to the generalized conditional gradient algorithm, does not apply to these modifications yet, which is a gap that we aim to fill in this section.

3.1 ProxConnect++

To address the above-mentioned issues, we propose to study the following regularized problem:

as a relaxation of the (equivalent) reformulation of (1):

In other words, T : R d → R d is some transformation that approximates P Q and the regularizer r : R d → R approximates the indicator function ι Q . Directly applying ProxConnect in (3) we obtain 4 :

Introducing the forward and backward proximal quantizers:

we can rewrite the update in (5) simply as:

It is clear that the original ProxConnect corresponds to the special choice

Of course, one may now follow the recipe in (6) to design new forward-backward quantizers. We call this general formulation in (7) ProxConnect++ (PC++), which covers a broad family of algorithms.

Conversely, the complete characterization of proximal quantizers in Dockhorn et al. [13] allows us also to reverse engineer T and r from manually designed forward and backward quantizers. As we will see, most existing forward-backward quantizers turn out to be special cases of our proximal quantizers, and thus their empirical success can be justified from an optimization perspective. Indeed, for simplicity, let us restrict all quantizers to univariate ones that apply component-wise. Then, the following result is proven in Appendix C.

Corollary 1. A pair of forward-backward quantizers ( F , B ) admits the decomposition in (6) (for some smoothing parameter µ and regularizer r ) iff both F and B are functions of P ( w ) := ∫ w -∞ 1 B ( ω ) d F ( ω ) , which is proximal (i.e., monotone, compact-valued and with a closed graph).

a

n

-

1

input to layer

n

w

∗

n

fp-weights

F

µ

r

Forward Quantizer

w

n

bn-weights

a

n

output of layer

n

∂a

n

∂

w

n

B

µ

r

evaluate gradient

×

back-prop

update

-

1

1

-

1

1

FP (forward)

-

1

1

-

1

1

FP (backward)

-

1

1

-

1

1

PC (forward)

-

1

1

-

1

1

PC (backward)

-

1

1

-

1

1

BNN (forward)

-

1

1

-

1

1

BNN (backward)

-

1

1

-

1

1

BNN+ (forward)

-

1

1

-

1

1

BNN+ (backward)

-

1

1

-

1

1

BNN++ (forward)

-

1

1

-

1

1

BNN++ (backward)

Importantly, with forward-backward proximal quantizers, the convergence results established by Dockhorn et al. [13] for PC directly carries over to PC++ (see Appendix C for details). Let us further illustrate the convenience of Corollary 1 by some examples.

Example 1 (BNN) . Hubara et al. [22] proposed BNN with the choice

which satisfies the decomposition in (6) . Indeed, let

Since B is constant over [ -1 , 1] , applying Corollary 1 we deduce that the proximal quantizer P µ r , if exists, must coincide with F over the support of B . Applying monotonicity of P µ r we may complete the reverse engineering by making the choice over | w | > 1 as indicated above. We can easily verify the decomposition in (6) :

Thus, BNN is exactly BinaryConnect applied to the transformed problem in (4) , where the transformation T is the so-called hard tanh in (8) while the regularizer r is determined (implicitly) by the proximal quantizer P µ r in (9) .

To our best knowledge, this is the first time the (regularized) objective function that BNN aims to optimize has been identified. The convergence properties of BNN hence follow from the general result of Dockhorn et al. [13] on ProxConnect, see Appendix C.

Example 2 (BNN+) . Darabi et al. [12] adopted the derivative of the sign-Swish (SS) function as a backward quantizer while retaining the sign function as the forward quantizer:

where µ is a hyperparameter that controls how well SS approximates the sign. Applying Corollary 1 we find that the derivative of SS (as backward) coupled with the sign (as forward) do not admit the decomposition in (6) , for any regularizer r . Thus, we are not able to find the (regularized) objective function (if it exists) underlying BNN+.

Weconclude that BNN+ cannot be justified under the framework of PC++. However, it is possible to design a variant of BNN+ that does belong to the PC++ family and hence enjoys the accompanying theoretical properties:

Example 3 (BNN++) . We propose that a simple fix of BNN+ would be to replace its sign forward quantizer with the sign-Swish (SS) function:

which is simply the primitive of B . In this case, the algorithm simply reduces to PC++ applied on (4) with r = 0 (and hence essentially stochastic gradient descent). Of course, we could also compose with a proximal quantizer to arrive at the pair ( F ◦ P µ r , B ◦ P µ r ) , which effectively reduces to PC++ applied on the regularized objective in (4) with a nontrivial r . We call this variant BNN++.

We will demonstrate in the next section that BNN++ is more desirable than BNN+ empirically.

More generally, we have the following result on designing new forward-backward quantizers:

Corollary 2. If the forward quantizer is continuously differentiable (with bounded support), then one can simply choose the backward quantizer as the derivative of the forward quantizer.

This follows from Corollary 1 since P ( w ) ≡ w is clearly proximal. Note that the BNN example does not follow from Corollary 2. In Appendix F, we provide additional examples of forward-backward quantizers based on existing methods, and we show that Corollary 2 consistently improves previous practices.

In summary: (1) ProxConnect++ enables us to design forward-backward quantizers with infinite many choices of T and r , (2) it also allows us to reverse engineer T and r from existing forwardbackward quantizers, which helps us to better understand existing practices, (3) with our theoretical tool, we design a new BNN++ algorithm, which enjoys immediate convergence properties. Figure 1 visualizes ProxConnect++ with a variety of forward-backward quantizers.

4 Experiments

In this section, we perform extensive experiments to benchmark PC++ on CNN backbone models and the recently advanced vision transformer architectures in three settings: (a) binarizing weights only (BW); (b) binarizing weights and activations (BWA), where we simply apply a similar forwardbackward proximal quantizer to the activations; and (c) binarizing weights, activations, with 8-bit dot-product accumulators (BWAA) [43].

Table 2: Binarizing weights (BW), binarizing weights and activation (BWA) and binarizing weights, activation, with 8-bit accumulators (BWAA) on CNN backbones. We consider the fine-tuning (FT) pipeline and the end-to-end (E2E) pipeline. We compare five variants of ProxConnect++ (BC, PC, BNN, BNN+, and BNN++) with FP, PQ, and rPC in terms of test accuracy. For the end-to-end pipeline, we omit the results for BWAA due to training divergence and report the mean of five runs with different random seeds.

4.1 Experimental settings

Datasets : We perform image classification on CIFAR-10/100 datasets [27] and ImageNet-1K dataset [28]. Additional details on our experimental setting can be found in Appendix D.

Backbone architectures: (1) CNNs : we evaluate CIFAR-10 classification using ResNet20 [18], and ImageNet-1K with ResNet-50 [18]. We consider both fine-tuning and end-to-end training; (2) Vision transformers : we further evaluate our algorithm on two popular vision transformer models: ViT [14] and DeiT [52]. For ViT, we consider ViT-B model and fine-tuning task across all models 5 . For DeiT, we consider DeiT-B, DeiT-S, and DeiT-T, which consist of 12, 6, 3 building blocks and 768, 384 and 192 embedding dimensions, respectively; we consider fine-tuning task on ImageNet-1K pre-trained model for CIFAR datasets and end-to-end training on ImageNet-1K dataset.

Baselines : For ProxConnect++, we consider the 6 variants in Table 1. With different choices of the forward quantizer F µ r and the backward quantizer B µ r , we include the full precision (FP) baseline and 5 binarization methods: BinaryConnect (BC) [11], ProxConnect (PC) [13], Binary Neural Network (BNN) [22], the original BNN+ [12], and the modified BNN++ with F µ r = SS. Note that we linearly increase µ in BNN++ to achieve full binarization in the end. We also compare ProxConnect++ with the ProxQuant and reverseProxConnect baselines.

Hyperparameters : We apply the same training hyperparameters and fine-tune/end-to-end training for 100/300 epochs across all models. For binarization methods: (1) PQ (ProxQuant): similar to Bai et al. [2], we apply the LinearQuantizer (LQ), see (10) in Appendix A, with initial ρ 0 = 0 . 01 and linearly increase to ρ T = 10 ; (2) rPC (reverseProxConnect): we use the same LQ for rPC; (3) ProxConnect++: for PC, we apply the same LQ; for BNN+, we choose µ = 5 (no need to increase µ as the forward quantizer is sign ); for BNN++, we choose µ 0 = 5 and linearly increase to µ T = 30 to achieve binarization at the final step.

Across all the experiments with random initialization, we report the mean of three runs with different random seeds. Furthermore, we provide the complete results with error bars in Appendix G.

4.2 CNN as backbone

We first compare PC++ against baseline methods on various tasks employing CNNs:

Binarizing weights only (BW), where we simply binarize the weights and keep the other components (i.e., activations and accumulations) in full precision.

Table 4: Results on binarizing vision transformers (BW, BWA, and BWAA) on DeiT-T. We compare 5 variants of ProxConnect++ (BC, PC, BNN, BNN+, and BNN++) with FP, PQ, and rPC. End-toend training tasks are marked as bold (i.e., ImageNet-1K), where we omit the results for BWAA due to training divergence and the reported results are the mean of five runs with different random seeds.

Binarizing weights and activations (BWA), while keeping accumulation in full precision. Similar to the weights, we apply the same forward-backward proximal quantizer to binarize activations.

Binarizing weights, activations, with 8-bit accumulators (BWAA). BWAA is more desirable in certain cases where the network bandwidth is narrow, e.g., in homomorphic encryption. To achieve BWAA, in addition to quantizing the weights and activations, we follow the implementation of WrapNet [43] and quantize the accumulation of each layer with an additional cyclic function. In practice, we find that with 1-bit weights and activations, the lowest bits we can successfully employ to quantize accumulation is 8, while any smaller choice would raise a high overflow rate and cause the network to diverge. Moreover, BWAA highly relies on a good initialization and cannot be successfully trained end-to-end in our evaluation (and hence omitted).

Note that for the fine-tuning pipeline, we initialize the model with their corresponding pre-trained full precision weights. For the end-to-end pipeline, we utilize random initialization. We report our results in Table 2 and observe: (1) the PC family outperforms baseline methods (i.e., PQ and rPC), and achieves competitive performance on both small and larger scale datasets; (2) BNN++ performs consistently better and is more desirable among the five variants of PC++, especially on BWA and BWAA tasks. Its advantage over BNN+ further validates our theoretical guidance.

4.3 Vision transformer as backbone

Next, we perform similar experiments on the three tasks on vision transformers.

Implementation on vision transformers : While network binarization is popular for CNNs, its application for vision transformers is still rare 6 . Here we apply four protocols for implementation:

(1) We keep the mean s n of full precision weights w ∗ n for each layer n as a scaling factor (can be thus absorbed into F µ t r ) for the binary weights w n . Such an approach keeps the range of w ∗ n during binarization and significantly reduces training difficulty without additional computation.

(2) For binarized vision transformer models, LayerNorm is important to avoid gradient explosion. Thus, we add one more LayerNorm layer at the end of each attention block.

(3) When fine-tuning a pre-trained model (full precision), the binarized vision transformer usually suffers from a bad

initialization. Thus, a few epochs of pre-training on the binarized vision transformer is extremely helpful and can make fine-tuning much more efficient and effective.

(4) We apply the knowledge distillation technique in BiBERT [45] to boost the performance. We use full precision pre-trained models as the teacher model.

Main Results : We report the main results of binarizing vision transformers in Table 3 (BW) and Table 4 (BW, BWA, BWAA), where we compare ProxConnect++ algorithms with the FP, PQ, and rPC baselines on fine-tuning and end-to-end training tasks. We observe that: (1) ProxConnect++ variants generally outperform PQ and rPC and are able to binarize vision transformers with less than 10% accuracy degradation on the BW task. In particular, for end-to-end training, the best performing ProxConnect++ algorithms achieve ≈ 5% accuracy drop; (2) Among the five variants, we confirm BNN++ is also generally better overall for vision transformers. This provides evidence that our Corollary 1 allows practitioners to easily design many and choose the one that performs best empirically; (3) With a clear underlying optimization objective, BNN++ again outperforms BNN+ across all tasks, which empirically verifies our theoretical findings on vision transformers; (4) In general, we find that weight binarization achieves about 30x reduction in memory footprint, e.g., from 450 MB to 15 MB for ViT-B.

Ablation Studies : We provide further ablation studies to gain more insights and verify our binarization protocols for vision transformers.

Post-training Binarization : in Figure 2, we verify the difference between PTB (post-training binarization) and BAT (binarization-aware training) on the fine-tuning task on CIFAR-10 across different models. Note that we use BNN++ as a demonstration of BAT. We observe that without optimization during fine-tuning, the PTB approach fails in general, thus confirming the importance of considering BAT for vision transformers.

Effect of binarizing protocols : here we show the effect of the four binarizing protocols mentioned at the beginning, including scaling the binarized weights using the mean of full precision weights

50

100

FP

BNN++

PTB

DeiT-T DeiT-S DeiT-B ViT-B

Figure 2: Comparison between Full Precision (FP) model, BNN++, and Post-training Binarization (PTB) on the fine-tuning task on CIFAR-10.

Block Architecture

?

MLP Head

MLP

Block 6

MLP

MLP

MSA

Block 12

MSA

Norm

MLP

Block 5

MSA

?

MLP

Block 11

MSA

+

?

MSA

Norm

Block 4

MLP

?

MLP

+

0%

MSA

Block 10

+

MSA

20%

Block3

MLP

+

MLP

MSA

Block 9

40%

MSA

+

Block 2

MLP

MSA

?

MLP

+

Block 8

+

60%

MSA

+

MLP

Block 1

+

80%

MLP

MSA

Block 7

MSA

Embedded

100%

Patches

(scaling), adding additional LayerNorm layers (normalization), BAT on the full precision pre-trained models (pre-train) and knowledge distillation. We report the results in Table 5 and confirm that each protocol is essential to binarize vision transformers successfully.

Which block should one binarize : lastly, we visualize the sensitivity of each building block to binarization in vision transformers (i.e., ViT-B) on CIFAR-10 in Figure 3. We observe that binarizing blocks near the head and the tail of the architecture causes a significant accuracy drop.

5 Conclusion

In this work we study the popular approximate gradient approach in neural network binarization. By generalizing ProxConnect and proposing PC++, we provide a principled way to understand forwardbackward quantizers and cover most existing binarization techniques as special cases. Furthermore, PC++ enables us to easily design the desired quantizers (e.g., the new BNN++) with automatic theoretical guarantees. We apply PC++ to CNNs and vision transformers and compare its variants in extensive experiments. We confirm empirically that PC++ overall achieves competitive results, whereas BNN++ is generally more desirable.

Broader impacts and limitations

We anticipate our work to further enable training and deploying advanced machine learning models to resource limited devices and environments, and help reducing energy consumption and carbon footprint at large. We do not foresee any direct negative societal impact. One limitation we hope to address in the future is to build a theoretical framework that will allow practitioners to quickly evaluate different forward-backward quantizers for a variety of applications.

Acknowledgments and Disclosure of Funding

We thank the reviewers and the area chair for thoughtful comments that have improved our final draft. We thank Arash Ardakani, Ali Mosleh and Marzieh Tahaei for their early participation in this project. YY gratefully acknowledges NSERC and CIFAR for funding support.

References

M. Alizadeh, A. Behboodi, M. van Baalen, C. Louizos, T. Blankevoort, and M. Welling. 'Gradient L1 Regularization for Quantization Robustness'. arXiv e-prints. 2020.

Y. Bai, Y.-X. Wang, and E. Liberty. 'ProxQuant: Quantized Neural Networks via Proximal Operators'. In: International Conference on Learning Representations . 2018.

R. Banner, Y. Nahshan, and D. Soudry. 'Post training 4-bit quantization of convolutional networks for rapid-deployment'. In: Advances in Neural Information Processing Systems . 2019, pp. 7948-7956.

Y. Bengio, N. Léonard, and A. Courville. 'Estimating or propagating gradients through stochastic neurons for conditional computation'. arXiv preprint arXiv:1308.3432. 2013.

Y. Bhalgat, J. Lee, M. Nagel, T. Blankevoort, and N. Kwak. 'LSQ+: Improving low-bit quantization through learnable offsets and better initialization'. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops . 2020, pp. 2978-2985.

S. Bhojanapalli, A. Chakrabarti, A. Veit, M. Lukasik, H. Jain, F. Liu, Y.-W. Chang, and S. Kumar. 'Leveraging redundancy in attention with Reuse Transformers'. arXiv preprint arXiv:2110.06821. 2021.

S. Biderman et al. 'Pythia: A suite for analyzing large language models across training and scaling'. arXiv preprint arXiv:2304.01373 (2023).

T. Brown et al. 'Language models are few-shot learners'. Advances in neural information processing systems , vol. 33 (2020), pp. 1877-1901.

Y. Cai, Z. Yao, Z. Dong, A. Gholami, M. W. Mahoney, and K. Keutzer. 'Zeroq: A novel zero shot quantization framework'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2020, pp. 13169-13178.

J. Choi, Z. Wang, S. Venkataramani, P. I.-J. Chuang, V. Srinivasan, and K. Gopalakrishnan. 'Pact: Parameterized clipping activation for quantized neural networks'. arXiv preprint arXiv:1805.06085. 2018.

M. Courbariaux, Y. Bengio, and J.-P. David. 'Binaryconnect: Training deep neural networks with binary weights during propagations'. In: Advances in Neural Information Processing Systems . 2015.

S. Darabi, M. Belbahri, M. Courbariaux, and V. P. Nia. 'Regularized binary network training'. In: NeurIPS Workshop on Energy Efficient Machine Learning and Cognitive Computing . 2019.

T. Dockhorn, Y. Yu, E. Sari, M. Zolnouri, and V. Partovi Nia. 'Demystifying and Generalizing BinaryConnect'. In: Advances in Neural Information Processing Systems . 2021, pp. 1320213216.

A. Dosovitskiy et al. 'An image is worth 16x16 words: Transformers for image recognition at scale'. In: International Conference on Learning Representations . 2021.

S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha. 'Learned step size quantization'. arXiv preprint arXiv:1902.08153. 2019.

A. Ghaffari, M. S. Tahaei, M. Tayaranian, M. Asgharian, and V. P. Nia. 'Is Integer Arithmetic Enough for Deep Learning Training?' In: Advances in Neural Information Processing Systems . 2022.

S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan. 'Deep Learning with Limited Numerical Precision'. In: Proceedings of the 32nd International Conference on Machine Learning (ICML) . 2015, pp. 1737-1746.

K. He, X. Zhang, S. Ren, and J. Sun. 'Deep residual learning for image recognition'. In: Proceedings of the IEEE conference on computer vision and pattern recognition . 2016, pp. 770778.

Y. He, Z. Lou, L. Zhang, W. Wu, B. Zhuang, and H. Zhou. 'BiViT: Extremely Compressed Binary Vision Transformer'. arXiv preprint arXiv:2211.07091. 2022.

K. Helwegen, J. Widdicombe, L. Geiger, Z. Liu, K.-T. Cheng, and R. Nusselder. 'Latent weights do not exist: Rethinking binarized neural network optimization'. Advances in neural information processing systems , vol. 32 (2019).

Z. Hou and S.-Y. Kung. 'Multi-Dimensional Model Compression of Vision Transformer'. arXiv preprint arXiv:2201.00043. 2021.

I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio. 'Binarized neural networks'. In: Advances in Neural Information Processing Systems . 2016.

B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. 'Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference'. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 2018.

S. R. Jain, A. Gural, M. Wu, and C. H. Dick. 'Trained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks'. arXiv preprint arXiv:1903.08066. 2019.

S. Jung, C. Son, S. Lee, J. Son, J.-J. Han, Y. Kwak, S. J. Hwang, and C. Choi. 'Learning to quantize deep networks by optimizing quantization intervals with task loss'. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 2019, pp. 4350-4359.

H. Kim, J. Park, C. Lee, and J.-J. Kim. 'Improving accuracy of binary neural networks using unbalanced activation distribution'. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2021, pp. 7862-7871.

A. Krizhevsky. 'Learning multiple layers of features from tiny images'. Tech. rep. University of Toronto, 2009.

A. Krizhevsky, I. Sutskever, and G. E. Hinton. 'Imagenet classification with deep convolutional neural networks'. In: Advances in Neural Information Processing Systems . 2012, pp. 1097-1105.

Y. Li, R. Gong, X. Tan, Y. Yang, P. Hu, Q. Zhang, F. Yu, W. Wang, and S. Gu. 'BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction'. In: International Conference on Learning Representations . 2021.

Z. Li, T. Yang, P. Wang, and J. Cheng. 'Q-ViT: Fully Differentiable Quantization for Vision Transformer'. arXiv preprint arXiv:2201.07703. 2022.

M. Lin, R. Ji, Z. Xu, B. Zhang, Y. Wang, Y. Wu, F. Huang, and C.-W. Lin. 'Rotated binary neural network'. Advances in neural information processing systems , vol. 33 (2020), pp. 7474-7485.

Y. Lin, T. Zhang, P. Sun, Z. Li, and S. Zhou. 'FQ-ViT: Fully Quantized Vision Transformer without Retraining'. arXiv preprint arXiv:2111.13824. 2021.

C. Liu, P. Chen, B. Zhuang, C. Shen, B. Zhang, and W. Ding. 'SA-BNN: State-aware binary neural network'. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 35. 3. 2021, pp. 2091-2099.

Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. 'Swin transformer: Hierarchical vision transformer using shifted windows'. In: Proceedings of the IEEE/CVF International Conference on Computer Vision . 2021.

Z. Liu, Z. Shen, S. Li, K. Helwegen, D. Huang, and K.-T. Cheng. 'How do adam and training strategies help bnns optimization'. In: International conference on machine learning . PMLR. 2021, pp. 6936-6946.

Z. Liu, Z. Shen, M. Savvides, and K.-T. Cheng. 'Reactnet: Towards precise binary neural network with generalized activation functions'. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16 . Springer. 2020, pp. 143-159.

Z. Liu, B. Wu, W. Luo, X. Yang, W. Liu, and K.-T. Cheng. 'Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm'. In: Proceedings of the European conference on computer vision (ECCV) . 2018, pp. 722-737.

Z. Liu, Y. Wang, K. Han, W. Zhang, S. Ma, and W. Gao. 'Post-training quantization for vision transformer'. Advances in Neural Information Processing Systems (2021), pp. 28092-28103.

C. Louizos, M. Reisser, T. Blankevoort, E. Gavves, and M. Welling. 'Relaxed Quantization for Discretized Neural Networks'. In: International Conference on Learning Representations (ICLR) . 2019.

B. Martinez, J. Yang, A. Bulat, and G. Tzimiropoulos. 'Training binary neural networks with real-to-binary convolutions'. In: International Conference on Learning Representations . 2019.

M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort. 'Up or down? Adaptive rounding for post-training quantization'. In: International Conference on Machine Learning . 2020, pp. 7197-7206.

M. Nagel, M. v. Baalen, T. Blankevoort, and M. Welling. 'Data-free quantization through weight equalization and bias correction'. In: Proceedings of the IEEE International Conference on Computer Vision . 2019, pp. 1325-1334.

R. Ni, H.-m. Chu, O. Castañeda Fernández, P.-y. Chiang, C. Studer, and T. Goldstein. 'Wrapnet: Neural net inference with ultra-low-precision arithmetic'. In: International Conference on Learning Representations ICLR 2021 . OpenReview. 2021.

B. Pan, R. Panda, Y. Jiang, Z. Wang, R. Feris, and A. Oliva. 'IA-RED 2 : InterpretabilityAware Redundancy Reduction for Vision Transformers'. In: Advances in Neural Information Processing Systems . 2021, pp. 24898-24911.

H. Qin, Y. Ding, M. Zhang, Q. Yan, A. Liu, Q. Dang, Z. Liu, and X. Liu. 'Bibert: Accurate fully binarized bert'. arXiv preprint arXiv:2203.06390. 2022.

H. Qin, R. Gong, X. Liu, M. Shen, Z. Wei, F. Yu, and J. Song. 'Forward and backward information retention for accurate binary neural networks'. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2020, pp. 2250-2259.

Y. Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsieh. 'Dynamicvit: Efficient vision transformers with dynamic token sparsification'. In: Advances in Neural Information Processing Systems . 2021, pp. 13937-13949.

M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. 'Xnor-net: Imagenet classification using binary convolutional neural networks'. In: European conference on computer vision . Springer. 2016, pp. 525-542.

H. Ren, H. Dai, Z. Dai, M. Yang, J. Leskovec, D. Schuurmans, and B. Dai. 'Combiner: Full attention transformer with sparse computation cost'. Advances in Neural Information Processing Systems (2021), pp. 22470-22482.

R. T. Rockafellar and R. J.-B. Wets. 'Variational Analysis'. Springer, 1998.

M. S. Ryoo, A. Piergiovanni, A. Arnab, M. Dehghani, and A. Angelova. 'TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?' arXiv preprint arXiv:2106.11297. 2021.

H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. 'Training dataefficient image transformers & distillation through attention'. In: International Conference on Machine Learning . 2021.

Z. Tu, X. Chen, P. Ren, and Y. Wang. 'Adabin: Improving binary neural networks with adaptive binary sets'. In: European conference on computer vision . Springer. 2022, pp. 379-395.

P. Wang, Q. Chen, X. He, and J. Cheng. 'Towards accurate post-training network quantization via bit-split and stitching'. In: International Conference on Machine Learning . 2020, pp. 9847-9856.

S. Xu, Y. Li, T. Ma, B. Zeng, B. Zhang, P. Gao, and J. Lu. 'TerViT: An Efficient Ternary Vision Transformer'. arXiv preprint arXiv:2201.08050. 2022.

Z. Xu, M. Lin, J. Liu, J. Chen, L. Shao, Y. Gao, Y. Tian, and R. Ji. 'Recu: Reviving the dead weights in binary neural networks'. In: Proceedings of the IEEE/CVF international conference on computer vision . 2021, pp. 5198-5208.

J. Zhang, H. Peng, K. Wu, M. Liu, B. Xiao, J. Fu, and L. Yuan. 'MiniViT: Compressing Vision Transformers with Weight Multiplexing'. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022, pp. 12145-12154.

X. Zhao, Y. Wang, X. Cai, C. Liu, and L. Zhang. 'Linear symmetric quantization of neural networks for low-precision integer hardware'. In: International Conference on Learning Representations . 2019.

Appendix for Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers

A More on Proximal Quantizers

Dockhorn et al. [13] gave a complete characterization of the proximal quantizer P r : a (multi-valued) mapping P is a proximal quantizer (of some underlying regularizer r ) iff it is monotone, compactvalued and with a closed graph. We now give a few examples to illustrate the ubiquity of proximal quantizers, as well as the generality of PC:

Identity function: apparently, choosing P µ t r as the identity function recovers the full precision training.

P µ t r = P Q : as Q = {± 1 } , this choice recovers exactly BC in (2).

P µ t r = L ϱ ρ : This is the general piecewise linear quantizer designed by Dockhorn et al. [13]. Recall that Q = { q k } 2 k =1 , where q 1 = -1 , q 2 = +1 , such that p 2 = 0 is the middle point. By introducing two parameters ρ, ϱ ≥ 0 , we can define two shifts:

Then, we define L ϱ ρ as the piece-wise linear map (that simply connects the points by straight lines):

For the middle points, L ϱ ρ ( w ∗ ) can be regarded as the intermediate state between the identity function and P Q such that, where L ϱ ρ ( w ∗ ) may take any value within the two limits. Note that ρ controls the discretization vicinity, such that in practice, ρ is linearly increased over time to fulfill binary weights in the end. We visualize examples of L ϱ ρ ( w ∗ ) in Figure 4.

-

1

1

-

1

1

p

-

2

p

+

2

w

∗

L

ϱ

ρ

(

w

∗

)

(a)

ρ

= 0

, ϱ

= 0

.

2

.

-

1

1

-

1

1

p

-

2

p

+

2

w

∗

L

ϱ

ρ

(

w

∗

)

(b)

ρ

=

ϱ

= 0

.

2

.

-

1

1

-

1

1

w

∗

L

ϱ

ρ

(

w

∗

)

(c)

ρ

= 0

.

2

, ϱ

= 0

.

B Related works

Vision Transformer. In computer vision, vision transformers have become one of the most popular backbone architectures. Dosovitskiy et al. [14] is the first to modify the transformer model to enable images as input, namely the ViT model. Specifically, Dosovitskiy et al. [14] translates an image to a sequence of flattened image patches as input, and applies a self-attention mechanism to retrieve patch-wise information in the feature representation. Touvron et al. [52] further equips ViT with knowledge distillation and proposes DeiT that generalizes well on smaller models and datasets. Liu et al. [34] further proposes Swin as a hierarchical vision transformer that computes representation with shifted windows.

In vision transformers, the main computation overhead is the multi-head attention layer, whose cost is quadratic with the length of the image patches. As a result, such models are in general expensive to train. To reduce the computational cost, different compression techniques have been explored. For instance, Pan et al. [44] performs dynamic pruning for less important patches; Bhojanapalli et al. [6] reuses attention scores computed for one layer in multiple building blocks; Hou and Kung [21] applies multi-dimensional model compression. In this paper, we focus on an alternative approach, namely network quantization.

Network Quantization. We consider two possible scenarios of network quantization:

(1) Post-training Quantization: We first discuss the easier post-training quantization methods. Such approaches usually quantize the full-precision pre-trained model and directly apply it for inference. Post-training quantization is widely used in CNNs [1, 3, 9, 29, 41, 42, 54]. Liu et al. [38] is the first to explore PTQ for vision transformers. It optimizes the quantization intervals and considers ranking information in the loss function. However, it only considers quantization to a 6-bit model without severe performance degradation. For lower-bit quantization, it is essential to leverage training.

(2) Quantization-Aware Training (QAT): different from post-training quantization, quantizationaware training leverage quantization during pre-training or fine-tuning. Thus it can be formulated as an optimization problem for learning the optimal quantized weights [5, 10, 15, 17, 23-25, 39, 58]. Compared with PTQ, QAT can obtain less accuracy drop in low-bit quantization compared to the full-precision model. Li et al. [30] and Xu et al. [55] demonstrate that QAT requires a unique design to quantize vision transformers and it is possible to perform quantization to 3 bit without severe performance degradation. [19] further performs binarization with softmax-aware binarization and information preservation.

Binarization Techniques: (1) Here we summarize existing binarization approaches that can be either justified or improved by PC++: Some existing implementations [31, 37, 46] set the forward quantizer as sign and design the backward quantizer in an ad hoc fashion (based on graphic approximation of the sign function), e.g., Liu et al. [37] applies a piece-wise polynomial approximation; Lin et al. [31] improves [37] with a dynamic polynomial approximation; Qin et al. [46] designs a dynamic error decay estimator based on the tanh function. These methods cannot be justified with PC++, but could be improved by designing new forward quantizers using our Corollary 2. We discuss these new variants in Appendix F and compare them with our methods. Other implementations [36, 53] applies shift transformation on both forward and backward quantizers, which belongs to the PC++ family.

(2) Architecture design that can be further integrated into our PC++ as future work: Xu et al. [56] designs a rectified clamp unit to address "dead weights"; Rastegari et al. [48] applies the absolute mean of weights and activations; Martinez et al. [40] uses real-to-binary attention matching and data-driven channel re-scaling; Kim et al. [26] proposes a shifted activation function.

(3) Optimization refinement, again could be integrated into our framework as future work: Liu et al. [33] utilizes state-aware gradient state; Liu et al. [35] provides a weight decay scheme; and Helwegen et al. [20] proposes Bop, a new optimizer for BNNs.

C Additional Theoretical Results

Corollary 1. A pair of forward-backward quantizers ( F , B ) admits the decomposition in (6) (for some smoothing parameter µ and regularizer r ) iff both F and B are functions of P ( w ) := ∫ w -∞ 1 B ( ω ) d F ( ω ) , which is proximal (i.e., monotone, compact-valued and with a closed graph).

Proof. We first recall the decomposition in (6):

Suppose first that ( F , B ) satisfies the above decomposition. Clearly, both F and B are functions of P = P µ r . Moreover,

and thus which is clearly proximal.

Conversely, let P ( w ) := ∫ w -∞ 1 B ( ω ) d F ( ω ) be proximal. Taking (generalized) derivative we obtain

Since B is a function of P , say B = T ′ ◦ P , performing integration we obtain

up to some immaterial constant (that can be absorbed into T ). Thus, ( F , B ) satisfies the decomposition (6).

The following convergence guarantee for PC++ follows directly from the results in Dockhorn et al. [13]:

Theorem 1. Fix any w , the iterates in (7) satisfy:

where ∆ τ ( w ) := r τ ( w ) -r τ ( w τ +1 ) -〈 w -w τ +1 , w ∗ τ +1 〉 is the Bregman divergence induced by the (possibly nonconvex) function r τ ( w ) := µ τ +1 · r ( w )+ 1 2 ∥ w ∥ 2 2 . (Recall that µ t := 1+ ∑ t -1 τ =1 η τ .)

The summand on the left-hand side of (11) is related to the duality gap, which is a natural measure of stationarity for the nonconvex problem (4). Indeed, it reduces to the familiar ones when convexity is present:

Theorem 2. For convex ℓ ◦ T and any w , the iterates in (7) satisfy:

If r is also convex, then

and

where w t = ∑ t τ = s η τ w τ ∑ t τ = s η τ , and f := ℓ ◦ T + r is the regularized and transformed objective.

The right-hand sides of (13) and (14) diminish iff η t → 0 and ∑ t η t = ∞ (assuming boundedness of the stochastic gradient). We note some trade-off in choosing the step size η τ : both the numerator and denominator of the right-hand sides of (13) and (14) are increasing w.r.t. η τ . The same conclusion can be drawn for (12) and (11), where ∆ τ also depends on η τ (through the accumulated magnitude of w ∗ τ +1 ).

D Additional Experimental Settings

Hardware and package: All experiments were run on a GPU cluster with NVIDIA V100 GPUs. The platform we use is PyTorch. Specifically, we apply ViT and DeiT models implemented in Pytorch Image Models (timm) 7 .

-

1

1

-

1

1

Bi-Real(forward)

-

1

1

-

1

1

-

1

1

-

1

1

R-BNN (forward)

-

1

1

-

1

1

-

1

1

-

1

1

Poly+ (forward)

-

1

1

-

1

1

-

1

1

-

1

1

EDE (forward)

-

1

1

-

1

1

-

1

1

-

1

1

EDE+ (forward)

-

1

1

-

1

1

-

1

1

-

1

1

ReActNet (forward)

-

1

1

-

1

1

Bi-Real(backward)

R-BNN (backward)

Poly+(backward)

EDE (backward)

EDE+ (backward)

ReActNet (backward)

Pre-trained models: In this work, we applied pre-trained full precision models for fine-tuning tasks. Here we specify the links to the models we used (note that we choose patch size equal to 16 across all models):

ViT-B (ImageNet-1K): https://storage.googleapis.com/vit_models/ augreg/B_16-i21k-300ep-lr_0.001-aug_medium1-wd_0.1-do_0.0-sd_0. 0--imagenet2012-steps_20k-lr_0.01-res_224.npz ;

ViT-B (ImageNet-21K): https://storage.googleapis.com/vit_models/augreg/ B_16-i21k-300ep-lr_0.001-aug_medium1-wd_0.1-do_0.0-sd_0.0.npz ;

DeiT-T (ImageNet-1K): https://dl.fbaipublicfiles.com/deit/deit_tiny_ patch16_224-a1311bcf.pth ;

DeiT-S (ImageNet-1K): https://dl.fbaipublicfiles.com/deit/deit_small_ patch16_224-cd65a155.pth ;

DeiT-B (ImageNet-1K): https://dl.fbaipublicfiles.com/deit/deit_base_ patch16_224-b5f2ef4d.pth .

E Comparison with BiViT

He et al. [19] propose BiViT, which considers the same binarization task on vision transformers (specifically, Swin-T and Nest-T). He et al. [19] follow a different implementation with softmaxaware binarization and information preservation. To fairly compare with this work, we follow the same setting and run PC++ on Swin-T and NesT-T on ImageNet-1K. We observe that BNN++ achieves 71.3% Top-1 accuracy (BiViT:70.8%) and 69.3% Top-1 accuracy (BiViT:68.7%) respectively on Swin-T and NesT-T. Note that BiViT simply applies BNN as the main algorithm and may be further improved with PC++ algorithms.

F Additional forward-backward quantizers

In this section, we summarize additional forward-backward quantizers, which can be either improved or justified with our PC++ framework. Specifically, we find that:

Bi-Real Net [37]/R-BNN [31]: F ( w ) = sign, B ( w ) = ∇ F ( w ) , where F ( w ) is a piewise polynomial function. We simply choose F = F ( w ) and arrive at our legitimate variant Poly+. Note that we gradually increase the coefficient of F ( w ) such that we ensure full binarization at the end of the training phase.

EDE in IR-Net [46]: F ( w ) = sign, B ( w ) = ∇ g ( w ) = kt (1 -tanh 2 ( t w )) , where k and t are control variables varying during the training process, such that g ( w ) ≈ sign at the end of training. Again, we choose F = g ( w ) and arrive at our new legitimate variant EDE+.

ReActNet [36] can be well justified and is a special case of PC++.

We visualize these forward-backward quantizers and our new variants in Figure 5. Moreover, we perform experiments on vision transformers to examine the performance of additional quantizers and their modified variants. We report the results in Table 6 and observe that (1)Our new proposed Poly+ and EDE+ always outperform the original algorithms and further confirm that our PC++ framework merits theoretical and empirical justifications; (2)BNN++ still outperforms other algorithms on all tasks.

G Additional results for end-to-end training

Finally, we provide the error bars for our main experiments in Table 7 and Table 8 for CNN backbones and vision transformer backbones, respectively.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper generalizes ProxConnect with forward-backward quantizers and introduces ProxConnect++ that includes some binarization techniques as special cases. With the derived ProxConnect++, the paper proposes BNN++ to illustrate the effectiveness of ProxConnect++. Experiments show the advantages of BNN++ on image classification benchmarks with CNNs and vision transformers.\n\nSTRENGTHS:\nThe paper generalizes ProxConnect and presents the unified framework ProxConnect++, which includes some binarization techniques as special cases. With ProxConnect++, one may design new forward-backward quantizers. Extensive experiments on CNNs and vision transformers demonstrate the advantages of the proposed ProxConnect++.\n\nWEAKNESSES:\nThe paper claims that it develops a unified framework ProxConnect++, which allows to design various forward-backward quantizers. However, the forward and backward quantizers used in the paper are limited to the existing ones. To show the advantages of ProxConnect++, it should design new forward and backward quantizers, which outperform the existing ones. The paper only gives an example BNN++, which is also inspired the existing forward and backward quantizers.\n\nQUESTIONS:\nThe paper presents a unified framework ProxConnect++. It is claimed that ProxConnect++ can include many binarization techniques as special cases. I have two concerns for the unified framework.\n\nFirst, it is desirable to present a unified framework. So one can analyze existing algorithms in a unified way. As a unified framework, the most important thing is to propose new superior algorithms following the framework. However, the paper does not give more insights into developing new algorithms. Note that BNN++ is also inspired by the exiting forward and backward quantizers. \n\nSecond, the paper compares with some variants of ProxConnect++ and shows that BNN++ performs the best. It is better to include other state-of-the-art binarization techniques in the experiments.\n\nLIMITATIONS:\nThe authors addressed the limitations of their work. There is no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThanks to authors for submitting their work to NeuRips 2023. After nicelly flowing introduction the lines 47-58 lay goals of the paper and its contributions out in the context of recent advances, cf. Dockhorn et al. [13], PC (ProxConnect). In particular, the paper generalizes PC to forward-backward binarization, as opposed to PC using only forward binarization, labelling this more general approach PC++. As a practical result of presented theory the fully binarized (8-bit integers) BNN++ is shown to be competitive against existing approaches on most tasks, including experiments on several datasets and architectures (incl. CNN and ViT). It achieves 30x reduction in memory and storage with a modest 5-10% accuracy drop compared to full precision training. Binarization results on vision transformers, rare up until now, is especially timely and promising given the recent deployment of the transformers in deep learning. Architecture ablation experiments conducted demonstrate practicality of the hereby developed theory and increase potential impact of the paper.\n\nBesides Theorem 1, being rather a straightforward extension of previous work Dockhorn et al. [13], including convergence guarantees applied on PC++ in appendix, I value authors used this general result to present a unifying and pragmatic framework for binarization of the NN in response to increasing energy demands for training and fine-tuning the recent large transformer models. I find paper timely and relevant contribution to research community.\n\nSTRENGTHS:\n+ $\\textbf{[Accessibility, Approach]}$ The goal of the paper is achieved in very accessible and neat form. Proving Theorem 1 (the main result) introduces a compelling and general framework for designing binarization algorithms for NNs.\n\n+ ProxConnect (3) ++  >>> nice idea of formally adding transformation T that “cancels out” (formally in the Theorem 1 sense) in backward step, so the original theory of PC, Dockhorn et al. [13], applies.\n\n+ $\\textbf{[Unifying framework]}$ anAdditional and very impactful merit of the paper is its unifying approach of currently quite scattered and uneasy to access research of neural networks binarization. Especially natural inclusion of the transformers is very timely, convenient and impactful addressed by only few previous works, e.g. Y. He et al. [19].\n\n+ $\\textbf{[Demonstration of practical usefulness]}$ (Architecture) Ablation experiments present practical usefulness of hereby developed theory (Theorem 1) (line: 283-284) and increasing the impact of the paper on further research.\n\nWEAKNESSES:\n- Is the extension of previous work of Dockhorn et al. [13] novel enough? As noted above in Summary and Strength sections, formally adding transformation $T$ that “cancels out” (formally in the Theorem 1 sense) in backward step, so the original theory of PC, Dockhorn et al. [13], applies is smart yet rather straightforward (also demonstrated in the proof in Appendix). Some may see this rather a direct consequence of the previous work, that should be fully contributed to the respective authors :-)... .Despite this caveat I believe the paper's focus on application of this theory outweighs the theoretical merits and yields a work with potential high impact on further advances in the field. I leave to authors without impacting my decision whether or not to recognise previous work of Dockhorn et al. [13] more and consider renaming the Theorem 1 to \"Corollary\" to refer directly to previous work. \n\n- Extend of experiments in terms of datasets and modes is well selected. Yet number of runs to avoid statistical error, especially in Tables $3$ and $4$, is insufficient and should be higher. See Questions section.\n\nQUESTIONS:\n- Would it be possible to Increase an extend of randomly seeded pool of experiments for Table 3,4? - reported averages on 3 runs together with rather diminishing differences between columns may not be significant. Because the results are used to draw performance claims, as on line 250, 251, “In particular, for end-to-end training, the best performing ProxConnect++ algorithms achieve ≈ 5% accuracy drop”,  the extended experiments are suggested to be reported in camera ready version. Especially as it concerns ViT, treated rarely before and thus of high impact and interest to community.\n\n- The Figure 2., p.8, presents well why more involved techniques that post-training binarization (PTB) are needed. Would it be worth to use it in the beginning of the paper as a motivation figure? Accompanied with similar figure of memory and storage requirements, perhaps.\n\n- What does Table 2 show? (accuracy?) While it is commented in the text it is not clear from Table captions. I recommend to factor it in for the sake of higher clarity.\n\n- Line 136: “proved” —> “proven”\n\nLIMITATIONS:\nSuggestion: For a camera ready version notes on limitation of the method are suggested to be brought back to main body of the paper from the Appendix G.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper proposes a new framework for training neural networks with binary weights, which generalizes ProxConnect and takes it as a special case. In the new framework, forward and backward quantizers are defined. A consistency result of the two quanziters is derived (Theorem 1). Extensive experiments are conducted to evaluate the usefulness of the new framework.

STRENGTHS:
1. The paper is very well-written and easy to the follow. I love the clarity of the paper.
2. The theoretical framework is sensible and the analysis is rigorous. I did not find a technical flaw.
3. The empirical evaluation is sufficient. The code is provided for reproduction.

WEAKNESSES:
1. The proposed framework seems a direct generalization of ProxConnect. Thus, it's novelty is somewhat limited.
2. It is not straightforward for practitioners how to use the framework to design new proximal quantizers. Please provide practical guidelines and tips.

QUESTIONS:
See above.

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied binary neural networks which extend the existing theory of ProxConnect(PC) to ProxConnect++ and explored the fully binarized scenario, where the dot-product accumulators are also quantized to 8-bit integers. The authors also proposed BNN++ with non-linear forward and backward approximation to the sign function. Authors did experiments on CIFAR10 and ImageNet datasets.\n\nSTRENGTHS:\nBinarizing weights and activations in neural networks is a challenging problem worth studying. This paper proposed an in-depth study of forward and backward functions for binary neural networks and proposed a new method that performs outstanding accuracy on CIFAR10 and ImageNet datasets.\n\nWEAKNESSES:\nThe authors only compared their method with several BNN methods. But many classic and famous BNN works are not even mentioned, such as XNOR-Net. And some papers studying the backward approximation are missing, e.g., the second-order approximation to the sign function proposed in Bi-Real Net should be compared in Fig.1. I would suggest authors compare or at least mention the binary neural networks to make the paper more comprehensive.\n\nTo name a few: \n\n[1] Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. (CVPR)\n\n[2] Zhijun Tu, Xinghao Chen, Pengju Ren, and Yunhe Wang. Adabin: Improving binary neural networks with adaptive binary sets. (ECCV)\n\n[3] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. (ECCV)\n\n[4] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. (ECCV)\n\n[5] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. (CVPR)\n\n[6] Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binary convolutions. (ICLR)\n\n[7] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. (ECCV)\n\n[8] Chunlei Liu, Peng Chen, Bohan Zhuang, Chunhua Shen, Baochang Zhang, and Wenrui Ding. Sa-bnn: State-aware binary neural network. (AAAI)\n\n[9] Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Yan Wang, Yongjian Wu, Feiyue Huang, and Chia-Wen Lin. Rotated binary neural network. (NeurIPS) \n\n[10] Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang-Ting Cheng. How do adam and training strategies help bnns optimization. (ICML)\n\n[11] Hyungjun Kim, Jihoon Park, Changhun Lee, and Jae-Joon Kim. Improving accuracy of binary  neural networks using unbalanced activation distribution. (CVPR)\n\n[12] Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang-Ting Cheng, and Roeland Nusselder. Latent weights do not exist: Rethinking binarized neural network optimization. (NeurIPS)\n\nQUESTIONS:\nThe authors use the non-linear approximation of the sign function in the forward pass, will that yield other real-valued outputs besides the binary values?\n\nLine 170 is confusing: “BNN++ is more desirable than BNN+ empirically.” The author means BNN++ is better than PC++?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}