SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} \left[ f (\bar {\mathbf {w}} _ {t}) - f (\mathbf {w}) \right] \leq \frac {1}{\sum_ {\tau = s} ^ {t} \eta_ {\tau}} \cdot \mathbb {E} \left[ \Delta_ {s - 1} (\mathbf {w}) + \sum_ {\tau = s} ^ {t} \frac {\eta_ {\tau} ^ {2}}{2} \| \widetilde {\nabla} \ell (\mathbf {w} _ {\tau}) \| _ {2} ^ {2} \right], \tag {14}
$$
where  $\mathbf{w}_t = \frac{\sum_{\tau=s}^t \eta_\tau \mathbf{w}_\tau}{\sum_{\tau=s}^t \eta_\tau}$ , and  $f := \ell \circ \mathsf{T} + \mathsf{r}$  is the regularized and transformed objective.
The right-hand sides of (13) and (14) diminish iff  $\eta_t \to 0$  and  $\sum_{t} \eta_t = \infty$  (assuming boundedness of the stochastic gradient). We note some trade-off in choosing the step size  $\eta_{\tau}$ : both the numerator and denominator of the right-hand sides of (13) and (14) are increasing w.r.t.  $\eta_{\tau}$ . The same conclusion can be drawn for (12) and (11), where  $\Delta_{\tau}$  also depends on  $\eta_{\tau}$  (through the accumulated magnitude of  $\mathbf{w}_{\tau + 1}^{*}$ ).
D Additional Experimental Settings
Hardware and package: All experiments were run on a GPU cluster with NVIDIA V100 GPUs. The platform we use is PyTorch. Specifically, we apply ViT and DeiT models implemented in Pytorch Image Models (timm) 7.
<sup>7</sup>https://timm.fast.ai/
Figure 5: Forward and backward pass for 6 additional ProxConnect++ algorithms.
Table 6: Results for additional ProxConnect++ algorithms on binarizing vision transformers (binarizing weights only), where the results are the mean of five runs with different random seeds.
Model	Dataset	FP	ProxConnect++
Bi-Real	R-BNN	Poly+	EDE	EDE+	ReAct	BNN++
DeiT-T	CIFAR-10	94.85%	84.11%	84.54%	85.31%	84.99%	85.57%	85.35%	86.41%
CIFAR-100	72.37%	59.01%	59.02%	60.00%	59.32%	60.04%	60.11%	60.33%
ImageNet-1K	72.20%	64.55%	64.59%	64.97%	64.28%	65.01%	65.37%	67.34%
DeiT-S	CIFAR-10	95.09%	85.01%	86.07%	84.99%	85.37%	85.33%	85.91%	86.19%
CIFAR-100	73.19%	59.66%	59.75%	60.14%	60.09%	61.17%	61.09%	62.98%
ImageNet-1K	79.90%	70.51%	70.87%	71.36%	70.66%	72.99%	72.53%	73.53%
Pre-trained models: In this work, we applied pre-trained full precision models for fine-tuning tasks. Here we specify the links to the models we used (note that we choose patch size equal to 16 across all models):
ViT-B (ImageNet-1K): https://storage.googleapis.com/vit_models/augreg/B_16-i21k-300ep-lr_0.001-aug_medium1-wd_0.1-do_0.0-sd_0.0--Imagenet2012-steps_20k-lr_0.01-res_224.npz;
ViT-B (ImageNet-21K): https://storage.googleapis.com/vit_models/augreg/B_16-i21k-300ep-lr_0.001-aug_medium1-wd_0.1-do_0.0-sd_0.0.npz;
- DeiT-T (ImageNet-1K): https://dl.fbaipublicfiles.com/deit/deit_tiny_batch16_224-a1311bcf.pth;
- DeiT-S (ImageNet-1K): https://dl.fbaipublicfiles.com/deit/deit_smallpatch16_224-cd65a155.pth;
- DeiT-B (ImageNet-1K): https://dl.fbaipublicfiles.com/deit/deit_basepatch16_224-b5f2ef4d.pth.
E Comparison with BiViT
He et al. [19] propose BiViT, which considers the same binarization task on vision transformers (specifically, Swin-T and Nest-T). He et al. [19] follow a different implementation with softmax-aware binarization and information preservation. To fairly compare with this work, we follow the same setting and run PC++ on Swin-T and NesT-T on ImageNet-1K. We observe that BNN++ achieves 71.3% Top-1 accuracy (BiViT:70.8%) and 69.3% Top-1 accuracy (BiViT:68.7%) respectively on Swin-T and NesT-T. Note that BiViT simply applies BNN as the main algorithm and may be further improved with PC++ algorithms.
F Additional forward-backward quantizers
In this section, we summarize additional forward-backward quantizers, which can be either improved or justified with our PC++ framework. Specifically, we find that:
Table 7: Error bar for binarizing weights (BW), binarizing weights and activation (BWA) and binarizing weights, activation, with 8-bit accumulators (BWAA) on CNN backbones. We consider the end-to-end (E2E) pipeline. We compare five variants of ProxConnect++ (BC, PC, BNN, BNN+, and BNN++) with FP, PQ, and rPC. For the end-to-end pipeline, we report the mean of five runs with different random seeds.
Dataset	Task	FP	PQ	rPC	ProxConnect++
BC	PC	BNN	BNN+	BNN++
CIFAR-10	BW	92.01% ±0.19	81.59% ±0.11	81.82% ±0.16	87.51% ±0.07	88.05% ±0.05	89.92% ±0.11	89.39% ±0.13	90.03% ±0.06
BWA	92.01% ±0.13	81.51% ±0.16	81.60% ±0.09	86.99% ±0.11	87.26% ±0.23	89.15% ±0.08	89.02% ±0.16	89.91% ±0.09
ImageNet-1K	BW	78.87% ±0.06	63.23% ±0.11	66.39% ±0.22	67.45% ±0.04	67.51% ±0.09	67.49% ±0.12	66.99% ±0.26	68.11% ±0.02
BWA	78.87% ±0.18	61.19% ±0.22	64.17% ±0.19	65.42% ±0.22	65.31% ±0.17	65.29% ±0.21	65.98% ±0.15	66.08% ±0.13
Table 8: Error bar on binarizing vision transformers (BW and BWA) on ImageNet-1K. We consider the end-to-end (E2E) pipeline. We compare five variants of ProxConnect++ (BC, PC, BNN, BNN+, and BNN++) with FP, PQ, and rPC. The results are the mean of five runs with different random seeds.
Model	Task	FP	PQ	rPC	ProxConnect++
BC	PC	BNN	BNN+	BNN++
DeiT-T	BW	72.20% ±0.11	61.23% ±0.07	60.35% ±0.19	63.22% ±0.21	66.15% ±0.11	65.00% ±0.15	66.67% ±0.09	67.34% ±0.07
BWA	72.20% ±0.13	60.01% ±0.12	58.77% ±0.08	62.13% ±0.06	65.29% ±0.19	63.75% ±0.18	65.29% ±0.06	65.65% ±0.03
DeiT-S	BW	79.91% ±0.21	69.87% ±0.26	68.74% ±0.16	73.16% ±0.19	73.51% ±0.22	73.77% ±0.08	73.23% ±0.11	73.53% ±0.13
DeiT-B	BW	81.81% ±0.17	72.54% ±0.15	70.11% ±0.23	76.55% ±0.07	76.61% ±0.24	75.60% ±0.17	76.63% ±0.13	76.74% ±0.07
- Bi-Real Net [37]/R-BNN [31]:  $\mathsf{F}(\mathbf{w}) = \mathrm{sign}, \mathsf{B}(\mathbf{w}) = \nabla F(\mathbf{w})$ , where  $F(\mathbf{w})$  is a piewise polynomial function. We simply choose  $\mathsf{F} = F(\mathbf{w})$  and arrive at our legitimate variant Poly+. Note that we gradually increase the coefficient of  $F(\mathbf{w})$  such that we ensure full binarization at the end of the training phase.
- EDE in IR-Net [46]:  $\mathsf{F}(\mathbf{w}) = \mathrm{sign}, \mathsf{B}(\mathbf{w}) = \nabla g(\mathbf{w}) = kt(1 - \tanh^2 (tw))$ , where  $k$  and  $t$  are control variables varying during the training process, such that  $g(\mathbf{w}) \approx \mathrm{sign}$  at the end of training. Again, we choose  $\mathsf{F} = g(\mathbf{w})$  and arrive at our new legitimate variant EDE+.
- ReActNet [36] can be well justified and is a special case of  $\mathrm{PC} + +$ .
We visualize these forward-backward quantizers and our new variants in Figure 5. Moreover, we perform experiments on vision transformers to examine the performance of additional quantizers and their modified variants. We report the results in Table 6 and observe that (1)Our new proposed Poly+ and EDE+ always outperform the original algorithms and further confirm that our PC++ framework merits theoretical and empirical justifications; (2)BNN++ still outperforms other algorithms on all tasks.
G Additional results for end-to-end training
Finally, we provide the error bars for our main experiments in Table 7 and Table 8 for CNN backbones and vision transformer backbones, respectively.