SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Comparing the proposed architecture LiResNet using the proposed loss EMMA against baseline methods. Verifiably Robust Accuracy (VRA) is the percentage of test points on which the model is both correct and certifiably robust so a higher VRA is better. An  $\ell_2$ -ball with a radius of  $\epsilon = 36/255$  is used for all experiments following the conventions in the literature. “+DDPM” means we use DDPM generators [36] to augment the training set (no external data used).
Dataset	Method	Architecture	Specification	#Param. (M)	Clean (%)	VRA (%)
CIFAR-10	BCOP [33]	ConvNet	6C2F	2	75.1	58.3
GloRo [32]	ConvNet	6C2F	2	77.0	60.0
Local-Lip Net [24]	ConvNet	6C2F	2	77.4	60.7
Cayley [48]	ConvNet	4C3F	3	75.3	59.2
SOC (HH+CR) [43]	ConvNet	LipConv-20	27	76.4	63.0
CPL [35]	ResNet	XL	236	78.5	64.4
SLL [2]	ResNet	XL	236	73.3	65.8
GloRo + EMMA (ours)	LiResNet	6L128W	5	78.7	64.4
GloRo + EMMA (ours)	LiResNet	12L512W	49	81.3	66.9
+ DDPM	LiResNet	12L512W	49	82.1	70.1
CIFAR-100	BCOP [33]	ConvNet	6C2F	2	45.4	31.7
Cayley [48]	ConvNet	4C3F	3	45.8	31.9
SOC (HH+CR) [43]	ConvNet	LipConv-20	27	47.8	34.8
LOT [54]	ConvNet	LipConv-20	27	49.2	35.5
SLL [2]	ResNet	XL	236	46.5	36.5
GloRo + EMMA (ours)	LiResNet	6L128W	5	52.1	36.3
GloRo + EMMA (ours)	LiResNet	12L512W	49	55.2	38.3
+ DDPM	LiResNet	12L512W	49	55.5	41.5
Tiny-ImageNet	GloRo [32]	ConvNet	8C2F	2	35.5	22.4
Local-Lip Net [24]	ConvNet	8C2F	2	36.9	23.4
SLL [2]	ResNet	XL	236	32.1	23.2
GloRo + EMMA (ours)	LiResNet	6L128W	5	40.8	29.0
GloRo + EMMA (ours)	LiResNet	12L512W	49	44.6	30.6
+ DDPM	LiResNet	12L512W	49	46.7	33.6
ImageNet	GloRo + EMMA (ours)	LiResNet	12L588W	86	45.6	35.0
predicts class  $i$  over the ground truth class  $y$  (i.e.,  $f_{y}(x) < f_{i}(x)$  thus  $\kappa_{i}(x) < 0$ ), we clip  $\tilde{\epsilon}_{i}$  to 0 and EMMA loss reduces to a non-robust cross-entropy loss (for class  $i$ ). In this case, the training focuses on improving clean accuracy first. As the training progresses,  $\tilde{\epsilon}_{i}$  may grow to the same magnitude as the expected radius  $\epsilon$ . Once this happens, we shift the model's focus towards certifying other instances, rather than continuously increasing the robust radius for  $x$ .
As an implementation detail, we need to stop the gradient back-propagation from  $\tilde{\epsilon}_i$ , i.e., we treat it as a constant during optimization. Otherwise, the denominator term  $f_{i}(x) + \tilde{\epsilon}_{i}K_{yi}$  is just  $f_{y}(x)$  and the logit for class  $i$  is removed from the computation graph of the loss.
Notice that EMMA introduces a dynamic adjustment to the robust radius using  $\tilde{\epsilon}_i$ . This allows the strength of the robustness regularization to adapt in accordance with the training progress. Specifically, EMMA initially encourages the model to make correct predictions when  $\tilde{\epsilon}_i = 0$ , then it gradually shifts the model towards making more robust predictions as  $\tilde{\epsilon}_i \to \epsilon$ . The idea of balancing the standard accuracy and robustness perhaps also agrees with the design of TRADES (for adversarial training) [59] and its GloRo counterpart [32]. Further discussion on this point is included in Appendix A.
Notice also that the proposed EMMA loss only affects the training step, so we still use Equation 1 to construct the  $\perp$  logit to certify the local robustness during inference, as proposed by Leino et al. [32].
In this section, we provide an empirical evaluation of LiResNet and EMMA loss in comparison to certifiably robust training approaches in prior works. We begin by comparing the best VRAs we achieve against the best VRAs reported in the literature in Section 5.1. Next, in Section 5.2, we run head-to-head comparisons between EMMA and the standard GloRo losses as an ablation study to measure the empirical benefits of EMMA loss. Section 5.3 presents experiments on networks of different depths, to shed light on the unique depth scalability of the LiResNet architecture. In
Appendix F, we dive deeper into the ImageNet results, providing another novel observation regarding the impact of the number of classes on VRA.
The models in our evaluation either follow the architecture choices of prior work, or belong to a family of LiResNet architectures shown in Figure 2a. Because the stem, neck, and head are essentially fixed in our LiResNet architectures, we refer to a particular architecture variation by the number of backbone blocks,  $\mathbf{L}$ , and the number of input/output channels,  $\mathbf{W}$ , in convolution layers of each backbone. For example, 6L128W refers to a LiResNet with 6 linear residual blocks with 128 channels each. Other architecture specifications can be found in Appendix B.
5.1 Improved VRA with LiResNet
We compare GloRo LiResNets trained with EMMA loss with the following baselines from the literature: GloRo Nets with ConvNet architecture and cross-entropy/TRADES loss [32], BCOP [33], Cayley [48], Local-Lip Net [24], and SOC with Householder and Certification Regularization (HH+CR) [43], CPL [35] and SLL [2], which are selected for having been shown to surpass other approaches. We experiment on CIFAR-10/100 and Tiny-ImageNet using  $\ell_2$  perturbations with  $\epsilon = 36 / 255$ , the standard datasets and radii used in prior work. Additionally, we demonstrate the scalability of GloRo LiResNets by training on and certifying ImageNet. We report the clean accuracy, VRA, and the model size (by # of parameters) in Table 1. Further details on training are in Appendix B.
On CIFAR-10/100 and Tiny-/ImageNet, we find that a small LiResNet, 6L128W, is able to match or outperform all baselines. This architecture is about  $40 \times$  smaller than the largest $^{4}$  and highest-performing baseline model (SLL), but comes close to the same accuracy on CIFAR-10/100, and outperforms on Tiny-Imagenet. Our larger LiResNet (12L512W) surpasses all prior approaches, setting the state-of-the-are VRAs (without DDPM augmentation) to  $66.9\%$ ,  $38.3\%$  and  $30.6\%$  on CIFAR-10, CIFAR-100 and Tiny-/ImageNet, respectively.
Recent work on promoting empirical robustness has found that augmenting the training set with examples generated by Denoising Diffusion Probabilistic Models (DDPMs) [23, 36] can further boost the empirical robust accuracy [20]. Thus, we also experiment to see if the same success can be found for certifiable robustness. DDPMs are trained with the training set, so the generated images do not leak information of the test set and do not use external data. Additional rows with “+DDPM” in Table 1 show that with this augmentation method we further improve VRAs to  $70.1\%$ ,  $41.5\%$  and  $33.6\%$  on CIFAR-10, CIFAR-100 and Tiny ImageNet, respectively (further details in Appendix H). Because diminishing returns have been observed for improving empirical robustness when augmenting the training set of ImageNet with DDPM data [20], we did not use DDPM for LiResNet on ImageNet.
Scaling to ImageNet with GloRo. There is no theoretical limitation to the other approaches considered in our evaluation that limit us from training them on ImageNet; however, practical resource constraints prevent us from training until convergence with non-GloRo approaches. For example, baselines using orthogonalized kernels—e.g., Cayley, BCOP, and SOC—do not easily fit into memory with  $224 \times 224 \times 3$  images, and local Lipschitz computation—e.g., Local-Lip Net—is both time and memory intensive. To the best of our knowledge, we are the first to report the VRA on ImageNet with a deterministic robustness guarantee and the first to calculate that VRA using the entire validation set, in contrast to the prior work which only evaluates  $1\%$  [25, 39, 40] or  $2\%$  [5] of the validation set using Randomized Smoothing [6] (also because of efficiency concerns).
5.2 Improved VRA with EMMA
This section demonstrates the empirical gains obtained by switching the typical loss used with standard GloRo losses—i.e., GloRo cross-entropy (CE) and GloRo TRADES—to EMMA. For both ConvNets and LiResNets, we experiment on CIFAR-10, CIFAR-100 and Tiny-ImageNet and report
<sup>4</sup>We observe that the SLL XL models have orders-of-magnitude more parameters than the other baselines, coming from a non-standard architecture with many wide dense layers. When we use the 12L512W architecture but with the LiResNet layers replaced with SLL layers, we find the performance is much lower (59.5% VRA on CIFAR-10 and 30.2% VRA on CIFAR-100), suggesting that the primary factor in performance for SLL XL is the large number of parameters rather than the approach to bounding the Lipschitz constant of the residual layers. Capacity has been shown to be an important factor for obtaining high VRA with Lipschitz-based approaches [30].
Architecture	dataset	TRADES	EMMA
ConvNet	CIFAR-10	58.8	59.2
CIFAR-100	34.0	35.0
TinyImageNet	26.6	27.4
LiResNet	CIFAR-10	66.2	66.3
CIFAR-100	37.3	37.8
TinyImageNet	28.8	30.3
(a)
Dataset	L	ConvNet	ResNet	LiResNet
CIFAR-10	6	64.0	60.3	65.5
12	59.2	60.0	66.3
18	×	60.1	66.6
CIFAR-100	6	36.5	33.5	37.2
12	35.0	33.5	37.8
18	×	33.6	38.0
Table 2: (a) VRA performance  $(\%)$  of a ConvNet and a LiResNet on three datasets with different loss functions. (b) VRA  $(\%)$  performance on CIFAR-10/100 with different architectures ( $L$  is the number of blocks in the model backbone). We use EMMA loss for Gloro training. All models in this table use 256 channels in the backbone. A value of  $\times$  indicates that training was unable to converge.
VRAs in Table 2a. The ConvNets we use are modified from the baseline 6C2F architecture used by Leino et al. [32] to have the same (wider) channel width as our 6L256W architecture, which is why we achieve higher VRAs than originally reported in [32]. The remaining implementation details and the clean accuracy of each model can be found in Appendix C.
We see in Table 2a that the performance gain from switching TRADES to EMMA becomes clear when the number of classes increases from 10 (CIFAR-10) to 200 (Tiny-Imagenet). This observation aligns with our hypothesis used to motivate EMMA loss, discussed in Section 4, namely, that the rotating threatening class phenomenon observed during training (see Figure 2b) may contribute to suboptimal learning.
5.3 Going Deeper with LiResNet
As depicted in Figure 1 from the introduction, the VRA obtained using GloRo LiResNets scales well as the depth of the model increases, while prior work has failed to further improve the best achievable VRA through additional layers. To further validate that the ability to successfully go deeper primarily comes from the structural improvement of linear residual branch in the LiResNet architecture—as opposed to being an advantage of the framework, GloRo, itself—we run head-to-head comparisons on CIFAR-10 and CIFAR-100 of GloRo Nets using (1) a feed-forward ConvNet architecture, (2) a conventional ResNet architecture, and (3) a LiResNet architecture. We train all three architectures with EMMA loss at three different depths. We report VRAs of these models in Table 2b (implementation details are given in Appendix D).
We see in Table 2b that very deep ConvNets may not be able to converge even on small-scale datasets like CIFAR-10 and CIFAR-100. Moreover, the VRA of both ConvNets and conventional ResNets do not benefit from the increasing network depth—in fact performance decreases as the network is made significantly deeper. By contrast, LiResNet is the only architecture under the same conditions that benefits from more layers, showing its unique promise for scalability. In Appendix D, we include more results with even deeper LiResNets on CIFAR-10, CIFAR-100 and Tiny-ImageNet.
6 Related Work and Discussion
Tightening Lipschitz Bound. Work closely related to enhancing VRA with architectural redesign includes the use of orthogonalized convolutions [2, 35, 42, 48, 54], which are 1-Lipschitz by construction. In a similar vein, we introduce the linear residual branch to solve the overestimation of Lipschitz Constant in the conventional ResNet. Our linear residual layer compares favorably to orthogonalized layers with a few key advantages. Specifically, although there is no technical limitation that would prevent us from using orthogonalized convolutions in LiResNets, GloRo regularization performs better in our experiments, and is significantly less expensive than training with orthogonalized kernels.
Model Re-parameterization. The LiResNet block is re-parameterization of a convolution layer, to make it trainable at large depth. Prior to this work, there are several studies use the same technique: DiracNet [58], ACB [11], RepVGG [12], RepMLP [13], etc. Our work has different motivations
from these works, which use re-parameterization to achieve better optimization loss landscapes. Our work follows a similar approach to obtain tighter estimates of Lipschitz bounds.
Towards Certifying Transformers. Although our current experiments (see Appendix I) indicate that Vision Transformers—utilizing linear residual blocks and Lipschitz-attention [9, 28]—do not yield VRAs comparable to those of LiResNets, we conjecture this discrepancy primarily arises from the loose Lipschitz upper-bounds inherent to most Lipschitz-attention mechanisms, in contrast to the exact Lipschitz constants of convolutions in LiResNets. To fully harness the potential of the linear residual block, future work should focus on tightening the Lipschitz bound for Lipschitz-attention.
Randomized Smoothing. As opposed to the deterministic robustness guarantee focused on in this work, probabilistic guarantees, based on Raondmized Smoothing (RS) [6], have been long studied at ImageNet-scale [5, 25, 39, 40]. Despite RS's reported higher certification results on ImageNet compared to GloRo LiResNet for the same  $\epsilon$ , it has two primary limitations. First, RS can provide false positive results, where adversarial examples are given robustness certificates. This, in many real-world security and financial contexts, can be untenable, even at a  $0.1\%$  false positive rate (FPR). Additionally, the computational cost of RS-based certification is orders of magnitude higher than Lipschitz-based certification—to certify one instance with FPR=0.1%, RS requires 100,000 extra inferences, and this number increases exponentially for lower FPR [6]. This resource-consuming nature of RS-based certification limits the type of applications one can deploy it in. Even in academic research, methods on certifying ImageNet with RS only report results using  $1\%$  of the validation images (i.e. 500 images) [25, 39, 40]; however, with the Lipschitz-based approach employed in our work, we can certify the entire validation set (50,000 images) in less than one minute.
Limitations for Robust Classification. The development of EMMA and empirical studies on class numbers' impact on VRA highlight the challenge of scaling robustness certification to real-world datasets. One challenge is that an increasing number of classes places increasing difficulty on improving VRA; we provide more details on this in Appendix F. In addition, ambiguous or conflicting labels can further exacerbate the issue, indicating a possible mismatch between the data distribution and the objective of categorical accuracy, especially in its robust form. A number of unlabeled images have been identified in ImageNet [3, 37, 50, 57], raising obstacles for robust classification. In addition, real-world images often contain multiple semantically meaningful objects, making robust single-label assignment problematic and limiting the network's learning potential unless alternate objectives (e.g., robust top- $k$  accuracy [31] and robust segmentation [17]) are pursued.
In this work, we propose a new residual architecture, LiResNet, for training certifiably robust neural networks. The residual blocks of LiResNet admit tight Lipsshitz bounds, allowing the model to scale much deeper without over-regularization. To stabilize robust training on deep networks, we introduce Efficient Margin Maximization (EMMA), a loss function that simultaneously penalizes worst-case adversarial examples from all classes. Combining the two improvements with GloRo training, we achieve new state-of-the-art robust accuracy on CIFAR-10/100 and Tiny-ImageNet under  $\ell_2$ -norm-bounded perturbations. Furthermore, our work is the first to scale up deterministic robustness guarantees to ImageNet, showing the potential for large scale deterministic certification.
Machine learning's vulnerability to adversarial manipulation can have profound societal implications, especially in applications where robust and reliable AI systems are paramount. This work is an important step towards increasing the robustness of neural networks to adversarial attacks, and more broadly, towards ensuring the trustworthiness of AI systems. Our ability to scale up fast deterministic robustness guarantees to ImageNet—a dataset more reflective of the complexity and diversity of real-world images—indicates that our approach to robust learning can be applied to practical, real-world applications. Nevertheless, while these advancements are promising, they also emphasize the need for ongoing vigilance and research in the face of increasingly sophisticated adversarial attacks. Ensuring that AI systems are robust and trustworthy will remain a critical task as these technologies continue to permeate society.
The work described in this paper has been supported by the Software Engineering Institute under its FFRDC Contract No. FA8702-15-D-0002 with the U.S. Department of Defense, as well as DARPA and the Air Force Research Laboratory under agreement number FA8750-15-2-0277.
[46] Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. In ICLR, 2019.
[47] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[48] Asher Trockman and J Zico Kolter. Orthogonalizing convolutional layers with the cayley transform. In ICLR, 2021.
[49] Yusuke Tsuzuki, Issei Sato, and Masashi Sugiyama. Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. In NIPS, 2018.
[50] Vijay Vasudevan, Benjamin Caine, Raphael Gontijo-Lopes, Sara Fridovich-Keil, and Rebecca Roelofs. When does dough become a bagel? analyzing the remaining mistakes on imagenet. arXiv preprint arXiv:2205.04596, 2022.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, 2017.
[52] Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In ICML, 2018.
[53] Xiaojun Xu, Linyi Li, Yu Cheng, Subhabrata Mukherjee, Ahmed Hassan Awadallah, and Bo Li. Certifiably robust transformers with 1-lipschitz self-attention.
[54] Xiaojun Xu, Linyi Li, and Bo Li. Lot: Layer-wise orthogonal training on improving 12 certified robustness. arXiv preprint arXiv:2210.11620, 2022.
[55] Yao-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R. Salakhutdinov, and Kamalika Chaudhuri. A closer look at accuracy vs. robustness. In NIPS, 2020.
[56] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In CVPR, 2022.
[57] Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. Re-labelingImagenet: From single to multi-labels, from global to localized labels. In CVPR, 2021.
[58] Sergey Zagoruyko and Nikos Komodakis. Diracnets: Training very deep neural networks without skip-connections. arXiv preprint arXiv:1706.00388, 2017.
[59] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
[60] Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. *ICLR*, 2019.
[61] Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton. Lookahead optimizer: k steps forward, 1 step back. NIPS, 2019.
A Discussions on EMMA Loss
One important aspect of EMMA loss is the dynamic adjustment to the robustness radius for each class when determining the loss. The robust radius used in EMMA is  $\tilde{\epsilon}$  instead of the radius  $\epsilon$  that is used at run time. By the definition of EMMA,  $0 \leq \tilde{\epsilon} \leq \epsilon$  and  $\tilde{\epsilon}$  is 0 if the model is not labeling the input correctly.  $\tilde{\epsilon}$  grows as the model becomes more robust at the corresponding input. As a result, when the model is not sufficiently robust at the input, EMMA uses  $\tilde{\epsilon} < \epsilon$  and imposes milder Lipschitz regularization. On the other hand, if using a fixed margin, the loss function turns out to be:
$$
\ell_ {\text {f i x e d}} = - \log \frac {f _ {y} (x)}{\sum_ {i} f _ {i} (x) + \epsilon K _ {y i}} \tag {3}
$$
This bears similarities to Lipschitz Margin loss, which has been used in prior work for certified training [49], although Lipschitz Margin loss typically uses  $\sqrt{2}\epsilon K$ , which gives a looser approximation than directly using the Lipschitz constant for each margin, i.e.,  $K_{yi}$ . The fixed margin loss  $\ell_{\mathrm{fixed}}$  in Equation 3 penalizes the Lipschitz-adjusted margin between the ground truth class and all other classes. Therefore, this loss function imposes a stronger regularization on the Lipschitz constant of the model than EMMA loss, and limits the model capacity more. We find that models trained with the fixed margin loss require weaker data augmentation or smaller training  $\epsilon$  to avoid underfitting. However, this can instead lead to robust overfitting. The gap between clean accuracy and VRA is notably higher for models trained with the fixed margin loss, and the overall performance is worse compared to models trained with the dynamic margin loss, i.e., EMMA loss.
B Implementation Details for Table 1
B.1 Training details
Dataset details The input resolution is 32 for CIFAR10/100, 64 for Tiny-ImageNet and 224 for ImageNet respectively. We apply the following data augmentation to CIFAR datasets: random cropping, RandAugment [8], random horizontal flipping. For Tiny-ImageNet, we find this dataset is easy to overfit and add an extra Cutout [10] augmentation. For data augmentation hyper-parameters, we use the default PyTorch setting.
Platform details Our experiments were conducted on an 8-GPU (Nvidia A100) machine with 64 CPUs (Intel Xeon Gold 6248R). Each experiment on CIFAR10/100 and Tiny-ImageNet takes one GPU and each experiment on ImageNet takes 8 GPUs. Our implementation is based on PyTorch [38].
Training details On the first 3 datasets, all models are trained with the NAdam [15] with the Lookahead optimizer wrapper [61] with a batch size of 256 and a learning rate of  $10^{-3}$  for 800 epochs. We use a cosine learning rate decay [34] with linear warmup [21] in the first 20 epochs. On ImageNet, we only change the batch size to 1024 and training epochs to 400.
During training, we schedule the training  $\epsilon$  to ramp up from small values and slightly overshoot the test epsilon. Let the total number of epochs be  $T$  and the test certification radius be  $\epsilon$ , we use
$$
\epsilon_ {\text {t r a i n}} (t) = \left(\min  \left(\frac {2 t}{T}, 1\right) \times 1. 9 + 0. 1\right) \epsilon , \quad \epsilon = 3 6 / 2 5 5.
$$
at epoch  $t$ . As a result,  $\epsilon_{\mathrm{train}}(t)$  begins at  $0.1\epsilon$  and increases linearly to  $2\epsilon$  before arriving halfway through the training. Later,  $\epsilon_{\mathrm{train}}$  remains  $2\epsilon$  to the end.
B.2 Model architecture details