SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Model stem is used to convert the input images into feature maps. On CIFAR10/100, we use a convolution with kernel size 5, stride 2, and padding 2, followed by a MinMax activation as the stem. On Tiny ImageNet, we use a convolution with kernel size 7, stride 4, and padding 3, followed by a MinMax activation as the stem. On ImageNet, we follow the ViT-like patching [14] and use a convolution with kernel size 14, stride 14, and padding 0, followed by a MinMax activation as the stem. Thus the output feature map size from the stem layer is  $16 \times 16$  for all 4 datasets. The number of filters used in the convolution is equal to the model width  $W$ .
Table 3: Clean accuracy and VRA performance (%) of a ConvNet and a LiResNet on three datasets with different loss functions
loss	TRADES	EMMA
Clean (%)	VRA (%)	Clean (%)	VRA (%)
CIFAR-10 (ε = 36/255, 10 classes)
ConvNet	71.7	58.8	72.5	59.2
LiResNet	79.6	66.2	80.4	66.3
CIFAR-100 (ε = 36/255, 100 classes)
ConvNet	53.4	34.0	50.6	35.0
LiResNet	57.8	37.3	54.2	37.8
Tiny-ImageNet (ε = 36/255, 200 classes)
ConvNet	42.2	26.6	40.0	27.4
LiResNet	45.8	28.8	43.6	30.0
Model backbone is used to transform the feature maps. It is a stack of  $L$  LiResNet blocks followed by the MinMax activation, i.e., (LiResNet block  $\rightarrow$  MinMax)  $\times L$ . We keep the feature map resolutions and the number of channels constant in the model backbone. We find some tricks in normalization-free residual network studies [41, 60] can improve the performance of our LiResNet as our method is also a normalization-free residual network. Specifically, we add an affine layer  $\beta$  that applies channel-wise learnable multipliers to each channel of the feature map (similar to the affine layer of batch normalization) and a scalar of  $1 / \sqrt{L}$  to the residual branch where  $L$  is the number of blocks:
$$
y = x + \frac {1}{\sqrt {L}} \beta \operatorname {C o n v} (x)
$$
Model neck is used to convert the feature maps into a feature vector. In our implementation, the model neck is a 2 layer network. The first layer is a convolution layer with kernel size 4, stride 4, and padding 0, followed by a MinMax activation. The number of input channels is the model width  $W$  and the number of output channels is  $2W$ . Then we reshape the feature map tensor into a vector. The second layer is a dense layer with output dimension  $d$  where  $d = 2048$  for the three small datasets (CIFAR10/100 and Tiny-ImageNet) and  $d = 4096$  for ImageNet.
Model head is used to make classification predictions. We apply the last layer normalization (LLN) proposed by [43] to the head.
B.3 Metric details
We report the clean accuracy, i.e., the accuracy without verification on non-adversarial inputs and the verified-robust accuracy (VRA), i.e., the fraction of points that are both correctly classified and certified as robust. Our results are averaged over 5 runs for CIFAR10/100 and TinyImageNet and 3 runs for ImageNet.
C Details for Table 2a
In Table 2a, we use an L12W256 configuration, i.e., the backbone has 12 blocks and the number of filters is 256. For ConvNet, the only difference is that the LiResNet block is replaced by a convolution of kernel 3, stride 1, and padding 1. All other settings are the same. Table 3 is a more detailed version of Table 2a with the clean accuracy.
D Details for Table 2b
In Table 2b, we use the configuration of W256, i.e., the number of channels in the backbone is 256. The only difference between conventional ResNet and LiResNet is the block. The block for
Table 4: Clean accuracy and VRA (\%) performance on CIFAR-10/100 with different architectures ( $L$  is the number of blocks in the model backbone). We use EMMA loss for Gloro training.  $\times$  stands for not converging at the end.
Dataset	L	ConvNet	ResNet	LiResNet
Clean(%)	VRA(%)	Clean(%)	VRA(%)	Clean(%)	VRA(%)
CIFAR-10	6	77.9	64.0	74.2	60.3	79.9	65.5
12	72.5	59.2	74.0	60.0	80.4	66.3
18	×	×	73.9	60.1	81.0	66.6
CIFAR-100	6	51.8	36.5	48.4	33.5	53.6	37.2
12	50.6	35.0	48.1	33.5	54.2	37.8
18	×	×	48.2	33.6	54.3	38.0
Table 5: Clean accuracy and VRA (%) performance of LiResNet of different depths ( $L$  is the number of blocks in the model backbone).
L	CIFAR10	CIFAR100	Tiny-ImageNet
Clean(%)	VRA(%)	Clean(%)	VRA(%)	Clean(%)	VRA(%)
6	79.9	65.5	53.6	37.2	43.1	29.8
12	80.4	66.3	54.2	37.8	43.6	30.3
18	81.0	66.6	54.3	38.0	43.9	30.6
24	81.2	66.8	55.0	38.2	44.2	30.7
30	81.3	66.9	54.9	38.4	44.2	30.6
36	81.2	66.9	55.0	38.3	44.3	30.4
conventional ResNet is
$$
y = x + \beta \operatorname {C o n v} (\operatorname {M i n M a x} (\operatorname {C o n v} (x)))
$$
where  $\beta$  is the affine layer. We find use zeros to initialize  $\beta$  works the best for conventional ResNet. The number of input and output channels of the two convolution layers are the same as that of the LiResNet block. Table 4 is a more detailed version of Table 2b with clean accuracy.
E Details for Figure 1
We make LiResNet further deeper and study how network depth influences the performance on CIFAR-10/100 and Tiny-ImageNet. Table 5 shows the clean accuracy and VRA of LiResNet (with EMMA loss) on three datasets. All models use a W256 configuration, i.e., the number of convolutional channels is 256. On CIFAR-10/100, the VRA performance of the LiResNet generally improves with depth. On Tiny-ImageNet, the performance remains with the increase of depth.
Figure 1 compares the VRA performance of LiResNet with some existing method for verification robustness on CIFAR-100 (i.e., the 5th of Table 5). The numbers of these methods are taken from their best-reported configurations. The VRA performance of these methods degrades at certain depths, limiting the maximum model capacity of the methods.
F Number of Classes vs. VRA
Despite the fact that EMMA loss improves the ability of GloRo Nets to handle learning problems with many classes, datasets with a large number of classes still stand out as particularly difficult for certified training. In principle, a data distribution with less classes is not guaranteed to have more separable features than more classes—indeed, the state-of-the-art clean accuracy for both CIFAR-10 and CIFAR-100 are comfortably in the high 90's despite the large difference in the number of classes. However, training a certifiably robust model with many classes appears more difficult in practice (as observed, e.g., by the large performance gap between CIFAR-10 and CIFAR-100). To test this
Figure 4: Plot of LiResNet performance on subsets of ImageNet with different number of classes with  $\epsilon = 1$
Table 6: VRA (%) of LiResNet of different widths (W).
W	CIFAR-10	CIFAR-100	Tiny-ImageNet
64	64.6	36.5	28.7
128	65.6	37.5	29.8
256	66.3	37.8	30.0
512	66.9	38.3	30.6
observation further, we provide an empirical study on various class-subsets of ImageNet to study the relationship between the number of classes and VRA.
We randomly shuffle the 1000 classes of ImageNet and select the first  $100 \cdot k$  classes, where  $k \in [10]$ , to build a series of subsets for training and testing. For each value of  $k$ , we train a GloRo LiResNet with EMMA loss  $(\epsilon = 1)$  and report the clean accuracy and VRA (at  $\epsilon = 1$ ) on the test set. For reference, we also train a standard (i.e., not robust) LiResNet with Cross Entropy and report its clean accuracy on the test set. The final results are shown in Figure 4 with additional details in Appendix F. Compared to the clean accuracy of a standard model, increasing the number of classes leads to a steeper drop in both the VRA and the clean accuracy of the robustly trained models. Specifically, while the performance of the standard model differs only by  $10\%$  between a 100-class subset and the full ImageNet, the performance of the GloRoNet (both clean accuracy and VRA), drops by  $30\%$ .
These results add weight to the observation that, even when mitigated by EMMA loss, large numbers of classes present a particular challenge for certifiably robust learning. This may arise from the need to learn a  $2\epsilon$ -margin between all regions with different labels, which becomes progressively more challenging as boundaries between a growing number of classes become increasingly difficult to push off the data manifold.
G Going Wider with LiResNet
We study how network width (i.e., the number of channels in the model backbone) can influence the performance of LiResNet on CIFAR-10, CIFAR-100 and Tiny-ImageNet. Table 6 shows the results. All models use a L12 configuration. Unlike the network depth, increasing the width can stably improve the model performance within a certain range.
H Extra data from DDPM
We use codes from the improved DDPM [36] to train generative models on CIFAR10, CIFAR100 and Tiny-ImageNet. The models are only trained on the training set of each dataset and no external data is used. We use the recommended hyper-parameters from [36] and the models are conditional, i.e., generated samples are with labels. We generate 1 million samples for each dataset.
During the training of Gloro Net, we sample 256 samples from the original dataset and 256 samples from the generated data for each batch. Due to the large total number of generated data, we do not need strong data augmentation on the generated data. Compared to the original dataset, we do not use the RandAugment augmentation for the generated data. All other settings are the same for the original dataset and the generated data.
I Certifiable Robustness with Transformers
Transformers-based models [51] have been shown to surpass existing convolution networks on major language tasks. To fully utilize the power of Transformer layers, and Vision Transformers [14] chunk images as small patches to convert one image as a sequence of patches, which is used as input to a Transformer-based network. A Transformer layer is a combination of self-attention blocks (SA) and feed-forward layers (FFN). Between the blocks, layer normalization layers are added to increase the stability of the learning. Because in this paper we mainly focus on certifying vision models, we will simplify refer Vision Transformers as Transformers in the rest of this section.
Non-Lipschitz Operations in Transformers. The fundamental challenge of certifying Transformers' prediction on images with Lipschitz-based approaches arises from the many modules in a Transformer layer are not Lipschitz-continuous, i.e. SA and Layer Normalization. One can simplify remove normalization layers from Transformer layer but this may lead to a serious performance degradation [56]. On the other hand, SA is also not Lipschitz because of the softmax calculation taken over the attention scores. Recent work focuses on designing Lipschitz-continuous alternatives for SAs, which includes OLSA [53] and L2-MHA [28]. Another idea is to use spatial MLP [47] that calculates static "attention weights" so they are bounded by construction.
Replacing with Linear Residual Connections. In terms of FFN, it is a standard residual block and thus suffers from the same problem identified in Section 3. Namely, the overall Lipschitz constant of FFN can be loose because of the skip connection. We therefore replace a standard FFN in a Transformer layer with our linear residual connection to tighten its Lipschitz bound.
Certifying Transformers. Our discussion above has shown that to certify the robustness of Transformer-based models, one needs to remove non-Lipschitz-continuous operations or replace them with Lipschitz-continuous alternatives. We will refer to a Transformer-based model modified to only contain Lipschitz-continuous operations as Lipschitz Transformers. We conduct experiments on CIFAR10 and CIFAR100 and provide VRAs of Lipschitz Transformers with different linear operations. That is, for the SA block, we replace it either with an OLSA or a spatial MLP. For the FFN layer, we can optionally it to our linear residual connection and will denote this modified FFN as LiFFN. We use the L12W256 configuration and Table 7 shows the results.
Results. As shown in Table 7, all variants of Lipschitz Transformers do not outperform LiResNets (with the same configuration). The combination of "Spatial MLP" and "LiFFN" performs the best among all variants of transformers. This architecture is also most aligned with the motivation of this paper - using linear operations for the weights to obtain tight Lipschitz constant estimation.
In conclusion, Transformers have potential applicability in Lipschitz-based robustness certification; however, their performance leaves much to be desired. This inadequacy stems primarily from two limitations. Firstly, layer normalization layers are nonviable, and secondly, the SA and FFN blocks struggle to attain a tight estimation of the Lipschitz constant. Although our work sheds a light on how to tightening the Lipschitz bound for FFN with LiFFN, the current Lipschitz alternatives for SAs are not tight enough to further promote the robustness of Transformers.
Table 7: VRAs (%) measured with different variants of Lipschitz Transformers. As a reference for the state-of-the-art performance for convolutional networks, we include our LiResNet results from Table 1.
dataset	replacement for SA	replacement for FFN	VRA(%)
CIFAR10	Spatial MLP	LiFFN	63.3
Spatial MLP	FFN	62.6
OLSA	FFN	56.6
LiResNet-L12W256	66.9
CIFAR100	Spatial MLP	LiFFN	36.5
Spatial MLP	FFN	33.7
OLSA	FFN	28.4
LiResNet-L12W256	38.3