SlowGuess/NeurIPS_2023_OCR huggingface dataset:
is clear by our supposition that  $X_{2} \subseteq X$  and  $X_{1} \subseteq \mathbb{R}^{d} \setminus X$ . Therefore, the supposition implies that  $(X_{1}, X_{2})$  is convexly separable, and thus
$P \geq \mathbb{P}$  (there exists  $k \in \{1, 2, \ldots, d\}$  such that  $x_k^{(i)} < y_k^{(j)}$  for all pairs  $(i, j)$ )
and that  $\min \{y_k^{(1)},\ldots ,y_k^{(N)}\} <  \max \{y_k^{(1)},\ldots ,y_k^{(N)}\})$
$= 1 - \mathbb{P}$  (for all  $k\in \{1,2,\ldots ,d\}$ , it holds that  $x_{k}^{(i)}\geq y_{k}^{(j)}$  for some pair  $(i,j)$ )
or that  $\min \{y_k^{(1)},\ldots ,y_k^{(N)}\} = \max \{y_k^{(1)},\ldots ,y_k^{(N)}\}$
$= 1 - \prod_{k = 1}^{d}\mathbb{P}\left(x_k^{(i)}\geq y_k^{(j)}\text{for some pair} (i,j)\text{or}\min \{y_k^{(1)},\ldots ,y_k^{(N)}\} = \max \{y_k^{(1)},\ldots ,y_k^{(N)}\}\right),$
where the final equality follows from the independence of the coordinates of the samples. Since  $\min \{y_k^{(1)},\ldots ,y_k^{(N)}\} <  \max \{y_k^{(1)},\ldots ,y_k^{(N)}\}$  almost surely, we find that
$$
\begin{array}{l} P \geq 1 - \prod_ {k = 1} ^ {d} \left(\mathbb {P} \left(x _ {k} ^ {(i)} \geq y _ {k} ^ {(j)} \text {f o r s o m e p a i r} (i, j)\right) \right. \\ \left. + \mathbb {P} \left(\min  \left\{y _ {k} ^ {(1)}, \dots , y _ {k} ^ {(N)} \right\} = \max  \left\{y _ {k} ^ {(1)}, \dots , y _ {k} ^ {(N)} \right\}\right)\right) \\ = 1 - \prod_ {k = 1} ^ {d} \mathbb {P} \left(x _ {k} ^ {(i)} \geq y _ {k} ^ {(j)} \text {f o r s o m e p a i r} (i, j)\right) \\ = 1 - \prod_ {k = 1} ^ {d} \left(1 - \mathbb {P} \left(x _ {k} ^ {(i)} <   y _ {k} ^ {(j)} \text {f o r a l l p a i r s} (i, j)\right)\right) \tag {3} \\ = 1 - \prod_ {k = 1} ^ {d} \left(1 - \mathbb {P} \left(\max _ {i \in \{1, 2, \dots , M \}} x _ {k} ^ {(i)} <   \min _ {j \in \{1, 2, \dots , N \}} y _ {k} ^ {(j)}\right)\right) \\ = 1 - \prod_ {k = 1} ^ {d} \left(1 - \mathbb {P} \left(\left(x _ {k} ^ {(1)}, \dots , x _ {k} ^ {(M)}, y _ {k} ^ {(1)}, \dots , y _ {k} ^ {(N)}\right) \in \bigcup_ {\sigma \in S} E _ {\sigma}\right)\right), \\ \end{array}
$$
where we define  $S$  to be the set of permutations on  $\{1, \ldots, M + N\}$  whose restriction to  $\{1, \ldots, M\}$  is also a permutation;
$$
S = \left\{\sigma \in S _ {M + N}: \sigma | _ {\{1, \dots , M \}} \in S _ {M} \right\},
$$
and where, for a permutation  $\sigma \in S_{M + N}$ ,  $E_{\sigma}$  is the event where an  $(M + N)$ -vector has indices ordered according to  $\sigma$ ;
$$
E _ {\sigma} = \left\{z \in \mathbb {R} ^ {M + N}: z _ {\sigma (1)} <   \dots <   z _ {\sigma (M + N)} \right\}.
$$
We note that the final equality in (3) relies on the fact that  $\mathbb{P}(x_k^{(i)} = x_k^{(i')}) = \mathbb{P}(y_k^{(j)} = y_k^{(j'})}) = 0$  for all  $i^{\prime}\neq i$  and all  $j^{\prime}\neq j$ , which is specific to our uniform distribution at hand.
Now, since  $E_{\sigma}, E_{\sigma'}$  are disjoint for distinct permutations  $\sigma, \sigma' \in S_{M + N}$ , the bound (3) gives that
$$
P \geq 1 - \prod_ {k = 1} ^ {d} \left(1 - \sum_ {\sigma \in S} \mathbb {P} \left(\left(x _ {k} ^ {(1)}, \dots , x _ {k} ^ {(M)}, y _ {k} ^ {(1)}, \dots , y _ {k} ^ {(N)}\right) \in E _ {\sigma}\right)\right). \tag {4}
$$
Since  $x_{k}^{(1)}, \ldots, x_{k}^{(M)}, y_{k}^{(1)}, \ldots, y_{k}^{(N)}$  are independent and identically distributed samples, they define an exchangeable sequence of random variables, implying that  $\mathbb{P}((x_{k}^{(1)}, \ldots, x_{k}^{(M)}, y_{k}^{(1)}, \ldots, y_{k}^{(N)}) \in E_{\sigma}) = \mathbb{P}(x_{k}^{(1)} < \dots < x_{k}^{(M)} < y_{k}^{(1)} < \dots < y_{k}^{(N)})$  for all permutations  $\sigma \in S_{M + N}$ . Since, under the uniform distribution at hand,  $(x_{k}^{(1)}, \ldots, x_{k}^{(M)}, y_{k}^{(1)}, \ldots, y_{k}^{(N)}) \in E_{\sigma}$  for some  $\sigma \in S_{M + N}$  almost
surely, it holds that
$$
\begin{array}{l} 1 = \mathbb {P} \left(\left(x _ {k} ^ {(1)}, \dots , x _ {k} ^ {(M)}, y _ {k} ^ {(N)}, \dots , y _ {k} ^ {(N)}\right) \in \bigcup_ {\sigma \in S _ {M + N}} E _ {\sigma}\right) \\ = \sum_ {\sigma \in S _ {M + N}} \mathbb {P} ((x _ {k} ^ {(1)}, \dots , x _ {k} ^ {(M)}, y _ {k} ^ {(1)}, \dots , y _ {k} ^ {(N)}) \in E _ {\sigma}) \\ = | S _ {M + N} | \mathbb {P} (x _ {k} ^ {(1)} <   \dots x _ {k} ^ {(M)} <   y _ {k} ^ {(1)} <   \dots <   y _ {k} ^ {(N)}). \\ \end{array}
$$
$$
\mathbb {P} ((x _ {k} ^ {(1)}, \dots , x _ {k} ^ {(M)}, y _ {k} ^ {(1)}, \dots , y _ {k} ^ {(N)}) \in E _ {\sigma}) = \frac {1}{| S _ {M + N} |} = \frac {1}{(M + N) !}
$$
for all permutations  $\sigma \in S_{M + N}$ . Hence, our bound (4) becomes
$$
P \geq 1 - \prod_ {k = 1} ^ {d} \left(1 - \frac {| S |}{(M + N) !}\right) = 1 - \left(1 - \frac {| S |}{(M + N) !}\right) ^ {d}.
$$
Finally, we immediately see that that map  $\Gamma \colon S_M \times S_N \to S_{M + N}$  defined by
$$
\Gamma (\sigma , \sigma^ {\prime}) (i) = \left\{ \begin{array}{l l} \sigma (i) & \text {i f} i \in \{1, \ldots , M \}, \\ \sigma^ {\prime} (i - M) + M & \text {i f} i \in \{M + 1, \ldots , M + N \}, \end{array} \right.
$$
is injective and has image  $S$ , implying that  $|S| = |S_M \times S_N| = |S_M||S_N| = M!N!$ . Thus,
$$
P \geq 1 - \left(1 - \frac {M ! N !}{(M + N) !}\right) ^ {d},
$$
which proves (2).
The unit probability of  $\hat{\mathcal{F}}_{\mathrm{Id}}$  containing a classifier that classifies all  $x^{(i)}$  into class 1 and all  $y^{(j)}$  into class 2 for large  $d$  follows immediately from Theorem 3.7.
D CIFAR-10 Cats-versus-Dogs Convex Separability
In order to establish that the cat and dog images in CIFAR-10 are convexly separable, we experimentally attempt to reconstruct an image from one class using a convex combination of all images in the other class (without augmentation such as random crops, flips, etc.). Namely, if  $x$  is drawn from one class and  $y^{(1)},\ldots ,y^{(N)}$  represent the entirety of the other class, we form the following optimization problem:
$$
\underset {\alpha \in \mathbb {R} ^ {N}} {\text {m i n i m i z e}} \quad \left\| x - \sum_ {j = 1} ^ {N} \alpha_ {j} y ^ {(j)} \right\| _ {2}
$$
$$
\begin{array}{l l} \text {s u b j e c t t o} & \alpha \geq 0, \end{array}
$$
$$
\sum_ {j = 1} ^ {N} \alpha_ {j} = 1.
$$
The reverse experiment for the other class follows similarly. We solve the optimization using MOSEK [6], and report the various norms of  $x - \sum_{j=1}^{N} \alpha_j y^{(j)}$  in Figure 6. Reconstruction accuracy is generally very poor, with no reconstruction achieving better than an  $\ell_1$ -error of 52. A typical reconstructed image is shown in Figure 7.
Yousefzadeh [74] and Balestriero et al. [9] showed a related empirical result for CIFAR-10, namely, that no test set image can be reconstructed as a convex combination of training set images. However, we remark that their findings do not necessarily imply that a training set image cannot be reconstructed via other training set images; our new finding that the CIFAR-10 cats-versus-dogs training set is convexly separable is required in order to assert Fact 3.8.
Figure 6: Reconstructing CIFAR-10 cat and dog images as convex combinations. The label "Dogs  $\rightarrow$  cat" indicates that a cat image was attempted to be reconstructed as a convex combination of all 5000 dog images.
Figure 7: Reconstructing a CIFAR-10 cat image (left) from a convex combination of dog images (right). The reconstruction error norms are 294.57, 6.65, and 0.38 for the  $\ell_{1} -$ ,  $\ell_{2} -$ , and  $\ell_{\infty}$ -norms, respectively. These are typical, as indicated by Figure 6.
E Experimental Setup
We include a detailed exposition of our experimental setup in this section, beginning with general details on our choice of epochs and batch size. We then discuss baseline methods, architecture choices for our method, class balancing, and data processing.
Epochs and batch size. Exempting the randomized smoothing baselines, for the MNIST 3-8 and Fashion-MNIST shirts experiments, we use 60 epochs for all methods. This is increased to 150 epochs for the Malimg dataset and CIFAR-10 cats-dogs experiments. The batch size is 64 for all datasets besides the  $512 \times 512$  Malimg dataset, where it is lowered to 32.
To ensure a fair comparison, the randomized smoothing baseline epochs are scaled larger than the aforementioned methods according to the noise value specified in the sweeps in Section I. The final epochs and smoothing noise values used are reported in Table 2. Note that as classifiers are typically more robust to the noise from splitting smoothing, larger values of  $\sigma$  are used for only this smoothing method in the MNIST 3-8 and Malimg datasets. For Malimg, we find experimentally that even noise values of up to  $\sigma = 100$  are tractable for the splitting method, outside the sweep range considered in Section I. As verification at that  $\sigma$  already takes several minutes per sample and runtime scales linearly with  $\sigma$ , we do not explore larger values of  $\sigma$ .
Table 2: Randomized smoothing final noise and epoch hyperparameters.
Dataset	Laplacian, Uniform, Gaussian Parameters	Splitting Parameters
MNIST 3-8	(σ,n) = (0.75,60)	(σ,n) = (0.75·4,60·4)
Maling	(σ,n) = (3.5·4,150·4)	(σ,n) = (100,150·4)
Fashion-MNIST shirts	(σ,n) = (0.75,60)	(σ,n) = (0.75,60)
CIFAR-10 cats-dogs	(σ,n) = (0.75·2,600·2)	(σ,n) = (0.75·2,600·2)
Hardware. All experiments were conducted on a single Ubuntu 20.04 instance with an Nvidia RTX A6000 GPU. Complete reproduction of the experiments takes approximately 0.08 GPU-years.
We introduce the various datasets considered in this work. MNIST 3-8 and Malimg are relatively simple classification problems where near-perfect classification accuracy is attainable; the Malimg dataset falls in this category despite containing relatively large images. Our more challenging settings consist of a Fashion-MNIST shirts dataset as well as CIFAR-10 cats-versus-dogs dataset.
For consistency with [77], we augment the MNIST and Fashion-MNIST training data with 1-pixel padding and random cropping. The CIFAR-10 dataset is augmented with 3-pixel edge padding, horizontal flips, and random cropping. The Malimg dataset is augmented with 20-pixel padding and random  $512 \times 512$  cropping.
For CIFAR-10, MNIST, and Fashion-MNIST, we use the preselected test sets. For Maling we hold out a random  $20\%$  test dataset, although this may not be entirely used during testing. The training set is further subdivided by an  $80\% - 20\%$  validation split. For all experiments, we normalize pixel values into the interval [0, 1], and we use the first 1000 test samples to evaluate our methods.
MNIST 3-8. For our MNIST binary classification problem, we choose the problem of distinguishing between 3 and 8 [37]. These were selected as 3 and 8 are generally more visually similar and challenging to distinguish than other digit pairs. Images are  $28 \times 28$  pixels and greyscale.
Maling. Our malware classification experiments use greyscale, bitwise encodings of raw malware binaries Nataraj et al. [50]. Each image pixel corresponds to one byte of data, in the range of 0–255, and successive bytes are added horizontally from left to right on the image until wrapping at some predetermined width. We use the extracted malware images from the seminal dataset Nataraj et al. [50], normalizing pixel values into  $[0,1]$ , as well as padding and cropping images to be  $512 \times 512$ . Note that licensing concerns generally prevent the distribution of "clean" executable binaries. As this work is focused on providing a general approach to robust classification, in the spirit of reproducibility we instead report classification results between different kinds of malware. Namely, we distinguish between malware from the most numerous "Allaple.A" class (2949 samples) and an identically-sized random subset of all other 24 malware classes. To simulate a scenario where we must provide robustness against evasive malware, we provide certificates for the latter collection of classes.
Fashion-MNIST shirts. The hardest classes to distinguish in the Fashion-MNIST dataset are T-shirts vs shirts, which we take as our two classes [33, 70]. Images are  $28 \times 28$  pixels and greyscale.
CIFAR-10 cats-dogs. We take as our two CIFAR-10 classes the cat and dog classes since they are relatively difficult to distinguish [26, 43, 30]. Other classes (e.g., ships) are typically easier to classify since large background features (e.g., blue water) are strongly correlated with the target label. Samples are  $32 \times 32$  RGB images.
E.2 Baseline Methods
We provide additional details on each of the baseline methods below.
Randomized smoothing. Since the certification runtime of randomized smoothing is large, especially for the  $512 \times 512$  pixel Maling images, we evaluate the randomized smoothing classifiers over  $10^{4}$  samples and project the certified radius to  $10^{5}$  samples by scaling the number fed into the Clopper-Pearson confidence interval, as described in [18]. This allows for a representative and improved certified accuracy curve while dramatically reducing the method's runtime. We take an initial guess for the certification class with  $n_{0} = 100$  samples and set the incorrect prediction tolerance parameter  $\alpha = 0.001$ . For CIFAR-10 we use a depth-40 Wide ResNet base classifier, mirroring the choices from Cohen et al. [18], Yang et al. [72]; for all other datasets we use a ResNet-18. All networks are trained using SGD with an initial learning rate of 0.1, Nesterov momentum of 0.9, weight decay of  $10^{-4}$ , and cosine annealing scheduling as described in Yang et al. [72]. Final smoothing noise values are selected as in Table 2, and are determined from the noise level comparison sweeps in Appendix I.
Splitting noise. As this method is a deterministic derivative of randomized smoothing, it avoids the many aforementioned hyperparameter choices. We use the same architectures described above for the other randomized smoothing experiments.
Cayley convolutions. To maintain consistency, we use a two-hidden-layer multilayer perceptron with  $(n_{1}, n_{2}) = (200, 50)$  hidden features, CayleyLinear layers, and GroupSort activations for the MNIST experiment. For the more challenging Fashion-MNIST and CIFAR-10 experiments, we use the ResNet-9 architecture implementation from [63]. Following the authors' suggestions, we train these networks using Adam with a learning rate of 0.001.
$\ell_{\infty}$ -distance nets. As the architecture of the  $\ell_{\infty}$ -distance net [77] is substantially different from traditional architectures, we use the authors' 5-layer MNIST/Fashion-MNIST architecture and 6-layer CIFAR-10 architecture with 5120 neurons per hidden layer. Unfortunately, the classification accuracy on the CIFAR-10 cats-dogs experiment remained near  $50\%$  throughout training. This was not the case when we tested easier classes, such as planes-versus-cars, where large features (e.g., blue sky) can be used to discriminate. We therefore only include this model in the MNIST and Fashion-MNIST experiments, and use the training procedure directly from the aforementioned paper's codebase.
$\alpha, \beta$ -CROWN. As  $\alpha, \beta$ -CROWN certification time scales exponentially with the network size, we keep the certified networks small in order to improve the certification performance of the baseline. For all datasets, we train and certify a one-hidden-layer network with 200 hidden units and ReLU activations. All networks are adversarially trained for a  $\ell_{\infty}$ -perturbation radius starting at 0.001 and linearly scaling to the desired  $\epsilon$  over the first 20 epochs, as described in Kayed et al. [33], which trained the models used in Wang et al. [66]. The desired final  $\epsilon$  is set to 0.3 for MNIST, 0.1 for Fashion-MNIST and Malimg, and 2/255 for CIFAR-10. The adversarial training uses a standard PGD attack with 50 steps and step size  $2\epsilon / 50$ . Other optimizer training details are identical to Wang et al. [66]. The branch-and-bound timeout is set to 30 seconds to maintain comparability to other methods, and robustness is evaluated over a dataset-dependent range of discrete radii for each adversarial norm.
E.3 Feature-Convex Architecture and Training
In this section, we provide more details of the feature-convex architectures used in our experiments of Section 4. For the more challenging datasets (Fashion-MNIST shirts and CIFAR-10 cats-dogs), we use various instantiations of a convex ConvNet (described below) where successive layers have a constant number of channels and image size. This allows for the addition of identity residual connections to each convolution and lets us remove the passthrough connections altogether. Convexity is enforced by projecting relevant weights onto the nonnegative orthant after each epoch and similarly constraining BatchNorm  $\gamma$  parameters to be positive. We initialize positive weight matrices to be drawn uniformly from the interval  $[0,\epsilon]$ , where  $\epsilon = 0.003$  for linear weights and  $\epsilon = 0.005$  for convolutional weights. Jacobian regularization is also used to improve our certified radii [31].
The convex ConvNet architecture consists of a sequence of convolutional layers, BatchNorms, and ReLU nonlinearities. The first convolutional layer is unconstrained, as the composition of a convex function with an affine function is still convex [2]. All subsequent convolutions and the final linear readout layer are uniformly initialized from some small positive weight interval ([0, 0.003] for linear weights, [0, 0.005] for convolutional weights) and projected to have nonnegative weights after
each gradient step. We found this heuristic initialization choice helps to stabilize network training, as standard Kaiming initialization assumptions are violated when weights are constrained to be nonnegative instead of normally distributed with mean zero. More principled weight initialization strategies for this architecture would form an exciting area of future research. Before any further processing, inputs into the network are fed into an initial BatchNorm—this enables flexibility with different feature augmentation maps.
Since the first convolutional layer is permitted negative weights, we generally attain better performance by enlarging the first convolution kernel size (see Table 3). For subsequent convolutions, we set the stride to 1, the input and output channel counts to the output channel count from the first convolution, and the padding to half the kernel size, rounded down. This ensures that the output of each of these deeper convolutions has equivalent dimension to its input, allowing for an identity residual connection across each convolution. If  $C_i(z)$  is a convolutional operation on a hidden feature  $z$ , this corresponds to evaluating  $C_i(z) + z$  instead of just  $C_i(z)$ . The final part of the classifier applies MaxPool and BatchNorm layers before a linear readout layer with output dimension 1. See Figure 8 for a diagram depicting an exemplar convex ConvNet instantiation.
Figure 8: An example convex ConvNet of depth 4 with a  $C_1$  stride of 2, pool size of 4, and  $32 \times 32$  RGB images. There are 6 input channels from the output of the feature map  $\varphi \colon x \mapsto (x - \mu, |x - \mu|)$ .
For training, we use a standard binary cross entropy loss, optionally augmented with a Jacobian regularizer on the Frobenius norm of the network Jacobian scaled by  $\lambda > 0$  [31]. As our certified radii in Theorem 3.1 vary inversely to the norm of the Jacobian, this regularization helps boost our certificates at a minimal loss in clean accuracy. We choose  $\lambda = 0.0075$  for CIFAR-10,  $\lambda = 0.075$  for Maling and  $\lambda = 0.01$  for MNIST and Fashion-MNIST. Further ablation tests studying the impact of regularization are reported in Appendix G. All feature-convex networks are trained using SGD with a learning rate of 0.001, momentum 0.9, and exponential learning rate decay with  $\gamma = 0.99$ .
Table 3: Convex ConvNet architecture parameters.  $C_1$  denotes the first convolution, with  $C_{2,\ldots}$  denoting all subsequent convolutions. The "Features" column denotes the number of output features of  $C_1$ , which is held fixed across  $C_{2,\ldots}$ . The "Pool" column refers to the size of the final MaxPool window before the linear readout layer. The MNIST and Malimg architectures are simple multilayer perceptrons and are therefore not listed here.
Dataset	Features	Depth	C1 size	C1 stride	C1 dilation	C2,... size	Pool
Fashion-MNIST	4	3	5	1	1	3	1
CIFAR-10	16	5	11	1	1	3	1
E.4 Class Accuracy Balancing
As discussed in Section 1.1, a balanced class 1 and class 2 test accuracy is essential for a fair comparison of different methods. For methods where the output logits can be directly balanced, this is easily accomplished by computing the ROC curve and choosing the threshold that minimizes  $|\mathrm{TPR} - (1 - \mathrm{FPR})|$ . This includes both our feature-convex classifiers with one output logit and the Cayley orthogonalization and  $\ell_{\infty}$ -Net architectures with two output logits.
Randomized smoothing classifiers are more challenging as the relationship between the base classifier threshold and the smoothed classifier prediction is indirect. We address this using a binary search balancing procedure. Namely, on each iteration, the classifier's prediction routine is executed over the test dataset and the "error" between the class 1 accuracy and the class 2 accuracy is computed. The sign of the error then provides the binary signal for whether the threshold should be shifted higher or lower in the standard binary search implementation. This procedure is continued until the error drops below  $1\%$ .
F  $\ell_2$ - and  $\ell_{\infty}$ -Certified Radii
This section reports the counterpart to Figure 2 for the  $\ell_{2}$ - and  $\ell_{\infty}$ -norms. Across all experiments, we attain substantial  $\ell_{2}$ - and  $\ell_{\infty}$ -radii without relying on computationally expensive sampling schemes or nondeterminism. Methods that certify to another norm  $\| \cdot \|_p$  are converted to  $\ell_q$ -radii at a factor of 1 if  $p > q$  or  $d^{1/p-1/q}$  otherwise.
Certified  $\ell_2$ -radii are reported in Figure 9. Our  $\ell_2$ -radii are moderate, generally slightly smaller than those produced by Gaussian randomized smoothing.
Certified  $\ell_{\infty}$ -radii are reported in Figure 10. For the MNIST 3-8 experiment, the  $\ell_{\infty}$ -distance nets produce exceptional certified radii. Likewise, the  $\ell_{\infty}$ -distance net certificates are dominant for the Fashion-MNIST dataset, despite achieving slightly inferior clean accuracy. We note however that the applicability of  $\ell_{\infty}$ -distance nets for sophisticated vision tasks is uncertain as the method is unable to achieve better-than-random performance for CIFAR-10 cats-dogs (Section E.2). Our method is comparable to randomized-smoothing and  $\alpha$ ,  $\beta$ -CROWN in all  $\ell_{\infty}$  experiments.
(c) Fashion-MNIST shirts.
Figure 9: Class 1 certified radii curves for the  $\ell_2$ -norm.
(c) Fashion-MNIST shirts.
Figure 10: Class 1 certified radii curves for the  $\ell_{\infty}$ -norm.
G Ablation Tests
We conduct a series of ablation tests on the CIFAR-10 cats-dogs dataset, examining the impact of regularization, feature maps, and data augmentation.
G.1 Regularization
Figure 11 examines the impact of Jacobian regularization over a range of regularization scaling factors  $\lambda$ , with  $\lambda = 0$  corresponding to no regularization. As is typical, we see a tradeoff between clean accuracy and certified radii. Further increases in  $\lambda$  yield minimal additional benefit.
G.2 Feature Map
In this section, we investigate the importance of the feature map  $\varphi$ . Figure 12 compares our standard feature-convex classifier with  $\varphi(x) = (x - \mu, |x - \mu|)$  against an equivalent architecture with  $\varphi = \mathrm{Id}$ . Note that the initial layer in the convex ConvNet is a BatchNorm, so even with  $\varphi = \mathrm{Id}$ , features still get normalized before being passed into the convolutional architecture. We perform this experiment across both the standard cats-dogs experiment (cats are certified) in the main text and the reverse dogs-cats experiment (dogs are certified).