A Recipe for Improved Certifiable Robustness
Authors: Kai Hu
Klas Leino
Zifan Wang
Matt Fredrikson
Venue: ICLR 2024 poster (2024)
Keywords: adversarial robustness
certificated robustness
Lipschitz-based
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce several techniques to improve the model capacity for Lipschitz based robust training and introduce a better pipeline to use generated data augmentation.

Abstract: Recent studies have highlighted the potential of Lipschitz-based methods for training certifiably robust neural networks against adversarial attacks.
A key challenge, supported both theoretically and empirically, is that robustness demands greater network capacity and more data than standard training. 
However, effectively adding capacity under stringent Lipschitz constraints has proven more difficult than it may seem, evident by the fact that state-of-the-art approach tend more towards \emph{underfitting} than overfitting.
Moreover, we posit that a lack of careful exploration of the design space for Lipshitz-based approaches has left potential performance gains on the table.
In this work, we provide a more comprehensive evaluation to better uncover the potential of Lipschitz-based certification methods.
Using a combination of novel techniques, design optimizations, and synthesis of prior work, we are able to significantly improve the state-of-the-art VRA for deterministic certification on a variety of benchmark datasets, and over a range of perturbation sizes.
Of particular note, we discover that the addition of large ``Cholesky-orthogonalized residual dense'' layers to the end of existing state-of-the-art Lipschitz-controlled ResNet architectures is especially effective for increasing network capacity and performance.
Combined with filtered generative data augmentation, our final results further the state of the art deterministic VRA by up to 8.5 percentage points.

Published as a conference paper at ICLR 2024
A RECIPE FOR IMPROVED CERTIFIABLE ROBUSTNESS
Kai Hu, Klas Leino, Zifan Wang, Matt Fredrikson Carnegie Mellon University Pittsburgh, PA 15213, USA (kaihu, kleino, zifanw, mfredrik]ecs. cmu.edu
ABSTRACT
Recent studies have highlighted the potential of Lipschitz- methods for training cer- tifiably robust neural networks against ad adversarial attacks. A key challenge, supported both theoretically and empirically, is that robustness demands greater network capacity and more data than standard training. However, effectively adding capacity under strin- gent Lipschitz constraints has proven more difficult than it may seem, evident by the fact that tate-oof-the-at approach tend more towards underfitting than overfitting. Moreover, we posit that lack of careful exploration of the design space for Lipshitz-basee approaches has left potential performance gains the table. In this work, we provide a more comprehensive evaluation to better uncover the potential of Lipschitz Lipschitz-based based certification methods. Using combination of novel techniques, design optimizations, and synthesis of prior work, we are able to significantly improve the state-of-the-ar VRA for deterministic certification on a variety of benchmark datasets, and over a range of perturbation sizes. Of particular note, we discover that the addition of large moollekk--othogonaliie residual dense" layers the end of existing state-of the-aa art ResNet architectures is especially effective for increasing network capacity and performance. Combined with filtered generative data augmentation, our final results further the state of the art deterministic VRA by up to 8.5 percentage points.
INTRODUCTION
Intentionally crafted perturbations (adversarial examples) have the potential to alter the predictions made by neural networks (Szegedy et al. 2014). Many methods have been proposed to improve the robustness of deep networks, either empirically or provably. In safety-critical domains especially, guarantees against adversarial examples are indispensable. Commonly, provable defenses provide certificates of local robustness to accompany a model's prediction; i.e., predictions should be guaranteed to be consistent within Ey-norm-bounder e-ball around the input. The success of robustness certification techniques is measured by the verified robust accuracy (VRA) the fraction of points with correct predictions that proven to be e-locally robust.
To date, look at the public robustness certification leaderboard (accessed Sept. 2023) shows that the best results are achieved by variants of Randomized Smoothing (RS) (Carlini et al. 2022 Cohen et al. 2019b Jeong et al. 2021 (Salman et al. 2019; Yang et al. 2021). However, there are two primary limitations associated with RS To begin with, RS only offers a probabilistic guarantee, typically configured have a 0.1%% 1.1% false positive certification rate. Perhaps more importantly, the inference of RS involves substantial computational overhead- this limitation is significant enough that these methods are typically tested on only 1% subset of the ImageNet validation dataset due to timing constraints.
Another successful family of methods perform certification using Lipschitz bounds (Araujo al. 2023) Hu et al. 2023 Leino et al. [02T] Trockman & Kolter 202T [aang & Manchester 2023). Essentially, the Lipschitz constant of the neural network provides bound on the maximum change output for given input perturbation, making it possible to certify local robustness. Compared with RS-based methods, Lipschitz-based methods can provide deterministic certification, and are efficient enough to perform robustness certification at scale, e.g., on the full ImageNet (Hu et al. 2023). While Lipschitz-based methods are promising in terms of both deterministic certification and efficiency, there a noticeable performance gap between these methods and based methods. It is not established, however, that this diserepancy is tied to a fundamental limitation of deterministic certification. In this work, we aim to narrow the gap between Lipschitz-basee based and RS-based methods.

---

Published as a conference paper at ICLR 2024
One important avenue for improving the performance of Lipschitz- certification is through increasing model capacity (ability to fit data). Bubeck & Sellke (202)) have shown that robust classification requires more capacity than is necessary for standard learning objectives, and Leino (2023) has shown more specifically that further capacity is required for tight Lipschitz-based certification. But while increasing model capacity for standard training is and using self-attention mechanisms are all possible approaches in Lipschitz- the picture more nuanced because the network's 's Lipschitz constant is tightly controlled, limiting the function's 's expressiveness. Thus, even models with many parameters may still underfit the training objective.
more blocks/layers, increasing the network width
certified training,
In addition, we find that an apparent limitation preventing prior work from discovering the full potential of besscitz-based certification stems from the framing and evaluation setup. Specifically, most prior work is framed around a particular novel technique intended to supersede the state-of-the-an necessitating evaluations centered on standardized benchmark hyperparameter design spaces, rather than exploring more general methods for improving performance (e.g., architecture choice, data pipeline, etc.). Although we introduce several of our own innovations, we present this work as more of a "master class" on optimizing Lipschitz-bascd robustness certification that draws from and synthesizes many techniques from prior work to achieve the best overall performance. This angle lets explore design choices meant to be synergistic to the overall Lipschitz- basee approach, rather than restricting us to choices tailored for head-to to-head comparisons.
This work provides more comprehensive evaluation to illuminate the potential of Lipschitz-baig based certification methods. First and foremost, we find that by delving more thoroughly into the design space of Lipschitz-based approaches, we can improve the state-of-the-art VRA for deterministic certification significaanly on a variety of benchmark datasets, and over a range of perturbation sizes, In the process, we propose a number of additional techniques not already used by the prior literature that contribute to these large performance improvements. That is, our results are achieved using a combination of design optimization, novel techniques, and synthesis of prior work.
After covering the relevant background in Section 2 we begin in Section with a brief survey of the design space for Lipschitz- based certified training, focusing on three key components: (I) architecture choice, (2) methods for controlling the Lipschitz constant, and (3) data augmentation. First, we cover the various architecture innovations and building blocks that have been used in the prior literature. Based on an analysis of the challenges faced by existing work, and motivated by the goal of efficiently increasing network capacity, we propose additional directions to explore along the architecture axis, including two novel network building blocks. Next, provide an overview of the existing methods used for controlling the Lipschitz constant during training, and propose one of our own that can be combined with other approaches. Third we discuss the role data augmentation plays in training high-capacity models. Specifically, DDPM (Karras et al. 2022). which prior work has found helpful for certified training, and propose an alteration to the typical augmentation strategy that find further boosts performance. Section 44 provides an in-depth evaluation that explores along the three dimensions identified in Section [3] shedding light on the most promising design choices, and demonstrating the significant performance improvements we achieve in this work. Finally, Section 5 concludes the paper.
cover
2 BACKGROUND
The problem of providing provable guarantees against adversarial examples is typically formalized using local robustness, which guards against the specific class of small-norm adversarial examples. A classification model, (org)aaaaxx//f (z);, is e-locally robust at point if
Vx'. -/..11--2 ||x1=211 2'11, Se --- -e=> F(z)=F(2)).
Given an upper bound, K, of the Lipschitz constant of f, which is an upper bound of (==))=((=))l//ll=== 11/1|==1111 we can bound the maximum impact of an fy-norn-buunded perturbation. Namely, if = F(x) is the network's 's prediction at x,, F is locally robust at x if (iijj-j f(x), - V2K>> f(x); Note that tighter variant of this certification procedure has been proposed in the literature, e.g., by Leino et al. (2021). Of course, this certification procedure may be very loose, both because it may be hard to obtain tight bound on the Lipschitz constant, and because the Lipschitz constant provides only a worst-case analysis of the model's 's local behavior at x. The key to Lipschitz- based certification is that both of these issues are mitigated by the fact that the Lipschitz constant is tightly controlled (either by hard constraints or by regularization), and the certification procedure is incorporated into the model's loss function.
2

---

Published as a conference paper at ICLR 2024
Lipschitz-based based certification methods in the literature have few common elements. First, all use essentially the same aforementioned certification procedure. Second, the Lipschitz bound for the entire network is bounded using product of the layer-wise Lipschitz constants. The key variation between methods is how they perform what we will call Lipschitz control, which ensures that (1) the Lipschitz bound does not explode, and (2) that the learned function can be (reasonably) tightly certified using the procedure above. We characterize these variations further in Section 3.2
3 DESIGN SPACE
We now turn to brief survey and analysis of the design space for Lipschitz-based certified training. We focus three primary axes of the design space: (1) architecture choice, (2) methods for controlling the Lipschitz constant, and (3) data augmentation, covered in sections sections[3.1 We include discussion of what prior work has done in each axis, as well as our analysis and proposals for further exploration.
3.1 ARCHITECTURES
Lipschitz-based certification has typically made use of a small set of architecture building blocks that are compatible with the overall approach described in Section 2 This includes I-Lipschitz activation functions, dense layers, convolutional layers, and residual layers (with few variations). Modules such as pooling, batch normalization, and attention are not frequently used, either because they lead to loose Lipschitz bounds, or because they are not Lipschitz all. While ReLU activations are I-Lipschitz, in the context of Lipschitz-based certification, they have been ubiquitously replaced by MinMax activations, whose gradient norm preservation property has been shown to be invaluable for this setting by Anil et al. (2019).
Unfortunately, while the space of architectures may be important to explore for maximizing deterministic VRA, prior work has had relatively little exploration here, often using benchmark architectures first proposed half a decade ago. On the other hand, new methods for performing Lipschitz control that present results on larger architectures may come across as misleading, becomes unclear if the performance benefits come from the added capacity or the Lipschitz control method. In this work we resolve this by exploring these axes more independently.
We begin our exploration with the LiResNet architecture (Hu et al. performs best on CIFAR10/100 and Tiny-ImageNet datasets. The LiResNet architecture is composed of 4 parts: (1) the stem layer, a single convolution layer, to convert images in to feature maps; (2) the backbone, a stack several residual convolutional blocks, to extract features from the feature maps; (3) the neck, 2~2 layers to convert the feature map into a flattened vector; and (4) the classification head for predictions. Prior to recent innovations that made residual blocks effective for prnschitz-based certification, deep architectures were not practical. However, with the LiResNet architecture, Hu et al, were able to increase the model capacity by increasing the number of blocks and the number of channels used by the block D. Unfortunately, they report diminishing returns beyond a dozen or blocks (and D 5122), at which point the network capacity is not even enough to overfit the training dataset of CIFAR-10.
2023) as reference point because
We posit that stacking the same block is less effective for adding capacity in Lipschitz- based training, where the network Lipschitz is tightly controlled. Specifically, since the Lipschitz constant is bounded by the product of all blocks' Lipschitz bound, hypothesize that any looseness in the layer-wise bounds compounds, causing overly-deep models to become over-regularized, effectively destroying its capacity. We therefore propose exploration of additional architecture features that can more effectively add capacity beyond the baseline LiResNet architecture.
Attention-like Mechanisms. Attention mechanisms (Dosovitskiy et al., 2020 Vaswani et al. 201 7) have shown excellent ability to improve model capabilities in standard training. However, attention cannot be directly applied to Lipschitz based training since it does not have a Lipschitz bound. One alternative is Spatial-MLP (Touvron et al. 2022 Yu et al., (022). Convolution layers extract local features while the Spatial-MLP can extract local features. A combination of the two different operations may allow richer features. Let X E CXSSS denote the feature map with C channels, and height and width of S; and let W 5 x denote the weights. The formulation of a Spatial-MLP block is (bias ignored):
X(chh,-= (ichw)==[chhaaa -[(ch, + ΣΣ w whhS+w.pS+ [[p..]).
(I)
p=19=1
3

---

Published as a conference paper at ICLR 2024
The lipschitz constant of this operation is || W| 22 We also consider using group of Spatial-MLPP with more parameters to increase model capacity. Supposing we use G groups (the number channels C' should be divisible by G), we would have E. ="5 1sis G as the weights. The formulation of group Res-MLP block is (Ie is the integer in c.G//C,,G/C+
8 s
X [,ch,,)x((hh,uu
W: t (cp-d)
(2)
p=1
The lipschitz constant of this operation is max; 7 W 22).
Dense Layers. Another solution is to add large fully connected (i.e., dense) layers after the neck. Early deep architectures like VGG employ this practice and recent work in Lipschitz based training also gets mileage from many large dense layers (Araujo et al. 2023
We also propose variation standard dense layers inspired by the LiResNet block Hf al. which adds residual connections to a single convolutional by modifying the layer as f(x) = x + conv(r). Analogously for dense layer with weight matrix W, we can add residual connections to form what call a residual dense layer as fxx)=((1 = (wt)=(WW3>) +/)x.
3.2 LIPSCHITZ CONTROL
Lipschitz- -ased certification requires the network to have low Lipschitz constant since upper bound on the Lipschitz constant is used to approximate output changes from input perturbations, and it too large, certification becomes difficult. There are two primary categories of Lipschitz control used in the literature: (I) Lipschitz regularization and (2) Lipschitz constraints.
The prevailing Lipschitz regularization approach is GloRo training proposed by Leino et al. (202I). In this approach, the layer-wise Lipschitz constant are computed as part of the forward pass and used to incorporate Lipschitz-base certification into the training objective. Thus the gradient provides feedback to keep the Lipschitz constant under control and optimized for certification. GloRo regularization is used by Hu et al. (2023), who achieve the current state-of-the-art VRA.
wide variety of Lipschitz constraint approaches exist, typically using special ce-parameterizations that force each linear layer's weights be orthogonal (the Lipschitz constant of orthogonal transformation 1). We consider several of these approaches our design space, described below.
Cayley transformation. (Trockman & Kolter
2021
For skew-symmetric matrix V.
W (T(++V)-(I-VV is orthogonal, thus (
W
1-Lipschitz.
Matrix exponential.
Singla & Feizi 2021 For skew-symmetric matrix V. W =e)p(V) is orthogonal,
thus f<x:V)=1 -ats/>=W>>>>> Wxik is 1- ipschitz.
Layer-wise Orthogonal Training Layer
Xu al. 2022) For non-singular matrix V, (VVT)- - is orthogonal. To obtain a differentiable inverse square root, Newton's iteration steps are performed.
Almost Orthogonal Layer I-Lipschitz.
Prach & Lampert
2022 showed that f<x;V=== diag Ei
V/k))'22 is
SDP-hased Lipschitz Layer.
Araujo et al.
2023
showed that
h(z;W.g) = 2W diag W W -llo(-o)))
1-Lipschitz with I-Lipschitz activation (-)).
Sandwich Layer.
Wang Manchester (2023) showed that
M(,AA,B,0 V2A" TTTa( Bx))
4

---

Published as a conference paper at ICLR 2024
is 1 I-Lipschitz with 1 I-Lipschitz activation of-) if ||2 T < a long orthogonal matrix using Cayley transformation.
holds. The condition is obtained by construct
In addition to the above Lipschite-conntrainee layers from prior work, we propose an approach to orthogonalize weights using the Cholesky decomposition. If E is symmetric positive definite matrix, there exists unique lower triangular matrix L =-Cholesky such that LL =t. Then for non-singular matrix V, SolveTri Solvefriangulaat aagglarryssteeeeoo Cholesky-based orthogonalization comes from the Gram- Sehm-S process for obtaining orthogonal matrix. Namely, Sol process result of orthogonalizing V. Cholesky-based orthogonalization is more numerically stable and efficient. Cholesky-based orthogonalization is typically twice as fast as Cayley transformation to obtain an orthogonal matrix. We propose the following I-Lipschitz layer:
(cholesky(VVT)).V is orthogonal. The motivation of this
SolveTri ve Snnglarryssten (aaestteeemhhcchyhe (Cholesy(VVV ))V) ) is the same as the Gram-Schmidt
Ghoolssy--Orhhoonnaliie Residual Layer. Let V E &R be the parameter, and W is the Cholesky- based orthogonalization result of + V where the identity matrix. The layer is formulated as [a:VV)= = Wa. Note that With the residual formula (analogous to the residual dense layer proposed in Section 3.1. the training of the model can be more effective in the case of stacking multiple such layers.
Although some studies may find certain approaches can approximate certain functions more smoothly, there is no direct theory showing one method has general advantages over others for Lipschitz control. Thus we conduct a fair comparison of all above approaches to find an optimal method empirically. We note that it is also possible to combine various methods of Lipschitz control. Although we do not try all combinations, in our experiments, we use GloRo regularization for convolutional layers while combining different Lipschitz control techniques for the dense layers. See Section 4.2 for details.
3.3 DATA AUGMENTATION WITH GENERATED MODELS
Prior work (Hu et al., 2023) uses IDDPM (Nichol & Dhariwal 2021 (obtain FID of 3.27 for CIFAR-10) to generate samples. We would like to know the performance of certificated robustness can be improved if using generative samples of better quality. We use the elucidating diffusion model (EDM) (Karras et al. 2022) to generate new samples, which obtain a FID of 1.79 for CIFAR-10. For each dataset (CIFARIO, CIFAR1O 110 and Tiny-ImageNet) we train the diffusion models on the corresponding training set using the settings recommended by EDM. Unless otherwise specified, the diffusion models are class-conditional thus the generated images have pseudo-labels.
We also train a standard (non-robust) classification model on each dataset. We use the Re (32x48d) model weakly supervised pre-trained on 940 million (224 x 222) images (Mahajan et al. 2018 We freeze the backbone and only fine-tune the last classification layer with the training dataset. This model archives 94%, 86% and 82% test accuracy on CIFAR-10, CIFAR- 100, and Tiny-ImageNet respectively. We use this classification model's prediction probability of the pseudo-label to score every generated image. Images with the least 20% scores are filtered.
EVALUATION
We now present our evaluation, which includes an exploration of the axes of the design space discussed in Section 3 We begin by showcasing our final result-ie.., the best configuration discovered from our design space exploration-and compare its performance the best VRA results reported in prior work (Section 4.1) To summarize, our ultimate configuration is based on the L12W512 LiResNet architecture proposed by (Hu al. 2023), i.e., its backbone contains 12 linear residual convolution blocks with 512 channels each. We modify this architecture by using Cholesky-based orthogonalization for the dense layers in the neck (which Huu et al.] controlled using GloRo regularization on standard dense layers), and adding 8
residual dense (CHORD) layers with 2,048 neurons each to the end of the neck. We refer to this proposed architecture configuration as "CHORD LiResNet." Note that we still use GloRo regularization for Lipschitz control the convolutional layers. This model is trained using our improved generated data augmentation pipeline (see Table for details). We provide further details on the exact training parameters in Appendix B
we provide an overview breaking down the various improvements we applied to reach our final configuration, followed by more detailed ablation studies comparing various Lipschitz
Next, in section 4.2
5

---

Published as a conference paper at ICLR 2024
Table I: This table presents the clean and verified robust accuracy (VRA) of several concurrent works and
our GloRo CHORD LiResNet models on CIFAR-101100, TinylmageNet and ImageNet datasets.
Clean Acc. (%)
VRA (%) at
Dataset
Method
258
255
GloRo (Leino 2021
77.0
58.4
Local-Lip-B (+MaxMin)) [Tuang el
2021
77.4
39.0
Cayley Large Trockman & Kolter 2021
74.6
61.4
46.4
32.1
SOC 20 (ingll Feizi
2021
76.3
62.6
48.7
36.0
CPL XL Meunier
2022
78.5
64.4
48.0
33.0
CIFAR-10 10
AOL Large Prach & Lamperi
Prach
2023
71.6
56.4
SLL X-Large/Arrnjoo et al. 2023
73.3
65.8
58.4
51.3
GloRo LiResNet (+DDPM) Hu
2023
82.1
70.0
GloRo CHORD LitesNet (+DDPM)
87.0
78.1
66.6
53.5
Cayley Large [Trokkaan Kolter
2021
43.3
29.2
18.8
11.0
SOC 20 (iigll & Foizi
2021
47.8
34.8
23.7
15.8
CPL XL Meunier or al.
2022
47.8
33.4
20.9
12.6
AOL Large (Prach Lampent
2022
43.7
33.7
26.3
20.7
CIFAR-100
X-Large Araujo
46.5
36.5
29.0
23.3
Sandwich (Wang & Manchester
2023
46.3
35.3
26.3
20.3
GloRo LiResNet (+DDPM)
Huuttal
2023
55.5
41.5
GloRo CHORD LitesNet (+DDPM)
62.1
50.1
38.5
29.0
GloRo (Leino 2021
35.5
22.4
Local-LippB (+MaxMMn)) (Huang ctal.
2021
36.9
23.4
12.7
SL.I. X-Large Anmio et al.
2023
32.1
23.2
16.8
12,0
TinylmageNet
Sandwich Wang & Munchester
2023
33.4
24.7
18.1
13.4
GloRo LiResNet (+DDPM) Hurttll
2023
46.7
33.6
GloRo CHORD LiResNet (+DDPM)
48.4
37.0
26.8
18.6
GloRo LiResNet Hu al 2023
45.6
35,0
ImageNet
GloRo CHORD ee (+DDPM)
49.0
38.3
control methods and data augmentation generation pipelines. Finally, in Section 4.3 method with randomized smoothing based methods, demonstrating that work brigees the gap between deterministic and stochastic certification.
we compare our
4.1 COMPARISON WITH PRIOR WORKS
We compare our GloRo CHORD LiResNet with the following works from the literature: Vanilla GloRo Nets with TRADES loss, Cayley, Local-Lip Net, SOC with Householder and Certification Regularization (HH+CR), CPL, SLL, Sandwich, and GloRo LiResNet, which are selected for having been shown surpass other approaches. Table presents the clean and certified accuracy with different radii of certification 72/255, and 108 108/2a5 on CIFAR- and Tiny- Liny-mmaeNNe. We can see that our approach outperforms all existing architectures with significant margins on clean accuracy and certified accuracy for all values of e. On CIFAR- 10/100, our model improves the certificated accuracy at < == 36/255 by more than 8%. We also compare the empirical robustness of the proposed method with some recent work Section A
To date, Hu et al. 2023 is the only work to report results on ImageNet. However they do not use generated data on ImageNet. We generated 2 million samples using guided diffusion boost model. Other settings are the same those on CIFAR- 10/100. With the improved model and generated data, we further improve the certification accuracy on ImageNet by 3.3%.
4.2 ABLATION STUDIES
Cumulative Effects from Modifications. Since CHORD LiResNet is based on LiResNet (Hu et al., 2023). one would be interested to see the breakdown of the improvements from each modification. Table shows the results. Modifications in the grey not applied to the final model.

---

Published as a conference paper at ICLR 2024
Table 2: This table presents the breakdown improvement of each modification. Numbers are repored in certificated accuracy (6=36/255) on CIFAR-10/100 datasets.
Modification
CIFAR-10
CIFAR-100
Baseline (LiResNet L12W512) w/o modifications
70.0
41.5
Adding Spatial-MLP blocks after the last convolution layers
70.2 (+0.2)
41.3
Inserting 1 Spatial-MLP block after every convolution layer
70.3 (40.3)
41.7 (40.2)
Adding dense layers before classification head
73.4 (+3.4)
43.6 (+2.1)
Adding 16 dense layers before classification head Using Cholesky-based orthogonalization in the neck
73.7 (43.7)
44,0 (42.5)
74.5 (+1.1)
44.9 (+1.3
Using CHORD layers in place of the 8 dense layers
75.6 (+1.1)
46.5 (+1.6)
Adopting better pipeline to use generated data for augmentation
78. (42.5)
50.1 (3.5)
Table 3: This table presents the clean accuracy and VRA using different dense layers CIFAR-10/100
datasets. All other settings are the same.
Clean Accuracy
VRA (%) at
Dataset
Layer Choice
36
72
108
255
233
253
Regular Dense Layer (w/ GloRo regularization)
86.1
77.0
65.6
52.4
Cayley Dense Layer
86.3
77.2
65.5
52.1
Almost Orthogonal Layer (AOL)
85.1
75.4
63.7
50.0
SDP-based Lipschitz Layer (SLL)
85.5
75.6
65.6
52.3
CIFAR-10
Sandwich Layer
85.4
76.1
64.0
51.0
Matrix Exponential Cholesky-orthogomaliiee Residual Dense (CHORD) Layer
87.0
78.1
66.7
53,6
87.0
78.1
66.6
53.5
Regular Dense Layer (w/ GloRo regularization)
60.4
48.2
36.9
27.0
Cayley Dense Layer
61.3
49.1
38.0
28.3
SDP-based Lipschitz Layer (SLL)
61.5
48.9
37.3
27.3
CIFAR-100
Sandwich Layer
61.3
48.3
36.6
27.1
Matrix Exponential
61.7
49.8
38.6
28.8
Cholesky-orthoggnalizee Residual Dense (CHORD) Layer
62.1
50.1
38.5
29.0
We explore different ways of using spatial MLPs. As shown in Table 2 we can either add all spatial-MLP blocks after the last convolution layer or insert one spatial-MLP block after every convolution layer. However, the improvement is very limited. We also consider using spatial MLPs with groups so that can obtain larger model capacity from more parameters. We experiment with 2,4,".,32 groups, and there no great difference. Although spatial-MLP shows comparable performance to transformers and conv-net in standard training, the capacity of spatial-MLP may not be enough for certificated training in Lipschitz bounded setting.
Adding dense layers to the model can increase the model capacity significantly. Dense layers introduce millions of parameters with fairly small computational cost. In standard training, this property can make large dense layers prone to overfitting. However, in Lipschitz-based certified training, the smoothness of the model is controlled by the network's 's Lipschitz constant, thus large dense layers improve the capacity (and thus VRA) without significantly overfitting. Adding more dense layers (16 layers) exhibited diminishing returns, thus we choose 8 dense layers in our final configuration.
Orthogonalizing the dense layers is another effective way to improve the model's certified robustness. However, as we mentioned earlier, we do not apply orthogonalization to convolutions since this requires applying FFTs and inverse FFTs the feature maps, which can become expensive in large models discuss this further in our presentation of our ablation study on Lipschitz control mechanisms.
An Ablation Study on the Lipschitz Control Mechanism. We conduct fair comparison between the various approaches to Lipschitz control discussed in Section 3.2 to find an optimal method for controlling the Lipschitz constant the many dense layers in our modified LiResNet architecture. We use the same backbone and train the model with the same settings the same data (original dataset plus generated data). changing only the Lipschitz control mechanism for the dense layers- for convolutions we still use Gloro-based regularization for Lipschitz control. Table shows the results CIFAR-10/100 datasets and
7

---

Published as a conference paper at ICLR 2024
Table 4: This table shows the effect of different generated data augmentation pipelines. VRA of GloRo CHORD LiResNet at radius <= 336255 is reported.
Sample Generator
Filtered by classification score
Real/Generated
CIFAR-10
CIFAR-100
sample ratio a batch
IDDPM
75.6
46.5
IDDPM
1:
75.9
47.6
EDM
X
76.1
47.3
EDM
76.5
48.5
1:
77.1
49.2
1:3
78.1
50.1
EDM
1: 0
69.2
39.0
0:
75.4
47.6
Figure showws the learning curve of certificated accuracy on the test set. We find that performance is better when parameterizing the dense layer weights orthogonal matrices (Cayley, Matrix Exp, Cholesky). Cholesky-based and Matrix Exponential orthogonalizatton perform similarly, but Matrix Exponential slower. It takes 32.4, 37.8 and 51.2 seconds train epoch with CHORD, Cayley and Matrix Exp respectively on CIFAR-IO using the same A100 machine. We conclude that CHORD layers are the optimal Lipschitz control choice for the dense layers considering both performance and efficiency.
From Figure I. we see that using a regular dense layer and applying GloRo-style Lips- chitz regularization for the final layers per- forms comparatively poorly in the early train- ing stages but surpasses SLL and Sandwich orthogonalization after ~300-600 epochs. Be- cause the dense layers are not constrained to be orthogonal this case, the model requires more steps to learn nearly orthogonal trans- formation. Note that this particularly pro- nounced in the case of large dense layers, as opposed to the convolutional layers, which also rely on GloRo regularization. Leino 2022) has shown that a reliable gradient signal for or- thogonalizing linear transformation requires more power iterations as the dimension of the eigenvector increases. For large dense layers, the eigenvectors are high-dimensional, as com- pared those of convolutions, which depend only on the size of the kernel (which is typically small). Thus we expect GloRo regularization to converge more slowly on dense layers than on convolutional ones.
Certifi cated accuracy on CIFAR 10 using different dense layer
76
74
Accurcy
Glote
65-
Matri Exp
SL
Cayley
100 200
30c
400
500
600
700
80C
Epoch
Figure 1: Certified accuracy (i.e., VRA) our modified LiResNet architecture CIFAR-IO using different Lips- chitz control methods on the dense layers during training.
Ablation Study on the Generated Data Augmentation. As shown in Table better pipeline to apply generated data augmentation can improve certification robustness significantly. Table shows detailed study on the effects of different pipelines. Switching to a better generator provides consistent improvements on both datasets whether or not the sampled images are filtered. Using a stronger classification model to remove samples with least 20% low confidence pseudo-labels can also help. On CIFAR-100, the improvement is more significant. The reason is that CIFAR 100 has more categories and therefore the diffusion model generates a higher proportion of images with unmatched pseudo-labede Using these labels can harm robust training. In the second part of the table, we study the ratio real and generated samples in batch. We find that seeing more generated samples can significantly improve the model's certification robustness. However, we only use generated samples to train the model (real/generated sample ratio = 0: 1), it suffers from overfitting and the performance is decreased. From this experiment, think the reason the generated data helps is not that the generated data is better quality, but generated data are easier to classify average (training generated data has faster training accuracy convergence). As
8

---

Published as a conference paper at ICLR 2024
Table 5: This table presents the clean and certificated robust accuracy of several probabilistic works and our deterministic GloRo CHORD I. LiResNet on CIFAR-10 dataset.
VRA (%) measured at at e
method
c= 0.25
0.5
0.75
1.0
RS Cohen et al.
2019b)
61.0
47..0)43.0
(6..0)
(86.0)
SmoothAdv (Salman et al.
2019
(75.6) 67.4
(7..6)57.6
(74.8)
(57.4)38.3
AmoothAdv Salman et al.
2019
(84.3)74.9
(80.1)63.4
(1.9
(62.2)99.6
Consistency (Jeong & Shin
2020
(77.8) 68.8
(75.8)58.1
(72.9)
(52.3))7.8
MACER ZZai et al. 2020
(81.0)71.0
(8..0)59.0
(66.0)
(66.0)38.0
DRT Yang et al. 2021)
78..5)70.4
(72.6)60.2
(56.1)39.8 50.5
SmoothMix (Jeong et al.
2021
(77.
577.1)
(61.8) 47.72)
Denoised (Salman al.
2020
56.00)
(62.0)41.0
(62.0)
(41.0) 19.0
DDS Carlini al. 2022)
79.3
65.5
(877
(5,55)35.5
GloRo CHORD LiRes (Ours)
(875
52.23)
(1070)
(08.1)35.1
we mentioned before, Lipschitz-based training suffers from underfitting and much of the model capacity is used to remember hard samples, including outliers and samples very close to the decision boundary. Learning from these hard samples does not improve robustness since these samples are not naturally non-robust. When trained with generated samples (which are easier), the percent of hard samples in the dataset is decreased and the model can focus more on learning a robust decision boundary. As a contrary, generated data not always improve the performance of standard accuracy (Azizi et al. (023) even when SOTA diffusion models are used. In the standard training setting, neural network can fit hard samples more easily. Adding too many generated samples (In Azizi et al. (223)'s 's setting 6 times of the original training set), the test accuracy would decrease since most hard samples help generalization and their proportion has decreased with generated data added.
4.3 COMPARISON WITH RANDOMIZED SMOOTHING
To date, the methods that achieve the best certified performance are derived from randomized smoothing (RS) Cohen et al. (2019a). As we discussed, Lipschitz-based methods demonstrate advantages over RS has in terms of their efficiency and the guarantce that they provide. We provide the first comparison between these methods in Table Notably, we are able to outperform several recent RS-based approaches on some or all certification radii.
5 CONCLUSION
Our primary objective in this work is to enhance the certified robustness of neural networks. We contend that significant problem of existing Lipschitz-based models is their limited capacity, which hinders their ability to even overfit small datasets. To address this challenge, we have reexamined network architectures and basic building blocks to control network Lipschitz and have proposed three solutions to mitigate this issue. First, we showed that combination of dense layers and convolutions can effectively expand the model's 's capacity. While conventional non-robust models use fewer dense layers, we find that dense layers are ideal for efficient capacity augmentation in the robust setting, which suffers less from overfitting and has a greater need for extreme capacity. Second, we introduced the Cholesky Cholesky-orthogomee -othhgoonalized Residual Desne (CHORD) I Layer, which serves as an efficient building block for achieving orthogonal weights in dense layers. Although I. Lipschitz regularization remains best suited for Lipschitz control on convolutional layers, we find dense layers-which which have very high- high-dimension eigenvectors bennefi benefit from orthogonalization. Third, we explored an improved pipeline for utilizing generated data to enhance Lipschitz-based certified training. Through extensive experiments, we have demonstrated the effectiveness of our design choices. Our final results have pushed the boundaries of deterministic certified accuracy CIFAR- 10/100 datasets, surpassing the state of the art by up to 8.5 percentage points. Our method opens up a promising avenue to bridge the gap between probabílistic and deterministic certification methods, which we believe crucial to making certification practical.

---

Published as a conference paper at ICLR 2024
REFERENCES
Cem Anil, James Lucas, and Roger Gross. Sorting out Lipschitz function approximation. In International Conference on Machine Learning (ICML.), 2019.
Alexandre Araujo, Aaron Havens, Blaise Delattre, Alexandre Allauzen, and Bin Hu. A unified algebraic perspective on lipschitz neural networks. In The Eleventh International Conference on Learning Representations, 2023.
net weee://oormmmddk1111iia
Shekoofeh Azizi, Simon Korblith, Chitwan Saharia, Mohammad Norouzi, and David Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint (2t:::300000866,, 2023.
Sebastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. In Advances in Neural Information Processing Systems, 2021.
Nicholas Carlini, Florian Tramer, Zico Kolter, et al. (certified!!) adversarial robustness for free! arXiv preprint arXiv:2206. 10550, 2022.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning (ICML), 2019a.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on muchine leurning, pp. 1310-1320. PMLR, M, 2019b.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning (ICML), 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Kai Hu, Andy Zou, Zifan Wang. Klas Leino, and Matt Fredrikson. Scaling in depth: Unlocking robustness certification imagenet. arXiv preprint arXiv:2301. /2549, 2023.
Yujia Huang, Huan Zhang, Yuanyuan Shi, J. Zico Kolter, and Anima Anandkumar. Training certifiably robust neural networks with efficient local lipschitz bounds. In NIPS, 2021.
Jongheon Jeong and Jinwoo Shin. Consistency regularization for certified robustness of smoothed classifiers. Advances in Neural Information Processing Systems, 1011005515555 2020.
Jongheon Jeong. Sejun Park, Minkyu Kim, Heung-Chang Lee, Do-Guk Kim, and Jinwoo Shin. Smooth- mix: Training confidence-caliivatte smoothed classifiers for certified robustness. Advances in Neural Information Processing Systems, 14:30153-30166 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-26577 2022.
Klas Leino. Identifying, analyzing, and addressing weaknesses in deep networks: Foundations for conceptually sound neural networks, 2022.
Klas Leino. Limitations of piecewise linearity for efficient robustness certification. arXiv preprint arX1v:2301.08842, 2023.
Klas Leino, Zifan Wang, and Matt Fredrikson. Globally-robust neural networks. In International Conference on Machine Learning, pp. 6212-6222. PMLR, 2021.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li. Ashwin Bharambe, and I Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pp. 181-196, 2018.
Laurent Meunier, Blaise J Delattre, Alexandre Araujo, and Alexandre Allauzen. A dynamical system perspective for Lipschitz neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 15484-15500 15500. PMLR, 17-23 Jul 2022. _RR.tttt.p
htttttsss/pprcceediiieeeeo http::///occeeeddeeeee ml press/vi 62/mmun er22a htm
10

---

Published as a conference paper at ICLR 2024
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. P.L. 2021.
Bemd Prach and Christoph H Lampert. Almost-orthogonal layers for efficient general-purpose lipschitz networks. In European Conference on Computer Vision, pp. 350-365. Springer, 2022.
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Advances in Neural Information Processing Systems, 32, 2019.
Hadi Salman, Mingjie Sun, Greg Yang. Ashish Kapoor, and J Zico Kolter. Denoised smoothing: A provable defense for pretrained classifiers. Advances in Neural Information Processing Systems, 33: 21945-21957, 2020.
Sahil Singla and Soheil Feizi. Skew orthogonal convolutions. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9756-9766. PMLR, 18-24 Jul 2021.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations (ICLR). 2014.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
Asher Trockman and J Zico Kolter. Orthogonalizing convolutional layers with the cayley transform. In International Conference on Learning Representations (ICLR), 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Ruigang Wang and Ian Manchester. Direct parameterization of lipschitz-buunded deep networks. In International Conference Machine Learning, pp. 36093-361 10. PMLR, 2023.
Xiaojun Xu, Linyi Li, and Bo Li. Lot: Layer-wise orthogonal training on improving 12 certified robustness. arXiv preprint arXiv:2210. 1620, 2022.
Zhuolin Yang, Linyi Li, Xiaojun Xu, Bhavya Kailkhura, Tao Xie, and Bo Li. On the certified robustness for ensemble models and beyond. arXiv preprint arXiv:2107.10873 2021.
Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. S2-mlp: Spatial-shift mlp architecture for vision. In Proceedings of the IEEF/CVF winter conference applications of computer vision, pp. 297-306,
2022.
Runtian Zhai, Chen Dan, Di He, Huan Zhang, Boqing Gong, Pradeep Ravikumar, Cho-Jui Hsieh, and Liwei Wang. Macer: Attack-free and scalable robust training via maximizing certified radius. arXiv preprint arXiv:2001.02377 2020.

---

Published as a conference paper at ICLR 2024
A EMPIRICAL EVALUATION OF OUR MODEL
As an evaluation the empirical robustness GloRo CHORD LiResNet in comparison with recent work, we use AutoAttack (Croce & Hein 22200, an ensemble of diverse parameler-free attacks. Table presents the results. Results of SLL and WandWich come from Wang & Manchester (2023). We find that not only does our model outperform prior work in terms of empirical robust accuracy, but additionally, when comparing Ta Table [ in Section 4.1 our model's ' ceritified robust performance (VRA) is higher than the empirical robust performance of the prior work under comparison.
Table 6: This table presents the clean and empirical robust accuracy several concurrett works and our GloRo CHORD LiResNet models CIFAR- 10/100 and TinyImageNet datasets.
Clean Accuracy
AutoAttack (e)
Datasets
Models
36
72
108
255
CPL XL Meunier et al. 2022
78.5
71.3
SLL X-Large Araujo 2025
73.3
70.3
65,4
60.2
CIFAR-10
GloRo CHORD LiResNet (Ours)
87.0
82.3
76.4
69.6
CPL XL Meunier ot al.
2022
47.8
39.1
SLL Large (Araujo 2023
54.8
44.0
34.9
27.5
CIFAR-100
Sandwich Wany & Manchester 2023
57.5
48.5
40.2
32.9
GloRo CHORD LiResNet (Ours)
62.1
55,5
48.7
42.2
SLL Medium Armjo et al, 20233
30.3
24.6
19.8
15.7
TinyImageNet
Sandwich Medium /Wang Manchester
2023
35.5
29.9
25.3
21.4
GloRo CHORD LiResNet (Ours, +DDPM)
48.4
42.9
38.1
33.7
B TRAINING DETAILS
We follow most the training settings used by Hu (2023) for their GloRo LiResNet model. The first difference is that we change the maximum training perturbation radius fuain = 108/255 as need to report certificated robustness performance of this radius during inference. The original setting is Etaiin = 72/235 as Hu al. only report the certificated robustness performance at E 33666625, The second difference is the choice of generated data. We generated the generated data as described in (Hu et al.) [2023). We train from the original dataset and the rest 768
the model with batch size of 1024 where 256 samples samples are generated. We do not change other settings including the learning rate.
C A NOTE ABOUT SLL RESULTS IN TABLE 3
We discovered an issue in the publicyy-avilable SLL implementation that causes the SLL layer to be non-L.pschitz. After communicating with the authors over email, devised implementation that addressed the issue. Our results using SLL layers in Table are based on this fix, and do not reflect the performance of the current public implementation of SLL.
12


Reviews:
[{"review_id": "KTjkw4nbuO", "reviewer": "Reviewer_BQAE", "summary": "This paper improves L2 deterministic certifiably robust training from three aspects: 1) adding additional layers; 2) using Cholesky-based orthogonal layers in the neck; 3) data augmentation with a newer diffusion model.", "questions": "* How does the Cholesky-base orthogonalization help on VRA?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* By combining technical improvements on three aspects as mentioned in the summary, the paper shows a significant empirical improvement over previous works across all the datasets (e.g., +8% on CIFAR-10). \n* This work provides suggestions on better settings for the robust training, in terms of model architecture with additional layers, building orthogonal layers with Cholesky decomposition, and data augmentation with a newer diffusion model.", "weaknesses": "* The paper looks like manually searching for settings (model architecture, orthogonal layers, diffusion model). It has engineering merits. But it does not have much novel contribution by adding more dense layers and replacing the diffusion model already used in Hu et al,, 2023 with a newer diffusion model.\n* The benefits of the best choices found by the paper are not well explained. For example, the paper only explains that the Cholesky-base orthogonalization is more numerically stable and faster, but it does not explain why it can improve VRA. Therefore, the paper provides limited insights, in its current form.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QsRQUbKIKG", "reviewer": "Reviewer_9Qkw", "summary": "The work proposed some new architectures to enhance the certified robustness of neural network by reducing the Lipschitz constant. Empirical experiments showed some improvement.", "questions": "Can the authors include training time and inference time for the new architecuture?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work studies the limitation for Lipschitz-based certification and proposed new architectures to mitigate the issue.\n2. Strong empirical result: experiments showed noticeable improvement over the baseline models.", "weaknesses": "The authors need to include some intuitions when designing the layers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "80kkNJGRT9", "reviewer": "Reviewer_TVPw", "summary": "It provides a more comprehensive evaluation to better uncover the potential of Lipschitz-based certification methods. Using a combination of novel techniques, design optimizations, and synthesis of prior work, it is able to improve the state-of-the-art VRA for deterministic certification on a variety of benchmark datasets, and over a range of perturbation sizes.", "questions": "see the weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It finds that an apparent limitation preventing prior work from discovering the full potential of Lipschitz-based certification stems from the framing and evaluation setup. Specifically, most prior work is framed around a particular novel technique intended to supersede the state-of-the-art, necessitating evaluations centered on standardized benchmark hyperparameter design spaces, rather than exploring more\ngeneral methods for improving performance (e.g., architecture choice, data pipeline, etc.).\n\n\nIt discovers that the addition of large “Cholesky-orthogonalized residual dense” layers to the end of existing state-of-the-art Lipschitz-controlled ResNet architectures is especially effective for increasing network capacity and performance. \n\nThis work provides a more comprehensive evaluation to illuminate the potential of Lipschitz-based certification methods. It finds that by delving more thoroughly into the design space of Lipschitz-based approaches, it can improve the state-of-the-art VRA for deterministic certification significantly on a variety of benchmark datasets, and over a range of perturbation sizes. Combined with filtered generative data augmentation, the final results further the state of the art deterministic VRA by up to 8.5 percentage points.\n\nIt provides an overview of the existing methods used for controlling the Lipschitz constant during training, and propose its own method that can be combined with other approaches.\n\nIt discusses the role data augmentation plays in training high-capacity models. It covers DDPM, which prior work has found helpful for certified training, and proposes an alteration to the typical augmentation strategy that further boosts performance.", "weaknesses": "In section 4.3, it seems to mainly discuss the comparison with RS based methods. But Table 5 shows several other works which can achieve better performance. It is better to also discuss the comparison with these works. Currently it seems that table 5 only shows the results without detailed discussions for these works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)