CTBench: A Library and Benchmark for Certified Training

## Abstract

Abstract Training certifiably robust neural networks is an important but challenging task. While many algorithms for (deterministic) certified training have been proposed, they are often evaluated on different training schedules, certification methods, and systematically under-tuned hyperparameters, making it difficult to compare their performance. To address this challenge, we introduce CTBench , a unified library and a high-quality benchmark for certified training that evaluates all algorithms under fair settings and systematically tuned hyperparameters. We show that (1) almost all algorithms in CTBench surpass the corresponding reported performance in literature in the magnitude of algorithmic improvements, thus establishing new state-of-the-art, and (2) the claimed advantage of recent algorithms drops significantly when we enhance the outdated baselines with a fair training schedule, a fair certification method and well-tuned hyperparameters. Based on CTBench , we provide new insights into the current state of certified training and suggest future research directions. We are confident that CTBench will serve as a benchmark and testbed for future research in certified training.

## 1 Introduction

As a crucial component of trustworthy artificial intelligence, adversarial robustness (SzegedyZSBEGF13; GoodfellowSS14), \ie, resilience to small input perturbations, has established itself as an important research area. While initially the community focused on heuristic methods to craft adversarial examples and defenses against these, it turned out that such defenses are often brittle and can be evaded by adaptive adversaries (AthalyeC018; TramerCBM20). Thus, neural network certification has emerged as a method for providing provable guarantees on the robustness of a given network (GehrMDTCV18; WongK18; ZhangWCHD18; SinghGPV19).

Two families of neural network certification methods have been proposed: complete methods (KatzBDJK17; TjengXT19) which compute the exact bounds but are extremely computationally expensive, and convex-relaxation based methods (ZhangWCHD18; SinghGPV19) which provide approximate bounds but are more scalable. State-of-the-art (SOTA) verifiers (XuZ0WJLH21; FerrariMJV22; ZhangWXLLJ22) combine both approaches, by using convex relaxations to speed up the solving of complete methods via Branch-and-Bound (BunelLTTKK20).

However, the scalability of neural network certification is still a major challenge since the computational complexity of SOTA verifiers grows exponentially with network size. To tackle this issue, certified training (MirmanGV18; GowalIBP2018) was proposed in order to train neural networks that are amenable to certification. Such methods are typically categorized into two groups: (1) training with a sound upper bound of the robust loss (ZhangCXGSLBH20; ShiWZYH21), and (2) training with an unsound surrogate loss that aims to approximate the exact robust loss (MuellerEFV22; MaoM0V23; palma2024expressive). The latter group has been shown to be more effective.

While certified training has made significant advances, there is currently no benchmark that can be used to fairly evaluate the effectiveness of the different certified training methods. Specifically, the literature often compares against previous methods using quoted numbers due to high computational costs, although the verifier and certification budget differ. These unfair comparisons ultimately hinder the community from drawing reasonable conclusions on the effectiveness of certified training methods. In addition, existing works systematically under-tune hyperparameters, in order to show effectiveness against baselines, thus establishing a weaker SOTA. Further, there is no unified codebase for these methods, making future development and comparison difficult.

Figure: Figure 1: Reduction in certified error on MNIST $\epsilon=0.3$ (lower is better).
Refer to caption: /html/2406.04848/assets/x1.png

#### This work: a unified library and high-quality benchmark for certified training

We address these challenges, for the first time unifying SOTA certified training methods into a single codebase called CTBench. This enables a fair comparison between certified training methods and re-establishes a much stronger SOTA by fixing problematic implementations and systematically tuning hyperparameters. As shown in [Figure 1](#S1.F1), these steps lead to significant improvements uniformly.
In addition, we show that the claimed advantage of recent SOTA reduces significantly when we apply the same budget and hyperparameter tuning to all methods. Based on our released model checkpoints, we provide an extensive analysis of the model properties, highlighting many new insights on its loss landscape, mistake patterns, regularization strength, model utilization, and out-of-distribution generalization. We are confident that CTBench will serve as a benchmark and testbed for future work in certified training.

## 2 Related Work

We now briefly review key developments most related to our work.

#### Benchmarking Certified Robustness

LiXL23 provides the first benchmark for certified robustness, covering not only deterministic certified training but also randomized certified training and certification methods. However, it is outdated and thus provides little insight into the current SOTA methods. For example, it reports 89% and 51% best certified accuracy for MNIST $\epsilon=0.3$ and CIFAR-10 $\epsilon=\frac{2}{255}$ in its evaluation, respectively, while recent methods have achieved more than 93% and 62% (MuellerEFV22; MaoM0V23; palma2024expressive).

#### Certified Training

DiffAI (MirmanGV18) and IBP (GowalIBP2018) apply box relaxation to upper bound the worst-case loss for training. Efforts have been made towards applying more precise approximations: WongSMK18 and BalunovicV20 apply DeepZ (SinghGMPV18), while ZhangCXGSLBH20 incorporate linear relaxations (ZhangWCHD18; SinghGPV19). While these approximations are more precise (baader2024expressivity), they often lead to worse training results, attributed to non-smoothness (LeeLPL21), discontinuity and sensitivity (jovanovic2022paradox) of the loss surface. Some recent work (balauca2024overcoming) aim to mitigate these problems, however, the most effective training approximation is still the least precise box relaxation. In this regard, the focus of the community has shifted towards improving IBP: ShiWZYH21 propose a new regularization and initialization paradigm to speed up IBP training; PalmaIBPR22 apply IBP regularization to make adversarial training certifiable; MuellerEFV22, MaoM0V23 and palma2024expressive propose unsound but more effective IBP-based surrogate losses for training; mao2024understanding propose to use wider models instead of deeper models for IBP-based methods. These methods achieve universal advantages and are thus the focus of our work.

## 3 Background

We now introduce the necessary background for our work, both training concepts and algorithms.

### 3.1 Training for Robustness

We present the mathematical notations on adversarial and certified training here. We consider a neural network classifier $f_{\theta}(x)$ that estimates the log-probability of each class and predicts the class with the highest estimated log-probability.

#### Adversarial Training

A classifier $f_{\theta}(x)$ is said to be *adversarially robust* with radius $\epsilon$ w.r.t. $L_{\infty}$ perturbation if $f_{\theta}(x+\delta)=y$ for all $\|\delta\|_{\infty}\leq\epsilon$, where $y$ is the ground truth label of $x$. For brevity, we will omit the perturbation type in the rest of the paper. Finding an adversarially robust classifier is formally defined to solve a min-max problem $\theta=\operatorname*{arg\,min}_{\theta}\mathbb{E}_{x,y}\max_{\|\delta\|\leq\epsilon}L(x+\delta)$. In this regard, adversarial training solves the inner maximization problem by generating adversarial examples during training, and the outer minimization problem by optimizing the empirical loss of adversarial examples.

#### Certified Training

A classifier $f_{\theta}(x)$ is said to be *certifiably robust* if it is adversarially robust and there exists a sound verifier that certifies the robustness. A verifier typically computes an upper bound on the margin $f_{i}(x+\delta)-f_{y}(x+\delta)$ and certifies its robustness if the upper bound is negative for all $i\neq y$. Certified training thus replaces the inner maximization problem with an upper bound and minimizes the upper bound during training instead.

#### Metrics

The main metric for certified training is *certified accuracy*, defined to be the ratio of certifiably robust samples in the dataset. The ratio of correctly classified samples in the dataset is thus called *natural accuracy*. For reference, we include *adversarial accuracy* as well, defined to be the ratio of adversarially robust samples in the dataset. We apply one of the most widely used SOTA certification methods, MN-BaB (FerrariMJV22), as the verifier. To compute adversarial accuracy, we apply the strong AutoAttack (Croce020a) for adversarial training, and a combination of PGD attack and branch-and-bound attack from MN-BaB for certified training.

### 3.2 Algorithms in CTBench

Here, we briefly introduce the core algorithms built into CTBench. Concepts behind these algorithms are visualized in [Figure 2](#S3.F2).

Figure: Figure 2: Conceptual overview of core algorithms built into CTBench.

#### PGD and EDAC

Projected Gradient Descent (PGD) (MadryMSTV18) is the most widely recognized adversarial training method. Starting from a random initialization, PGD solves the inner maximization problem by iteratively taking a step towards the gradient direction and clipping the result into the valid perturbation set. Then, it uses the generated adversarial input $x^{\prime}$ to compute the worst case loss as $L(x^{\prime})$. Croce020a find that PGD remains effective against strong attacks, thus is popular as an integrated part of many certified training methods (MuellerEFV22; MaoM0V23; palma2024expressive). To further improve adversarial robustness, zhang2023generating improves adversarial generalization via an extra-gradient method called EDAC, which remains one of the SOTA methods in adversarial training. These methods achieve good but uncertifiable adversarial robustness, hence we use them as adversarial robustness baselines in CTBench.

#### IBP

Interval Bound Propagation (IBP) (GowalIBP2018) uses interval analysis to approximate the output range of each layer. For example, for the toy network $y=2-\operatorname*{ReLU}(x_{1}+x_{2})$ with input bounds $x_{1},x_{2}\in[-1,1]$, it first computes the output range of the first layer as $x_{1}+x_{2}\in[-1,1]+[-1,1]\subseteq[-2,2]$, the second layer as $\operatorname*{ReLU}([-2,2])\subseteq[0,2]$ and then final layer as $2-[0,2]\subseteq[0,2]$, thus proving $y\geq 0$ for all possible $x_{1},x_{2}\in[-1,1]$. Similarly, IBP computes the layer-wise bounds and then derives the worst-case loss based on the output bounds of the final layer. To stably train models with IBP, ShiWZYH21 propose to rescale the parameter initialization to ensure constant growth of IBP bounds and a specialized regularization to control the activation status of neurons. They also show that adding a batch norm (IoffeS15) layer before every ReLU can improve IBP training. These training tricks are adopted by every IBP-based method introduced below. For brevity, we refer to this variant as IBP in the rest of the paper unless otherwise stated, since it improves the original IBP universally with tricks that facilitate training.

#### CROWN-IBP

CROWN-IBP (ZhangCXGSLBH20) tightens the imprecise interval analysis with linear relaxations of ReLU layers based on IBP bounds and only solves the linear constraints for the final layer output based on CROWN (ZhangWCHD18), avoiding prohibitive costs during training. To further reduce the cost of solving the bounds for each class, XuS0WCHKLH20 propose a loss fusion trick to only solve for the final loss, thus reducing the asymptotic complexity by a factor equal to the number of classes. For brevity, we refer to this variant as CROWN-IBP in the rest of the paper unless otherwise stated, since the original CROWN-IBP cannot scale to datasets with many classes.

#### SABR

Since IBP is often criticized for the increasingly strong regularization w.r.t. input radius imposed on the neural network, SABR (MuellerEFV22) proposes to use IBP only for a carefully chosen small box inside the original input box for IBP training. More specifically, it first conducts a PGD attack in the full input box to find an adversarial input, and then takes the surrounding small box with radius $\lambda\epsilon$ around the adversarial input as the input box for IBP training, where $\lambda$ is a pre-defined ratio. For exceptional cases (specifically CIFAR-10 $\epsilon=\frac{2}{255}$), SABR further shrinks the output box of every ReLU towards zero by a pre-defined constant to further reduce the regularization.

#### TAPS and STAPS

Observing that IBP relaxation error grows exponentially w.r.t. model depth (MuellerEFV22; mao2024understanding), TAPS (MaoM0V23) proposes to split the network into two subparts, using IBP for the first subpart and PGD for the other. This way, the over-approximation from IBP and the under-approximation from PGD partially cancel out, yielding a more precise approximation of the worst-case loss. Further, TAPS uses a separate PGD attack to estimate the bounds for every class to align better with the certification objective. STAPS (MaoM0V23) combines TAPS with SABR by using the adversarial small box for TAPS training, thus further reducing regularization.

#### MTL-IBP

palma2024expressive formalizes a family of surrogate loss functions that interpolate between PGD and IBP training. We study MTL-IBP, one of the most effective algorithms in this family. MTL-IBP linearly interpolates between PGD loss and IBP loss, \ie, ${\mathcal{L}}=(1-\tau){\mathcal{L}}_{\textsc{PGD}}+\tau{\mathcal{L}}_{\textsc{IBP}}$, where $\tau$ is the pre-defined IBP coefficient. To recover the re-weighing between PGD and IBP as SABR does with box shrinking, MTL-IBP uses a larger input radius for a PGD attack in the same setting (specifically, CIFAR-10 $\epsilon=\frac{2}{255}$).

## 4 A Unified Library and High-quality Benchmark for Certified Training

We now discuss CTBench, both the unified library and the corresponding benchmark.

### 4.1 The CTBench library

Table: Table 1: CTBench results with comparison to the literature. We include the natural accuracy of standard training and adversarial training, and the adversarial accuracy of adversarial training for reference. The best numbers are in bold and those exceeding the literature results are underlined.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. It is already well established by previous works that robustness training increases local smoothness. What is unique about the findings presented in this paper?
2. It is also previously established that adversarially robust trianing methods tend to have higher sample complexity, and therefore are more likely to overfit (less regularization). Other than the choice of metric, what is unique about the findings in Section 5.4.?
3. Is there an explanation for why the model performs worse for certain corruptions? How will these results be affected if we use different L_p norms? For example, I would expect a model trained to be robust in the L_2 space to be better resistant to Gaussian noise and less resistant to salt and pepper noise.
soundness: 4
strengths: 1. The paper proposes a new benchmark for certified robustness methods for image classifiers.
2. Authors implement several prominent certified robustness methods in a unified framework, thereby standardizing the implementations to facilitate future research.
3. Furthermore, authors correct implementation mistakes and perform systematic hyperparameter tuning to fully realize the potential of all methods.
4. Authors present several interesting findings regarding the properties of ceritified robustness methods, for example, models trained using distinct methods have a high overlap in the examples they succeed and fail on, uncovering a sample-specific inherent difficulty level that can be leveraged to improve training. And, these methods can boost OOD generalization for specific corruptions, and hurt generalization for others.
summary: The paper presents CTBENCH, a standardized library and benchmark designed to fairly evaluate certified training algorithms for neural networks, addressing the inconsistency in previous evaluations due to varied training schedules, certification methods, and under-optimized hyperparameters. By testing all algorithms under consistent conditions with tuned hyperparameters, CTBENCH reveals that most certified training methods perform better than previously reported, setting new benchmarks. Through CTBENCH, authors uncover several interesting properties of models training with certified methods.
weaknesses: 1. Authors incorrectly state that the benchmark from Li et al. is not up to date as "it reports 89% and 51% best certified accuracy for MNIST epsilon = 0.3 and CIFAR-10 epsilon = 2/255 in its evaluation, respectively, while recent methods have achieved more than 93% and 62%". However, at the time of this review, the numbers on Li et al.'s leaderboard (https://sokcertifiedrobustness.github.io/leaderboard/) are even higher than 93% and 62%, they are 94.02% and 68.2%. Furthermore, the leaderboard toppers are defenses from 2019/2021. It appears that the authors might have pulled their numbers from a stale source.
2. In order to be an improvement over the existing benchmark (of Li et.al.), one important requirement is comprable or improved comprehensiveness. Based on the results in the paper, the proposed benchmark is significantly less comprehensive than Li et. al. on two important directions: (i) number of defenses evaluated, (ii) number of diverse models used during evaluation. While I understand that the proposed work can be made more comprehensive by running more experiments, this is not the case currently and so is worth poining out.
3. Furthermore, as stated in the limitations section, the propsoed benchmark only focuses on deterministic certified robustness in the L_infinity space. Whereas, Li et. al.'s benchmark uses both determinisitc and probabilistic certified methods, and covers all the popularly used norms in literature (i.e., L_1, L_2, L_infinity). Thereby further hurting the comprehensiveness of the proposed benchmark.
4. Some of the findings presented in this paper are expected and already established by prior works (see Questions).
5. The main contribution of the paper is a unified code-based (and benchmark) for promiment certified robustness methods. Even though authors uncover several interesting findings while reproducing and tuning sota methods, the nature of the contributions of this paper are heavily empirical (not enough technical novelty). As such, this paper is much better suited for venues like TMLR that put emphasis on contributions of such nature.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 4
questions: Some questions I had while reading:

-- Why do methods like TAPS and MTL-IBP achieve better accuracy while deactivating more neurons?

-- Is there a theoretical framework to explain the relationship between neuron deactivation and robustness? 

-- Is there a way to understand and leverage the shared mistakes patterns to improve certified training? Or is it natural that mistakes would overlap (similar to how mistakes overlap in natural training)?
soundness: 3
strengths: -- The paper is well-written and typeset well

-- Tackles an important problem in the field: the inconsistent evaluation of different certified training methods. I think the field needed this kind of paper. 

-- It's not only a benchmark paper but provides some analysis into certified model behavior in loss fragmentation (showing certified models reduce fragmentation compared to adversarial training), have shared mistake patterns, model utilization metrics, and generalization performance (showing certified training provides benefits for certain types of corruptions).
summary: The paper introduces a benchmark for certified robust training. The goal is to standardize the hyperparameters, training schedules (& other training configurations) between competing methods in certified training. The purported advantages of newer methods are lower when older baselines were given equivalent optimization and testing conditions. The work covers several popular approaches like PGD, IBP, and CROWN-IBP.
weaknesses: -- The novelty of the paper is limited since it's just focused on benchmarking existing methods. Certified robustness is a relatively new field and the field needs methods as much as unifying benchmarks. I do believe the lack of novelty is mitigated to an extent by the analysis provided in Section 5.

-- I wonder about the sustainability of the benchmark since there are other leaderboards for adversarial training (e.g. RobustBench). Others may want to submit their work to an existing leaderboard rather than standardize to adopt your settings.

-- I'm a bit confused about the purpose of the fragmentation experiments. Robust models lead to fewer flipped neurons in the presence of noise, but why should we care? This is after all expected given they are more robust in general to input noise. I believe these experiments may be valuable but the authors should articulate why.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: The main concerns about the experiments are raised in the weaknesses section. If these can be addressed, I would be happy to change my opinion.
soundness: 2
strengths: - The paper raises an important question of fairly assessing the algorithmic improvements of recent certified training methods compared to older IBP-based training. Since the evaluation depends on many factors and components, the paper proposes to fix some of them to the best-known ones and to properly tune the rest. 
- The writing is clear (except for the presentation of Table 1), the code for benchmarking, and the weights of pre-trained models are provided. 
- The analysis of training methods leads to interesting conclusions. Particularly, the relationship between propagation tightness and certified accuracy at larger epsilon, i.e. the absence of correlation, is surprising.
summary: This paper proposes a library for benchmarking certified training methods under unified settings. It uses the best practices for certified training from (Shi et al., 2021), such as CNN7 architecture with batch normalization, IBP initialization, warm-up schedule and warm-up regularizers. To improve generalization, it uses L1 regularization and stochastic weight averaging (Izmailov et al., 2018). From the implementation perspective, the authors propose to use full batch statistics to address problems with batch normalization when gradient accumulation or PGD attack is performed. The paper claims that the improvements of recent methods in certified training drop significantly compared to older IBP training method under the same settings with proper hyperparameter tuning. Further, the authors analyze different aspects of the training methods: regularization strength, model utilization, loss fragmentation, OOD generalization and shared mistakes.
weaknesses: I believe the **experiments are insufficient** to support the main claims of the paper. Particularly:

1. **Accuracy-robustness tradeoffs are not considered**. Improvements in robustness can be due to decreased natural accuracy, and vice versa [a, b, c]. For example, in Table 1 for CIFAR-10 at 2/255 the implementations of following methods choose a different point at accuracy-robustness tradeoff curve compared to the one in literature, getting higher robustness at the cost of reduced accuracy: CROWN-IBP, SABR, STAPS, MTL-IBP, making claims about technical improvements unsupported. In this regard, the baselines such as ACERT [a], and ACE [b] are missing. Accuracy-robustness tradeoff curves and metrics such as ART-score [a] can be used to capture the improvements in the tradeoff.
2. **Error bars are missing**.  The presented improvements over the results in the literature could be statistically insignificant. For example, the experimental results for CIFAR-10 at 8/255 in paper by Shi et al. (2021) show standard deviation of $\pm0.3$ for certified accuracy and of $\pm0.4-0.7$ for natural accuracy, which makes improvements in both accuracy and robustness in Table 1 for SABR and TAPS within the error of standard deviation. 
3. **Training costs are not considered**. Different methods require different amount of computational costs for training, which could be an important factor to consider in benchmarking.
4. **Certification costs are not considered**. Since some certified training methods allow computing tight certified bounds using efficient "online" certification methods, such as IBP (Gowal et al., 2018, Mao et al., 2024), the IBP-based certified accuracy or IBP-based certified radius [a] could also be compared. The cost of test-time verification might be an important factor in choosing a training method.

Since this is a paper proposing a benchmark, it **lacks original** contributions. In terms of evaluation setting, most of the components were already used consistently in previous works.

Smaller comments:
- The main results in Table 1 are hard to parse and analyze due to large amount of numbers to compare. Accuracy-robustness plots could help with improving clarity.
- Due to shared mistakes, the paper claims that "_... there could be an intrinsic difficulty score for each input_". The certified radius of robustness of each point, described in [a, d], could serve as such score. The average certified radius and/or the histogram of radii [d] can be compared in the benchmark. The adaptive training methods can be discussed in this regard.

[a] Nurlanov, Z., Schmidt, F.R., Bernard, F. (2024). Adaptive Certified Training: Towards Better Accuracy-Robustness Tradeoffs. In: Bifet, A., et al. Machine Learning and Knowledge Discovery in Databases. Research Track and Demo Track. ECML PKDD 2024. Lecture Notes in Computer Science(), vol 14948. Springer, Cham. https://doi.org/10.1007/978-3-031-70371-3_8

[b] Müller, M. N., Balunović, M., & Vechev, M. (2021). Certify or predict: Boosting certified robustness with compositional architectures. In International Conference on Learning Representations (ICLR 2021).

[c] Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., & Madry, A (2019). Robustness May Be at Odds with Accuracy. In International Conference on Learning Representations (ICLR 2019).

[d] Bosman, A. W., Hoos, H. H., & van Rijn, J. N. (2023). A preliminary study of critical robustness distributions in neural network verification. In Proceedings of the 6th workshop on formal methods for ML-enabled autonomous systems.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: See above; I would love to hear the authors comments on each of the weakness above along with a response to the general comment.
soundness: 4
strengths: 1. The paper is well presented, well written, with clear goals and objectives.

2- While this is for a non-expert is not obvious, but the amount of experiments and computation required in this paper is beyond impressive.

3- The insights of the paper are particularly helpful. I personally did not expect that current SOTA methods are under performing. However, it was not that surprising that the improvements over IBP for larger epsilons are not that big.

4- Paper sheds light on a relatively good problem.
summary: The The paper introduces CTBENCH, a unified library and benchmark for evaluating certified training methods for neural networks. It addresses the challenges in comparing existing certified training algorithms by standardizing training schedules, certification methods, and hyperparameter tuning. The authors demonstrate that most algorithms in CTBENCH surpass previously reported results, revealing that much of the perceived advantage of newer methods diminishes when outdated baselines are properly tuned. The benchmark provides insights into certified training methods, encouraging future research by offering a consistent framework and re-establishing state-of-the-art performance.
weaknesses: 1. The paper focuses solely on deterministic certified training, overlooking advancements in randomized certified robustness. I believe the paper should have cited works like Cohen et al., Matthias et al. (l1 certification with differential privacy -- early works from 2019), Greg Yang ("All Shapes and Sizes" paper), among many others.

2. The paper only considers infinity ball, neglecting other perturbation sets. While this is generally okay, some insights with a few experiments in other perturbation sets might be helpful. It is not clear whether the proposed tricks in the library as part of the unified certified training would work for other perturbation sets (e.g., L2). If they do not, it raises the question of whether we would need a separate library for each perturbation set. The next steps are unclear if that is the case.

3. Some conclusions on the impact of tuning and modifications, while valid, lack formal decomposition, making it difficult to quantify individual contributions. No clarity on the contribution of each individual component (batch norm, etc) towards the final performance. A small systematic study will be very helpful.

4. The evaluation is based on a single model architecture (CNN7); the paper should demonstrate that the library and recommendations hold across different architectures.


General comment: Interest in certified models has significantly declined over the past two years. At ECCV, for example, there were notably fewer submissions and accepted papers on adversarial attacks, even though this topic was previously very popular in vision conferences. One reason for this decline could be the uncertainty around where such certifications can be practically deployed, especially given the massive scale of current models, which are thousands of times larger than the CNNs discussed here. Furthermore, as models shift towards generative architectures, it’s unclear who will find this domain relevant. While the paper makes valuable contributions, this direction feels somewhat outdated by about two years and the question of the benefit for it is very unclear and vague, at least to me. I would love to hear the authors take on this.

Minor Comments:
1. Cite "is NP-complete" line 321.
2.  Is not typical robust accuracy (adv acc) for PGD around 48% on 8/255 CIFAR10? Or is because you use CNN7.
3. adversarial accuracy is not well defined in line 135. You need to say that it is empirical and serves as an upper bound to the robust accuracy.
4. certified accuracy defined in 133 is not correct. It should be the portion of *correctly* classified samples that are certifiably robust.

Meta-review: This paper develops a benchmark and library for several representative deterministic certified training algorithms for fair comparison on hyperparameters, training schedules and (exact) certification methods. The reviewers generally agree that this paper is well-written, however, lacking technical novelty due to the nature of the topic. The submission will benefit from incorporating error bars of the results on all the dataset reported to show informative comparison. On a side note, the L2 (and more generally, Lp, p>=1) certified training can be directly extended from the Linf certified training method, since propagating the bounds are essentially the same and the only difference would be at the input when using holder's inequality. The authors are urged to include other Lp results to further strengthen the paper.

Decision: Reject