CTBench: A Library and Benchmark for Certified Training
Authors: Yuhao Mao
Stefan Balauca
Martin Vechev
Venue: ICML 2025 poster (2025)
Keywords: certified training
certification
library
benchmark
Primary area: deep_learning->robustness
TL;DR: We develop a library unifying certified training algorithms, achieve SOTA universally and significantly by correcting exsiting implementation mistakes, gain new insights on many questions of interest to the area, and point out future work directions.

Abstract: Training certifiably robust neural networks is an important but challenging task. While many algorithms for (deterministic) certified training have been proposed, they are often evaluated on different training schedules, certification methods, and systematically under-tuned hyperparameters, making it difficult to compare their performance. To address this challenge, we introduce CTBench, a unified library and a high-quality benchmark for certified training that evaluates all algorithms under fair settings and systematically tuned hyperparameters. We show that (1) almost all algorithms in CTBench surpass the corresponding reported performance in literature in the magnitude of algorithmic improvements, thus establishing new state-of-the-art, and (2) the claimed advantage of recent algorithms drops significantly when we enhance the outdated baselines with a fair training schedule, a fair certification method and well-tuned hyperparameters. Based on CTBench, we provide new insights into the current state of certified training, including (1) certified models have less fragmented loss surface, (2) certified models share many mistakes, (3) certified models have more sparse activations, (4) reducing regularization cleverly is crucial for certified training especially for large radii and (5) certified training has the potential to improve out-of-distribution generalization. We are confident that CTBench will serve as a benchmark and testbed for future research in certified training.

CТВЕNсH: A Library and Benchmark for Certified Training
Yuhao Mao Stefan Balauca 2
Martin Vechev!
Abstract
established itself as an important research area. While ini- tially the community focused on heuristic methods to craft adversarial examples and defenses against them, it turned out that such defenses are often brittle and can be evaded by adaptive adversaries (Athalye et al., 2018; Tramèr et al., 2020). Thus, neural network certification has emerged as method for providing provable guarantees on the robustness of a given network (Gehr et al., 2018; Wong & Kolter, 2018; Zhang et al., 2018; Singh et al., 2019).
Training certifiably robust neural networks is an important but challenging task. While many algorithms for (deterministic) certified training have been proposed, they are often evaluated on different training schedules, certification meth- ods, and systematically under-tuned hyperparam- eters, making it difficult to compare their perfor- mance. To address this challenge, we introduce CTBENCH, a unified library and a high-quality benchmark for certified training that evaluates all algorithms under fair settings and systemati- cally tuned hyperparameters. We show that (I) almost all algorithms in СТВЕNCH surpass the corresponding reported performance in literature in the magnitude of algorithmic improvements. thus establishing new state-of-the-art, and (2) the claimed advantage of recent algorithms drops sig- nificantly when we enhance the outdated baselines with a fair training schedule, a fair certification method and well-tuned hyperparameters. Based CTB ENCH, we provide new insights into the current state of certified training, including (I) certified models have less fragmented loss sur- face, (2) certified models share many mistakes, (3) certified models have more sparse activations, (4) reducing regularization cleverly crucial for certified training especially for large radii and (5) certified training has the potential to improve out- of-distribution generalization. We are confident that CT will serve as benchmark and testbed for future research in certified training.
Two families of neural network certification methods have been proposed: complete methods (Katz al., 2017; Tjeng et al., 2019) which compute the exact bounds but are ex- tremely computationally expensive, and convex-relaxation based methods (Zhang et al., 2018; Singh et al., 2019) which are more scalable but provide approximate bounds. State- of-the-art (SOTA) verifiers (Xu et al., 2021; ; Ferrari et al., 2022; Zhang et al., 2022) combine both approaches, by us- ing convex relaxations to speed up the solving of complete methods via Branch-and- Boundddd (Bunel al., 2020).
However, the scalability of neural network certification is still a major challenge since the computational complexity of SOTA verifiers grows exponentially with network size. To tackle this issue, certified training (Mirman et al., 2018: Gowal al., 2018) was proposed train neural networks that are amenable certification. Such methods are typi- cally categorized into two groups: (1) training with sound upper bound of the robust loss (Gowal et al., 2018; Zhang et al., 2020; Shi et al., 2021), and (2) training with an un- sound surrogate loss that approximates the exact robust loss (Müller et al., 2023; Mao et al., 2023; De Palma et al., 2024). The latter group has been shown to be more effective.
While certified training has made significant advances, there is currently no benchmark that can be used to fairly evaluate the effectiveness of the different certified training methods. Specifically, the literature often compares against previous methods using quoted numbers due to high computational costs, although the verifier and certification budget differ. These unfair comparisons ultimately hinder the community from drawing reasonable conclusions the effectiveness of certified training methods. In addition, existing works systematically under-tune hyperparameters, in order to show effectivvnesss against baselines, thus establishing a weaker SOTA. Further, there is no unified codebase for these meth-
1. Introduction
As crucial component of trustworthy artificial intelligence, adversarial robustness (Szegedy et al., 2014; Goodfellow et al., 2015), i.e., resilience to small input perturbations, has
'Department of Computer Science, ETH Zürich, Switzerland "INSAIT. Soña University "St. Kliment Ohridski". Sofia, Bulgaria. Correspondence to: Yuhao Mao eyuhao. inf..ettn..cccc....
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 202.5 by the author(s).

---

CТBBNNH: A Library and Benchmark for Certified Training
Certified Error (%)
(Singh et al., 2018), and Zhang et al. (2020) incorporate linear relaxations (Zhang al., 2018; Singh al., 2019). While these approximations are more precise (Baader et al., 2024; Mao et al., 2025), they often lead to worse training results, attributed to non-smoothness (Lee et al., 2021), dis- continuity and sensitivity (Jovanović et al., 2022) of the loss surface. Some recent work (Balauca et al., 2024) aims to mitigate these problems. however, the most effective train- ing approximation is still the least precise box relaxation. In this regard, the focus of the community has shifted towards improving IBP: Shi et al. (2021) propose new regulariza- tion and initialization paradigm to speed up IBP training; De Palma et al. (2022) apply IBP regularization to make adversarial training certifiable; Müller et al. (2023), Mao et al. (2023) and De Palma et al. (2024) propose unsound but more effective IBP-based surrogate losses for training; Mao et al. (2024) propose to use wider models instead of deeper models for IBP-based methods. These methods achieve universal advantages over non-IBP-based methods, and are thus the focus of our work.
7.2
Lieerature
L 7Z
Ours
6.6
6.0
77
IRP
CROWN-IBP SABR
STAPS
MTL-IBP
Figure I: Reduction in certified error on MNIST 0.3 (lower is better).
ods, making future development and comparison difficult.
This work: Unified Library and High-quality Bench- mark for Certified Training We address these challenges, for the first time unifying SOTA certified training methods into single codebase called CТВЕNCH. This enables a fair comparison between certified training methods and re- establishes much stronger SOTA by fixing problematic implementations and systematically tuning hyperparame- ters. As shown in Figure 1, these steps lead to significant improvements uniformly. In addition, we show that the claimed advantage of recent SOTA reduces significantly when we apply the same budget and hyperparameter tuning to all methods. Based on our released model checkpoints, provide extensive analysis of the model properties, highlighting many new insights on its loss landscape, mis- take patterns, regularization strength, model utilization, and out-of-distribution generalization. We are confident that CTB will serve as a benchmark and testbed for future work in certified training.
3. Background
We now introduce the necessary background for our work, both concepts and training algorithms.
3.1. Training for Robustness
We present the mathematical notations on adversarial and certified training here. We consider neural network classi- fier fx(x)) that estimates the log-probability of each class and predicts the class with the highest estimated log- bog-probability.
Adversarial Training A classifier fe(z) is said to be adversarially robust with radius w.r.t. Tnp perturbation if fo(x + 8) y for all ||0|| c, where y is the ground truth label of x. Finding an adversarially robust classifier is formally defined to solve a min-max problem 0 = arg ming Ex.y max Me L(x + 5). In this regard, ad- versarial training solves the inner maximization problem by generating adversarial examples during training, and the outer minimization problem by optimizing the empirical loss of adversarial examples.
2. Related Work
We now briefly review works mostly related to ours.
Benchmarking Certified Robustness et al. (2023) pro- vides the first benchmark for certified robustness, covering not only deterministic certified training but also random- ized certified training and certification methods. However, it is outdated and thus provides little insight into the cur- rent SOTA methods. For example, it reports 89% and 51% best certified accuracy for MNIST e = 0.3 and CIFAR-10 <= 255 in its benchmark, respectively, while recent meth- ods have achieved more than 93% and 62% (Müller et al., 2023; Mao al., 2023; De Palma et al.. 2024).
Certified Training A classifier fe(x) is said to be cer- tifiably robust if it adversarially robust and there ex- ists a sound verifier that certifies the robustness. ifier typically computes an upper bound on the margin f((z + 6) fy( + 5) and certifies its robustness if the upper bound is negative for all i + y. Certified training thus replaces the inner maximization problem with an up- per bound and minimizes the upper bound during training instead. Since existing certified training algorithms focus solely on Tezo distance, we only consider T.o perturbations
ver-
Certified Training DIFFAI (Mirman et al., 201 8) and IBP (Gowal et al., 2018) apply box relaxation to upper bound the worst-case loss for training. Efforts have been made towards applying more precise approximations: Wong et al. (2018) and Balunovic & Vechev (2020) apply DEFPZ
2

---

CTBENCH: A Library and Benchmark for Certified Training
I Legend
PGD
IBP
Input Space
Latent
Output
Space
Space
SABR
TAPS
MTI.-IBP
Forward Pass
L = (1 (1-170 -T)CPG
Gradient Ascent
Box Relaxation
+TLIHP
Figure 2: Conceptual overview of core algorithms built into CTBENCH.
in this work and omit the distance type in the notation.
extra-gradiens method called EDAC, which remains one of the SOTA methods in adversarial training. These methods achieve good but uncertifiable adversarial robustness, hence we use them as adversarial robustness baselines.
Metries The main metric for certified training is certified accuracy, defined to be the ratio of certifiably robust sam- ples in the dataset; certified error is defined as one minus the certified accuracy. The ratio of correctly classified samples in the dataset is thus called natural accuracy. For reference, we include adversarial accuracy as well, defined to be the ratio of adversarially robust samples in the dataset. We ap- ply one of the most widely used SOTA certification methods, MN-BAB (Ferrari et al., 2022). as the verifier. To compute adversarial accuracy, we apply the strong AUTOATTACK (Croce& & Hein, 2020) for adversarially trained models, and at-
IBP Interval Bound Propagation (IBP) (Mirman et al., 2018; Gowal et al., 2018) uses interval analysis to approx- imate the output range of each layer. For example, for the toy network y = 2 ReLU(xr + x2) with input bounds 21, 22 E [-1, 1]. it first computes the output range of the first layer as æ + 222 E [-1, [] S 2,, the second layer as 2e S [0. and then final layer as 2 - 00.2 2] C [0, 2). thus proving 2 0 for all possible I1, 22 E [-1, Similarly, IBP computes the layer-wise bounds and then derives an upper bound of the worst-case loss based on the output bounds of the final layer. To stably train models with IBP, Shi et al. (2021) propose rescale the parameter initialization to ensure constant growth of IBP bounds and specialized regularization to control the activation status of neurons. They also show that adding a batch norm (loffe& & Szegedy, 2015) layer before every ReLU layer can improve IBP training. These training tricks are adopted by every IBP-based method introduced below. For brevity, we refer to this variant IBP in the rest of the paper unless otherwise stated, since it improves the original IBP universally with tricks that facilitate training.
combination of PGD attack and tack from MN- BAB for certifiably trained models. Both attacks have similar empirical strengths, with the latter be- ing slightly stronger on models trained by certified training algorithms due to the completeness of the branch-and-boo attack.
3.2. Algorithms in CTВENCH
Now we briefly introduce the core algorithms built into CT- BENCH. Concepts behind them are visualized in Figure 2. theoretical complexity analysis is provided in Table 11 in App. C.6.
PGD and EDAC Projected Gradient Descent (PGD) (Madry et al., 2018) is the most widely rec- ognized adversarial training method. Starting from a randomly initialized point, PGD solves the inner maxi- mization problem by iteratively taking a step towards the gradient ascent direction and clipping the solution into the valid perturbation set. Then, uses the generated adversarial input x' to lower bound the worst case loss as [xx). Croce& & Hein (2020) find that PGD- models remains effective against strong attacks, thus it is popular as an integrated part of many certified training methods (Müller et al., 2023; Mao et al., 2023; De Palma et al., 2024). To further improve adversarial robustness, Zhang et al. (2023) improves adversarial generalization via an
CROWN-IBP (Zhang et al., 2020) tight- ens the imprecise interval analysis with linear relaxations of ReLU layers based on IBP bounds and only solves the lin- ear constraints for the final layer output based on CROWN (Zhang et al., 2018), avoiding prohibitive costs during train- ing. To further reduce the cost of solving the bounds for each class, Xu et al. (2020) propose a loss fusion trick to only solve for the final loss, thus reducing the asymptotic complexity by a factor equal to the number of classes. For brevity, we refer to this variant as CROWN-IBP in the rest of the paper unless otherwise stated, since the original CROWN-IBP cannot scale to datasets with many classes, such as TINY TAAGNEET.
CROWN-IBP
3

---

CTBENCH: A Library and Benchmark for Certified Training
Since IBP is often criticized for the increasingly strong regularization w.r.t. input radius imposed on the neural network, SABR (Müller et al., 2023) proposes to use IBP only for a carefully chosen small box inside the original input box for IBP training. More specifically, it first conducts a PGD attack in the input domain to find an approximately worst-case input, and then takes the sur- rounding small box with radius de around the found input as the input box for IBP training, where À is a pre-defined ra- tio. For exceptional cases (specifically CIFAR-10+ e = 255 SABR further shrinks the output box of every ReIJU to- wards zero by pre-defined constant to further reduce the regularization.
SABR
regularization proposed by Shi et al. (2021) to improve cer- tified training methods. The IBP initialization (Shi et al., 2021) is applied for every certified training method, while adversarial training is initialized with Kaiming uniform (He et al., 2015). Every method has a warmup phase where e is increased from 0 to the target value and a fine-tuning phase where the model continues to train at the targeted to con- verge. The learning rate is held constant during the warmup phase and decayed twice in the fine-tuning phase with constant multiplier. We use CNN7 as the model architecture, in agreement with recent literature (Shi et al., 2021; Müller et al., 2023; Mao et al., 2023; De Palma et al., 2024).
Due to the importance of batch norm in certified training, we consider as native part of CTBENCH. Specifically, the best practice so far is to set batch norm statistics based the clean input and use this for computing IBP bounds. However, we find several problematic implementations of batch norm in the literature: (1) when gradient accumula- tion is involved, the batch norm statistics are not updated correctly, as sub-batch statistics are applied for training; (2) batch norm statistics change more than once before taking gradient step, as typically the exponentially accumulated statistics are used for conducting PGD attack and thus evaluating Loob, while Lat is evaluated with batch statistics. The first problem makes gradient accumulation ineffective since the quality of batch statistics depends highly on the batch size, and the second problem prevents training with Whob E (0, 1) due to the varied parameters. To address the first problem, we propose to use full batch statistics during gradient accumulation, which leads slim overheads but allows arbitrary gradient accumulation, as forward pass is usually much cheaper than full batch update in certified training. To address the second problem, we conduct PGD attacks with the batch statistics as well and evaluate every- thing with the current batch statistics. This way, the batch norm statistics are set once per batch just like standard train- ing, allowing training with the combination of Lnat and Loo.. We remark that the identified problems are systematically ignored in the literature, thus may only be discovered by carefully reading the implementations, which is infeasible for most researchers.
TAPS and STAPS Observing that IBP relaxation error grows exponentially w.r.t. model depth (Mäller al., 2023; Mao et al., 2024), TAPS (Mao et al., 2023) proposes to split the network into two subparts. using IBP for the first subpart and PGD for the other. This way, the aperappproinationn from IBP and the under-approximation from PGD partially cancel out, yielding a more precise approximation of the worst-case loss. Further, TAPS uses a separate PGD attack to estimate the bounds of every class to align better with the certification objective. STAPS (Mao et al., 2023) combines TAPS with SABR by using the adversarial small box for TAPS training, thus further reducing regularization.
De Palma et al. (2024) formalizes a family of surrogate loss functions that interpolate between PGD and IBP training. We study MTL-IBP, one of the most effective algorithms in this family. MTL-IBP linearly interpolates between PGD loss and IBP loss, i.e., L = (1 CPGD + 7 Lim,, where is the pre-defined IBP coefficient. To allow more fine- uineggraneed control of the interpolation, MTL-IBP uses a larger input radius for the PGD attack for CIFAR-10 when e 255
MTL-IBP
4. A Unified Library and High-quality Benchmark for Certified Training
We now discuss CTBENCH, both the unified library and the corresponding benchmark.
In addition, we find that models trained with the hyperpa- rameters reported in the literature frequently show strong overfitting patterns. To remediate this, we conduct magni- tude search for L1 regularization until the train and valida- tion performance roughly match. To further aid generaliza- tion, we apply Stochastic Weight Averaging (Izmailov et al., 2018) for methods that cannot provide metrics for model selection, e.g., MTL-IBP. A more detailed description of the implementation can be found in App. C.
4.1. The CТВЕNCH library
We implement every algorithm described in Section 3.2 in a unified framework. The training loss is composed of three components: the natural loss which measures per- formance clean inputs, the robust loss which measures robust performance depending on the concrete algorithms and regularization losses which are used to stabilize training and improve generalization. Formally, the training loss is defined as L = (1- Wrob + Coo + Leg. We mainly use L. regularization to reduce overfitting and the warmup
4

---

CТВЕNNH: A Library and Benchmark for Certified Training
Table 1: CТВЕENII results with comparison to the literature. We include the natural accuracy of standard training with CNN7 on each dataset for reference. The best numbers are in bold and those exceeding the literature results are underlined.
Dataset Std. Nat. [%]]
Nat.
Cert. [%]
Adv: [8]
Ero
Training Method
Source
Literature
CTBЕNCH
Literature
CTBENCH
CTBENCH
PGD
/
99.47
98.97
EDAC
99.58
==
98.95
IBP
Shi (2021)
98.84
98.87
97.95
98.28 26
98.27
0.1
CROWN-IBP
Xu et al. (2020)
98.83
98.94
97.76
98.21
98.23
SABR
Mäller et al. (2023)
99.23
99.08
98.22
98.40
98.47
TAPS
Mao et al. (2023)
99.19
99.16
98.39
98.52
98.58
STAPS
Mao et al. (2023)
99.15
99.11
98.37
98.47
98.50
MTL--BP
Palma et al. (2024)
99.25
99.18
98.38
98.37
98.44
99.50
PGD
/
99.43
=
93.83
EDAC
99.51
01
95.02
IBP
Shi et (2021)
97.67
98.54
93.10
93.80
94.30
0.3
CROWN-IBP
et (2020)
98.18
92.98
94.29
SABR
Müller (2023)
TAPS
Mao al. (2023)
97.94
98.56
93.62
93,95
94.66
STAPS
Mao al. (2023)
98.53
98.74
93.51
93.64
94.36
MTL-IBP
De Palma et al. (2024)
98.80
98.74
93.62
93.90
94.55
PGD
/
88.67
==
72.41
EDAC
89.18
0=
72.42
IBP
Shi (2021)
66.84
67.49
52.85
55.99
56.10
CROWN-IBP
Xu et al. (2020)
71.52
67.60
53.97
57.11
57.28
SABR
Müller et al. (2023)
79.24
77.86
62.84
63.61
65.56
TAPS
Mao et al. (2023)
75.09
74.44
61.56
61.27
62.62
STAPS
Mao al. (2023)
79.76
77.15
62.98
64.21
66.09
CIFAR-10
MTL-IBP
Palma et (2024)
80.11
78.82
63.24
64.41
67.69
91.27
PGD
/
78.71
35.93
EDAC
78.95
42.48
IBP
Shi et (2021)
48.94
48.51
34.97
35.28
35.48
CROWN-IBP
Xu et (2020)
48.25
33.38
32.59
32.77
205
SABR
Müller (2023)
52.38
35.13
TAPS
Mao al. (2023)
35.25
35.69
STAPS
Mao al. (2023)
52.82
51.49
34.65
35.11
35.54
MTL--BPP
De Palma (2024)
53.35
54.28
35.44
35.41
36.02
PGD
46.78
0=
33.16
EDAC
46.79
33.16
IBP
Shi et al. (2021)
25.92
26.77
17.87
19.82
19.84
TINYIMAGENET
CROWN-IBP
Xu et al. (2020)
25.62
28.44
17.93
22.14
22.31
47.96
SABR
Müller et al. (2023)
28.85
30.58
211.66
20.96
21.16
TAPS
Mao et al. (2023)
28.34
28.64
20.82
21.58
21.71
STAPS
Mao et al. (2023)
28.98
30.63
22.16
22.31
22.57
MTL-IBP
Palma et al. (2024)
37.56
35.97
26.09
27.73
28.49
1 None of the first 10 samples certified due to the time limit of 1000 seconds per sample.
4.2. The CТВЕNCH benchmark
statistics properly in certified training. We also observe the following: (1) when e is large, the claimed advantage of recent SOTA over IBP drops significantly, e.g., from (100 93. /000 93.62) - 8.15% relative certified error reduction MNIST 0.3; (2) when the model has sufficient ca- pacity, e.g., MNIST close to the natural accuracy of standard training 199.18% for MTL-IBP vs 99.50% for standard training), and they also get similar adversarial accuracy adversarial training (98.58% for TAPS vs 98. 85.5%% for EDAC), while certified accuracy is boosted (98.52% for TAPS almost 0% for EDAC); (3) when is large, certified training even gets better adversarial accuracy than PGD training (94.66% for
Table shows the results of CTBENCH using the method- ology described in Section 4.1. We further include the average and standard deviation obtained from independent runs in App. D.I, to validate the significance of results. We find that CТВЕNCH achieves consistent improvements in certified accuracy for almost all settings, accompanied by increases in natural accuracy in most cases. In par- ticular, it establishes the new SOTA by a margin match- ing algorithmic advances everywhere except CIFAR-10 255 where we have 0.03% lower certified accuracy compared De Palma al. (2024) but 0.93% higher nat- ural accuracy. This proves the effectiveness of im- plementation and the importance of setting batch norm
93.8) (1000) (100-13.995 - 2.48%
0.1, certified training can get
5

---

CTBENCH: A Library and Benchmark for Certified Training
TAPS 93.83% for PGD on MNIST 0.3 and 36.1 16% for SABR vs 35.93% for PGD on CIFAR-10 <= 255 ) but there is still gap between the adversarial accuracy of the SOTA adversarial training methods and that of the SOTA certified training methods, as well as a similar gap for natu- ral accuracy. We further include a comparison on another architecture, CNN5, between CTBЕENC and the implemen- tation of De Palma al. (2024) in App. D.2, to validate the stability of CTBENCH results across architectures.
Table 2: Observed count of common mistakes of models
on MNIST against their expected values assuming indepen- dence across model mistakes.
models succeeded
0
2
3
4
5
6
obs.
30
32
56
9743
0.1
exp.
0
0
0
1
37
900
9062
obs.
452
73
53
51
80
111
9180
c 0.3
exp.
0
0
2
39
445
2698
6816
5. Evaluating and Understanding Certified Models
boundary of the input box, where new fragments appear most likely. We find this sampling process extremely effec- tive; empirically the ratio of unstable neurons observed is very close to the upper bounds derived by IBP for certified models.
We now preform an extensive evaluation on models trained with CTBENCI. Our evaluation provides insights into the current state of certified training and addresses sev- eral key questions, including the loss fragmentation (Sec- tion 5.1), shared mistakes (Section 5.2), model utilization (Section 5.3), regularization strength (Section 5.4), and out- of-distribution generalization (Section 5.5).
Figure shows the result of certified models trained at < < 0.1 and e = 0.3 on MNIST, respectively. We eval- uate the fragmentation of every model at both e 0.1 and e 0.3. First, we observe that both adversarial train- ing and certified training greatly reduce loss fragmentation compared to standard training, even though many certified training algorithms involve no adversarial attacks. Second, comparing different training methods within each group of and Z. we observe that certified training consistently has significantly less fragmentation than adversarial train- ing when evaluated at the train radius, e-g-, IBP reduces fragmentation by 3x compared to EDAC when trained and evaluated at e = 0.1, facilitating the approximation of the worst case loss via adversarial attacks. This is consistent with the practice where a weak single-step attack is adopted in certified training (De Palma et al., 2024), resulting in sim- ilar performance as strong attacks but improved efficiency. Third, comparing models trained at different (0 vs 22 and A), we observe that further increasing training does not necessarily reduce fragmentation, yet the trend is con- sistent with adversarial training. These observations prove that certified training can further boost the fragmentation reduction effect of adversarial training, thus introducing more local smoothness into the model. Consistent results on CIFAR- 10 are included in App. D.4 as Figure 9.
5.1. Loss Fragmentation
ReLU networks are known to have a fragmented loss sur- face over the input space. due to the activation switch of neurons. Fragmentation leads non-smooth loss surface and increases the difficulty of finding a good approximation of the worst-case loss via gradient-based methods like PGD. Further, SOTA complete certification algorithms relies on branching on different linear regions, and reducing the num- ber of linear regions reduces the certification difficulty. Due to these reasons, in this section, we investigate the frag- mentation of loss surfaces in certified models. Specifically, we answer: (1) do certified models have less fragmenta- tion, thus easing adversarial search, and (2) how does the fragmentation change w.r.t. e?
Fragmentation is closely related to the number of unstable neurons, i.e., neurons that switch activation status in the neighborhood, as all fragments are defined by a group of unstable neurons. Vice versa, in most cases, switching neuron introduces at least one fragmentation since every activation pattern defines a local linear function. Therefore, we can quantify the fragmentation by the ratio of unstable neurons. Since the exact ratio is NP-complete to compute, we use a heuristic but effective method to estimate it: first, a group of inputs is sampled in the input box; second, these inputs are fed into the model to get the corresponding activa- tion pattern; finally, we count the ratio of unstable neurons observed in the sampled activations. This method always es- tablishes a lower bound of the true ratio and gets arbitrarily close when sample size is large enough. In our experiments, we sample the noise 50 times from a standard Gaussian clipped to [-1, 1] and rescale it by 6. This sampling fo- cuses more on the neighborhood of the clean input and the
5.2. Shared Mistakes
We now study the correlation between certified models, specifically: do certified models share common mistakes?
We consider models trained by six certified training methods on MNIST at E 0.1 and e = 0.3 and calculate the dis- tribution of their common mistakes. Specifically, we count the number of models that successfully certify the input, for each sample in the test set containing 10k samples. The observed value compared with the expected value, defined
6

---

CТВЕNNH: A Library and Benchmark for Certified Training
evaluated 0.1
trained trained
coalusted = 0.3
trained 0.3
Unstable (%)
10
10
109.
10-1
EDAC
IRP
CROWN-IBP
SABR
TAPS
STAPS
MIL-IBP
Figure 3: Ratio of unstable neurons for models trained on MNIST with different methods and e.
as expected counts when models with the same certified ac- curacy make mistakes independently (rounded to the closest integer necessary). e.g., two models with 90% certified accuracy are expected to have 81% % of inputs certified by both. The result is shown in Table 2. Accordingly, certified models share many mistakes, the number of samples that are certified by none of these models systematically exceeds the expected value. In addition, the number of samples that are certified by all six models is much larger than the corre- sponding expected value as well. These facts suggest that there could be an intrinsic difficulty score for each input, thus curriculum learning (Bengio al., 2009; Ionescu et al., 2016) could be promising direction to improve certified training. This phenomenon is also observed across differ- ent certification methods, different model architectures and different datasets, as shown in App. D.3, confirming that this is not context-specific property but rather an intrinsic property of certified models.
to TAPS and MTL.-IBP, and gets higher adversarial accu- racy than PGD. This suggests that the ability to adaptively keep necessary utilization could be crucial both adver- sarial and certified robustness. Since dying neurons (Lu et al., 2019) are hard to activate again, future work on better warmup (Shi et al., 2021) could be beneficial, as IBP still struggles to keep necessary model utilization even with the improvements by Shi et al. (2021). More results on CIFAR- are included in App. D.4 as Figure 10. There, however, all certified training methods cannot activate more neurons when large, just like IBP in MNIST, while adversarial training methods show similar behavior MNIST. This suggests that the current certified training methods have not fully solved the utilization problem, especially when e is large.
5.4. Regularization Strength
Previous work (Mao et al., 2024) has shown that IBP bounds are close to optimal bounds for IBP-based certified training, and this condition is established via strong constraints on the model parameters. They quantify this regularization effect by propagation tightness, defined be the ratio between the optimal bound radius and the IBP bound radius, approx- imating the ReLU network locally with linear replacement. Intuitively, a close-to-1 propagation tightness means IBP bounds approximately match the exact bounds, and a close- to-0 propagation tightness means IBP bounds are far from the exact bounds. In addition, a high propagation tightness implies strong regularization itself. In this section, we ex- tend the study of propagation tightness more certified training methods and investigate how it interacts with certi- fied accuracy. Specifically, using propagation tightness as the representative of regularization strength, we answer: (1) does less regularization lead to better certified models, and (2) how does the input radius e affect this?
5.3. Model Utilization
Model utilization represents how much the model capacity utilized for the task. IBP is shown to systematically deac- tivate neurons (Shi al., 2021) gain precision. However, it is not yet clear whether more advanced certified training methods deactivate fewer neurons, thus utilizing the model capacity better.
We define model utilization to be the ratio of neurons acti- vated by the clean input. Figure 4 visualizes the result for models trained on MNIST at 0.1 and 0.3. Surpris- ingly, we find that more advanced certified training methods, TAPS and MTL-IBB deactivate more neurons than IBP on MNIST mental (Shi et al., 2021), ), but these models achieve better natural and certified accuracy than IBP. Interestingly, these methods, but not IBP, can retain more utilization on c 0.3 for better performance where IBP struggles to keep high natural accuracy. Further, we observe that the advanced ad- versarial training method, EDAC, shows similar behaviors
0.1. This is previously believed to be detri-
Figure 5 shows the interaction between certified accuracy and propagation tightness for certified models trained on MNIST and 10.. When is small (Figure 5a and

---

CТBЕNNH: A Library and Benchmark for Certified Training
Utilization %%|
40-
trained at 0.1
trained 0.3
30
20
10
PGD
EDAC
IBP
CROWN-IP SADR
TAPS
STAPS
MTL-IBP
Figure 4: Model utilization for models trained on MNIST with different methods and c. We note that standard training has 42.99% utilization.
Prop. Tightness
Prop. Tightness
Prop. Tightness
Prop. Tightness
0.8
0.85
1.0
0.97- -
0.6
0.75 -
0.5
0.92 -
0.4
0.2
0.65 >
0.0 55
0.87 -
98.00
98.3
98.6
93.6
93.8
94.0
60
65
35.0
35.3
35.6
Cert. Accu. [56]
Cert. Accu. [%6]
Cert. Accu. [%]
Cert. Accu. [%]]
(a) MNIST 0.1
(b) MNIST 6 0.3
(c) CIFAR-I0 10
(d) CIFAR-10 10
255
255
Figure 5: Certified accuracy vs. propagation tightness for models trained on MNIST and CIFAR-10.
Figure 5c), certified accuracy has a perfectly negative corre- lation with propagation tightness, i.e., better certified mod- els exhibits less regularization consistently; the best models have close-to-O propagation tightness. However, when large (Figure 5b and Figure 5d), the correlation is not as clear, and the best model in certified accuracy does not necessarily have the lowest propagation tightness. Instead, models with similar propagation tightness can have signif- icantly different certified accuracy. Nevertheless, models trained for larger radius exhibits much higher propagation tightness. Therefore, we conclude that reducing regulariza- tion strength cleverly is crucial for certified training, and the effect more pronounced when is small, while improper reduction could hurt certified accuracy, especially when is large. This is consistent with the observation made in Müller et al. (2023) and De Palma et al. (2024) that the best models for small often require much less regularization.
compare to adversarial training?
We use MNIST-C (Mu & Gilmer, 2019) to evaluate OOD generalization, defined to be the ratio between OOD accu- racy and natural accuracy. MNIST-C includes 15 carefully chosen corruptions, covering a broad range of corruptions that are not covered by adversarial robustness while preserv- ing the semantics. We evaluate models trained with both adversarial training and certified training under e = 0.1 and = 0.3, and report the corresponding OOD accuracy of the model trained via standard training. We note that none of the models has seen these corruptions during training.
Figure 6 depicts the result of OOD generalization for each model on all corruptions. We observe the following: (1) certified training improves OOD generalization compared to standard training except on the brightness corruption where both adversarial and certified training fail; (2) certified train- ing shows different OOD generalization patterns to adversar- ial training, e.g., certified training boosts generalization on the canny edges corruption while adversarial training wins the stripe corruption. In general, we find that certified training either greatly boosts the OOD generalization sig- nificantly downgrades the OOD generalization depending on the corruption, and the failure cases are usually those in which adversarial training performs worse than or similarly to standard training. Therefore, we hypothesize that these
5.5. Out-of-Distributio Generalization
Out-of-distribution (OOD) generalization is closely related to adversarial robustness (Gilmer et al., 2019). However, the interaction between certified robustness and OOD gen- cralization is not yet clear. We thus investigate the OOD generalization of certified models and answer: (1) do certi- fied models generalize OOD data, and (2) how does this
8

---

CTBBNNH: A Library and Benchmark for Certified Training
large radius settings. Overall, future work is still required improve the learning process of certified training. Despite the challenges, Section 5.5 finds that certified models can have surprising and qualitatively different improvement on OOD generalization, which could be a promising applica- tion for certified training beyond certified robustness.
FGD
27
6.6
818
1.7
110
788
96
96
98
59
.9
97
56
70
90
EDAC
19
899
477
515
948
945
IBP-
10
91
022
944
CROWN-IBP
10
82
85
42
04
70
94
02
033
98
90
96
40
70
SABRR
7
61
96
40
04
71
0.7
02
94
99
90
97
47
36
81
TAPS
10
81
96
61
04
66
077
02
94
96
90
97
18
STAPS
10
72
MTL-IBP
10
81
06
42
04
71
977
92
94
98
90
97
42
Limitations We only consider deterministic certified train- ing in CTBENCH, while randomized certified robustness (Cohen et al., 2019) has also made substantial progress. Moreover, we only consider the adversarial robustness, while other types of robustness, such as robustness against patch attacks (Salman et al., 2022) is also important. Finally, we only focus on T. robustness because there exists no scalable deterministic certified training algorithm regarding other norms, and leave them as future work.
PGD
15
55
199
300
895
553
995
96
965
99
96
999
56
72
89
EDAC
19
61
99
31
90
555
955
95
95
999
50
IBP
7
60
33
94
CROWN IBP
7
79
97
94
76
9.7
93
94
99
99
98
54
37
82
SABR
10
77
97
15
9.4
91
946
944
94
999
99
999
68
60
83
TAPS
77
97
022
95
7.8
977
94
95
STAPS
10
MTI-IBP
10
80
89
28
95
88
97
94
95
00
00
100
Standard
30
89
16
87
96
96
00
96
brightuss
canny exders
drstes Live
insputie. inspulse.tune Iimbe
robatte
abowr
shot Monde
spatter
stajpe
Uncodate
aigrng
Rookko
Connection to Randomized Certified Training The is- sue of unfair comparison highlighted in this work may gen- eralize to the domains of randomized certified robustness, particularly Randomized Smoothing (RS) as introduced by Cohen et al. (2019). Recent RS literature shows consider- able difference in evaluation regarding important aspects such as network architectures, training procedures, hyperpa- rameter configurations, and the noise distributions employed for certification. These inconsistencies suggest that unfair comparisons may also be prevalent in RS studies. There- fore, we believe that a similar unified library and benchmark for randomized certified training would be beneficial to the community, allowing for fair comparisons and systematic hyperparameter tuning. We leave this as future work.
gdeen
Figure 6: fitto---istribution generalization evaluated on
MNIST-C for models trained on .NISTT at c = 0.3 (top), e = 0.1 (middle) and standard training (bottom).
corruptions are odds with adversarial robustness. Further, different training does not significantly affect the OOD generalization except few cases, and ranking in certified accuracy does not show strong correlations with the ranking in OOD generalization. Overall, these results suggest that certified training has the potential to improve OOD gen- eralization to corruptions that standard training struggles with, and the effect is exaggerated when adversarial train- ing improves over standard training. Consistent results CIFAR-10-C (Hendrycks & Dietterich, 2019) are included in App. D.4 as Figure 11.
7. Conclusion
We introduced CTBENCH, a unified library and high-quality benchmark for deterministic certified training on noom robustness. Based on CТBЕNCH, we extensively evalu- ated certified models trained via state-of-the-art methods, analyzing their regularization strength and utilitics. Our analysis reveals that certified training schemes can reduce loss fragmentation, adaptively keep model utilization, make shared mistakes, and generalize well on data with certain corruptions. We are confident that the insights and tools provided by CTBENCH will facilitate future research on certified training and its applications.
6. Discussion
Future Directions Section 5.1 shows that certified train- ing consistently reduces loss fragmentation, which also ben- efits adversarial attacks. Therefore, future works may ex- plore architectures and training algorithms that explicitly have little loss fragmentation. In addition, Section 5.2 shows that certified models share mistakes on some hard samples, thus curriculum learning with some well-defined difficulty ranking could facilitate training, where optimization has been known to be particularly hard (Jovanović et al., 2022). Moreover, Section 5.3 shows that even the most advanced certified training method, MTL-IBP, struggles to keep nec- essary model utilization on large e. Further, Section 5.4 finds that reducing regularization has a different consequence in small and large radius settings; while reducing regulariza- tion benefits small radius, it risks decreased performance in
Reproducibility Statement
We release the complete codebase of CTBENCH, includ- ing the implementation of all certified training methods and the model checkpoints for the benchmark. The code- base is available at https://githu..com/// wtwpa/uu..o//ttttt///cectoc wti.s//ee/.C........ A complete description of the experiment setup and hyperpa- rameters is provided in App. C.

---

CТВЕNNH: A Library and Benchmark for Certified Training
Acknowledgements
De Palma, A., Bunel, R., Dvijotham, K., Kumar, M. and Stanforth, R. IBP regularization for verified adversarial robustness via branch-and- 2022.
This work has been done as part of the EU grant ELSA (Eu- ropean Lighthouse on Secure and Safe AI, grant agreement no. 101070617) and the SERI grant SAFEAI (Certified Safe, Fair and Robust Artificial Intelligence, contract MBB2.00088). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or European Commission. Neither the European Union nor the European Commission can be held responsible for them.
De Palma, A., Bunel, R., Dvijotham, K. D., Kumar, M. P., Stanforth, R., and Lomuscio, A. Expressive losses for verified robustness via convex combinations. In Proc. of ICLR, 2024.
Ferrari, C., Müller, M. N., Jovanovic, N., and Vechev, M. T. Complete verification via multi-neuron relaxation guided branch-and-bound. In Proc. of ICLR, 2022.
This research was partially funded by the Ministry of Educa- tion and Science of Bulgaria (support for INSAIT, part of the Bulgarian National Roadmap for Research Infrastructure).
Gehr, T., Mirman, M., Drachsler- Chaudhuri, S., and Vechev, M. T. A12: safety and ro- bustness certification of neural networks with abstract interpretation. In Proc. of S&P, 2018.
D., Tsankov, P.,
The work has received funding from the Swiss State Secre- tariat for Education, Research and Innovation (SERI).
Gilmer, J., Ford, N., Carlini, N., and Cubuk, E. D. Adver- natural consequence of test error in
sarial examples noise. In Proc. of ICML, 2019.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In Proc. of ICLR, 2015.
Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Arandjelovic, R., Mann, T. A., and Kohli, P. On the effectiveness of interval bound propagation for training verifiably robust models. 2018.
References
Athalye, A., Carlini, N., and Wagner, D. A. Obfuscated gradients give a false sense of security: Circumventing defenses adversarial examples. In Proc. ICML, 2018.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proc. of ICCV, 2015.
Baader, M., Mueller, M. N., Mao, Y., and Vechev, M. Ex- pressivity of unetwoksss under convex relaxations. In Proc. ICLR, 2024.
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness common corruptions and perturba- tions. In Proc. of ICLR, 2019.
Balauca, S., Müller, M. N., Mao, Y., Baader, M., Fischer, M., and Vechev, M. Overcoming the paradox of certified training with gaussian smoothing. 2024.
loffe, and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. ICML, 2015.
Balunovic, M. and Vechev, M. T. Adversarial training and provable defenses: Bridging the gap. In Proc. of ICLR, 2020.
Ionescu, R. T., Alexe, B., Leordeanu, M., Popescu, M., Papadopoulos, D. P., and Ferrari, How hard can be? estimating the difficulty of visual search in an image. In Proc. of ICLR, 2016.
Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. In Proc. of ICML, 2009.
Bunel, R., Lu, J., Turkaslan, I., Torr, P. H. S., Kohli, P., and Kumar, M. P. Branch and bound for piecewise linear neural network verification. J. Mach. Learn. Res., 2020.
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D. P., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. In Proc. of UAI, 2018.
Cohen, J. M., Rosenfeld, F., and Kolter, Z.. Certified adversarial robustness via randomized smoothing. In Proc. of ICML, 2019.
Jovanović, N., Balunović, M., Baader, M., and Vechev, M. On the paradox of certified training. Proc. of ICML, 2022.
Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In Proc. of ICMI., 2020.
Katz, G., Barrett, C. W., Dill, D. L.,, Julian, K., and Kochen- derfer, M. J. Reluplex: An efficient SMT solver for veri- fying deep neural networks. CAV (f), volume 10426 of
10

---

CТВЕNNH: A Library and Benchmark for Certified Training
Lecture Notes in Computer Science, pp. 97-117. SpriierSr 2017.
Salman, H., Jain, S., Wong, E., and Madry, A. Certified patch robustness via smoothed vision transformers. In Proc. of CVPR, 2022.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Proc. of ICLR, 2015.
Shi, Z.,, Wang, Y., Zhang, H., Yi, J., and Hsieh, C. Fast certified robust training with short warmup. In Proc. of NeurIPS, 2021.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Singh, G., Gehr, T., Mirman, M., Püschel, M., and Vechev, M. T. Fast and effective robustness certification. In Proc. of NeurlPS, 2018.
Le, Y. and Yang, X. S. Tiny imagenet visual recognition challenge, 2015.
Singh, G., Gehr, T., Püichell, M., and Vechev, M. T. An abstract domain for certifying neural networks. In Proc. of POPL, 2019.
LeCun, Y., Cortes, C., and Burges, C. Mnist hand- written digit database. ATT Labs [Online]. Available: 2010.
http://aan.lecuu..co//eeee//
Soletskyi, R. and Dairympie, D. Training safe neural networks with global sdp bounds, 2024. URL https:
Lee, S., Lee, W. Park, J., and Lee, J. Towards better un- derstanding of training certifiably robust models against adversarial examples. In Proc. NeurlPS, 2021.
//arxiv.orgaas//2400 09687.
Szegedy, C., Zaremba, W. Sutskever, I., Bruna, J., Erhan, D., Goodfellow. J., and Fergus, R. Intriguing properties of neural networks. Proc. of ICLR, 2014.
Li, L., Xie, T., and Li, B. Sok: Certified robustness for deep neural networks. SP, pp. 1319-1310. IEEE, 2023.
Lu, L.. Shin, Y., Su, Y. and Karniadakis, G. Dying relu and initialization: Theory and numerical examples, 2019.
Tjeng, V., Xiao, K. Y., and Tedrake, R. Evaluating robust- ness of neural networks with mixed integer programming. In Proc. of ICLR, 2019.
Lyu, S., Shaikh, S., Shpilevskiy, F., Shelhamer, E., and Lécuyer, M. Adaptive randomized smoothing: Certi- fied adversarial robustness for multi-step defences, 2024. URL https::/arxiv.or http://aaxiiv..rrg///////0/00 10427.
Tramèr, F., Carlini, N., Brendel, W., and Madry, A. On adaptive attacks to adversarial example defenses. In Proc. of NeurlPS, 2020.
Wang. Z., Jha, S., Krishnamurthy, and Dvijotham. Efficient symbolic reasoning for neural-network verification, 2023. URL httphtttss///ai.....
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In Proc. of ICLR, 2018.
/2303. 13588.
Wong. E. and Kolter, Z. Provable defenses against adver- sarial examples via the convex outer adversarial polytope. In Proc. of ICML, 2018.
Mao, Y., Müller, M. N., Fischer, M., and Vechev, M. T. Connecting certified and adversarial training. In Proc. of NeurlPS, 2023.
Wong, E., Schmidt, F. R., Metzen, J. H., and Kolter, J. Z. Scaling provable adversarial defenses. In Proc. of NeurIPS, 2018.
Mao, Y., Mueller, M. N., Fischer, M., and Vechev, M. Un- derstanding certified training with interval bound propa- gation. In Proc. of ICLR, 2024.
Wu, Y. and Johnson, J. Rethinking "batch" in batchnorm, 2021.
Mao, Y., Zhang, Y., and Vechev, M. On the expressiveness of multi-neuron convex relaxations, 2025. URL https: wttrxiw.org/abb/24110 https////11111111111.11
Xu, K., Shi, Z., Zhang, H., Wang, Y., Chang, K., Huang,
M., Kailkhura, B., Lin, X., and Hsieh, C. Automatic perturbation analysis for scalable certified robustness and beyond. In Proc. of NeurIPS, 2020.
Mirman, M., Gehr, T., and Vechev, M. T. Differentiable ab- stract interpretation for provably robust neural networks. In Proc. of ICML, 2018.
Xu, K., Zhang, H., Wang, S., Wang, Y., Jana, S., Lin, X., and Hsich, C. Fast and complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers. In Proc. of ICLR, 2021.
Mu, N. and Gilmer, J. MNIST-C: A robustness benchmark for computer vision, 2019.
Müller, M. N., Eckert, F., Fischer, M., and Vechev, M. T. Certified training: Small boxes are all you need. In Proc. of ICLR, 2023.
Zhang, H., Weng, T., Chen, P., Hsieh, C., and Daniel, L. Effi cient neural network robustness certification with general activation functions. In Proc. of NewrIPS, 2018

---

CTBENCH: A Library and Benchmark for Certified Training
Zhang, H., Chen, H., Xiao, C.,, Gowal, S., Stanforth, R., Li, B., Boning, D. S., and Hsieh, C. Towards stable and efficient training of verifiably robust neural networks. In Proc. of ICLR, 2020.
Zhang, H., Wang. S., Xu, K., Li, L., Li, B., Jana, S., Hsieh, C., and Kolter, J. Z. General cutting planes for bound- propagation-based neural network verification, 2022.
Zhang, M., Backes, M., and Zhang, X. Generating less cer- tain adversarial examples improves robust generalization, 2023.
12

---

CТВЕNNH: A Library and Benchmark for Certified Training
A. Improvement Decomposition
Decomposition of the universal modifications we made such as batch norm fixes and the hyperparameter tuning is not always possible, as these modifications allow additional vectors of hyperparameter for tuning. For example, we fix batch norm statistics in one batch rather than reset it multiple times as done in some original implementations, allowing Whob be tuned within [0, 1], while in the literature Whob has to be fixed to 1. Therefore, we cannot formally decompose the effects of hyperparameter tuning and batch norm behaviors, as they are closely dependent each other.
The literature results are run with three different random seeds, and only the best results among them are reported. This prevents us from substituting our fine-tuned hyperparameter to the original implementation because merely using the same hyperparameters even based on the original implementation hardly reproduces the same number as reported in the literature. In contrast, we run every experiment with the same fixed random seed to allow fair and faithful comparison. Nevertheless, we can showcase the effect for one setting: IBP on MNIST e 0.3. The literature reports 93.1 % certified accuracy, while the same hyperparameter results in 93.18% % in implementation. Further tuning the hyperparameters as in the CTBench benchmark gets 93.8%. While this proves the effectiveness of both the implementation and our hyperparameter tuning, we would like to note that based on previous arguments, this does not faithfully decompose the effect of hyperparameter tuning and batch norm changes, and such decomposition efforts are doomed to fail.
While full disentanglement is infeasible, we conduct preliminary studies to separate implementation advantages from hyperparameter tuning. Table 18 compares CNN5 performance using CTBench and the SOTA codebase, applying CNN7- tuned hyperparameters to both to reduce tuning bias, showing CTBench's 's universal implementation benefits. Additionally, Table 3 (L1 regularization on IBP-trained networks) and Figure 7 (effects of varying À for SABR and STAPS and for MTL-IBP on the robustness-accuraay trade-off) illustrate hyperparameter impact.
B. Additional Discussions
B.1. Ablation on L, Regularization
Table shows the effect of Lı regularization on the certified accuracy of IBP-trained networks. We observe that L1 regularization tuned within a small range of hyperparameter choices can improve certified accuracy in most cases, especially for small perturbation sizes.
Table 3: Effect of L1 regularization for IBP-trained networks on different datasets and perturbation sizes.
Setting
L, weight
Nat. [%]
Cert. [%]
98 92
98.21
84
MNIST 0.1
2- 10
98.87
98.26
5 1-10
98.85
98.13
98.52
93.56
MNIST 0.3
2- 10-6
98.51
93.66
5- 10-
98.40 98.40
93.76
67.81
$5.45
CIFAR 2/255
2- 10-6
66.15
55,01
5- 10-
.1.41
54.
48.51
35.28
CIFAR 8/255
1 10-5
48.31
34.36
25.68
19.04
2- 10
26.26
19.82
TIN 1/255
26.37
19.80
1-10-
26.77
19.82
B.2. Robustness-Accuracy Trade-off
The robustness-accuracy trade-off is well-known, where higher certified accuracy often comes at the cost of natural accuracy. Most methods, including SABR and MTL-IBP, have hyperparameters (e.g., X, a) that directly regulate this trade-off. Our goal, like in prior work, is to maximize certified accuracy, with natural accuracy improvements seen as a bonus. For completeness, we further provide robustness- accuracy curves, as shown in Figure 7.
13

---

CTBBNCH: A Library and Benchmark for Certified Training
Consistent with prior work (Mäller et al. (2023, Figure 7) and De Palma al. (2024, Figure 1)), we observe that reducing regularization initially improves both robustness and natural accuracy, but beyond optimal point, further reduction severely hurts certifiability while increasing natural accuracy.
Certified Aocuracy [%]
SABR
64-
MTL.IBP
STAPS
76
77
78
80
Natural Accurny [X]
Figure 7: Robustness-Accuracy Trade-off for SABR, STAPS and MTL-IBP on CIFAR-I0, 10, 2/255.
Additionally, we provide a visualization of how the coouustnessacccuraa trade-off changes for all certification settings in Figure 8, comparing CТВЕECH results with literature results. The transparent points represent previously reported best literature results, while the solid points represent СТВЕNCH results.
B.3. Extending To Deterministic Certified Robustness to Other Norms
Our work focuses on deterministic certified training using bound propagation for the Los norm, as remains the most reliable and widely adopted approach for robustness guarantees. While Li et al. (2023) explores various norms for certification, also limits deterministic certified training T reflecting the current state of the field, with practical deterministic methods focused on L..
Certification under other norms, such as L2, faces scalability challenges. For example, Wang et al. (2023) evaluate L2 certification on small models with only 192 hidden nodes, while our CNN7 network has over 10M parameters, making their method impractical. Similarly, Soletskyi & Dalrymple (2024) use expensive SDP methods, limiting their approach to synthetic toy datasets (Spheres) and does not naturally extend to Lo Furthermore, while Soletskyi & Dalrymple (2024) address norm robustness, their methods do not naturally extend Los.
Bound propagation is difficult for norms other than L... For example, for the L2 norm, it is difficult to track the exact shape of the L2 ball after passing through a linear and ReI.U layer. To apply bound propagation, the typical L.2 ball with radius = 1 used in randomized smoothing must be over-approximated by the full [0, 1jdh input set, which guarantees meaningless results. Notably, Wang et al. (2023) do not attempt bound propagation. Developing new deterministic certification methods for other norms is out the scope of this work.
Exploring deterministic certified training for other norms is a valuable future direction. However, due to scalability limitations and the lack of effective methods for other norms (even on MNIST), our focus remains on Loc.
B.4. Comparison with Loo Randomized Certified Robustness
In this section, we conduct preliminary study on comparing -nom robustness certified by RS to our results based on deterministic algorithms. Specifically, we compare the numbers by the state-of-the-ar -nom RS algorithm of Lyu et al. (2024) on CIFAR- 10 at 2/255 and 8/255 with CTBench results in Table 4. We find that the current RS approaches yield lower certified accuracy compared to CTBench, in agreement with the literature where deterministic methods dominate the Los -oom robustness.
B.5. General Trends Across Datasets
Across the datasets considered in this work, several performance trends emerge, offering insights into how different certification and training methods generalize. For both MNIST and CIFAR-10, we observe that IBP shows decent performance at larger perturbation sizes, while other methods show limited improvement over IBP. This suggests that as perturbations increase in magnitude, stronger regularization is crucial for maintaining certifiability. In the context of corrupted datasets (MNIST-C and CIFAR-10-C), adversarial and certified training methods effectively enhance robustness
14

---

CТВЕNNH: A Library and Benchmark for Certified Training
Cortified Accuracy [X]
Certified Accuracy [%]
CTBBacch Lisenature
IBP
98.4
93.8
CROWN-IRP
SABR
IBP
93.6
CROWN-IB
STAPS
SABR
93.4
MTL-IIPP
98.0
TAPS
93.2
STAPS
CTBeachh Literature
97.8
83 MTL--IBP
93.0
98.8
98.9
99.0
99.1
99.2
07.8
98.0
98.2
98.4 4
98.6
98.8
Natural Ancuracy [96]
Natural Accuracy [X6]
(a) MNIST, c 0.1
(b) MNIST, = 0.3
Certified Accurscy [X]
Certified Accuracy [%
64
CTBeench
CTBBench Literature
Literature
35
62
60
IBP
IBP
CROWN-IBP
34
CROWN-IBP
58-
SABR
SABR
56-
TAPS
TAPS
STAPS
STAPS
54
MTL-IBP
B MTL-IBP
68
70
72
74
76
78
80
46
48
50
52
54
Natural Accuracy [%]]
Natural Accurary [%]]
(c) CIFAR-10, 2/255
(d) CIFAR- 10, 8/255
Certified Accuracy [%6]
28
CTBench
26-
Literature
24-
IBP
CROWN-II -BP
22-
SABR
20
STAPS
83 MTL-IBP
18
26
28
30
32
34
36
38
Natural Accuracy [%]]
(e) TINYIMAGENET, e 1/255
Figure 8: Visualization of CTВENCH improvements on the robustness-accuracy trade-off for each certification setting.
15

---

CТВЕENH: A Library and Benchmark for Certified Training
Table 4: Comparison between Deterministic Certified Training (this work) and Adaptive Randomised Smoothing (Lyu et al.,
2024) on CIFAR-10.
Method
Nat. [%]
Cert, al
[%]]
Cert.
[%]]
255
253
IBP
67.49
55.99
48.51
35.28
SABR
77.86
63.61
SABR
52.71
35.34
MTL-IBP
78.82
64.41
MTL- IBP
54.28
35.41
ARS v = 0.12
79.21
48.79
0.00
ARS a 0.25
73.61
48.79
0.00
ARS x 0.50
65.47
51.98
8.96
ARS V = 0.75
57.71
44.64
11.91
ARS x 1.00
49.96
40.71
14.07
ARS 1.50
39.86
31.41
14.57
against localized perturbations such as blur, noise, and pixelation. However, these methods remain less resilient to global transformations like brightness and contrast changes compared to standard training. This observation aligns with the intuition that adversarial and certified training primarily improve robustness in the immediate neighborhood of the original inputs, whereas global changes fall outside this region. More diverse data augmentation strategies may mitigate this problem, though this may also come at the cost of reduced certified adversarial accuracy.
When examining network-leves properties such as neuron instability and network utilization, trends across datasets are less straightforward. In all cases, standard training results in the highest neuron instability, expected due to the absence of regularization aimed at minimizing this effect. However, network utilization does not follow a consistent pattern. In some scenarios. certified training increases network utilization compared to adversarial training, indicating the learning of more complex patterns and relationships. Nevertheless, this trend is not universally observed, suggesting that the underlying dynamics of network utilization may be context-specific.
Overall, these findings highlight that while some performance trends persist across datasets, others are context-dependent, underscoring the need for context-specific analysis when evaluating the current stage of certified robustness methods.
B.6. Comparing the OOD Generalization of Certified and Adversarial Training
On corrupted datasets (MNIST-C, CIFAR-10-C), adversarial and certified training improve robustness against localized perturbations like blur, noise, and pixclation but struggle with global shifts like brightness and contrast changes. This aligns with the intuition that these methods enhance robustness mainly in the Euclidean neighborhood of the original inputs, whereas global changes fall outside this specification. Moreover, the stronger regularization induced by certified training when compared to adversarial training often exceeds what's 's needed for untargeted corruptions.
B.7. Comparing our Loss Fragmentation Result with Shi et al. (2021)
While our approach and the results of Shi et al. (2021) share common concepts and both target to quantify the difficulty of certification, we clarify that Shi et al. (2021) analyze only real instability of neurons. In contrast, our analysis applies an estimate the true number of unstable neurons. To illustrate this difference, we provide a comparison between the two variants in Table We observe that the gap between our lower bound estimate and IBP is larger for SOTA methods, which also reflects in the certification gap between IBP and MN-BAB for these models (Table 22).
instability, which is an wveraapproximation of the
Both approaches aimed quantifying neuron instability hard to generalize to non-ReLU networks. However, this is of critical importance to the certification of ReLUU networks, beyond measuring the difficulty of adversarial attacks which may also be indicated by other smoothness metrics. Concretely, aannchh-nd-buun (BaB), the dominating strategy for complete certification of ReLU networks, directly branches the unstable neurons; thus, the number of unstable neurons provides a direct metric for the difficulty of certification. Since ReLU networks dominate certified training, we adopt the number of unstable neurons as the main metric.
16

---

CТВЕNNH: A Library and Benchmark for Certified Training
Table 5: Unstable neuron estimates using our sampling method (lower bound) and IBP (upper bound) for networks trained on CIFAR-10 = 2/255.
Unstable Neurons [%]
Method
lower bound
upper bound (IBP)
IBP
0.97
1.26
CROWN-IBP
1.16
1.69
SABR
1.37
2,82
TAPS
1.00
1.68
STAPS
1.32
2.64
MTL-IBP
1.53
3.42
C. Experiment Details
C.1. Dataset
We use the MNIST (LeCun et al., 2010), CIFAR- 10 (Krizhevsky et al., 2009) and TINYIMAGENET (Le & Yang, 2015) datasets for our experiments. All are open-source and freely available with unspecified license. The data preprocessing mostly follows De Palma et al. (2024). For MNIST, we do not apply any preprocessing. For CIFAR-10 and TINYIMAGENET, we normalize with the dataset mean and standard deviation and augment with random horizontal flips. We apply random cropping to 32 x 32 after applying a 2 pixel zero padding at every margin for CIFAR-10, 10, and random cropping to 64 x 64 after applying 4 pixel zero padding at every margin for TINYIMAGENET. We train on the corresponding train set and certify the validation set, as adopted in the literature (Shi et al., 2021; Müller al., 2023; Mao al., 2023; De Palma al., 2024).
C.2. Model Architectures
We follow Shi al. (2021); Müller al. (2023) and use a CNN7 with Batch Norm for our main experiments. CNN7 is convolutional network with 7 convolutional and linear layers. All but the last linear layer are followed by Batch Norm and ReLU layer. This architecture is found to achieve uniformly better results across settings (Shi et al., 2021), and thus is adopted by the literature (Shi et al., 2021; Müller et al., 2023; Mao et al., 2023; De Palma et al., 2024). For TINYIMAGENET, the stride of the last convolution is doubled to reduce the cost.
C.3. Training Details
Initialization Adversarial training methods are initialized by Kaiming uniform (He et al., 2015), while certified training methods are initialized by IBP initialization (Shi et al., 2021).
Training Schedule We mostly follow the training schedule of (De Palma et al., 2024), but in some cases shorter schedule to reduce cost. Specifically, the warmup phase is 20 epochs for MNIST 0.1 and e 0.3, 80 epochs for CIFAR-10 2 120 epochs for CIFAR-10 and NINYIMAGENET, we use standard training for additional epoch at the beginning. We apply the IBP regularization proposed by (Shi et al., 2021), with weight equals 0.5 on MNIST and CIFAR-10, and 0.2 on TINYIMAGENET, during
253 and 80 epochs for TINYNMAGENET
111 In addition, for CIFAR-10
255
255
the warmup phase. In total, we train 70 epochs for MNIST E = 0.1 and = 0.3, 160 epochs for CIFAR-10 e 255
2 240
epochs for CIFAR-10 e 255
8 and 160 epuchs for TINYIMAGENET 255
Optimization We use Adam (Kingma & Ba, 2015) with learning rate of 0.0005. The learning rate is decayed by factor of 0.2 epoch 50 and 60 for MNIST 0.1 and 0.3, at epoch 120 and 140 for CIFAR-10 We use a batch size of 256 for
255 at epoch 200
and 220 for CIFAR-10 MNIST, and 128 for CIFAR-10 and TINYIMAGENER Gradients of each step are clipped to 10 in Le norm. No weight decay is applied and L, regularization only on weights of linear and convolution layers is used. Further, Wu & Johnson (2021) find that rumning statistics lag behind the population statistics and propose to use the population statistics for testing. We adopt this strategy CTBENCH, since only needs to compute Lex and is much cheaper than the computation of Lub.
8 255. and at epoch 120 and 140 for TINYIMAGENET
255
17

---

СТВЕNNE: A Library and Benchmark for Certified Training
Table 6: Best hyperparameter for MNIST e 0.1.
PGD
EDAC
IBP
CROWN-IBP
SABR
TAPS
STAPS
MTL-IBP
11 regularization
X 10-3
1 X 10-3
2 X 10-6
2
10-6
1 10
1 10-6
X 10
X 10-3
Whob
1.0
1.0
1.0
1.0
0.7
0.7
0.7
0.7
Train
0.2
0.2
0.2
0.2
0.2
0.2
0.2
0.2
e shrink ratio
/
0.4
0.4
/
Classifier size
3
TAPS gradient scale
4
4
ReLU shrink ratio
/
IBP coefficient
1
0.02
C.4. Tuning Scheme
We conduct hyperparameter tuning for each method ensure the best performance, and reduce the search space whenever appropriate based human knowledge. The search space for each hyperparameter
follows:
In regularization: (1 10 2,2x 10 6,5 x 10 10-5,22 ,2x x 10 5, 5 10 10-5). We include 10 sspeciiically
for CIFAR-IO 255 as this is the value reported by De Palma al. (2024).
Wnb: (0.7, 0.9, 1.0). Surprisingly, Whoh equal improve both certified and natural accuracy by large margin when is small.
Train 2x train for MNIST 0.1, and tune within {Ix, 1.25x, 1.5x) specifically for CIFAR-10 255 For others, we use the test c for training.
shrink ratio for SABR and STAPS: we mostly keep the value in the literature. When we observe large certifibility gap, we increase the shrink ratio by 0.1 until the performance fails to increase consistently.
Classifier size for TAPS and STAPS: we keep the value in the literature for TAPS, and include only 1 ReLU layer the classifier for STAPS universally.
TAPS gradient seale: (1,2,3,4,6, 6,
ReLU shrink ratio for SABR and STAPS: we keep the value the literature, thus shrinking the output box of each ReLU by multiplying 0.8 CIFAR-I0
2 and do not apply this in other settings.
255
IBP coefficient for MTL-IBP: (0.01, 0.02, 0.05) for MNIST = 0.1, CIFAR-10 255 and TINYIMAGENET and 10.4,0.5, 0.6) for MNIST 0.3, CIFAR-10 &
255
Attack Strength: we use 3 restarts everywhere for the attack. By default, we use 10 steps for MNIST 0.1, steps for MNIST 0.3, 8 steps for CIFAR-10 255
2 10 steps for CIFAR-10 255 and step for TINY IMAGEMTT 255 However, we find MTL.-IBP benefits from using only step everywhere, while more steps will hurt certified accuracy, thus we only use step specifically for MTL-IBP except CIFAR-10 (2024). We further only use 2x attack for MTL-IBP on CIFAR-10
8
255 consistent to De Palma et al.
255
We report the best hyperparameter for each method respectively Table 6, Table 7, Table 8, Table 9, and Table 10.
C.5. Certification Details
We combine IBP (Gowal et al., 2018), CROWN-IBP (Zhang et al., 2020), and MN-BAB (Ferrari et al., 2022) for certification running the most precise but also computationally costly MN-BA BAB only on samples not certified by the other methods. The timout for each input is set 1000 seconds.
18

---

CТВЕNNH: A Library and Benchmark for Certified Training
Table 7: Best hyperparameter for MNIST 0.3.
PGD
EDAC
IBP
CROWN-IBP
SABR
TAPS
STAPS
MTL-IBP
L, regularization
10
10
10
10
10
10
10
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
Train e
0.3
0.3
0.3
0.3
0.3
0.3
0.3
0.3
shrink ratio
0.8
0.8
Classifier size
TAPS gradient scale
ReLU shrink ratio
IBP coefficient
0.5
Table 8: Best hyperparameter for CIFAR-I0 2/255.
PGD
EDAC
IBP
CROWN-IBP
SABR
TAPS
STAPS
MTL-IBP
L1 regularization
x 10-3
5 10-6
1 X 10-6
1 x 10-6
X 10
x 10
X 10
10-6
Wrob
1.0
1.0
1.0
1.0
0.7
1.0
1.0
0.9
Train e
2/255
2/255
2/255
2/255
3/255
2/255
3/255
2/255
shrink ratio
0.1
0.1
Classifier size
5
TAPS gradient scale
5
ReLU shrink ratio
0.8
0.8
IBP coefficient
0.01
Table 9: Best hyperparameter for CIFAR-10 8/255.
PGD
EDAC
IBP
CROWN-IBP
SABR
TAPS
STAPS
MTL-IBP
L regularization
10
10-6
0
0
Wjob
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
Train
8/255
8/255
8/255
8/255
8/255
8/255
8/255
8/255
shrink ratio
0.7
/
0.9
Classifier size
/
1
1
TAPS gradient scale
/
/
2
2
ReLU shrink ratio
/
/
IBP coefficient
1
0.5
Table 10: Best hyperparameter for TINYIMAGENET 1/255.
PGD
EDAC
IBP
CROWN-IBP
SABR
TAPS
STAPS
MTL-IBP
bn regularization
5 x 10
x 10
x 10
1 10
1 10
1 10
X 10
5 x 10
Weob
1.0
1.0
1.0
1.0
1.0
1.0
1.0
0.7
Train
1/255
1/255
1/255
1/255
1/255
1/255
1/255
1/255
shrink ratio
0.4
0.6
Classifier size
TAPS gradient scale
/
8
4
ReLU shrink ratio
IBP coefficient
0.05
19

---

CТBЕNNH: A Library and Benchmark for Certified Training
C.6. Computation
We train and certify MNIST 0.1, MNIST E 0.3 and CIFAR-I0 e 2080 Ti with Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz and 530GB RAM. We train and certify CIFAR-10 2 and TINYIMAGENET RAM. The training and certification time for each method is reported in Table 12. We provide a detailed complexity analysis for each training method in Table 11.
255 models on single NVIDIA GeForce RTX
255 models on single NVIDIA L4 with Intel(R) Xeon(R) CPU @ 2.20GHz CPU and 377 GB
Table 11: Detailed breakdown of training costs for each Certified Training method.
Method
Training cost per batch
Details
Standard
T
Forward Backward
PGD / EDAC
(M+1)T
M attack steps Standard loss computation
IBP
2T
Lower and Upper Bounds propagation
CROWN-IBP LF
4T
IBP pass back-substitution of IBP bounds
SABR
(M+2yT
IBP + PGD
MTL--BP
(M+2yT
IBP + PGD
TAPS
(*(M+))*(T-)
for first network split PGD second split for each class
STAPS
21 <*(M+I)((T-) + (M+1)T
TAPS PGD
Legend
Time cost for Standard Training (Includes Forward + Backward Pass)
M
Number of adversarial attack steps (including repeats)
Number classes
I.
Time cost for Standard Training in the first network split (TAPS)
20

---

CТВЕNNH: A Library and Benchmark for Certified Training
Table 12: Training and certification time for each method different datasets and
Dataset
Method
Train Time (seconds)
Certification Time (seconds)
PGD
1.5 x 104
EDAC
3.1 x 104
IBP
2.1 X 10a
2.5 x 10
CROWN-IBP
5.6 10a
1.8 10
0.1
SABR
1.8 x 10*
6,0 x 10
TAPS
3.8 10*
6.0 103
STAPS
2.5 x 10*
6.9 x 103
MTL-IBP
6.8 10
6.8 10
MNIST
PGD
1.1 x 10a
EDAC
2.2 x 104
IBP
2.6 x 10a
3.2 104
CROWN-IBP
5.4 X 10s
2.6 X 104
SABR
9.7 X 10S
5,2 10a
TAPS
7.1 10S
4.7 X 10a
STAPS
1.4 X 10a
5.1 10
MTL-IBP
5.5 x 10S
44 x 10a
PGD
x 10*
EDAC
1.3 x 105
IBP
x 10a
1.3 10
CROWN-IBP
2.7 X 104
1.9 X 105
255
SABR
x 10a
10
TAPS
1.1 X 105
1.1 X 105
STAPS
4.5 10a
10
MTL-IBP
3.6 x 10*
2.7 x 10
CIFAR-10
PGD
6.4 x 10a
EDAC
1.3 x 10S
IBP
1.1 x 10a
1.9 x 104
CROWN-IBP
2.1 X 10a
2.0 10a
253
SABR
4.1 X 104
6.5 x 10a
TAPS
3.3 x 10a
4.0 10a
STAPS
9.9 x 10a
4.2 x 10a
MTL-IBP
2.2 x 10a
5.6 10
PGD
1.0 X 105
EDAC
x 105
IBP
6.7 x 104
4.9 x 10a
CROWN-IBP
x 10
1.3 104
TINYIMAGENET
SABR
1.1 X 105
1.8 x 104
TAPS
x 10
1.5 104
STAPS
3.3 x 105
2,6 X 10a
MTL-IBP
10
10
21

---

CТВЕNNH: A Library and Benchmark for Certified Training
D. Additional Results
D.I. Result Significance
In Tables 13-17, we present the natural and certified accuracy across certified training algorithms, datasets and perturbation levels. We report the average and standard deviation across 3 random seeds for each method. The results show that our improvements over previously reported values in the literature are significant, in most cases the statistical difference being larger than 30.
Table 13: Comparison of Natural and Certified Accuracy between CТВЕNCH and previous literature results on MNIST 0.1. We report average and standard deviation across random seeds for each method.
Method
Source
Nat. [%]
Cert. [%]
Literature
98.84
97.95
IBP
This work
98.86 + 0.06
98.25 +: 0.03
Literature
98.83
97.76
CROWN-IBP
This work
98.93 + 0.01
98.17 + 0,05
Literature
99.23
98.22
SABR
This work
99.15 + 0.08
98.42 + 0.03
Literature
99.19
98.39
TAPS
This work
99.20 1:: 0.05
98.5 +: 0.04
Literature
99.15
98.37
STAPS
This work
99.15 + 0.04
98.38 + 0.10
Literature
99.25
98.38
MTL-IBP
This work
99.16 + 0.03
98.31 + 0.06
Table 14: Comparison of Natural and Certified Accuracy between CTBENCH and previous literature results on MNIST é 0.3. We report average and standard deviation across random seeds for each method.
Method
Source
Nat. [%]
Cert. [%]
Literature
97.67
93.10
IBP
This work
98.55 + 0.02
93.82 + 0.10
Literature
98.18
92.98
CROWN-IBP
This work
98.46 + 0.03
93.84 + 0.12
Literature
98.75
93.40
SABR
This work
98.69 + 0.03
93.64 + 0.06
Literature
97.94
93.62
TAPS
This work
98.58 1: 0.03
93.90 1: 0.1
Literature
98.53
93.51
STAPS
This work
98.69 + 0.06
93.60 + 0.05
Literature
98.80
93.62
MTL--IBP
This work
98.75 + 0.02
93.82 + 0.21
22

---

CТВЕNNH: A Library and Benchmark for Certified Training
Table 15: Comparison of Natural and Certified Accuracy between CТВЕNCH and previous literature results CIFAR-10 2/255. We report average and standard deviation across random seeds for each method.
Method
Source
Nat. [%]
Cert. [%]
Literature
66.84
52.85
IBP
This work
66.85 + 0.72
55.32 = 0.68
Literature
71.52
53.97
CROWN-IBP
This work
67.56 0.04
56.69 = 0.58
Literature
79.24
62.84
SABR
This work
77.82 1 0.28
63.62 1: 0.22
Literature
75.09
61.56
TAPS
This work
74.76 + 0.34
61.37 + 0.09
Literature
79.76
62.98
STAPS
This work
76.88 + 0.15
63.96 + 0.27
Literature
80.11
63.24
MTL-IBP
This work
78.91 +: 0.16
64.00 + 0.37
Table 16: Comparison of Natural and Certified Accuracy between СТВЕΝСИ and previous literature results CIFAR-10 8/255. We report average and standard deviation across 3 random seeds for each method.
Method
Source
Nat. [%]
Cert. [%]
Literature
48.94
34.97
IBP
This work
48.74 1: 0.23
34.99 d: 0.28
Literature
46.29
33.38
CROWN-IBP
This work
48.24 0.09
32.49 + 0.18
Literature
52.38
35.13
SABR
This work
52.51 + 0.38
34.97 + 0.62
Literature
49.76
35.10
TAPS
This work
49.82 1: 0.28
34.89 1: 0.40
Literature
52.82
34.65
STAPS
This work
51.46 + 0.25
35.32 + 0.25
Literature
53.35
35.44
MTL-IBP
This work
53.72 + 0.49
35.23 + 0.18
23

---

CТВЕNNH: A Library and Benchmark for Certified Training
Table 17: Comparison of Natural and Certified Accuracy between CТВЕNCII and previous literature results on TINYIMA-
GENET ε 1/255. We report average and standard deviation across 3 random seeds for each method.
Method
Source
Nat. [%]
Cert. [%]
Literature
25.92
17.87
IBP
This work
26.4 =: 0.45
19.87 0.19
Literature
25.62
17.93
CROWN-IBP
This work
28.16 + 0.27
21.69 + 0.42
Literature
28.85
20.46
SABR
This work
30.96 + 0.41
21.14 + 0.2
Literature
28.34
20.82
TAPS
This work
28.59 + 0.09
21.54 = 0.22
Literature
28.98
22.16
STAPS
This work
30.25 + 0.33
22.03 + 0.25
Literature
37.56
26.09
MTL-IBP
This work
35.97 + 0.17
27.49 0.21
D.2. Architecture Generalization
In Table 18 we present the natural and certified accuracy on CNN5, which is five-layer CNN smaller than CNN7. We observe that the improvements are consistent across different settings even with CNN5, showing that the improvements are not specific certain architecture.
Table 18: Comparison on CNN5 between CTВЕNCH and the implementation of De Palma et al. (2024).
Method
Code and hyperparameters Nat. [%]
Cert. [%]
СТВЕNCH
98.19
92.88
IBP
(De Palma et al., 2024)
93.16
81.81
CТВЕNCH
98.41
92.62
SABR
(De Palma et al., 2024)
97.33
90.87
CТВЕNCH
98.41
92.49
MTL-IBP
(De Palma et al., 2024)
98.39
91.45
24

---

CТВЕNNH: A Library and Benchmark for Certified Training
D.3. Ablation Certification Algorithms & Additional Analysis Shared Mistakes
In Table 19 we present the correlation between the certification capabilities of SOTA verifiers (MN-BAB (Ferrari et al., 2022) and OVAL (De Palma al., 2022)). We observe that there is very high correlation between the two verifiers, which expected since both are based on the same underlying principles. This shows that the certification algorithms have reached certain level of maturity and are converging to similar results. Combining the verified sets of the two verifiers, we get marginal improvement in certified accuracy at the cost of a much larger certification time. Therefore, we only use MN-BAB for certification our main experiments.
Table 19: Observed count of common mistakes of certification algorithms (MN-BA) (Ferrari et al., 2022) and OVAL (De Palma et al., 2022)) on CIFAR-I0 against their expected values assuming independence across certification mistakes.
neither certify
one certifies
both certify
obs.
3549
15
6436
2/255
exp.
1264
4585
4151
obs. 6454
3537
8/255
exp.
4171
4575
1254
In Table we present the observed count of common mistakes that different certified training models make on CIFAR-10 against their expected values assuming independence across model mistakes. We observe that the observed count significantly higher than the expected count, indicating that the models are highly correlated in their mistakes.
Table 20: Observed count of common mistakes CIFAR-10 against their expected values assuming independence across model mistakes.
models succeeded
0
2
5
6
obs.
2350
653
520
564
708
894
4311
exp.
35
330
1296
2704
3163
1965
507
obs.
5206
679
487
388
387
585
2268
255
exp.
766
2457
3283
2339
937
200
18
Furthermore, Table 21 examines shared mistakes between CNN5 and CNN7, revealing common patterns across architectures.
Table 21: Additional results CNN5 and CNN7 shared mistakes for MNIST 0.3
Number of not certified samples
Models
Observed
Expected if independent
CNN5 IBP
771
CNN5 SABR
793
CNNS MTL-IBP
746
CNN7 IBP
620
CNN7 SABR
632
CNN7 MTL-IBP
610
/
CNN5, CNN7 IBP
526
48
CNN5, CNN7 SABR
541
50
CNN5, CNN7 MTL-IBP
516
46
All 3 CNN5 networks
593
5
D.4. Deferred Results on CIFAR-10
In Figures and 10, we present additional analyses on the neuron statistics for different models trained on CIFAR-10. We analyze the amount of unstable neurons and the model utilization for each model.
25

---

CТВЕNNH: A Library and Benchmark for Certified Training
evaluated 2/255
trained 2/255
trained at
evaluated 8/255
trained at 8/255
Unstable (%)
10a
10
109-
10-1-
PGD
EDAC
IRP
CROWN-IRP SABR
TAPS
STAPS
MTL-IBP
Figure 9: Ratio of unstable neurons for models trained on CIFAR-10 with different methods and
Utilization [%]
70
trained at 2/255
trained at 8/255
50
30
10
PGD
EDAC
IBP
CROWN-IBE -IPP SABR
TAPS
STAPS
MTL-IBP
Figure 10: Model utilization for models trained on CIFAR-10 with different methods and c. We note that standard training has 35.79% utilization.
In Figure 11, present the out-of-distribution generalization evaluated CIFAR- 10-C for models trained CIFAR-10 8/255, 2/2 /255 and standard training. We observe that the models trained with certified training methods have better out-of-distribution generalization compared to standard training.
D.5. Results with Incomplete Certification Algorithms
In Table 22, present the results of the certification with more efficient, but incomplete methods (IBP and CROWN-IBP) We observe that the incomplete methods have significant impact on the certified accuracy, and the results are consistent with the previous findings in the literature.
Table 22: comparison of incomplete certification (IBP and CROWN-IBP) against complete certification (MN-BAB) on CIFAR-10 2/255.
Cert [%)
Train Method
Nat [%]
Adv [%]
IBP
CROWN-IBP
MN-BAB
67.49
54.22
54.57
55.99
56.10
CROWN-IBP
67.60
49.92
54,86
57.11
57.28
SABR
77.86
12.12
44,79
63.61
65.56
TAPS
74.44
28.22
50.14
61.27
62.62
STAPS
77.05
0.72
30.92
64.21
66.09
MTL-IBP
78.82
0.62
23.31
64.41
67.69
26

---

CТBЕNNH: A Library and Benchmark for Certified Training
PGD
93
92
98
EDAC
96
49
95
93
69
86
92
96
92
91
98
90
98
95
97
96
97
93
IBP 88
57
97
94
68
69
95
91
95
77
98
94
98
92
93
92
91
95
CROWN-IBP
97
95
95
88
94
SABR
86
55
96
94
67
66
94
92
94
81
98
98
92
94
93
93
94
TAPS
88
57
96
93
69
70
94
90
95
77
98
93
98
91
92
92
90
94
STAPS
55
96
94
69
95
92
95
78
98
93
98
92
94
93
92
MTL-IBP
33
95
93
67
94
91
94
79
98
98
92
93
941
PGD
92
86
EDAC
97
53
92
92
74
90
88
90
86
82
97
86
96
94
93
94
92
90
IBP
91
17
91
90
63
76
88
85
90
67
98
86
97
92
89
91
89
88
CROWN-IBP
94
92
6-
91
94
93
89
98
98
92
96
96
SABR
93
48
93
92
66
82
90
96
92
91
98
87
97
92
97
94
97
TAPS
92
49
94
92
67
80
91
95
93
92
99
89
98
93
96
95
96
92
STAPS
49
94
92
66
82
91
97
92
93
98
98
94
98
95
98
91
MTL-IBP
92
92
96
Standard
97
72
84
87
89
80
75
41
55
55
86
78
80
93
55
87
59
77
contrast
defocts blur
elastic, transforn 1og
frost
gaussion. hlur
gaussian, noise glass blur
impulse noise
ipeg compression motion. blur
pixelate
saturate
shot noise
spatter
speckle. noise
20om. blu
bright
Figure 11: Out-of-distribution generalization evaluated on CIFAR-10-C for models trained on CIFAR-10 at 8/255 (top),
e 2/255 (middle) and standard training (bottom).
27


Reviews:
[{"review_id": "cZ0CRlUysc", "reviewer": "Reviewer_vyCo", "summary": "The authors proposed to do a new round of meta-research on the topic of certified training (because the previous one [1] became outdated), compared the top algorithms and baselines with a fair training pipeline, and are going to share a library and related benchmark for further usage.\n\n[1] Linyi Li, Tao Xie, and Bo Li. Sok: Certified robustness for deep neural networks. In SP, pp. 1289–1310. IEEE, 2023.\n\n## update after rebuttal\nAuthors provided more analysis on the diverse datasets so I will keep my original score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We $\\newcommand{\\Rv}{\\textcolor{green}{vyCo}}$thank Reviewer $\\Rv$ for their insightful and careful review. We are happy to hear that Reviewer $\\Rv$ finds our work interesting and well motivated, our experimental results convincing, and our ablation studies insightful. In the following, we address all concrete questions raised by Reviewer $\\Rv$. We include new results, named with Table S1 etc., in the [anonymized link](https://mega.nz/file/eZp1CYBC#hwOYJzm4U47TDzuCmQFTJhCust2pFz-8Wzzy-CmXd6Q).\n\n**Q1: How are the issues of unfair comparison highlighted in this work relate to the broader area of certified robustness, namely non-deterministic methods such as randomized smoothing (RS)?**\n\nWe agree that unfair comparison may also be present in the RS literature. After a brief survey, we find that recently published randomized smoothing techniques vary significantly in network architecture, training schedules, hyperparameter choices, and the noise distributions used for certification, all of which may contribute to unfair comparisons. Therefore, introducing a standardized fair benchmark as well as unified implementation for RS is also important. However, while a meta-survey on the issue may be easily performed, establishing a benchmark at the same level of CTBench requires a large amount of effort, and is way beyond the scope of this work. Concretely, one needs to unify all algorithms in a single library (which has never been developed, to the best of our knowledge), validate that all implementations match (or exceed, as in CTBench) the original reports, and then may start to evaluate them in fair settings. We remark that even the last step, which requires the least human effort, takes months’ compute on four GPUs in CTBench. Therefore, it is impractical for us to conduct the same study for RS in this work. However, we strongly believe that this is a good future work, and we will discuss this direction and the meta-survey in the revised manuscript.\n\nMeanwhile, we conduct a preliminary study on comparing $L_\\infty$-norm robustness certified by RS to our results based on deterministic algorithms. Specifically, we compare the numbers by the state-of-the-art $L_\\infty$-norm RS algorithm [1] on CIFAR-10 $\\epsilon=2/255$ and $\\epsilon=8/255$ with CTBench results in Table S1. We find that the current RS approaches yield lower certified accuracy compared to CTBench, in agreement with the literature that deterministic methods dominate the $L_\\infty$ robustness.\n\n**Q2: The paper includes results for multiple datasets, but each of them is analysed separately. Could the authors provide a further analysis and comparison of performance trends across the datasets?**\n\nSure, we provide a preliminary analysis below, and will include more in the revised manuscript.\n\nAcross the datasets considered in this work, several performance trends emerge, offering insights into how different certification and training methods generalize. For both MNIST and CIFAR-10, we observe that Interval Bound Propagation (IBP) demonstrates great performance at larger perturbation sizes, while other methods show limited improvement over IBP. This suggests that as perturbations increase in magnitude, stronger regularization is crucial for maintaining certifiability. In the context of corrupted datasets (MNIST-C and CIFAR-10-C), adversarial and certified training methods effectively enhance robustness against localized perturbations such as blur, noise, and pixelation. However, these methods remain less resilient to global transformations like brightness and contrast changes compared to standard training. This observation aligns with the intuition that adversarial and certified training primarily improve robustness in the immediate neighborhood of the original inputs, whereas global changes fall outside this region. Addressing this limitation could involve more diverse data augmentation strategies, though this may come at the cost of reduced certified adversarial accuracy.\n\nWhen examining network-level properties such as neuron instability and network utilization, trends across datasets are less straightforward. In all cases, standard training results in the highest neuron instability, as expected due to the absence of regularization aimed at minimizing this effect. However, network utilization does not follow a consistent pattern. In some scenarios, certified training increases network utilization compared to adversarial training, indicating the learning of more complex patterns and relationships. Yet this trend is not universally observed, suggesting that the underlying dynamics of network utilization are context-specific and not easily generalizable.\n\nOverall, these findings highlight that while some performance trends persist across datasets, others are context-dependent, underscoring the need for context-specific analysis when evaluating the current stage of certified robustness methods.\n\n**References**\n\n[1] arxiv.org/abs/2406.10427", "claims_and_evidence": "Main (meta) claims:\n1. almost all algorithms in CTBENCH surpass the corresponding reported performance in literature in the magnitude of algorithmic improvements, thus establishing new state-of-the-art, and \n2. the claimed advantage of recent algorithms drops significantly when we enhance the outdated baselines with a fair training schedule, a fair certification method and well-tuned hyperparameters.\n\nAdditionally, the following insights are provided:\n1. certified models have less fragmented loss surface, \n2. certified models share many mistakes, \n3. certified models have more sparse activations, \n4. reducing regularization cleverly is crucial for certified training especially for large radii and \n5. certified training has the potential to improve out-of-distribution generalization.\n\nActually, all of the claims are supported by careful experimentation pipeline and analysis.", "methods_and_evaluation_criteria": "Although the datasets have the low resolution (like 32x32), taking into account the difficulty of formal verification for deep neural nets, it's the trade-off between realistic data/NN and meaningful performance.", "theoretical_claims": "No really theoretic concepts were covered in the paper. There were some hypotheses that were to some extent proved by experiments (like fragmented loss surface and others) but nothing more.", "experimental_designs_or_analyses": "A very good experimentation part - actually, the whole paper is one big careful experimentation pipeline.", "relation_to_broader_scientific_literature": "The work is related to a more broad certified area of research that comprises both deterministic certified training (the scope of the reviewed paper) and randomized certified robustness [1]. It would be really great to provide a meta-research in this area as well, because highly likely the problems with unfair comparison do exist there as well.\n\n[1] Jeremy M. Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. In Proc. of ICML, 2019.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "In addition to meta-research on randomized certified robustness, I think the following this is missing in the paper: there are a lot of comparisons of methods and other finer grained details *inside* one dataset, but almost zero comparisons of the differences of performance and trends on different datasets - like OOD performance difference on MNIST-C vs CIFAR-10-C vs TinyImageNet, etc. Such sort of insights would be very important to have a meta-view on the datasets as well, not only algorithms.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eMwbivs5DU", "reviewer": "Reviewer_mj6P", "summary": "This paper introduces a novel benchmark for certified training, addressing the inconsistencies in evaluating certifiably robust neural networks. Existing methods suffer from unfair comparisons due to varying training schedules, certification techniques, and under-tuned hyperparameters, leading to misleading claims of improvement. CTBench standardizes evaluations by integrating state-of-the-art certified training algorithms into a single codebase, systematically tuning hyperparameters, and correcting implementation issues, thus reestablishing a stronger state-of-the-art. The study reveals several key insights: (1) certified models have less fragmented loss surface, (2) certified models share many mistakes, (3) certified models have more sparse activations, (4) reducing regularization cleverly is crucial for\ncertified training especially for large radii, and (5) certified training has the potential to improve out-of-distribution generalization.\n\n## update after rebuttal\nThe rebuttal helped improve the soundness of empirical results. Hence, I preserve my positive recommendation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We $\\newcommand{\\Rm}{\\textcolor{blue}{mj6P}}$thank Reviewer $\\Rm$ for their insightful review. We are happy to hear that Reviewer $\\Rm$ finds our work interesting and well motivated, our library and benchmark useful and comprehensive, and our experimental results insightful. In the following, we address all concrete questions raised by Reviewer $\\Rm$. We include new results as Table S1 etc., in the [anonymized link](https://mega.nz/file/eZp1CYBC#hwOYJzm4U47TDzuCmQFTJhCust2pFz-8Wzzy-CmXd6Q).\n\n**Q1: Can the findings of this study be generalised to other architectures than CNN7?**\n\nOur study focuses on the CNN7 architecture, which is consistently adopted by SOTA works to enable direct comparison. Table 14 in Appendix C2 compares CTBench training with the baseline training code from [1] using a smaller CNN5 on MNIST with $\\epsilon = 0.3$. Results show that CTBench improves IBP, SABR, and MTL-IBP performance on CNN5, proving that the benefits extend beyond CNN7. Moreover, it is confirmed on CNN5 that IBP can match SOTA methods under large perturbations with fair comparisons and well-chosen hyperparameters.\n\nFurthermore, Table S2 examines shared mistakes between CNN5 and CNN7, revealing common patterns across architectures. We will include additional results on generalizability in the revised manuscript.\n\n**Q2: The number of unstable neurons as the metric of smoothness may be challenging to generalize to non-ReLU networks. Could the authors comment on this and consider alternatives, such as measuring the change in loss value within a neighborhood of an input sample?**\n\nWe agree that the number of unstable neurons is hard to generalize to non-ReLU networks, as a metric of smoothness. However, it is of critical importance to the certification of ReLU networks, beyond measuring the difficulty of adversarial attacks which may also be indicated by other smoothness metrics. Concretely, branch-and-bound (BaB), the dominating strategy for complete certification of ReLU networks, directly branches the unstable neurons; thus, the number of unstable neurons provides a direct metric for the difficulty of certification. Since ReLU networks dominate certified training, we adopt the number of unstable neurons as the main metric. We will clarify this and comment on generalization to non-ReLU networks in the revised manuscript.\n\n**Q3: CTBench achieves SOTA performance by incorporating different strategies. Could the authors discuss the individual impact of these strategies?**\n\nWe acknowledge the importance of ablation studies and discuss in Appendix A the challenge of fully disentangling these effects due to their interconnected nature.\n\nWhile full disentanglement is infeasible, we conduct a preliminary study to separate implementation advantages from hyperparameter tuning. Table 14 compares CNN5 performance using CTBench and the SOTA codebase, applying CNN7-tuned hyperparameters to both to reduce tuning bias, showing CTBench's universal implementation benefits. Additionally, Table S3 (L1 regularization on IBP-trained networks) and Figure S3 (effects of varying $\\lambda$ for SABR and STAPS and $\\alpha$ for MTL-IBP on the robustness-accuracy trade-off) illustrate hyperparameter impact.\n\n**Q4: Complexity is a crucial metric, but the running time is only reported in the appendix. Could the authors provide more analysis and discussion on the complexity of different training and certifying methods?**\n\nWe agree that complexity is crucial. Therefore, in addition to the running times reported in Appendix (Table 8), we provide a more detailed complexity analysis for each training method in Table S6. \n\nFor certification, we use complete certification algorithms based on branch-and-bound techniques that have exponential complexity, depending on the number of unstable neurons that need to be analyzed. Due to this exponential growth, we use a timeout of 1000 s per sample. Also, we report a new ablation study on certification algorithms in Table S4, including IBP and CROWN-IBP (efficient but incomplete certification algorithms) certified accuracies.\n\n**Q5: The experiment on shared mistakes appears to provide dataset-specific results rather than general insights. Could the authors clarify why curriculum learning improves certified training?**\n\nWe would like to clarify that we do not claim curriculum learning directly improves certified training. Our claim is that data points exhibit varying levels of difficulty for certified training, as evidenced by the systematic deviation from independent errors. The connection between data point difficulty and curriculum learning is discussed in [2], which is why we suggest its potential benefit. Additionally, Appendix C.3 provides a more extensive evaluation across different datasets and certification algorithms, demonstrating that the results are not specific to a single dataset.\n\n**References**\n\n[1] arxiv.org/abs/2305.13991\n\n[2] arxiv.org/abs/1705.08280", "claims_and_evidence": "This paper proposes a novel benchmark for certified training, and its claims and insights are primarily supported by experimental results conducted within this benchmark. While the findings are well-documented, one notable concern is that all results are derived from a single CNN7 architecture, limiting the generalizability of the conclusions. Including more diverse network architectures would strengthen the validity of the results.", "methods_and_evaluation_criteria": "Yes. The evaluation metrics follow the previous studies on certified training, including certified accuracy, natural accuracy, and adversarial accuracy.", "theoretical_claims": "No theoretical claims are made in the paper.", "experimental_designs_or_analyses": "Most of the experimental designs make sense to me. Here are some of my concerns:\n\n1. CTBench uses the number of unstable neurons to represent the smoothness of the loss surface. This can pose difficulty in generalizing to non-ReLU neural networks such as Swish [1] and GELU [2]. A more explicit way is to focus on the change of the loss value within a neighborhood of an input sample.\n\n2. CTBench achieves a state-of-the-art certified training performance. To achieve this desirable performance, several strategies are taken, such as batch norm, hyperparameter tuning, and L1 regularization. It would be helpful to add an ablation study on these strategies. \n\n3. Complexity is an important metric for certified training. However, the running time is only provided in the appendix (Table 8). It would be helpful to add more analyses and discussions on the complexity of different certifying methods.\n\n4. The experiment of shared mistakes does not provide a general insight. The result is specific to the data. It is unclear to me why curriculum learning can improve certified training based on the results.\n\n[1] Ramachandran, Prajit, Barret Zoph, and Quoc V. Le. \"Searching for activation functions.\" arXiv preprint arXiv:1710.05941 (2017).\n\n[2] Hendrycks, Dan, and Kevin Gimpel. \"Gaussian error linear units (gelus).\" arXiv preprint arXiv:1606.08415 (2016).", "relation_to_broader_scientific_literature": "This paper contributes to the broader literature by standardizing the evaluation of certified training methods. While prior works have introduced individual certified training algorithms, there has been no unified benchmark to fairly compare these methods under standardized conditions. By providing a standardized benchmark and implementation framework, this paper ensures that future studies can fairly compare new certified training methods against a well-tuned set of baselines, preventing misleading claims due to unfair comparisons. This aligns with broader machine learning efforts in reproducibility, benchmarking, and robustness evaluation.", "essential_references_not_discussed": "I don't see any missing references.", "other_strengths_and_weaknesses": "Strengths: \n- This paper proposes a comprehensive benchmark for certified training. This benchmark standardizes the evaluation of deterministic certified training methods, enabling fair comparisons. \n- Based on the proposed CTBench, this paper achieves state-of-the-art certified training performance by carefully adapting previous methods.\n- This paper also reveals new insights from the evaluation results of existing methods.\n- This paper is overall well organized and easy to follow.\n\nWeaknesses: please see my concerns in the Claims And Evidence and Experimental Designs Or Analyses sections. I list them below:\n- While the findings are well-documented, one notable concern is that all results are derived from a single CNN7 architecture, limiting the generalizability of the conclusions. Including more diverse network architectures would strengthen the validity of the results.\n- CTBench uses the number of unstable neurons to represent the smoothness of the loss surface. This can pose difficulty in generalizing to non-ReLU neural networks such as Swish [1] and GELU [2]. A more explicit way is to focus on the change of the loss value within a neighborhood of an input sample.\n- CTBench achieves a state-of-the-art certified training performance. To achieve this desirable performance, several strategies are taken, such as batch norm, hyperparameter tuning, and L1 regularization. It would be helpful to add an ablation study on these strategies. \n- Complexity is an important metric for certified training. However, the running time is only provided in the appendix (Table 8). It would be helpful to add more analyses and discussions on the complexity of different certifying methods.\n- The experiment of shared mistakes does not provide a general insight. The result is specific to the data. It is unclear to me why curriculum learning can improve certified training based on the results.\n\n[1] Ramachandran, Prajit, Barret Zoph, and Quoc V. Le. \"Searching for activation functions.\" arXiv preprint arXiv:1710.05941 (2017).\n\n[2] Hendrycks, Dan, and Kevin Gimpel. \"Gaussian error linear units (gelus).\" arXiv preprint arXiv:1606.08415 (2016).", "other_comments_or_suggestions": "Please see Strengths And Weaknesses", "questions_for_authors": "Please see Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8XOJYd9DAB", "reviewer": "Reviewer_VJvs", "summary": "While a number of algorithms for verifying the robustness of neural networks have been developed, it has also been shown that models trained using standard training approaches are often not robust and difficult to certify. Certified Training aims at developing methods which encourage verifiability and robustness during training while maintaining acceptable standard accuracy. This work introduces CTBench, a benchmark for comparing different certified training algorithms which were developed in recent years. The authors further include two adversarial training algorithms for completeness to enable a comparison of the different methods and their standard, empirical and certified accuracies. The authors further identify some issues in existing implementations of algorithm which they fix in their code base, and further unify the hyperparameter tuning methodology for different algorithms. The algorithms are evaluated on different standard datasets from the literature where the new implementation is found to lead to improved results in multiple cases. The analysis is complemented by a number of experiments on the effects that certified training has on the trained models.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We $\\newcommand{\\Rj}{\\textcolor{purple}{VJvs}}$are happy to hear that Reviewer $\\Rj$ finds our work interesting and well motivated, our library and benchmark useful and comprehensive, and our experimental results insightful. Due to the word limit, we address major questions raised by Reviewer $\\Rj$ below and are happy to discuss more in the follow-up. We include new results as Table S1 etc., in the [anonymized link](https://mega.nz/file/eZp1CYBC#hwOYJzm4U47TDzuCmQFTJhCust2pFz-8Wzzy-CmXd6Q).\n\n\n**Q1: Could the authors comment on why the improvement of certified accuracy sometimes accompanies a decrease in natural accuracy?**\n\nThe robustness-accuracy trade-off is well-known, where higher certified accuracy often comes at the cost of natural accuracy. Most methods, including SABR and MTL-IBP, have hyperparameters (e.g., $\\lambda$, $\\alpha$) that directly regulate this trade-off. Our goal, like in prior work, is to maximize certified accuracy, with natural accuracy improvements seen as a bonus. For completeness, we further provide robustness-accuracy curves, as shown in Figures S1–S3. \n\n**Q2: Could the authors provide an ablation study to evaluate the individual impact of CTBench strategies?**\n\nPlease refer to our reply to Q4 of Reviewer $\\textcolor{blue}{mj6P}$.\n\n**Q3: Can differences in implementation change the test-time certification?**\n\nNo. Test-time certifications are conducted using third-party tools like MNBaB and OVAL. Our changes only affect the training process, while the final trained network can be exported and verified independently, ensuring direct comparability with the literature.\n\n**Q4: Could the authors discuss the connection between this work and probabilistic certification methods such as Randomised Smoothing?**\n\nPlease refer to our reply to Q1 of Reviewer $\\textcolor{green}{vyCo}$.\n\n**Q5: Could the authors discuss the possibility of extending this work to training and certification for other norms?**\n\nOur work focuses on deterministic certified training using bound propagation for the $L_\\infty$ norm, as it remains the most reliable and widely adopted approach for robustness guarantees. While [1] explores various norms for certification, it also limits deterministic certified training to $L_\\infty$, reflecting the current state of the field, with practical deterministic methods focused on $L_\\infty$.\n\nCertification under other norms, such as $L_2$, faces scalability challenges. For example, [2] evaluates $L_2$ certification on small models with only 192 hidden nodes, while our CNN7 network has over 10M parameters, making their method impractical. Similarly, [4] uses expensive SDP methods, limiting their approach to synthetic toy datasets (Spheres) and does not naturally extend to $L_\\infty$. Furthermore, while [4] addresses $L_2$-norm robustness, their methods do not naturally extend to $L_\\infty$.\n\nWe acknowledge that exploring deterministic certified training for other norms is a valuable future direction. However, due to scalability limitations and the lack of effective methods for other norms (even on MNIST), our focus remains on $L_\\infty$. If Reviewer $\\Rj$ knows of scalable methods for other norms, we would be happy to include them in our study. In addition, we will revise our statement to say that “we only focus on $L_\\infty$ robustness because there exists no *scalable* deterministic certified training algorithm regarding other norms”.\n\n**Q6: Could the authors clarify the difference between the experiment with loss fragmentation in Section 5.1 and the findings of Shi et al. [3]?**\n\nWe clarify that Shi et al. [3] analyze only IBP-based instability, which is an over-approximation of the real instability of neurons. In contrast, our analysis applies an estimate of the true number of unstable neurons. To illustrate this difference, we provide a comparison between the two variants in Table S5. We observe that the gap between our lower bound estimate and IBP is larger for SOTA methods, which also reflects in the certification gap between IBP and MN-BAB for these models (Table S4).\n\n**Q7: Do the authors have an intuition as to why certified training performs so poorly (and even worse than adversarial training, which is much less expensive) for some OOD corruptions?**\n\nOn corrupted datasets (MNIST-C, CIFAR-10-C), adversarial and certified training improve robustness against localized perturbations like blur, noise, and pixelation but struggle with global shifts like brightness and contrast changes. This aligns with the intuition that these methods enhance robustness mainly in the immediate neighborhood of the original inputs, whereas global changes fall outside this region. Moreover, the stronger regularization induced by certified training when compared to adversarial training often exceeds what's needed for untargeted corruptions. Addressing this may require diverse augmentations, though it could reduce certified adversarial accuracy.", "claims_and_evidence": "- The experimental results only partially support the claims made by the authors. While the certified accuracy is improved by the authors' implementation in most cases, the improvements seem marginal in a number of cases.\nThe increase in robust accuracy is often accompanied by a decrease in standard accuracy which, in practical applications, would not be desirable (see e.g. the results for CIFAR10, $\\epsilon=\\frac{2}{255}$ for this). This issue is not sufficiently discussed by the authors and the discussion mostly focuses on the certified accuracies. Therefore, the experimental results don't seem to support statements such as \"CTBench achieves consistent improvements in both certified and natural accuracies\"\n- The CTBench implementation of the algorithms shows improved performance in some cases. However, the authors change multiple aspects of the implementation (e.g. regarding BatchNorm statistics) and separately tune the methods' performance according to a specific scheme. There seem to be a number of moving parts in this work and it is not always clear to me which of the performance improvements are induced by which of the changes made by the authors. They discuss in the appendix that separating the changes is not always feasible, but in the current state of the paper this makes it hard to evaluate which changes make sense/have a positive effect. I assume that fixing mistakes in the implementation might also have effects on the way networks are evaluated at test time which makes it difficult to compare the results presented by the authors to previous results.", "methods_and_evaluation_criteria": "The authors include an evaluation of a number of state-of-the-art certified training algorithms in their evaluation. However, there are multiple aspects that are not considered by the authors, including e.g. probabilistic methods such as randomised smoothing or robustness to other norms such as $\\ell_2$ or $\\ell_1$. All of these are considered by the existing benchmark [1] but not discussed by the authors. Works such as [2] show that extending bound propagation to different norms is doable so the authors should extend their evaluation to also cover these other aspects when proposing a new benchmark.\n\n[1] Li, L., Xie, T., & Li, B. (2023, May). Sok: Certified robustness for deep neural networks. In 2023 IEEE symposium on security and privacy (SP) (pp. 1289-1310). IEEE.\n\n[2] Wang, Z., & Jha, S. (2023). Efficient symbolic reasoning for neural-network verification. arXiv preprint arXiv:2303.13588.", "theoretical_claims": "The paper has no theoretical claims which would require checking.", "experimental_designs_or_analyses": "The experimental setup and the hyperparameter tuning approach presented by the authors makes sense and I couldn't find any issues related to the experimental design.", "relation_to_broader_scientific_literature": "The paper basically implements a number of existing robust training algorithms in a common framework and fixes issues in the implementation of some of the algorithms. The novelty of the work is therefore quite limited. Section 5 analyses some other aspects of models trained with certified training algorithms, but some of the insights analysed are less surprising to readers familiar with the field. For example, the loss fragmentation discussed in section 5.1 seems closely related to the fact that certified training forces a number of neurons in the network to be either stably active or stably inactive as previously found by [3]. I also found the results on OOD generalisation to be somewhat difficult to parse since there seems to be no clear tendency as to when a specific certified training is actually helpful, which would make it quite difficult to decide which certified training algorithm to use in practice if improving OOD generalisation is an aim.\n\n[3] Shi, Z., Wang, Y., Zhang, H., Yi, J., & Hsieh, C. J. (2021). Fast certified robust training with short warmup. Advances in Neural Information Processing Systems, 34, 18335-18349.", "essential_references_not_discussed": "The authors state that \"we only focus on $\\ell_\\infty$ robustness because there exists no deterministic certified training algorithm regarding other norms\". Although the approach presented by [4] will struggle to scale to large network architectures and perturbations, it does explicitly address training for $\\ell_2$ robustness, so this statement is incorrect. Since it should not be hard to extend bound propagation to other norms and since a number of works in verification do address robustness in these other norms, it would strengthen the paper if these were also considered.\n\n[4] Soletskyi, R., & Dalrymple, D. (2024). Training Safe Neural Networks with Global SDP Bounds. arXiv preprint arXiv:2409.09687.", "other_strengths_and_weaknesses": "- The paper is quite polished and well-written, the explanations are clear\n- Having a unified implementation of different algorithms will make it easier to benchmark new methods against existing ones", "other_comments_or_suggestions": "Small typo: Line 439 \"training algorithms that explicitly has little loss fragmentation\" --> training algorithms that explicitly **have** little loss fragmentation", "questions_for_authors": "- Do the authors have an intuition as to why certified training performs so poorly (and even worse than adversarial training, which is much less expensive) for some OOD corruptions?\n\n## Post-rebuttal edit\nSome points that should be addressed:\n- The novelty of the work is severely limited, the paper provides an experimental evaluation which is similar to the evaluation section in any of the certified training papers that were published in recent years.\n- Having a unified implementation seems useful, but the authors overstate the contributions that are made (e.g. in phrases such as \"CTBench achieves consistent improvements in both certified and natural accuracies\"), such statements should be toned down.\n- Results on e.g. other networks or norms, or e.g. techniques such as randomised smoothing should be added to the paper to strengthen it", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces CTBench, a unified library and benchmark for deterministic certified training of neural networks. CTBench re-implements existing certified training algorithms under a shared codebase with systematically tuned hyperparameters, enabling fair and consistent evaluation. The authors observe that, under this standardized setting, existing algorithms perform better than originally reported. Moreover, the improvements claimed by recent methods diminish when compared against baselines with carefully tuned training schedules. The paper also offers insights into certified training based on these findings.

All three reviewers appreciate the well-designed experiment pipeline, the contribution of having a unified implementation to compare different algorithms, and the well-organized experiment observations and the well-written paper. However, reviewers also raised concerns about the missing evaluations of randomized smoothing methods (reviewers vyCo, VJvs), limited generalizability due to the use of a single architecture (reviewer mj6P), missing general insights across different datasets (reviewers vyCo, mj6P), limited novelty beyond evaluation of existing algorithms (reviewer VJvs), and missing ablation study and discussion about training complexity (reviewer mj6P).

During the rebuttal phase, the authors provided additional experiments and analyses to address some of the reviewers’ concerns, including the preliminary study on randomized smoothing, the results of another architecture, the impacts of hyperparameter tuning, the training complexity, and the analyses across datasets. The reviewers acknowledged that these results addressed their original concerns. However, some concerns still remain, such as the limited novelty and overstated contributions. Considering all reviews and given the contribution of a unified evaluation pipeline, the AC recommends weak acceptance for the paper.
Decision: Accept (poster)