Accelerating Certified Robustness Training via Knowledge Transfer
Authors: Pratik Vaishnavi
Kevin Eykholt
Amir Rahmati
Venue: NeurIPS 2022 Accept (2022)
Keywords: Adversarial machine learning
certified robustness
randomized smoothing

Abstract: Training deep neural network classifiers that are certifiably robust against adversarial attacks is critical to ensuring the security and reliability of AI-controlled systems. Although numerous state-of-the-art certified training methods have been developed, they are computationally expensive and scale poorly with respect to both dataset and network complexity. Widespread usage of certified training is further hindered by the fact that periodic retraining is necessary to incorporate new data and network improvements. In this paper, we propose Certified Robustness Transfer (CRT), a general-purpose framework for reducing the computational overhead of any certifiably robust training method through knowledge transfer. Given a robust teacher, our framework uses a novel training loss to transfer the teacher’s robustness to the student. We provide theoretical and empirical validation of CRT. Our experiments on CIFAR-10 show that CRT speeds up certified robustness training by 8× on average across three different architecture generations while achieving comparable robustness to state-of-the-art methods. We also show that CRT can scale to large-scale datasets like ImageNet.

Accelerating Certified Robustness Training via Knowledge Transfer
Pratik Vaishnavi
Kevin Eykholt IBM Research kheykhol Srrrkholt@p t@i bm. com
Stony Brook University
praishnavi@cs. wtw.oeroostttoboooo.eoooooo
Amir Rahmati
Stony Brook University
amirêcs. www.sstoooooooooooooooooooooo edu
Abstract
Training deep neural network classifiers that are certifiably robust against adver- sarial attacks is critical to ensuring the security and reliability of Al-controlled systems. Although numerous state-of-the-a arte-artooo certified training methods have been developed, they are computationally expensive and scale poorly with respect to both dataset and network complexity. Widespread usage of certified training is fur- ther hindered by the fact that periodic retraining is necessary to incorporate new data and network improvements. In this paper, we propose Certified Robustness Transfer (CRT), a general-purpose framework for reducing the computational over- head of any certifiably robust training method through knowledge transfer. Given a robust teacher, our framework uses novel training loss to transfer the teacher's robustness the student. We provide theoretical and empirical validation of CRT. Our experiments on CIFAR-10 show that CRT speeds up certified robustness train- ing by 8x on average across three different architecture generations while achiev- ing comparable robustness to state-of-the-ar methods. We also show that CRT can scale large-scale datasets like ImageNet.
Introduction
Deep Neural Networks (DNNs) are susceptible to adversarial evasion attacks 31 9 that add a small amount of carefully crafted imperceptible noise to input to reliably trigger misclassification. As defense, numerous training methods have been proposed [25] 40]355 to grant empirical robustness to DNN. But in the absence of any provable guarantees for this robustness, these defenses were frequently broken I 332. These failures have motivated the development of training methods classifier, hence safeguarding it against all attacks (known or unknown) within a pre-determined threat model. Such methods are broadly categorized either deterministic or probabilistic (23]. Deterministic robustness training methods [12] 26 33
that grant certifiable/provable robustness
34 28
30) rely on computing provable bounds on the output neurons of a classifier for given perturbation budget in the input space. However, the deterministic robustness guarantees provided by these methods come at high computational cost. Probabilistic robustness training methods address this limitation by providing highly probable (e.g.. with 0.99 probability) robustness guarantees at greatly reduced computational cost. Within this category, randomized smoothing- based methods 19
[5) are considered the state-of-the-a for certifiable robustness in the Space. Even so, these training methods remain order of magnitude slower than standard training. In commercial applications where constant model re-deployment occurs to provide improvements (see Figure [T. re-training using computationally expensive methods is burdensome.
3 29
22
20
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

88
90
A
Number of Params (M)
Top-1 Accuracy (%)
84
-75
60
76
45
ResNet101 (2016)
ResNeXt101 (2017)
DLA-X (2018)
RegNetX-16GB (2020)
VIT-B/16 (2021)
Figure 1: Evolution of DNN architectures on the ImageNet dataset. We plot the performance (top-1 accuracy) and the number of parameters of few popular architectures (year of release is noted in brackets). Newer generations attempt to improve performance and/or reduce network parameters.
In this work, reduce the training overhead of randomized smoothing -baee robustness training methods with minimal impact on the robustness achieved. We propose Certified Robustness Transfer (CRT), knowledge transfer framework that significantly speeds up the process of training &2 certifiably robust image classifiers. Given a pre-trained classifier that is certifiably robust (i.e., teacher), CRT trains new classifier (i.e., student) that has comparable levels of robustness in fraction of the time required by state-of-the-ard methods. CRT brings down the cost of training certifiably robust image classifiers be comparable standard training while preserving state- of-the-art robustness. On CIFAR-10, CRT speeds up training by an average of 8x across three different architecture generations compared state-of-the-art robustness training method (T5). Furthermore, we show that state-of-the robustness training is only necessary to train the initial classifier. Afterward, CRT can be continuously reused transfer robustness in order to expedite future model re- re-deployments and greatly reduce costs associated with computational resources. Our contributions can summarized follows:
We present Certified Robustness Transfer (CRT), the first framework, to our knowledge, that student classifier.
can transfer the robustness of certifiably robust teacher classifier CRT greatly reduces the time required to train certifiably robust image classifiers relative to existing state-of-the-art methods while achieving comparable or better robustness.
We provide theoretical understanding of CRT, showing how our approach of matching outputs enables robustness transfer between the student and teacher irrespective of the certified robustness training method used to train the teacher.
On CIFAR-10, we show that CRT trains certifiably robust classifiers on average 8x faster than a state-of-the-art method while having comparable better Average Certified Radius (by 8% in the best case). Furthermore, CRT reduces the cumulative computational cost of training three classifiers by 87 87.84%.
We also show that CRT can be reused in recursive manner, thus supporting continuous deployment scenario (e.g., commercial applications). Finally, we show that CRT remains effective large-scale dataset, ImageNet.
2 Background
In this section, we briefly introduce certified robustness and discuss notable existing methods for training certifiably robust image classifiers using randomized smoothing.
2.1 Preliminaries
Problem Setup. Consider neural network classifier f parameterized by 0 (denoted fo) trained to map given input x E Rd to a set of discrete labels y using a set of i.i.d. samples S ( v1), (II,11) v1). (In, yn)) drawn from data distribution D. The output of the classifier can be written as fo(x) = arg max.ey =g(x). Here (xx is the softmax output of the classifier and =z)z) denotes the probability that image z belongs to class

---

Certified Robustness via Randomized Smoothing. The robustness of the classifier fo for given input pair (z. y) is defined using the radius of the largest &g ball centered at r within which fo has constant output This radius is referred to as robust radius and it can mathematically be expressed as:
= ={ fo(a fo(a) {
inf || x=112 when fa(
R(fo:z,y)
when fo(x) + y
Within this fz-neighborhood of fe is considered to be certifiably robust. Therefore. to improve the robustness of a classifier, one needs to maximize this robust radius corresponding to any point sampled from the given data distribution. Directly maximizing the robust radius of DNN classifier an NP-hard problem [17] Therefore, several prior works attempt to derive a lower bound for the robust radius 21
19 3 This lower bound, often termed as the certified radius, satisfies the following condition: O S CR(fa x, y) S. R[fo; y), for any fe, (I, y). In this paper, we utilize the
certified robustness framework derived by Cohen al. [3] using randomized smoothing. Given classifier So. they first define the smooth classifier Go as:
Definition 2.1. For a given (base) classifier fe and o > O, the smooth classifier go corresponding to fe is defined as follows:
(o arg max P (f) n)) c) cEy
Simply put, Go returns the class c, which has the highest probability mass under the Gaussian distribution (xx,, d² Using Theorem 2.2, they proved that if the smooth classifier correctly classifies given input v, is certifiably robust at æ. They also provided an analytical form of the l certified radius at 2C
Theorem 2.2. Let fe Rd + Y be a classifier and go be its smoothed version (as defined in Defimition 2.1p For a given input x E IR and corresponding ground truth y E y, if go correctly classifies as y, i.e.,
Pa(Jo( + n) = yy) max Pny( + =
(3)
5'
then ge is provably robust at within the certified radius R given by:
C'((gs x. y) 2 | 1 (Pr(fo(x 11) = y)) 4 (max Py (ff +n) y')i
(4)
of Fツ
where & is the c.d.f. of the standard Gaussian distribution.
This certified radius is a tight lower bound the robust radius defined in Equation I impossible to certify go at for radius larger than CR.
i.e., it is
2.2 Training Methods for Maximizing Certified Radius
In addition to the theoretical framework discussed above, Cohen et al. 3 also propose a simple yet effective method for training the base classifier in way that maximizes the la certified radius of the smooth classifier, as expressed in Equation 4 We include an evaluation of their method in Appendix ?? Following their work, several other works build upon the randomized smoothing frame- work and propose training methods that better maximize the & certified radius of the smooth classi- fier. Salman al. [29] proposed combining adversarial training
(25 with randomized smoothing
(called SmoothAdv). They adapted the vanilla PGD attack to target the smooth classifier ge instead of the base classifier fe. Zhai el al. [39] proposed a new robustness loss, hinge loss that enforces maxi- mization of the soft approximation of the certified radius. Their method (called MACER) is faster than SmoothAdv as it does not use adversarial training. More recently, Jeong et al. [13] proposed training with a convex combination of samples along the direction of adversarial perturbation for each input to regularize over-confident predictions. Their method (called SmoothMix) is the current state-of-tt ofie-of-tteerrtt in the domain of & certified robust image classifiers, Finally, we note the Consistency regularization method proposed by Jeong et al. [16), which adds a regularization loss to existing methods that helps better maximize the certified radius.
3

---

Table 1: Training on CIFAR-10 using ResNetl 10 classifier on a single Nvidia V100 GPU. State-of- the-art robustness training methods significantly slow down training compared standard training.
METHOD
TRAINING SLOWDOWN FACTOR
AMOOTHADV
46.20x
MACER
20.86x
SMOOTHMIX
4.97x
3 Maximizing Certified Radius via Knowledge Transfer
Although prior works have proposed methods for increasing the certified radius of the smooth classifier. their training overhead is significant, making them much slower than standard training. As we show in Table I training a certifiably robust ResNetl 10 classifier to convergence using SmoothAdv, MACER, and SmoothMix 46.20x, 20.86 and 4.97 slower, respectively, compared to training non-robust classifier with standard training.
Given constant innovations in architecture design (Figure 1 and the influx of new data, which may result in various tweaks to deployed networks that elicit retraining, the large overhead of state-of-the- art robustness training methods makes preserving certified robustness across model re-deployment difficult. Therefore, we propose Certified Robustness Transfer (CRT), a training method that improves the usability of certified robustness training methods by dramatically reducing their training overhead while preserving the certified robustness. Given the base classifier of pre-trained certifiably robust smooth classifier, we leverage the knowledge transfer framework guide the training of new base classifier (and its associated robust smooth classifier) In this section, we describe our method and provide theoretical justification for effectiveness.
3.1 Transferring Certified Robustness
From Equation 4 it follows that training the base classifier to maximize (Jos( + + y) for any given input z will result in the maximization of the certified radius associated with the smooth classifier, provided Equation is satisfied. Thus, for the base classifier fz(z), our goal is to maximize the following quantity over the training set:
n Σ F., + 11) = MAl R
n
E, (æ; +1)
(5)
In the above equation, like prior works 3
we leverage the fact that the softmax output of a classifier can be treated as a continuous and differentiable approximation of its arg max output. Methods like SmoothAdv (29), MACER 39 and SmoothMix [T5] that target &g certifiable robustness propose training objectives that maximize this term.
29
39
Now, suppose have pre- pre-trained base classifier fa. It follows that E, + 7)] 2 0. Through straightforward algebraic manipulations (see Appendix ??), we derive the following lower bound:
Σ [2 (x; + m)) 2
Fn (xr: + 1)) = (x; + +]]
(6)
That is to say that, for given input x4, if we minimize the difference between the softmax outputs of the teacher and the student (S& and fa) corresponding the correct label y,, we will maximize Equation for the student. However, to ensure that the student has a non-trivial certified radius, we must also ensure that Equation is satisfied. If we assume that Equation 3 holds true for the teacher (i.e., the base classifier of a certifially robust smooth classifier), this condition can also be achieved for the student by matching the overall softmax output of the student to that of the teacher.
3.2 Certified Robustness Transfer (CRT)
Based on the previous discussion, we now describe our method for training a certifiably robust classifier through knowledge transfer. First, we obtain pre-trained base classifier fa. which has been trained using a randomized smoothing based robustness training method as this maximizes
If no pre- trained classifier is available, we first train an architecture of lower complexity (ie., fast to train) compared to the target architecture 5.1I

---

Algorithm 1 Certified Robustness Transfer (CRT)
1: Input: Training data distribution D, certifiably robust teacher base classifier fe, noise level total training iterations T, learning rate a
2: Output: Certifiably robust student base classifier fo
3: < random initialization
4: i
5: while < 7
do do
6: From D, sample batch of inputs (2, J.
7:
From (100, I)),, generate a batch of Gaussian noise samples {?h, 12, "1,""" ;77
8:
4
- ( + mss) 20(2z + +,))ll2
9:
0-0-6 Vol;
10:
i 1-i+1
11:
end while
F [= + 7)]l Next, we use So as a teacher to train new student base classifier So. The student trained to match the output of the teacher. In doing so, can maximize the certified radius of the associated smooth classifier 9o (Equation 6). as well as ensure that Equation 3 is satisfied. We describe our implementation in Algorithm [] Given batch of inputs, we first perturb them with additive Gaussian noise. Next, we compute the &g distance between the student and the teacher's ' outputs for these Gaussian perturbed inputs. This distance serves as our loss function, and we update the parameters of the student to minimize this loss. At test time, the classifier So is converted to its smooth version go following Definition 2.1
3.3 Prior Works on Robustness Transfer
Several prior works have examined transferring adversarial robustness between classifiers, but these works have been limited to transferring empirical rather than certified robustness 22 8 14
42 43
Of note is the work by Goldblum et al. [8] in which they combine adversarial training 25 with knowledge distillation [3]. They show that distilling knowledge from a large network to a small
network improves its empirical robustness compared to training the small network on its own, but their method makes no effort to improve the computational cost of adversarial training.
4 Evaluation
Our goal is to improve the usability of randomized smoothing based robustness training methods. In this section, demonstrate how CRT enables the reuse of an existing certifiably robust classifier to train new certifiably robust classifiers at significantly reduced training cost compared to prior methods. In our first experiment, MixothMix [13), and use CRT to transfer its robustness to train several newer generation classifiers. In a second experiment, previous generation classifier that was also trained using CRT. In each experiment, we compare the certified robustness of classifier trained using CRT against classifier trained using SmoothMix (Section[..) We find that classifiers trained using CRT are similarly robust as when trained using SixothMix but only require fraction of training time (Section 4.2 Our main results are generated using the CIFAR-10 10 dataset [18). but we also demonstrate the effectiveness of CRT on ImageNet [5] Both these datasets are open-source and free for non-commereial use.
train ResNetl 10 classifier with state-of-the-art method, i.e.
recursively use CRT to train a newer generation classifier using the
(Section 5.3
Architectures. We use several popular DNN architectures that were proposed to either improve upon the visual recognition performance of the previous generation architectures preserve performance while requiring significantly fewer parameters (or both). For the CIFAR-10 experiments, we use ResNetl 10 [T], ResNeX129-2x644d (36). DLA 38) and RegNetX_200MFF 200MF 27
Training details. All SmoothMix classifiers were trained using the code made available by the author and the hyperparameters reported by them [15). All CRT classifiers were trained using Stochastic Gradient Descent till convergence (200 epochs), with a batch size of 128. Further hyperparameter details are available in Appendix ??. Unless specified, we report results for noise
We use the CIFAR-10 version of these architectures. code [MIT License]: knangliu/pyyorrhh- chacf SmmoohhMix code [MIT License]: attpsss//gtttuab.cmm...foo/////p
httpps://github.com
5

---

level σ = 0.25 in the main paper. Additional results for higher noise levels = 0.5 and 1.0 are reported in Appendix ??.
Evaluation Metrics. We report our results using two metrics. First, as done in prior work, we measure the certified robustness of classifier based on (1) the certified test accuracy at &z radius r0 which is defined as the fraction of test set inputs that the smooth classifier classifies correctly within an & ball of radius r centered each input, and (2) average certified radius (ACR), which is the average of the certified radius across all inputs in the test set:
AC
CR(go 1=1
Titest
On CIFAR-10, 10, we compute these metrics using the entire test set. Second, we measure training time of classifier based on the per-epoch time and total training time. The total training time is computed once the model's loss has converged. All classifiers were trained on the same machine with single Nvidia Titan GPU.
4.1 Certified Robustness Comparison
Standard CRT Training. Given ResNetl 10 classifier trained using SmoothMix, transfer its robustness to several newer generation classifiers. We compare the certified robustness of these classifiers with their SmoothMix trained versions. The results summarized in Table We observe that using CRT does not reduce the certified robustness of the trained classifier compared to training with SmoothMix. In fact, interestingly, CRT trained classifiers exhibit higher certified robustness compared to their SmoothMix baseline. Not only do CRT trained classifiers have higher ACR (improvement of 8.1 1% in the best case), they also exhibit higher certified accuracy different & radii. Furthermore, CRT remains effective even as the generation gap between the student and the teacher increases. This implies that the same teacher can potentially be reused indefinitely, amortizing the teacher's 's training cost to a constant. These results empirically validate our theoretical justification of CRT. Finally, we note that in Table 2 the accuracy on clean inputs and ACR of CRT trained classifiers follow the same trend as in Figure thus motivating the need for periodic model re-deployment to incorporate architectural improvements.
Table 2: The certified robustness of classifiers with different architectures trained on CIFAR-10 10 using SmoothMix [15 and CRT. We use CRT to transfer the robustness of a ResNetl 10 trained using SmxothMix. We report certified test accuracy at different values of &2 radius and the Average Certified Radius (ACR). The architectures are sorted chronologically based on published date. The noise level a is set to 0.25.
ARCHITECTURE
0.00
0.25
0.50
0.75
ACR
SMOOTHMIX [TS]
RES 10
11
76.89
68.25
57.42
46.26
0.550
RESNEXT29
36
75.98
65.40
53.78
41.03
0.516
DLA
38)
77.72
68.53
57.69
45.56
0.551
REGNETX_ 200MF
27
76.48
66.79
56.36
44.47
0.538
CRT (RESNET 10 TEACHER)
RESNEXT29
(3
77.57
69.00
58.31
47.16
0.558
DLA
3
77.31
68.91
58.26
46.34
0.554
REGNETX 220ME 200MF
27
77.89
69.57
59.36
47.28
0.564
Recursive CRT Training. We now explore the effectiveness of CRT if it is used recursively, i.e., the newest generation is trained using a CRT trained classifier from the previous generation as the teacher. We begin with ResNetl 10 trained using SmoothMix. Then, all subsequent classifiers using CRT recursively and report the results in Table The chain length measures the number of times CRT was used. For example, the DLA network, with chain length of 2, the result of using CRT
trained
NNote that the certified accuracy represents the clean accuracy of the smooth classifier.

---

Table 3: The certified robustness of classifiers with different architectures trained on CIFAR-10 using CRT recursively. We report certified test aceuracy at different values of la radius and the Average Certified Radius (ACR). Here, the previous generation classifier is used to train the current generation one. Chain length represents the number times CRT was used in training. The noise level set to 0.25. CRT remains effective despite recursive use.
ARCHITECTURE
CHAIN LENGTH
0.00
0.25
0.50
0.75
ACR
RESNEXT22-2X644
36
1
77.57
69.00
58.31
47.16
0.558
DLA
38
2
78.46
70.05
60.01
48.30
0.570
REGNETX_ CCGNETT_200MF 200MF
27
3
78.16
69.00
58.69
47.00
0.559
twice: once to transfer the SmoothMix trained ResNetl 10 network's performance to the ResNeXt29- 2x64d network and once to transfer the CRT trained ResNeX(29-22 DLA network. We observe that the certified robustness of the resulting classifiers remains high even with recursive use of CRT. The empirical results are to be expected given our theoretical understanding of CRT: In order to train a robust student, we only require that the teacher is already robust (i.e., satisfies the condition of Theorem 2.2 irrespective of the training method used to achieve robusiness. Thus, we expect CRT to remain effective even longer chain lengths. In Section 5.2 we will highlight the relationship between the teacher's 's training method and the robustness of a CRT trained student.
network's performance to the
4.2 Training Time Comparison
Having established that CRT effectively transfers certified robustness between classifiers, we now evaluate its training overhead. For comparison, we also evaluate the training overhead of SmoothMix. In Table we report the per-epoch time and total time of training different architectures with each method. For brevity, we only compare the training time for standard CRT We observe that the per- epoch time CRT is significantly lower than SmoothMix. Similarly. when trained until convergence, the total training time of CRT is significantly lower. Across the three architectures that we run our experiments on, CRT achieves an average epoch time speedup of 10.75x. Comparing overall training times, CRT speeds up training by, on average, 8.06x. If we consider the real-world scenario where the model has to be periodically redeployed to incorporate architectural improvements, the cumulative training time using SmoothMix is 96.21 1 hours as each new architecture is trained from scratch. With CRT, the cumulative time reduced to 11.70 hours representing a 87.84% savings in costs associated with computational resources.
Teacher's availability. So far. we assumed the availability of certifiably robust teacher (ResNet 10). We argue that this is a reasonable assumption as the amortized cost associated with the one-timg time training of robust teacher is negligible across many generations of the model. Regardless, in Section 1. we examine a scenario where the teacher is unavailable. Under this scenario, we demonstrate how CRT can be used to speedup the training of ResNetl for use as teacher.
Table 4: Training time statistics for SmoothMix and CRT. We report the mean and 95% confidence interval computed over all training epochs. CRT on average 8x faster than SmoothMix across all three architectures.
SMOOTHMIX 15
CRT (RESNET110 TEACHER)
ARCHITECTURE
EPOCH TIME (S)
TOTAL TIME (H)
EPOCH TIME (S)
TOTaL TIME (I)
RESNETL10
m
455.55 = 1.17
18.98
RESNEXT29-
3d
1085.09 09 + 0.50
45.21
86.41 + 0.11
4.80
DLA
38
854,41 = 0.09
35.60
62.24 + 0.40
3.46
REGNETX 200MF
369.42 0.51
15.39
61.92 + 0.30
3.44
Recursive CRT differs in time an insignificant factor due forward pass through different teacher.
7

---

Discussion
In this section, we address the standout concerns about CRT. The section layout is as follows: in Section[5.]] we discuss the scenario in which a certifiably robust teacher is not readily available and demonstrate how CRT can still speed up robustness training; in Section 5.2 we examine how the method used to train the teacher affects the robustness of the student; in Section 5.3 we study the scalability of CRT using the ImageNet dataset; in Section 5.4, we compare CRT with a closely related prior work fast certified robustness training. i.e. Consistency regularization [[6]; in Section [5.5]
we discuss the limitations of CRT; in Section 5.6. we address the broader impact of CRT.
5.1 Teacher Not Available
We've designed CRT under the assumption that certifiably robust teacher is already available. However, even if a certifiably robust teacher is not available, CRT can still speed training. Given certifiable robust training method and large network architecture, we can reduce the training overhead by robustly training a comparatively smaller network first. Then, we can use CRT to larger network. In Table [ we present results
transfer the robustness of the small network for such process. First, we trained a ResNet20 network using SmoothMix, then we used CRT to train a ResNetl 10 network. We compare the robustness of ResNetl 10 trained using this process with one trained using SmxothMix. As we can see, the CRT ResNetl 10 network has comparable robustness with the MmoothMix ResNetl 10 network. However, even when adding the teacher and student training times, CRT still speeds up training by approximately 2x relative to SioothMix.
Table 5: Certified robustness and total time of a ResNetl 10 classifier trained on CIFAR-16 10 using Mix and CRT. For CRT, train ResNet20 teacher first using SmoothMix and report total time as the time taken to train the teacher and the student. The noise level a is set to 0.25. The ResNetl 10 trained using CRT achieves ACR comparable to the SmoothMix ResNetl 10 while achieving ~ 2x speedup in total training time.
METHOD
0.00
0.25
0.50
0.75
ACR
TOTAL TIME (H)
SмOOTHMIX
TS
76.89
68.25
57.42
46.26
0.550
18.98
CRT (RESNET20 TEACHER)
75.68
67.20
56.30
44.83
0.540
10.07
5.2 Teacher Training Method
We train ResNet20 classifier using MACER [3)], SmoothAdv [29]. and SmoothMix 13 For MACER and SmoothAdv training, we use the code made available by the author and the hyperpa- rameters reported by them. Using CRT, we transfer the robustness of each teacher a ResNetl 10 clas- sifier. The results are reported in Table 6 For reference, we also report robustness of a ResNetl 10 net- work trained independently using the chosen robustness training methods. Overall, we observe a slight variation in the robustness of the CRT trained networks depending on the teachers training method. Based on Equation 6 this expected as maximizing the teacher's performance will in turn maximize the performance of the student. Our empirical results align with this reasoning: the MACER teacher was the least robust of the three methods, and its student is similarly the least robust of the students. However, in all cases, the CRT trained network obtained certified robustness comparable to its teacher.
and
5.3 Scalability
Here, we study the effectiveness of CRT on a large-scale dataset, i.e., ImageNet. For this purpose, we train ResNet18 classifiers using three certified robustness training methods (MACER, SmoothAdv, and SmoothMix). Next, we transfer their robustness to a ResNet50 classifier. The results were generated on a 500 sample test set (following prior works 22 39 15)) and are summarized in Table 7 For reference, we also report robustness of a ResNet50 network trained independently using the chosen robustness training methods. In all cases, we observe that students achieve certified robustness comparable to their respective teachers. Therefore, CRT remains effective even on more complex dataset.
'MACER code (No license available]: "moothAdy code [MIT License]: http:://ggthuu...c///caassaamsas a
8

---

Table 6: For CIFAR-10 10 dataset, certified robustness achieved on training the CRT teacher (ResNet20) with different methods. The student classifier is ResNetl 10. For reference, we also report robustness of ResNetl 10 trained independently using chosen methods. The noise level a is set to 0.25. Students attain comparable ACR to their respective teachers.
TEACHER (RT2NET20)
STUDENT (RESNET110)
TRAINING METHOD
ACR
TRAINING METHOD
ACR
SMOOTHADV 29
0.531
CRT
0.519
MACER 39
0.507
CRT
0.528
SMOOTHMIX
15
0.522
CRT
0.540
SMOOTHADV 29
0.547
STUDENT TRAINED DIRECTLY
MACER 39
0.531
SMOOTHMIX
0.550
Table 7: ImageNet results using CRT and three robustness training methods. We report both the ACR of the R teacher and its ResNet50 student. For reference, we also report robustness of ResNet50 trained independently using chosen methods. The noise level a set to 0.5. Students attain comparable ACR to their respective teachers.
TEACHER (RESNET18)
STUDENT (RESNET50)
TRAINING METHOD
ACR
TRAINING METHOD
ACR
SMOOTHADV 29
0.684
CRT
0.684
MACER 30
0.574
CRT
0.576
SMOOTHMIX 15
0.653
CRT
0.661
SMOOTHADV 29
0.820
STUDENT TRAINED DIRECTLY
MACER 30
0.653
SMOOTHMIX
0.799
5.4 Comparison with Consistency Regularization [16
In Section 4 we compared CRT against SmoothMix as it has state-of-the-an ACR. However, another closely related work was recently published by Jeong & Shin [16), which shows state-of-the-ar ACR and potential training time improvements. They proposed a consistency regularization loss that improves the certified robustness of smooth classifiers by enforcing the base classifier's soft outputs to be consistent across multiple noisy copies of given input. Therefore, their additional computational overhead scales linearly with the number of noisy samples used compute the consistency loss. With respect to computational overhead, CRT adds only one forward pass, i.e., the pass through the teacher. When paired with Gaussian data augmentation training, their regularization loss significantly improves the certified robustness of smooth classifier. By applying their regularization loss over only two noisy copies of the input, they can achieve better certified robustness than prior state-of-the- art robustness training methods like MACER 39 and SmoothAdy (29
The key difference between CRT and consistency regularization is in the training overhead when combined with other state-of-the-an certified training methods. Consistency regularization augments classifier training with additional loss term. Therefore, their training overhead is dominated by the training method selected. In their experiments, they focused on Gaussian data augmentation, which adds little training overhead relative standard training. However, if more computationally intensive method was selected (e.g., MACER), they remark their training overhead would dramatically increase. With respect CRT, if teacher is available (i.e., a previous generation model), the overhead of CRT is agnostic to the training method. If it is not available, we demonstrated in Section 5.1 that CRT can still greatly reduce training overhead. For interested readers, we include results for transferring robustness from teacher trained using Consistency regularization in Appendix ??.

---

5.5 Limitations
In this paper, probabilistic certified robustness methods they rely Theorem 2.2 and, thus, are designed to maximize the certified radius (Fquatiio[4) We found that deterministic methods (e.g. CROWN-IBP (41) impose stricter training requirement on the base teacher classifier. For a given input, deterministic training methods require the base classifier to be correct for all inputs within the lg-norm ball, rather than simply be likely correctly classify inputs within the 4n-nrrm ball. This restriction lowers the potential ACR of the smooth teacher classifier, which also lowers the ACR of the student trained using CRT. For example, when using CROWN-IBP 41 to train ResNeXt base classifier, the ACR for the corresponding smooth classifier is only 0.064. When transferring the robustness of this ResNeXt classifier to WideResNNe334 10 student, we get an ACR of 0.065.
Additionally, we note that the classifier architectures we present in the paper are restricted to CNNs. Recently, a new class of transformer- improved performance over CNN classifiers. We briefly studied the effectiveness of CRT when transferring robustness between CNN and transformer architectures using ViT [6] and present the results in Appendix ??, but further exploration is needed. Finally, CRT has only been studied using the Eg norm and image data due the limitations of current certified robustness training methods.
image classifiers 6 24 4 have been proposed that show
5.6 Broader Impacts
As we have shown, our work improves the efficiency of training certifiably robust classifiers, in an effort improve the security of Al-powered systems. Beyond the broad negative societal impacts of machine learning, we are not aware of any impacts specific to our work.
6 Conclusion
In this paper, we proposed the first general-purpose framework to speed up the training of certifiably robust classifiers using knowledge transfer and randomized smoothing. Our proposed method, Certified Robustness Transfer (CRT) enables transferring the certified robustness of a classifier to another classifier at cost comparable standard training. We provided theoretical understanding of CRT and provided empirical evidence of its effectiveness. On CIFAR-IO, we showed that across several generations of classifier architectures, CRT trained classifiers 8x faster than when using a state-of-the-art training method, while achieving comparable or better certified robustness. Furthermore, CRT can reduce the training overhead of certified robustness training methods even when an initial robust classifier is not present. The use of machine learning in security and safety critical environments motivates need for models with certifiably robust performance, but the training overhead of existing certified robustness training methods inhibits their usability. Our work addresses this issue, especially for commercial applications where periodical model re-deployment is inevitable.
Acknowledgement
This work was supported by the Office of Naval Research under grants 100014-22-1-2000 Air Force Research Lab under grant FA9550-22-1-0022 and NVIDIA 2018 GPU Grant. Any opinions, findings, or conclusions expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.
and
References
[1] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses adversarial examples. In International Conference on Machine Learning, 2018.
[2] Alvin Chan, Tay, and Yew-Soon Ong. What it thinks is important important: Robustness transfers through input gradients. In IEFF/CVF Conference on Computer Vision and Puttern Recognition, 2020.
[3] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, 2019.
[4] Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 2021.
[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In IEEF/CVE Conference on Computer Vision and Pattern Recognition, 2009.
10

---

[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[7] Krishnamurthy Dj Dvijotham, Jamie Hayes, Borja Balle, Zico Kolter, Chongli Qin, Andras Gy- orgy, Kai Xiao, Sven Gowal, and Pushmeet Kohli. A framework for robustness certification of smoothed classifiers using f-divergences. In International Conference on Learning Representa- tions, 2019.
[8] Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In AAAI Conference on Artificial Intelligence, 2020.
[9] lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In International Conference on Learning Representcatinns, 2014.
[10] Sven Gowal, Krishnamurthy Dj Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovie, Timothy Mann, and Pushmeet Kohli. Scalable verified training for provably robust image classification. In IEEE/CVF International Conference Computer Vision, pages 4842-4851, 2019.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
[12] Matthias Hein and Maksym Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. In NIPS, 2017.
[13] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network,
2015. 503.005....
[14] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Alcksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, 2019.
[15] Jongheon Jeong, Sejun Park, Minkyu Kim. Heung- Chang Lee, Do-Guk Kim, and Jinwoo Shin. Smoothmix: Training canfidence-calibratte smoothed classifiers for certified robustness. Advances in Neural Information P Processing Systems, 2021.
[16] Jongheon Jeong and Jinwoo Shin. Consistency regularization for certified robustness of smoothed classifiers. Advances in Neural Information Processing Systems, 2020.
[17] Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In International conference on computer aided verification, pages 97-117. Springer, 2017.
[18] A Krizhevsky. Learning multiple layers of features from tiny images. Master's 's thesis, University of Toronto, 2009.
[19] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In IEEE Symposium on Security and Privacy, 2019.
[20] Guang-He Lee, Yang Yuan, Shiyu Chang, and Tommi Jaakkola. Tight certificates of adversarial robustness for randomly smoothed classifiers. Advances in Neural Information Processing Systems, 32:4910-4921, 2019.
[21] Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. Second-order adversarial attack and certifiable robustness, 2019. In URL https://openreview net/forum, 2018.
[22] Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. Certified adversarial robustness with additive noise. In Advances in Neural Information Processing Systems, 2019.
[23] Linyi Li, Xiangyu Qi, Tao Xie, and Bo Li. Sok: Certified robustness for deep neural networks. arXiv preprint arXiv:2009.00: 00009/04433//// 2020.
[24] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In /EEF/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[25] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
11

---

[26] Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable abstract interpretation for provably robust neural networks. In International Conference on Machine Learning, pages 3578-3586. PMLR, 2018.
[27] Iija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. De- signing network design spaces. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[28] Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial examples. In International Conference on Learning Representationss 2018
[29] Hadi Salman, Greg Yang. Jerry Li, Pengchuan Zhang, Huan Zhang. Hya Razenshteyn, and Sébastien Bubeck. Provably robust deep learning via adversarialyy trained smoothed classifiers. In Advances in Neural Information Processing Systems, 2019.
[30] Sahil Singla and Soheil Feizi. Second-order provable defenses against adversarial attacks, In International Conference on Machine Learning, pages 8981-8991, PMLR, 2020.
[31] Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
[32] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in Neural Information Processing Systems, 33, 2020.
[33] Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pages 5286-5295. PMLR, 2018
[34] Eric Wong, Frank R Schmidt, Jan Hendrik Metzen, and Zico Kolter. Scaling provable adversarial defenses. In Advances in Neural Information Processing Systems, pages 8410-8419, 2018.
[35] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 501-509, 2019.
[36] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
[37] Greg Yang. Tony Duan, J Edward Hu, Hadi Salman, Ilya Razenshteyn, and Jerry Li. Randomized smoothing of all shapes and sizes. In International Conference on Machine Learning, pages 10693-10705. PMLR, 2020.
[38] Fisher Yu. Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[39] Runtian Zhai, Chen Dan, Di He, Huan Zhang, Boqing Gong, Pradeep Ravikumar, Cho-Jui Hsieh, and Liwei Wang. Macer: Attack-free and scalable robust training via maximizing certified radius. In International Conference on Learning Representations, 2020.
[40] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning. 2019.
[41] Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsieh. Towards stable and efficient training of verifiably robust neural networks. In International Conference on Learning Representations, 2019.
[42] Jianing Zhu, Jiangchao Yao, Bo Han. Jingfeng Zhang, Tongliang Liu, Gang Niu, Jingren Zhou, Jianliang Xu, and Hongxia Yang. Reliable adversarial distillation with unreliable teachers. arXiv preprint MXXi:2106.0499
2021.
[43] Bojia Zi, Shihao Zhao, Xingjun Ma, and Yu-Gang Jiang. Revisiting adversarial robustness distillation: Robust soft labels make student better. In /EEE/CVF International Conference on Computer Vision, pages 16443-16452, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section
(b) Did you describe the limitations of your work? [Yes] See Section 5.5
(c) Did you discuss any potential negative societal impacts of your work? [No] In this paper, propose a method to improve the usability of certified robustness training methods. Furthermore, beyond the broad negative societal impacts that results for ML, we are unaware of any specific to work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the ethics review guidelines and acknowledge that paper conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 2 for foundational theoretical results from prior work and Section for the CRT specific results.
(b) Did you include complete proofs of all theoretical results? [Yes] The complete theoreti- cal justification for CRT is included in Appendix ??. For proofs for other foundational results such as Theorem 2.2. see the respective paper.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] See Ap- pendix ??.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix ??.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We include 95% confidence interval for our timing re- sults. However, computing error bars for certified accuracy and ACR requires training multiple classifiers and performing the costly process of certification multiple times. Additionally, we observed that most prior works do compute such error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [No] While in Sections 4.2 and [.1 we include the total training time results to support our claims as well as the GPU model. We do include such information for all experiments.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] When specified by the asset creator,
we included the license.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] See Appendix ??.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "fKEElJf5QVc", "reviewer": "Reviewer_t68V", "summary": "In this work, the author study the transfer learning setting for training a robust classifier using randomized smoothing. It relies on a pretrained smooth model, such as MACER and SmoothAdv, as the teacher model, and proposes a simple loss function to make the student model also learn a similar smoothness. With a good teacher model, the student model can be trained very quickly, much faster than training from scratch. Adequate experiments on CIFAR-10 and ImageNet are conducted with ablation studies to show the effectiveness of this approach, where the improvement is mostly on training time. Even in the setting where a teacher model is not available, the authors show that the training a smaller teacher model first can still be beneficial.", "questions": "Although this paper proposes a quite simple algorithm but I feel it is solving a real problem and the proposed approach is effective. Thus, I tend to accept this paper. I have no major questions regarding this paper.\n", "limitations": "Limitations are well discussed and I like the discussions on page 9.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. Thorough empirical results with good speed up on training time, even on large datasets (ImageNet) and large models (ViT-B).\n2. Method is simple, effective, and sound.\n3. The paper is well presented and easy to understand.\n\nWeakness: \n\nThe novelty of this work is limited, because it simply combines transfer learning with robust training using a simple loss function, without further theoretical justifications. However, I feel it is solving a real problem that training a certifiably robust model can be slow, and using a teacher model can indeed speedup the training of large robust model by a large factor without sacrificing too much performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eHEW7Iel4Z", "reviewer": "Reviewer_iNdk", "summary": "This paper proposes certified Robustness Transfer (CRT) for reducing the computational overhead of any certifiably robust training method through knowledge transfer. The paper targets a scenario where different neural network architectures evolve, and one needs to retrain different architectures with time-consuming, certifiably robust training methods.", "questions": "1. In the recursive evaluation of CRT in table 3, the result of chain length 3 became worse than shorter chains. Does it imply that the effectiveness of CRT will decay with respect to the number of recursions? If it does decay, then it will harm the speed up brought by CRT because one needs to retrain after applying CRT two or three times.\n2. Why not directly minimizing $z^{y_i}_\\phi(x_i + \\eta) - z^{y_i}_\\theta(x_i + \\eta)$ but minimizing the distance between them?\n3. Algorithm 1 only minimizes $\\|z^{y_i}_\\phi(x_i + \\eta) - z^{y_i}_\\theta(x_i + \\eta)\\|_2$. How about incorporating other losses? ", "limitations": "Yes, the authors adequately addressed the limitations in section 5.5.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n+ The paper conducts solid experiments to demonstrate the effectiveness and efficiency of the CRT framework.\n\nWeakness:\n- The paper lacks novelty. The approach of using transfer learning to improve (certified) accuracy and training efficiency is not new. The fact that transfer learning can transfer the robustness is already explored by existing works, as mentioned in section 3.3. The paper provides a theatrical analysis of the transferability, but the analysis is trivial. \n- Another concern is that it is unclear whether the robust knowledge has been transferred to the student model apart from metrics like certified accuracy and ACR. In other words, the paper would be improved by theoretically analyzing the decay of the robustness transfer, e.g., after $r$ recursions, specific robust metrics would decay at rate $h(r)$.\n\n\nDetailed comments:\n1. For all tables presenting certified metrics and speeds, please show the baseline training method with data augmentation by Gaussian noise. Comparing this baseline method can help illustrate the necessity of fancy training methods and CRT.\n2. The paper does not explore how far transfer learning can go without considering the cost of training, e.g., is it possible to improve algorithm 1 such that it incorporates the existing loss (capturing the knowledge of the teacher) and another loss that uses fancy certifiably robust training methods or the baseline Gaussian noise data augmentation. \n\n\n===============================\n\nPost one round of rebuttal:\n\nChanged Contribution from 1 to 2\n\nChanged Rating from 3 to 4\n\n===============================\n\nAfter reading other reviewer and authors discussion about experiments \n\nChanged Rating from 4 to 5", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KgmOpN7qyFE", "reviewer": "Reviewer_aqwd", "summary": "I think the paper is well-motivated and the proposed method is straightforward. This topic is interesting. However, the empirical contribution is over-claimed to me by a wrong measurement and in fact they might be not significant and lack enough experiments. I therefore vote for rejection for now but am open to changing depending on the authors’ response. ", "questions": "\nMy questions are embedded in my comments about the weakness. I can summarize them again in this part but please find them in the context above. \n\n1) Why is this work significant as students are no significantly better than teachers and the robustness transfer is only between architectures?\n\n2) Why is the training cost of CRT only the cost of training students, but is used to compare with baselines that actually train teachers ? \n", "limitations": "The paper has a separate paragraph to discuss its limitations. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\nThe writing of this paper is pretty clean and easy to follow. The motivation is very clear and the presentation of the proposed method carries that clearness. \n\n### Weakness\nMy major concern for this paper is that the empirical contribution is over-claimed. However, Section 5.1 is the place I think the authors measure their work in a correct way but the corresponding results are neither significantly better nor comprehensive enough to support their claimed contribution. I will elaborate. \n\n**Training time comparisons are unfair.**  The cost of the CRT pipeline in terms of the per-epoch time is measured in the wrong way. The current way, excluding Section 5.1, to measure the training cost for CRT is total cost = student cost. If this paper is about to compare the training cost between baselines that train student networks, this is correct. However, all the baselines are methods that actually train the teacher network. Therefore, the correct cost of CRT should be *total cost = teacher cost + student cost*. It is okay to assume a robust teacher network exists but not to assume that the cost of having a teacher network is zero. The authors seem to have noticed this problem. I found the correct measurements in Section 5.1. In fact, if this paper follows the way Section 5.1 is designed, the results will be very impressive: only do robust training on a tiny/small network and use them as teachers for monster networks, which can save a significant amount of time. To do this, the current experiments in Section 5.1 is far from enough. As one may notice that the training cost of the teacher dominates the training budget of CRT because training students do not need robustness regularizations. If one ignores the training cost of teachers, this paper is just to compare the training cost between robustness regularization and standard training, which is not interesting. \n\n**The significance of the work needs more justifications.** All the results show that the student networks are just marginally better or worse than the teachers. This raises a question: why in practice does one want to do this robustness transfer? Why do we not directly use the teacher network? The transfer is only between architectures and not between datasets. Unless network architectures are limited in some sense or I missed something, I don’t see why we need CRT to produce a similar network. \n\n**The discussion on the scalability of CRT is confusing.** Section 5.3 aims to show the scalability of CRT, which is pretty confusing to me. My superficial understanding of the scalability of CRT is determined by the scalability of training teachers and the scalability of training students. The scalability of teachers is determined by the robustness training method proposed by other work. The training of student models is just standard training, which always scales up. However, if the authors take my advice to shift the focus of this paper to train large robust student networks from small teachers, then it is fair to claim that CRT is the way to scale robustness training for large networks. \n\n**Some figures and tables are not necessary.** I find Figure 1 and Table 1 are not necessary at all because 1) I don’t see any information related to the topic this paper tries to discuss in Figure 1. This is just a very general plot for deep learning, and 2) making the factors in a table does not help convey more messages than pure text. There is no more information at all. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CE-4rdSFkaH", "reviewer": "Reviewer_w1QE", "summary": "This paper presents a method for more efficient re-training of certifiably robust models by means of knowledge distillation. Specifically, the proposed method minimizes the l2 distance between the student and teacher softmax predictions on Gaussian-noise perturbed inputs, where the teacher is a certifiably robust model trained. ", "questions": "- How is clean accuracy affected? \n\nSuggestions:\n- The motivation of the paper could be discussed and evaluated in more detail. For example, the scenario considered in this paper is the need to re-deploy a different architecture using the same dataset, but it would also be interesting to explore fine-tuning the student network on new data. \n- It would still be helpful to include in Tables 6 and 7 the certified robustness of the student architecture when trained from scratch in order to better calibrate the results. \n- The consistency regularization method mentioned in section 5.4 should also be referenced in the background section at the beginning of the paper. \n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper empirically shows that the proposed method significantly speeds up training a certifiably robust model on CIFAR-10 (with the constraint of requiring access to a certifiably robust teacher model), and can scale to ImageNet. \n- The contributions and motivation of this paper are made clear.\n- The writing is straightforward. \n- The distilled models have better certified robustness than those trained from scratch given the same network architecture, even when the model is a “second-generation” distillation. \n- The transfer of certified robustness using distillation is a novel application of distillation. \n\nWeaknesses\n- The related work is lacking in terms of prior works on knowledge distillation - it would be useful for the reader to include a section on knowledge distillation and relate the technique used in this paper to existing works. \n- The experimental evaluation could be somewhat more rigorous, and the proposed method explicitly compared to the consistency regularization method referenced in section 5.4. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept